573 lines
41 KiB
Markdown
573 lines
41 KiB
Markdown
# Test Assurance Frontend Capability Implementation Plan
|
|
|
|
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
|
|
|
**Goal:** Extend `test-assurance-platform` so it is the sole authority for selecting, planning, executing, normalizing, evidencing, and assessing every frontend product suite, including tests bound to an immutable static-site candidate.
|
|
|
|
**Architecture:** The platform keeps v2 JVM documents readable, emits v3 documents for new work, and requires an explicit `SOURCE` or `ARTIFACT` phase plus exact input-artifact identities throughout the execution/evidence chain. A normalizer registry dispatches strict report adapters by capability. Each frontend capability has an independent catalog contract and readiness card; no aggregate frontend readiness is introduced.
|
|
|
|
**Tech Stack:** Java 21, Gradle Kotlin DSL, Jackson, JSON Schema 2020-12, Node.js 24.14.0, pnpm 11.17.0, TypeScript 7.0.2, Vitest 4.1.10, Playwright 1.62.0.
|
|
|
|
## Global Constraints
|
|
|
|
- Repository: `/home/donghyeon/workspace/desktop-server-git/test-assurance-platform`; every command runs from its isolated worktree root.
|
|
- Test assurance owns test meaning; it does not create Gitea jobs, allocate remote runners, publish builds, or promote releases.
|
|
- New frontend suites require `test-assurance.platform/v3`; there is no implicit phase or host-toolchain fallback.
|
|
- `requiredInputArtifacts` is an ordered canonical list of `{artifactId, mediaType, sha256}`. Paths are execution-local data and never artifact identity.
|
|
- v2 and v3 evidence cannot be bundled or assessed together.
|
|
- Missing, empty, oversized, malformed, symlinked, mismatched, zero-discovery, all-skipped, or retry-only-green evidence fails closed.
|
|
- The exact capability IDs are `unit-typescript-vitest`, `component-react-vitest`, `integration-http-msw`, `architecture-typescript`, `coverage-v8`, `e2e-playwright-chromium`, `e2e-playwright-firefox`, `e2e-playwright-webkit`, `accessibility-web`, and `visual-regression-web`.
|
|
|
|
---
|
|
|
|
### Task 1: Introduce the v3 artifact-bound execution identity
|
|
|
|
**Files:**
|
|
|
|
- Modify: `modules/assurance-model/src/main/java/io/testassurance/model/ApiVersion.java`
|
|
- Create: `modules/assurance-model/src/main/java/io/testassurance/model/ExecutionPhase.java`
|
|
- Create: `modules/assurance-model/src/main/java/io/testassurance/model/InputArtifact.java`
|
|
- Create: `modules/assurance-model/src/main/java/io/testassurance/model/ArtifactInputDeclaration.java`
|
|
- Create: `modules/assurance-model/src/main/java/io/testassurance/model/ArtifactSuiteTemplate.java`
|
|
- Create: `modules/assurance-model/src/main/java/io/testassurance/model/InputArtifactMap.java`
|
|
- Modify: `modules/assurance-model/src/main/java/io/testassurance/model/SuiteDefinition.java`
|
|
- Modify: `modules/assurance-model/src/main/java/io/testassurance/model/WorkItem.java`
|
|
- Modify: `modules/assurance-model/src/main/java/io/testassurance/model/ExecutionRequest.java`
|
|
- Modify: `modules/assurance-model/src/main/java/io/testassurance/model/ExecutionPlan.java`
|
|
- Modify: `modules/assurance-model/src/main/java/io/testassurance/model/RawSuiteResult.java`
|
|
- Modify: `modules/assurance-model/src/main/java/io/testassurance/model/RawResultSet.java`
|
|
- Modify: `modules/assurance-model/src/main/java/io/testassurance/model/NormalizedSuiteResult.java`
|
|
- Modify: `modules/assurance-model/src/main/java/io/testassurance/model/EvidenceBundle.java`
|
|
- Modify: `modules/assurance-model/src/main/java/io/testassurance/model/Assessment.java`
|
|
- Modify: `modules/assurance-model/src/main/java/io/testassurance/model/RepositoryManifest.java`
|
|
- Modify: `modules/assurance-model/src/main/java/io/testassurance/model/CompiledPolicy.java`
|
|
- Modify: `modules/assurance-model/src/main/java/io/testassurance/model/json/JsonReader.java`
|
|
- Modify: `modules/assurance-model/src/main/java/io/testassurance/model/json/JsonWriter.java`
|
|
- Create: `modules/assurance-model/src/main/java/io/testassurance/model/LegacyV2Decoder.java`
|
|
- Modify: `modules/assurance-model/src/test/java/io/testassurance/model/ModelRoundTripTest.java`
|
|
- Modify: `modules/assurance-model/src/test/java/io/testassurance/model/RepositoryFixtures.java`
|
|
- Preserve unchanged: all 26 current v2 files under `schemas/*.schema.json`
|
|
- Create: 27 complete v3 wire schemas under `schemas/v3/`, one for every current kind plus `artifact-suite-template.schema.json`
|
|
- Create: `schemas/v3/input-artifact-map.schema.json` and register it as an execution-only v3 kind
|
|
- Modify: `modules/assurance-schema/src/main/java/io/testassurance/schema/SchemaRegistry.java`
|
|
- Modify: `modules/assurance-schema/src/test/java/io/testassurance/schema/SchemaRegistryTest.java`
|
|
- Modify: `tools/validate_package.py`
|
|
- Modify: `machine/example-schema-map.json`
|
|
- Modify: `machine/adversarial-schema-map.json`
|
|
- Create: `examples/v3/artifact-suite-template.json`
|
|
- Create: `examples/v3/input-artifact-map.json`
|
|
- Modify: `modules/assurance-schema/src/main/java/io/testassurance/schema/SchemaRegistry.java`
|
|
- Modify: `modules/assurance-schema/src/test/java/io/testassurance/schema/SchemaRegistryTest.java`
|
|
- Create: `conformance/adversarial/v3-artifact-suite-missing-input.json`
|
|
- Create: `conformance/adversarial/v3-source-suite-with-input.json`
|
|
- Create: `conformance/adversarial/v2-v3-evidence-mix.json`
|
|
|
|
**Interfaces:**
|
|
|
|
```java
|
|
public enum ExecutionPhase { SOURCE, ARTIFACT }
|
|
|
|
public record InputArtifact(String artifactId, String mediaType, String sha256) {
|
|
public InputArtifact {
|
|
Identifier.require(artifactId, "inputArtifact.artifactId");
|
|
Values.requireText(mediaType, "inputArtifact.mediaType");
|
|
Digests.require(sha256, "inputArtifact.sha256");
|
|
}
|
|
}
|
|
|
|
public record ArtifactInputDeclaration(String artifactId, String mediaType) {}
|
|
|
|
public record ArtifactSuiteTemplate(
|
|
String id,
|
|
String capabilityId,
|
|
List<String> command,
|
|
int caseTimeoutSeconds,
|
|
int suiteTimeoutSeconds,
|
|
List<String> requiredArtifacts,
|
|
List<String> environmentAllowlist,
|
|
List<ArtifactInputDeclaration> requiredInputArtifacts,
|
|
Optional<ObjectNode> parameters) {}
|
|
|
|
public final class ApiVersion {
|
|
public static final String V2 = "test-assurance.platform/v2";
|
|
public static final String V3 = "test-assurance.platform/v3";
|
|
public static final String CURRENT = V3;
|
|
}
|
|
```
|
|
|
|
All nine execution-chain records expose `apiVersion()`, `executionPhase()`, and `requiredInputArtifacts()`; `SOURCE` requires an empty input list and `ARTIFACT` requires at least one item. `JsonWriter.envelope(String kind, String apiVersion)` requires an explicit version. Existing `toJson()` methods remain v2-compatible; every new v3 output path calls `toJson(ApiVersion.V3)`. `LegacyV2Decoder` reads the complete v2 repository graph explicitly and ordinary v3 readers never guess a phase. `ExecutionRequest.applicationArtifactDigest` exists only in the frozen v2 decoder and is absent from v3.
|
|
|
|
In v3, `RawSuiteResult.artifacts` is `List<ArtifactReference>` rather than path strings. The executor computes each digest immediately after the child exits and before results become visible to normalization. This makes raw artifact identity, normalized references, and evidence chain equality directly verifiable.
|
|
|
|
The committed repository cannot know a future candidate digest. Therefore the approved design amendment adds `RepositoryManifest.artifactSuiteTemplateFiles`: those files declare only artifact ID and media type and are not executable wire `SuiteDefinition` documents. During artifact planning, testctl matches them to concrete `ExecutionRequest.requiredInputArtifacts` and materializes v3 `SuiteDefinition`/`WorkItem` values containing the actual SHA-256. The compiler rejects a digest or digest placeholder in a template. `LegacyV2Decoder` is the only API that maps an approved v2 JVM graph into the legacy SOURCE compatibility path.
|
|
|
|
- [ ] Add model and schema tests for valid SOURCE v3, valid materialized ARTIFACT v3, absent phase, blank digest, duplicate artifact ID, SOURCE with artifacts, ARTIFACT without artifacts, a template containing any SHA field, v2 JVM graph acceptance, v2 frontend rejection, v2/v3 evidence mixing rejection, and byte-identical frozen v2 schema files.
|
|
- [ ] Run:
|
|
|
|
```bash
|
|
./gradlew :modules:assurance-model:test :modules:assurance-schema:test
|
|
```
|
|
|
|
Expected: RED because `ExecutionPhase`, `InputArtifact`, explicit-version envelopes, and the v3 schema registry do not exist.
|
|
|
|
- [ ] Implement explicit-version codecs and a complete v3 schema set. `SchemaRegistry` keys every schema by `(apiVersion, kind)`. `tools/validate_package.py` validates 26 v2 wire schemas, 28 v3 wire/execution schemas, and 11 report schemas as separate inventories rather than one hard-coded total.
|
|
- [ ] Run `./gradlew :modules:assurance-model:test :modules:assurance-schema:test`; expected PASS.
|
|
- [ ] Commit:
|
|
|
|
```bash
|
|
git add modules/assurance-model modules/assurance-schema schemas conformance/adversarial tools/validate_package.py machine examples/v3
|
|
git commit -m "feat(contracts): add artifact-bound execution v3"
|
|
```
|
|
|
|
---
|
|
|
|
### Task 2: Propagate phase and input identity through compile, selection, and planning
|
|
|
|
**Files:**
|
|
|
|
- Modify: `modules/assurance-compiler/src/main/java/io/testassurance/compiler/PolicyCompiler.java`
|
|
- Modify: `modules/assurance-compiler/src/test/java/io/testassurance/compiler/PolicyCompilerTest.java`
|
|
- Modify: `modules/assurance-selector/src/main/java/io/testassurance/selector/ChangeSelector.java`
|
|
- Modify: `modules/assurance-selector/src/test/java/io/testassurance/selector/ChangeSelectorTest.java`
|
|
- Modify: `modules/assurance-plan/src/main/java/io/testassurance/plan/PlanEngine.java`
|
|
- Modify: `modules/assurance-plan/src/main/java/io/testassurance/plan/ResourceProfiles.java`
|
|
- Modify: `modules/assurance-plan/src/test/java/io/testassurance/plan/PlanEngineTest.java`
|
|
- Modify: `modules/assurance-model/src/main/java/io/testassurance/model/CompiledPolicy.java`
|
|
- Modify: `modules/assurance-model/src/main/java/io/testassurance/model/SelectionResult.java`
|
|
|
|
**Contract:** The compiler rejects any new frontend capability on v2. The selector preserves the suite phase without changing selected obligations. The planner copies the exact ordered artifact tuple from the request into every selected ARTIFACT work item and rejects a request whose tuple differs from the suite declaration. Plan digest calculation includes phase and artifact tuple.
|
|
|
|
`PolicyCompiler` stores source `SuiteDefinition` values and artifact `ArtifactSuiteTemplate` values separately. `PlanEngine` exposes `materializeArtifactSuite(ArtifactSuiteTemplate, ExecutionRequest)`; it requires an exact one-to-one match on artifact ID/media type, supplies the request SHA-256, and returns an executable v3 `SuiteDefinition`. No template or unresolved artifact can be serialized into `ExecutionPlan`.
|
|
|
|
- [ ] Add RED tests named `frontendCapabilityRequiresV3`, `artifactTemplateRejectsDigestPlaceholder`, `artifactTemplateMaterializesConcreteSuite`, `selectorPreservesExecutionPhase`, `artifactPlanRejectsWrongCandidateDigest`, `sourcePlanRejectsArtifactInput`, and `planDigestChangesWithArtifactDigest`.
|
|
- [ ] Run:
|
|
|
|
```bash
|
|
./gradlew :modules:assurance-compiler:test :modules:assurance-selector:test :modules:assurance-plan:test
|
|
```
|
|
|
|
Expected: RED because the current compiler and planner do not carry phase/input identity.
|
|
|
|
- [ ] Implement exact propagation. Do not infer phase from capability name or from `applicationArtifactDigest`.
|
|
- [ ] Run `./gradlew :modules:assurance-compiler:test :modules:assurance-selector:test :modules:assurance-plan:test`; expected PASS.
|
|
- [ ] Commit:
|
|
|
|
```bash
|
|
git add modules/assurance-compiler modules/assurance-selector modules/assurance-plan modules/assurance-model
|
|
git commit -m "feat(plan): preserve source and artifact execution identity"
|
|
```
|
|
|
|
---
|
|
|
|
### Task 3: Register the frontend capability family and immutable toolchains
|
|
|
|
**Files:**
|
|
|
|
- Modify: `machine/capability-catalog.json`
|
|
- Modify: `machine/toolchain-catalog.json`
|
|
- Modify: `gradle/libs.versions.toml`
|
|
- Modify: `modules/assurance-catalog/src/main/java/io/testassurance/catalog/CapabilityCatalog.java`
|
|
- Modify: `modules/assurance-catalog/src/main/java/io/testassurance/catalog/CapabilityContract.java`
|
|
- Modify: `modules/assurance-catalog/src/main/java/io/testassurance/catalog/ToolchainCatalog.java`
|
|
- Create: `modules/assurance-catalog/src/main/java/io/testassurance/catalog/FrontendToolchainInspector.java`
|
|
- Create: `modules/assurance-catalog/src/test/java/io/testassurance/catalog/FrontendToolchainInspectorTest.java`
|
|
- Modify: `modules/assurance-catalog/src/test/java/io/testassurance/catalog/CatalogsTest.java`
|
|
- Modify: `modules/assurance-catalog/src/test/java/io/testassurance/catalog/ReadinessCardsTest.java`
|
|
- Modify: `modules/assurance-plan/src/main/java/io/testassurance/plan/ResourceProfiles.java`
|
|
- Modify: `modules/testctl-cli/src/main/java/io/testassurance/cli/Commands.java`
|
|
- Modify: `modules/testctl-cli/src/test/java/io/testassurance/cli/TestctlMainTest.java`
|
|
- Modify: `tools/validate_package.py`
|
|
- Modify: `docs/04-test-type-contracts.md`
|
|
- Modify: `README.md`
|
|
- Create: `readiness/unit-typescript-vitest.json`
|
|
- Create: `readiness/component-react-vitest.json`
|
|
- Create: `readiness/integration-http-msw.json`
|
|
- Create: `readiness/architecture-typescript.json`
|
|
- Create: `readiness/coverage-v8.json`
|
|
- Modify: `readiness/e2e-playwright-chromium.json`
|
|
- Create: `readiness/e2e-playwright-firefox.json`
|
|
- Create: `readiness/e2e-playwright-webkit.json`
|
|
- Create: `readiness/accessibility-web.json`
|
|
- Create: `readiness/visual-regression-web.json`
|
|
|
|
**Catalog entries:** The catalog grows from 16 to 25 capabilities: nine new entries plus the upgraded existing Chromium entry form the ten-capability frontend family. Every capability declares contract revision v3, execution phase, required raw artifact IDs, case/suite timeout bounds, isolation/resource claims, maximum evidence age, false-green rule IDs, and minimum readiness. Toolchain entries pin version plus observed SHA-256 for Node.js 24.14.0, pnpm 11.17.0, TypeScript 7.0.2, Vitest 4.1.10, Playwright 1.62.0, and independent Chromium/Firefox/WebKit payload IDs.
|
|
|
|
`FrontendToolchainInspector` reads `.nvmrc`, `package.json.packageManager`, exact devDependency versions, and the root importer in `pnpm-lock.yaml`. It rejects semver ranges, workspace/host fallbacks, missing artifact digests, and lock resolution drift with `TA-LOCK-002`/`TA-LOCK-003`. `Commands.lock()` uses this inspector for any selected frontend capability.
|
|
|
|
| Capability | Phase | Required raw artifacts |
|
|
|---|---|---|
|
|
| `unit-typescript-vitest` | SOURCE | JUnit XML, Vitest discovery JSON |
|
|
| `component-react-vitest` | SOURCE | JUnit XML, Vitest discovery JSON |
|
|
| `integration-http-msw` | SOURCE | JUnit XML, Vitest discovery JSON, typed HTTP scenario receipts |
|
|
| `architecture-typescript` | SOURCE | TypeScript architecture graph/violations JSON |
|
|
| `coverage-v8` | SOURCE | V8 coverage summary, production-module inventory |
|
|
| `e2e-playwright-chromium` | ARTIFACT | Playwright report, provider identity, trace/screenshot/console/network indexes, mutation receipts |
|
|
| `e2e-playwright-firefox` | ARTIFACT | Playwright report, provider identity, trace/screenshot/console/network indexes, mutation receipts |
|
|
| `e2e-playwright-webkit` | ARTIFACT | Playwright report, provider identity, trace/screenshot/console/network indexes, mutation receipts |
|
|
| `accessibility-web` | ARTIFACT | Playwright report, provider identity, accessibility findings, manual-review records |
|
|
| `visual-regression-web` | ARTIFACT | Playwright report, provider identity, baseline identity, image-diff results |
|
|
|
|
- [ ] Add tests that require exactly 25 catalog capabilities and one readiness card per capability, and fail on a missing digest, Node 22 fallback, semver range, pnpm lock drift, capability aliasing, a combined browser readiness card, or missing Firefox/WebKit resource profile.
|
|
- [ ] Run:
|
|
|
|
```bash
|
|
./gradlew :modules:assurance-catalog:test :modules:assurance-plan:test
|
|
python3 tools/validate_package.py
|
|
```
|
|
|
|
Expected: RED because the capabilities and toolchains are absent.
|
|
|
|
- [ ] Add independent catalog/resource entries and keep all nine new cards plus Chromium at R0 with explicit `nonGuarantees`. R1 is set only in Task 10 after real adapter and conformance evidence exists.
|
|
- [ ] Run `./gradlew :modules:assurance-catalog:test :modules:assurance-plan:test` and `python3 tools/validate_package.py`; expected PASS.
|
|
- [ ] Commit:
|
|
|
|
```bash
|
|
git add machine readiness gradle/libs.versions.toml modules/assurance-catalog modules/assurance-plan modules/testctl-cli tools/validate_package.py docs/04-test-type-contracts.md README.md
|
|
git commit -m "feat(catalog): register frontend assurance capabilities"
|
|
```
|
|
|
|
---
|
|
|
|
### Task 4: Replace hard-coded JUnit normalization with a strict registry
|
|
|
|
**Files:**
|
|
|
|
- Modify: `modules/assurance-normalizer/src/main/java/io/testassurance/normalizer/SuiteResultNormalizer.java`
|
|
- Modify: `modules/assurance-normalizer/src/main/java/io/testassurance/normalizer/NormalizationContext.java`
|
|
- Create: `modules/assurance-normalizer/src/main/java/io/testassurance/normalizer/NormalizerRegistry.java`
|
|
- Create: `modules/assurance-normalizer/src/main/java/io/testassurance/normalizer/NormalizationArtifactReader.java`
|
|
- Create: `modules/assurance-normalizer/src/test/java/io/testassurance/normalizer/NormalizerRegistryTest.java`
|
|
- Create: `modules/assurance-normalizer/src/test/java/io/testassurance/normalizer/NormalizationArtifactReaderTest.java`
|
|
- Create: `modules/assurance-schema/src/main/java/io/testassurance/schema/ReportSchemaRegistry.java`
|
|
- Create: `modules/assurance-schema/src/test/java/io/testassurance/schema/ReportSchemaRegistryTest.java`
|
|
- Create report schemas under `schemas/reports/`: `vitest-discovery-report.schema.json`, `http-scenario-receipt-set.schema.json`, `v8-coverage-summary.schema.json`, `production-module-inventory.schema.json`, `typescript-architecture-report.schema.json`, `playwright-suite-report.schema.json`, `browser-provider-report.schema.json`, `browser-mutation-receipt-set.schema.json`, `accessibility-report.schema.json`, `accessibility-manual-review.schema.json`, `visual-regression-report.schema.json`
|
|
- Modify: `modules/assurance-schema/build.gradle.kts`
|
|
- Modify: `modules/testctl-cli/src/main/java/io/testassurance/cli/Commands.java`
|
|
- Modify: `modules/testctl-cli/src/test/java/io/testassurance/cli/TestctlMainTest.java`
|
|
- Modify: `modules/testctl-cli/build.gradle.kts`
|
|
|
|
**Interfaces:**
|
|
|
|
```java
|
|
public interface SuiteResultNormalizer {
|
|
String adapterId();
|
|
Set<String> capabilityIds();
|
|
NormalizedSuiteResult normalize(NormalizationContext context);
|
|
}
|
|
|
|
public final class NormalizerRegistry {
|
|
public NormalizerRegistry(Collection<SuiteResultNormalizer> normalizers);
|
|
public SuiteResultNormalizer requireFor(String capabilityId);
|
|
}
|
|
|
|
public final class NormalizationArtifactReader {
|
|
public NormalizationArtifactReader(ReportSchemaRegistry schemas);
|
|
public byte[] readRequired(Path outputRoot, String repositoryRelativePath, long maximumBytes);
|
|
public JsonNode readRequiredJson(Path outputRoot, String repositoryRelativePath, long maximumBytes, ReportSchemaRegistry.ReportKind reportKind);
|
|
public ArtifactReference reference(Path outputRoot, String repositoryRelativePath, String mediaType, long maximumBytes);
|
|
}
|
|
```
|
|
|
|
The registry rejects duplicate adapter IDs, duplicate capability ownership, and zero matches. `Commands.normalize()` asks the registry for exactly one adapter and no longer constructs `JunitXmlNormalizer` directly. The common reader uses `NOFOLLOW_LINKS`, checks a regular file before reading, bounds bytes, rejects empty or malformed UTF-8, canonicalizes repository-relative paths, and validates JSON against `ReportSchemaRegistry` before adapters inspect fields.
|
|
|
|
- [ ] Add RED tests for no match, two owners, stable registration order, adapter exception mapping, preservation of v3 input artifacts, path escape, symlink, empty file, oversized file, malformed UTF-8, unknown report schema, and `additionalProperties` rejection.
|
|
- [ ] Run:
|
|
|
|
```bash
|
|
./gradlew :modules:assurance-normalizer:test :modules:testctl-cli:test
|
|
```
|
|
|
|
Expected: RED because dispatch is hard-coded.
|
|
|
|
- [ ] Implement registry wiring with explicit constructors in `testctl-cli`; do not use classpath scanning.
|
|
- [ ] Run `./gradlew :modules:assurance-normalizer:test :modules:testctl-cli:test`; expected PASS.
|
|
- [ ] Commit:
|
|
|
|
```bash
|
|
git add modules/assurance-normalizer modules/assurance-schema modules/testctl-cli schemas/reports
|
|
git commit -m "refactor(normalizer): dispatch by capability contract"
|
|
```
|
|
|
|
---
|
|
|
|
### Task 5: Normalize Vitest results and V8 production coverage
|
|
|
|
**Files:**
|
|
|
|
- Create: `adapters/vitest/build.gradle.kts`
|
|
- Create: `adapters/vitest/src/main/java/io/testassurance/adapter/vitest/VitestJunitNormalizer.java`
|
|
- Create: `adapters/vitest/src/main/java/io/testassurance/adapter/vitest/VitestDiscoveryReport.java`
|
|
- Create: `adapters/vitest/src/test/java/io/testassurance/adapter/vitest/VitestJunitNormalizerTest.java`
|
|
- Create fixture directories: `adapters/vitest/src/test/resources/valid/`, `adapters/vitest/src/test/resources/zero-tests/`, `adapters/vitest/src/test/resources/all-skipped/`, `adapters/vitest/src/test/resources/retry-green/`, `adapters/vitest/src/test/resources/missing-discovery/`, `adapters/vitest/src/test/resources/malformed/`, `adapters/vitest/src/test/resources/oversized/`, `adapters/vitest/src/test/resources/symlink/`
|
|
- Create: `adapters/v8-coverage/build.gradle.kts`
|
|
- Create: `adapters/v8-coverage/src/main/java/io/testassurance/adapter/coverage/V8CoverageNormalizer.java`
|
|
- Create: `adapters/v8-coverage/src/main/java/io/testassurance/adapter/coverage/ProductionModuleInventory.java`
|
|
- Create: `adapters/v8-coverage/src/test/java/io/testassurance/adapter/coverage/V8CoverageNormalizerTest.java`
|
|
- Create fixture directories: `adapters/v8-coverage/src/test/resources/valid/`, `adapters/v8-coverage/src/test/resources/missing-module/`, `adapters/v8-coverage/src/test/resources/zero-universe/`, `adapters/v8-coverage/src/test/resources/path-mismatch/`, `adapters/v8-coverage/src/test/resources/malformed/`, `adapters/v8-coverage/src/test/resources/oversized/`, `adapters/v8-coverage/src/test/resources/symlink/`
|
|
- Modify: `settings.gradle.kts`
|
|
- Modify: `build.gradle.kts`
|
|
- Modify: `modules/testctl-cli/build.gradle.kts`
|
|
- Modify: `modules/testctl-cli/src/main/java/io/testassurance/cli/Commands.java`
|
|
|
|
**Module dependencies:** Both modules expose the model/normalizer APIs, implement report schema support, and are added to root `javaModules`. Vitest additionally depends on `:adapters:junit-gradle` to reuse secure XML parsing.
|
|
|
|
**Rules:** `VitestJunitNormalizer` owns only `unit-typescript-vitest` and `component-react-vitest`; it exports `VitestReportReader` for HTTP composition. `V8CoverageNormalizer` alone owns `coverage-v8`. Missing production module is `INCOMPLETE_DELETED_TEST`; non-empty inventory with zero counter-bearing modules is `INCOMPLETE_ZERO_TESTS`; summary/inventory count or digest mismatch is `INVALID_RESULT`; configured threshold miss is `FAIL_PRODUCT`.
|
|
|
|
- [ ] Write all adapter tests before implementation and confirm RED:
|
|
|
|
```bash
|
|
./gradlew :adapters:vitest:test :adapters:v8-coverage:test
|
|
```
|
|
|
|
- [ ] Implement bounded regular-file reads, fatal UTF-8, secure XML, canonical repository-relative paths, cross-file count reconciliation, and no symlink following.
|
|
- [ ] Register the three capability owners in the CLI registry.
|
|
- [ ] Re-run focused tests plus `:modules:testctl-cli:test`; expected PASS.
|
|
- [ ] Commit:
|
|
|
|
```bash
|
|
git add adapters/vitest adapters/v8-coverage settings.gradle.kts build.gradle.kts modules/testctl-cli
|
|
git commit -m "feat(normalizers): add vitest and v8 coverage evidence"
|
|
```
|
|
|
|
---
|
|
|
|
### Task 6: Normalize HTTP scenarios and TypeScript architecture
|
|
|
|
**Files:**
|
|
|
|
- Create: `adapters/http-scenario/build.gradle.kts`
|
|
- Create: `adapters/http-scenario/src/main/java/io/testassurance/adapter/http/HttpScenarioNormalizer.java`
|
|
- Create: `adapters/http-scenario/src/test/java/io/testassurance/adapter/http/HttpScenarioNormalizerTest.java`
|
|
- Create fixture directories: `adapters/http-scenario/src/test/resources/valid/`, `adapters/http-scenario/src/test/resources/missing-receipt/`, `adapters/http-scenario/src/test/resources/duplicate-receipt/`, `adapters/http-scenario/src/test/resources/unknown-scenario/`, `adapters/http-scenario/src/test/resources/status-mismatch/`, `adapters/http-scenario/src/test/resources/malformed/`
|
|
- Create: `adapters/typescript-architecture/build.gradle.kts`
|
|
- Create: `adapters/typescript-architecture/src/main/java/io/testassurance/adapter/architecture/TypeScriptArchitectureNormalizer.java`
|
|
- Create: `adapters/typescript-architecture/src/test/java/io/testassurance/adapter/architecture/TypeScriptArchitectureNormalizerTest.java`
|
|
- Create fixture directories: `adapters/typescript-architecture/src/test/resources/valid/`, `adapters/typescript-architecture/src/test/resources/empty-graph/`, `adapters/typescript-architecture/src/test/resources/unresolved-import/`, `adapters/typescript-architecture/src/test/resources/cycle/`, `adapters/typescript-architecture/src/test/resources/violation/`, `adapters/typescript-architecture/src/test/resources/path-escape/`, `adapters/typescript-architecture/src/test/resources/malformed/`
|
|
- Modify: `settings.gradle.kts`
|
|
- Modify: `build.gradle.kts`
|
|
- Modify: `modules/testctl-cli/build.gradle.kts`
|
|
- Modify: `modules/testctl-cli/src/main/java/io/testassurance/cli/Commands.java`
|
|
|
|
**Module dependencies:** Both modules are added to root `javaModules`; HTTP depends on `:adapters:vitest`, and both depend on model, normalizer, and schema modules.
|
|
|
|
**Typed inputs:** `HttpScenarioNormalizer` is the sole owner of `integration-http-msw` and composes `VitestReportReader`. It compares exact declared/executed IDs; a missing receipt is `INCOMPLETE_DELETED_TEST`, duplicate/unknown receipt or catalog mismatch is `INVALID_RESULT`, and unmatched/catch-all behavior is `FAIL_TEST`. Architecture requires non-empty modules/dependencies; empty graph is `INCOMPLETE_ZERO_TESTS`, malformed/parser mismatch is `ERROR_TOOLING` or `INVALID_RESULT`, and unresolved import/cycle/rule violation is `FAIL_PRODUCT`.
|
|
|
|
- [ ] Add fixtures/tests and run focused Gradle tests; expected RED.
|
|
- [ ] Implement bounded `ReportSchemaRegistry` parsing, exact scenario/graph cross-field reconciliation, and sole capability ownership for both adapters.
|
|
- [ ] Re-run focused tests and CLI tests; expected PASS.
|
|
- [ ] Commit:
|
|
|
|
```bash
|
|
git add adapters/http-scenario adapters/typescript-architecture settings.gradle.kts build.gradle.kts modules/testctl-cli
|
|
git commit -m "feat(normalizers): add http and architecture evidence"
|
|
```
|
|
|
|
---
|
|
|
|
### Task 7: Upgrade Playwright into three independent browser capabilities
|
|
|
|
**Files:**
|
|
|
|
- Create: `adapters/playwright/src/main/java/io/testassurance/adapter/playwright/PlaywrightResultNormalizer.java`
|
|
- Create: `adapters/playwright/src/main/java/io/testassurance/adapter/playwright/BrowserArtifactIndex.java`
|
|
- Create: `adapters/playwright/src/main/java/io/testassurance/adapter/playwright/MutationReceipt.java`
|
|
- Modify: `adapters/playwright/src/main/java/io/testassurance/adapter/playwright/BrowserRunPolicy.java`
|
|
- Create: `adapters/playwright/src/test/java/io/testassurance/adapter/playwright/PlaywrightResultNormalizerTest.java`
|
|
- Create fixture directories: `adapters/playwright/src/test/resources/chromium/`, `adapters/playwright/src/test/resources/firefox/`, `adapters/playwright/src/test/resources/webkit/`, `adapters/playwright/src/test/resources/missing-project/`, `adapters/playwright/src/test/resources/retry-green/`, `adapters/playwright/src/test/resources/zero-tests/`, `adapters/playwright/src/test/resources/missing-trace/`, `adapters/playwright/src/test/resources/console-error/`, `adapters/playwright/src/test/resources/network-error/`, `adapters/playwright/src/test/resources/valid-mutation/`, `adapters/playwright/src/test/resources/missing-mutation-read/`, `adapters/playwright/src/test/resources/malformed/`
|
|
- Modify: `modules/testctl-cli/src/main/java/io/testassurance/cli/Commands.java`
|
|
|
|
**Rules:** The selected browser capability must match the report project exactly. Required trace/screenshot/console/network indexes bind repository-relative files by digest. Retry-only pass is FLAKY, not PASS. Write journeys require observed response, mutation read, and reload read receipts bound to the same scenario/candidate.
|
|
|
|
- [ ] Add RED tests for each browser and every false-green case.
|
|
- [ ] Run:
|
|
|
|
```bash
|
|
./gradlew :adapters:playwright:test
|
|
```
|
|
|
|
Expected: RED because only policy helpers exist.
|
|
|
|
- [ ] Implement one normalizer parameterized by three independently registered capability IDs; do not aggregate readiness or results.
|
|
- [ ] Run focused and CLI tests; expected PASS.
|
|
- [ ] Commit:
|
|
|
|
```bash
|
|
git add adapters/playwright modules/testctl-cli
|
|
git commit -m "feat(playwright): normalize independent browser evidence"
|
|
```
|
|
|
|
---
|
|
|
|
### Task 8: Add accessibility and visual-regression evidence
|
|
|
|
**Files:**
|
|
|
|
- Create: `adapters/accessibility-web/build.gradle.kts`
|
|
- Create: `adapters/accessibility-web/src/main/java/io/testassurance/adapter/accessibility/AccessibilityNormalizer.java`
|
|
- Create: `adapters/accessibility-web/src/test/java/io/testassurance/adapter/accessibility/AccessibilityNormalizerTest.java`
|
|
- Create fixture directories: `adapters/accessibility-web/src/test/resources/valid/`, `adapters/accessibility-web/src/test/resources/violation/`, `adapters/accessibility-web/src/test/resources/missing-provider/`, `adapters/accessibility-web/src/test/resources/missing-manual-review/`, `adapters/accessibility-web/src/test/resources/expired-review/`, `adapters/accessibility-web/src/test/resources/malformed/`
|
|
- Create: `adapters/visual-regression-web/build.gradle.kts`
|
|
- Create: `adapters/visual-regression-web/src/main/java/io/testassurance/adapter/visual/VisualRegressionNormalizer.java`
|
|
- Create: `adapters/visual-regression-web/src/test/java/io/testassurance/adapter/visual/VisualRegressionNormalizerTest.java`
|
|
- Create fixture directories: `adapters/visual-regression-web/src/test/resources/valid/`, `adapters/visual-regression-web/src/test/resources/diff/`, `adapters/visual-regression-web/src/test/resources/missing-baseline/`, `adapters/visual-regression-web/src/test/resources/wrong-baseline-digest/`, `adapters/visual-regression-web/src/test/resources/missing-provider/`, `adapters/visual-regression-web/src/test/resources/malformed/`
|
|
- Modify: `settings.gradle.kts`
|
|
- Modify: `build.gradle.kts`
|
|
- Modify: `modules/testctl-cli/build.gradle.kts`
|
|
- Modify: `modules/testctl-cli/src/main/java/io/testassurance/cli/Commands.java`
|
|
|
|
**Module dependencies:** Both modules are added to root `javaModules` and depend on model, normalizer, schema, and Playwright evidence types.
|
|
|
|
**Rules:** Accessibility evidence carries scanner/provider identity, rule-set digest, findings, and required manual-review records. Manual records require `reviewedAt`, `expiresAt`, `sourceRevision`, `candidateSha256`, `reviewerId`, and `rulesetDigest`; expiry is evaluated against the raw result's `finishedAt`, never wall-clock time. Visual evidence carries baseline ID/digest, provider identity, current image digest, diff digest, and threshold outcome. Absent identities can never PASS.
|
|
|
|
- [ ] Add RED tests `rejectsAccessibilityWithoutProvider`, `rejectsExpiredManualReviewAtFinishedAt`, `reportsUnwaivedAccessibilityViolation`, `rejectsVisualWithoutBaseline`, `rejectsWrongVisualProvider`, and `reportsVisualDiffOverThreshold`, each backed by the named fixture directory above.
|
|
- [ ] Implement and register both adapters.
|
|
- [ ] Run adapter and CLI tests; expected PASS.
|
|
- [ ] Commit:
|
|
|
|
```bash
|
|
git add adapters/accessibility-web adapters/visual-regression-web settings.gradle.kts build.gradle.kts modules/testctl-cli
|
|
git commit -m "feat(normalizers): add accessibility and visual evidence"
|
|
```
|
|
|
|
---
|
|
|
|
### Task 9: Enforce artifact materialization in bounded execution
|
|
|
|
**Files:**
|
|
|
|
- Modify: `modules/assurance-executor/src/main/java/io/testassurance/executor/WorkItemExecutor.java`
|
|
- Modify: `modules/assurance-executor/src/main/java/io/testassurance/executor/LocalProcessExecutor.java`
|
|
- Create: `modules/assurance-executor/src/main/java/io/testassurance/executor/InputArtifactVerifier.java`
|
|
- Modify: `modules/assurance-executor/src/test/java/io/testassurance/executor/LocalProcessExecutorTest.java`
|
|
- Modify: `modules/testctl-cli/src/main/java/io/testassurance/cli/Commands.java`
|
|
- Modify: `modules/testctl-cli/src/test/java/io/testassurance/cli/TestctlMainTest.java`
|
|
- Modify: `docs/06-testctl-external-executor-contract.md`
|
|
|
|
**External contract:** `testctl execute-one` retains `--plan`, `--work-item-id`, and `--output`, and v3 adds `--artifact-map`. The artifact map resolves artifact IDs to local regular archive files, while expected media type and SHA-256 come only from the signed v3 work item. Verification occurs before the product command starts.
|
|
|
|
The compatible public syntax is:
|
|
|
|
```text
|
|
testctl execute-one --plan <plan.json> --work-item-id <sha256> --artifact-map <input-artifact-map.json> --output <work-dir>
|
|
testctl execute-plan --plan <plan.json> --artifact-map <input-artifact-map.json> --output <run-dir>
|
|
```
|
|
|
|
SOURCE plans reject `--artifact-map`; ARTIFACT plans require it. V3 maps only regular archive files—directories must already be represented by a canonical archive subject. `InputArtifactVerifier` copies each file into a private staging directory while hashing it, then exports only the verified copy path to the child.
|
|
|
|
- [ ] Add RED tests for missing artifact, wrong digest, wrong media type, symlink, path escape, directory input, mutation between verification and use, SOURCE receiving an artifact map, ARTIFACT missing a map, unknown work-item ID, and work item not belonging to the plan digest.
|
|
- [ ] Run executor and CLI tests; expected RED.
|
|
- [ ] Implement verification with opened handles or verified private copies, bounded hashing, and exact environment projection; never trust a caller-supplied digest.
|
|
- [ ] Re-run tests; expected PASS.
|
|
- [ ] Commit:
|
|
|
|
```bash
|
|
git add modules/assurance-executor modules/testctl-cli docs/06-testctl-external-executor-contract.md
|
|
git commit -m "feat(executor): verify artifact-bound work inputs"
|
|
```
|
|
|
|
---
|
|
|
|
### Task 10: Prove the v3 chain, frontend fixture, and independent R1 readiness
|
|
|
|
**Files:**
|
|
|
|
- Modify: `modules/assurance-normalizer/src/main/java/io/testassurance/normalizer/FalseGreenRules.java`
|
|
- Modify: `modules/assurance-evidence/src/main/java/io/testassurance/evidence/EvidenceBundler.java`
|
|
- Modify: `modules/assurance-evidence/src/test/java/io/testassurance/evidence/EvidenceBundlerTest.java`
|
|
- Modify: `modules/assurance-assessor/src/main/java/io/testassurance/assessor/Assessor.java`
|
|
- Modify: `modules/assurance-assessor/src/test/java/io/testassurance/assessor/AssessorTest.java`
|
|
- Create: `modules/assurance-catalog/src/main/java/io/testassurance/catalog/ReadinessCatalog.java`
|
|
- Create: `modules/assurance-catalog/src/test/java/io/testassurance/catalog/ReadinessCatalogTest.java`
|
|
- Modify: `modules/assurance-catalog/build.gradle.kts`
|
|
- Modify: `modules/testctl-cli/src/main/java/io/testassurance/cli/ConformanceRunner.java`
|
|
- Modify: `modules/testctl-cli/src/main/java/io/testassurance/cli/Commands.java`
|
|
- Modify: `modules/testctl-cli/src/test/java/io/testassurance/cli/TestctlMainTest.java`
|
|
- Create: `conformance/golden-v3/source/case.json`
|
|
- Create: `conformance/golden-v3/source/01-execution-request.json`
|
|
- Create: `conformance/golden-v3/source/02-execution-plan.json`
|
|
- Create: `conformance/golden-v3/source/03-raw-result-set.json`
|
|
- Create: `conformance/golden-v3/source/normalized.json`
|
|
- Create: `conformance/golden-v3/source/04-evidence-bundle.json`
|
|
- Create: `conformance/golden-v3/source/05-assessment.json`
|
|
- Create: `conformance/golden-v3/artifact/case.json`
|
|
- Create: `conformance/golden-v3/artifact/01-execution-request.json`
|
|
- Create: `conformance/golden-v3/artifact/02-execution-plan.json`
|
|
- Create: `conformance/golden-v3/artifact/03-raw-result-set.json`
|
|
- Create: `conformance/golden-v3/artifact/normalized.json`
|
|
- Create: `conformance/golden-v3/artifact/04-evidence-bundle.json`
|
|
- Create: `conformance/golden-v3/artifact/05-assessment.json`
|
|
- Create: `conformance/adversarial/v3-wrong-candidate-normalized-result.json`
|
|
- Create: `conformance/adversarial/v3-missing-browser-result.json`
|
|
- Create: `fixtures/frontend-reference/test-assurance.yaml`
|
|
- Create: `fixtures/frontend-reference/test-assurance.lock.json`
|
|
- Create: `fixtures/frontend-reference/.nvmrc`
|
|
- Create: `fixtures/frontend-reference/package.json`
|
|
- Create: `fixtures/frontend-reference/pnpm-lock.yaml`
|
|
- Create: `fixtures/frontend-reference/config/test-assurance/risks/frontend.json`
|
|
- Create: `fixtures/frontend-reference/config/test-assurance/obligations/frontend.json`
|
|
- Create: `fixtures/frontend-reference/config/test-assurance/suites/source.json`
|
|
- Create: `fixtures/frontend-reference/config/test-assurance/suites/artifact-templates.json`
|
|
- Create: `fixtures/frontend-reference/config/test-assurance/change-surfaces.json`
|
|
- Create: `fixtures/frontend-reference/artifacts/frontend-static.tar`
|
|
- Create: `fixtures/frontend-reference/results/source-valid/raw-result-set.json`
|
|
- Create: `fixtures/frontend-reference/results/source-valid/normalized.json`
|
|
- Create: `fixtures/frontend-reference/results/artifact-valid/raw-result-set.json`
|
|
- Create: `fixtures/frontend-reference/results/artifact-valid/normalized.json`
|
|
- Create: `fixtures/frontend-reference/results/adversarial-zero-discovery/raw-result-set.json`
|
|
- Create: `fixtures/frontend-reference/results/adversarial-missing-browser/raw-result-set.json`
|
|
- Create: `fixtures/frontend-reference/results/adversarial-wrong-candidate/raw-result-set.json`
|
|
- Modify: all ten frontend files under `readiness/`
|
|
- Modify: `docs/03-manifest-schema-contracts.md`
|
|
- Modify: `docs/04-test-type-contracts.md`
|
|
- Modify: `docs/06-testctl-external-executor-contract.md`
|
|
- Modify: `docs/07-evidence-flaky-contract.md`
|
|
- Modify: `docs/09-implementation-handoff-contract.md`
|
|
- Modify: `README.md`
|
|
- Modify: `validation.json`
|
|
|
|
**Identity rule:** Bundling requires exact equality of API major, request digest, plan digest, execution phase, ordered input artifacts, work-item IDs, attempt identities, and raw artifact digests. Assessment emits the same tuple plus `readinessDigest` and never converts an integrity error into an unsatisfied product obligation. `ReadinessCatalog` packages the independent cards into the distribution; `Commands.assess()` supplies it to `AssessmentInputs`, and the assessor applies the higher of capability-contract minimum and obligation minimum readiness.
|
|
|
|
- [ ] Add RED chain tests for v2/v3 mixing, source/artifact mixing, wrong candidate, missing work item, duplicate attempt, retry-only green, absent terminal result, empty readiness map, readiness digest drift, and an aggregate frontend readiness card.
|
|
- [ ] Run:
|
|
|
|
```bash
|
|
./gradlew :modules:assurance-evidence:test :modules:assurance-assessor:test :modules:assurance-catalog:test :modules:testctl-cli:test
|
|
```
|
|
|
|
Expected: RED until the entire tuple is checked.
|
|
|
|
- [ ] Implement fail-closed chain verification, readiness binding, and a `ConformanceRunner` case format that validates the five canonical stages plus listed `normalizedFiles`.
|
|
- [ ] Build the frontend fixture with all ten capabilities. Compute the artifact SHA from the committed `frontend-static.tar` bytes and use it consistently in artifact request, materialized suites, plan, raw results, normalized results, bundle, and assessment; no repeated-character or hand-entered digest is allowed.
|
|
- [ ] Run focused tests; expected PASS.
|
|
- [ ] Install the CLI and run both golden cases:
|
|
|
|
```bash
|
|
./gradlew :modules:testctl-cli:installDist
|
|
modules/testctl-cli/build/install/testctl-cli/bin/testctl conformance --case conformance/golden-v3/source/case.json
|
|
modules/testctl-cli/build/install/testctl-cli/bin/testctl conformance --case conformance/golden-v3/artifact/case.json
|
|
```
|
|
|
|
- [ ] Run the fixture twice, once for SOURCE and once for ARTIFACT, through `validate → lock → compile → select → plan → normalize → bundle → assess`; assert five selected source capability IDs and five selected artifact capability IDs with separate plan/evidence/assessment digests.
|
|
- [ ] Update each frontend card from R0 to R1 only when its own adapter fixture and chain evidence digest exists. Keep missing evidence at R0 with explicit `nonGuarantees`.
|
|
- [ ] Add documentation checks naming every capability, exact toolchain pin, v3 CLI argument, phase/template/materialization rule, false-green rule, readiness digest, and v2 compatibility limit.
|
|
- [ ] Run full verification:
|
|
|
|
```bash
|
|
./gradlew clean build
|
|
python3 tools/validate_package.py
|
|
git diff --check
|
|
```
|
|
|
|
Expected: PASS. If a toolchain artifact cannot be observed locally, keep affected cards at R0; do not fabricate a digest.
|
|
|
|
- [ ] Commit:
|
|
|
|
```bash
|
|
git add modules/assurance-normalizer modules/assurance-evidence modules/assurance-assessor modules/assurance-catalog modules/testctl-cli conformance fixtures/frontend-reference readiness docs README.md validation.json
|
|
git commit -m "test(conformance): publish frontend v3 readiness"
|
|
```
|
|
|
|
## Handoff to CI/CD
|
|
|
|
The consuming CI/CD plan may begin only from the immutable commit produced by Task 11. It imports the v3 schemas and CLI distribution by digest, invokes only documented argument-array commands, treats normalized/evidence/assessment documents as opaque test semantics, and carries both source and artifact plan/evidence/assessment digests into release identity.
|