donghyeon.kang/tech-log-frontend
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
Files
034b702e8e0022cbf3b7015daf1db5cb01cdfe6e
tech-log-frontend/docs/superpowers/plans/2026-08-02-test-assurance-frontend-capabilities.md
T

41 KiB
Raw Blame History

Test Assurance Frontend Capability Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Extend test-assurance-platform so it is the sole authority for selecting, planning, executing, normalizing, evidencing, and assessing every frontend product suite, including tests bound to an immutable static-site candidate.

Architecture: The platform keeps v2 JVM documents readable, emits v3 documents for new work, and requires an explicit SOURCE or ARTIFACT phase plus exact input-artifact identities throughout the execution/evidence chain. A normalizer registry dispatches strict report adapters by capability. Each frontend capability has an independent catalog contract and readiness card; no aggregate frontend readiness is introduced.

Tech Stack: Java 21, Gradle Kotlin DSL, Jackson, JSON Schema 2020-12, Node.js 24.14.0, pnpm 11.17.0, TypeScript 7.0.2, Vitest 4.1.10, Playwright 1.62.0.

Global Constraints

  • Repository: /home/donghyeon/workspace/desktop-server-git/test-assurance-platform; every command runs from its isolated worktree root.
  • Test assurance owns test meaning; it does not create Gitea jobs, allocate remote runners, publish builds, or promote releases.
  • New frontend suites require test-assurance.platform/v3; there is no implicit phase or host-toolchain fallback.
  • requiredInputArtifacts is an ordered canonical list of {artifactId, mediaType, sha256}. Paths are execution-local data and never artifact identity.
  • v2 and v3 evidence cannot be bundled or assessed together.
  • Missing, empty, oversized, malformed, symlinked, mismatched, zero-discovery, all-skipped, or retry-only-green evidence fails closed.
  • The exact capability IDs are unit-typescript-vitest, component-react-vitest, integration-http-msw, architecture-typescript, coverage-v8, e2e-playwright-chromium, e2e-playwright-firefox, e2e-playwright-webkit, accessibility-web, and visual-regression-web.

Task 1: Introduce the v3 artifact-bound execution identity

Files:

  • Modify: modules/assurance-model/src/main/java/io/testassurance/model/ApiVersion.java
  • Create: modules/assurance-model/src/main/java/io/testassurance/model/ExecutionPhase.java
  • Create: modules/assurance-model/src/main/java/io/testassurance/model/InputArtifact.java
  • Create: modules/assurance-model/src/main/java/io/testassurance/model/ArtifactInputDeclaration.java
  • Create: modules/assurance-model/src/main/java/io/testassurance/model/ArtifactSuiteTemplate.java
  • Create: modules/assurance-model/src/main/java/io/testassurance/model/InputArtifactMap.java
  • Modify: modules/assurance-model/src/main/java/io/testassurance/model/SuiteDefinition.java
  • Modify: modules/assurance-model/src/main/java/io/testassurance/model/WorkItem.java
  • Modify: modules/assurance-model/src/main/java/io/testassurance/model/ExecutionRequest.java
  • Modify: modules/assurance-model/src/main/java/io/testassurance/model/ExecutionPlan.java
  • Modify: modules/assurance-model/src/main/java/io/testassurance/model/RawSuiteResult.java
  • Modify: modules/assurance-model/src/main/java/io/testassurance/model/RawResultSet.java
  • Modify: modules/assurance-model/src/main/java/io/testassurance/model/NormalizedSuiteResult.java
  • Modify: modules/assurance-model/src/main/java/io/testassurance/model/EvidenceBundle.java
  • Modify: modules/assurance-model/src/main/java/io/testassurance/model/Assessment.java
  • Modify: modules/assurance-model/src/main/java/io/testassurance/model/RepositoryManifest.java
  • Modify: modules/assurance-model/src/main/java/io/testassurance/model/CompiledPolicy.java
  • Modify: modules/assurance-model/src/main/java/io/testassurance/model/json/JsonReader.java
  • Modify: modules/assurance-model/src/main/java/io/testassurance/model/json/JsonWriter.java
  • Create: modules/assurance-model/src/main/java/io/testassurance/model/LegacyV2Decoder.java
  • Modify: modules/assurance-model/src/test/java/io/testassurance/model/ModelRoundTripTest.java
  • Modify: modules/assurance-model/src/test/java/io/testassurance/model/RepositoryFixtures.java
  • Preserve unchanged: all 26 current v2 files under schemas/*.schema.json
  • Create: 27 complete v3 wire schemas under schemas/v3/, one for every current kind plus artifact-suite-template.schema.json
  • Create: schemas/v3/input-artifact-map.schema.json and register it as an execution-only v3 kind
  • Modify: modules/assurance-schema/src/main/java/io/testassurance/schema/SchemaRegistry.java
  • Modify: modules/assurance-schema/src/test/java/io/testassurance/schema/SchemaRegistryTest.java
  • Modify: tools/validate_package.py
  • Modify: machine/example-schema-map.json
  • Modify: machine/adversarial-schema-map.json
  • Create: examples/v3/artifact-suite-template.json
  • Create: examples/v3/input-artifact-map.json
  • Modify: modules/assurance-schema/src/main/java/io/testassurance/schema/SchemaRegistry.java
  • Modify: modules/assurance-schema/src/test/java/io/testassurance/schema/SchemaRegistryTest.java
  • Create: conformance/adversarial/v3-artifact-suite-missing-input.json
  • Create: conformance/adversarial/v3-source-suite-with-input.json
  • Create: conformance/adversarial/v2-v3-evidence-mix.json

Interfaces:

public enum ExecutionPhase { SOURCE, ARTIFACT }

public record InputArtifact(String artifactId, String mediaType, String sha256) {
    public InputArtifact {
        Identifier.require(artifactId, "inputArtifact.artifactId");
        Values.requireText(mediaType, "inputArtifact.mediaType");
        Digests.require(sha256, "inputArtifact.sha256");
    }
}

public record ArtifactInputDeclaration(String artifactId, String mediaType) {}

public record ArtifactSuiteTemplate(
        String id,
        String capabilityId,
        List<String> command,
        int caseTimeoutSeconds,
        int suiteTimeoutSeconds,
        List<String> requiredArtifacts,
        List<String> environmentAllowlist,
        List<ArtifactInputDeclaration> requiredInputArtifacts,
        Optional<ObjectNode> parameters) {}

public final class ApiVersion {
    public static final String V2 = "test-assurance.platform/v2";
    public static final String V3 = "test-assurance.platform/v3";
    public static final String CURRENT = V3;
}

All nine execution-chain records expose apiVersion(), executionPhase(), and requiredInputArtifacts(); SOURCE requires an empty input list and ARTIFACT requires at least one item. JsonWriter.envelope(String kind, String apiVersion) requires an explicit version. Existing toJson() methods remain v2-compatible; every new v3 output path calls toJson(ApiVersion.V3). LegacyV2Decoder reads the complete v2 repository graph explicitly and ordinary v3 readers never guess a phase. ExecutionRequest.applicationArtifactDigest exists only in the frozen v2 decoder and is absent from v3.

In v3, RawSuiteResult.artifacts is List<ArtifactReference> rather than path strings. The executor computes each digest immediately after the child exits and before results become visible to normalization. This makes raw artifact identity, normalized references, and evidence chain equality directly verifiable.

The committed repository cannot know a future candidate digest. Therefore the approved design amendment adds RepositoryManifest.artifactSuiteTemplateFiles: those files declare only artifact ID and media type and are not executable wire SuiteDefinition documents. During artifact planning, testctl matches them to concrete ExecutionRequest.requiredInputArtifacts and materializes v3 SuiteDefinition/WorkItem values containing the actual SHA-256. The compiler rejects a digest or digest placeholder in a template. LegacyV2Decoder is the only API that maps an approved v2 JVM graph into the legacy SOURCE compatibility path.

  • Add model and schema tests for valid SOURCE v3, valid materialized ARTIFACT v3, absent phase, blank digest, duplicate artifact ID, SOURCE with artifacts, ARTIFACT without artifacts, a template containing any SHA field, v2 JVM graph acceptance, v2 frontend rejection, v2/v3 evidence mixing rejection, and byte-identical frozen v2 schema files.
  • Run:
./gradlew :modules:assurance-model:test :modules:assurance-schema:test

Expected: RED because ExecutionPhase, InputArtifact, explicit-version envelopes, and the v3 schema registry do not exist.

  • Implement explicit-version codecs and a complete v3 schema set. SchemaRegistry keys every schema by (apiVersion, kind). tools/validate_package.py validates 26 v2 wire schemas, 28 v3 wire/execution schemas, and 11 report schemas as separate inventories rather than one hard-coded total.
  • Run ./gradlew :modules:assurance-model:test :modules:assurance-schema:test; expected PASS.
  • Commit:
git add modules/assurance-model modules/assurance-schema schemas conformance/adversarial tools/validate_package.py machine examples/v3
git commit -m "feat(contracts): add artifact-bound execution v3"

Task 2: Propagate phase and input identity through compile, selection, and planning

Files:

  • Modify: modules/assurance-compiler/src/main/java/io/testassurance/compiler/PolicyCompiler.java
  • Modify: modules/assurance-compiler/src/test/java/io/testassurance/compiler/PolicyCompilerTest.java
  • Modify: modules/assurance-selector/src/main/java/io/testassurance/selector/ChangeSelector.java
  • Modify: modules/assurance-selector/src/test/java/io/testassurance/selector/ChangeSelectorTest.java
  • Modify: modules/assurance-plan/src/main/java/io/testassurance/plan/PlanEngine.java
  • Modify: modules/assurance-plan/src/main/java/io/testassurance/plan/ResourceProfiles.java
  • Modify: modules/assurance-plan/src/test/java/io/testassurance/plan/PlanEngineTest.java
  • Modify: modules/assurance-model/src/main/java/io/testassurance/model/CompiledPolicy.java
  • Modify: modules/assurance-model/src/main/java/io/testassurance/model/SelectionResult.java

Contract: The compiler rejects any new frontend capability on v2. The selector preserves the suite phase without changing selected obligations. The planner copies the exact ordered artifact tuple from the request into every selected ARTIFACT work item and rejects a request whose tuple differs from the suite declaration. Plan digest calculation includes phase and artifact tuple.

PolicyCompiler stores source SuiteDefinition values and artifact ArtifactSuiteTemplate values separately. PlanEngine exposes materializeArtifactSuite(ArtifactSuiteTemplate, ExecutionRequest); it requires an exact one-to-one match on artifact ID/media type, supplies the request SHA-256, and returns an executable v3 SuiteDefinition. No template or unresolved artifact can be serialized into ExecutionPlan.

  • Add RED tests named frontendCapabilityRequiresV3, artifactTemplateRejectsDigestPlaceholder, artifactTemplateMaterializesConcreteSuite, selectorPreservesExecutionPhase, artifactPlanRejectsWrongCandidateDigest, sourcePlanRejectsArtifactInput, and planDigestChangesWithArtifactDigest.
  • Run:
./gradlew :modules:assurance-compiler:test :modules:assurance-selector:test :modules:assurance-plan:test

Expected: RED because the current compiler and planner do not carry phase/input identity.

  • Implement exact propagation. Do not infer phase from capability name or from applicationArtifactDigest.
  • Run ./gradlew :modules:assurance-compiler:test :modules:assurance-selector:test :modules:assurance-plan:test; expected PASS.
  • Commit:
git add modules/assurance-compiler modules/assurance-selector modules/assurance-plan modules/assurance-model
git commit -m "feat(plan): preserve source and artifact execution identity"

Task 3: Register the frontend capability family and immutable toolchains

Files:

  • Modify: machine/capability-catalog.json
  • Modify: machine/toolchain-catalog.json
  • Modify: gradle/libs.versions.toml
  • Modify: modules/assurance-catalog/src/main/java/io/testassurance/catalog/CapabilityCatalog.java
  • Modify: modules/assurance-catalog/src/main/java/io/testassurance/catalog/CapabilityContract.java
  • Modify: modules/assurance-catalog/src/main/java/io/testassurance/catalog/ToolchainCatalog.java
  • Create: modules/assurance-catalog/src/main/java/io/testassurance/catalog/FrontendToolchainInspector.java
  • Create: modules/assurance-catalog/src/test/java/io/testassurance/catalog/FrontendToolchainInspectorTest.java
  • Modify: modules/assurance-catalog/src/test/java/io/testassurance/catalog/CatalogsTest.java
  • Modify: modules/assurance-catalog/src/test/java/io/testassurance/catalog/ReadinessCardsTest.java
  • Modify: modules/assurance-plan/src/main/java/io/testassurance/plan/ResourceProfiles.java
  • Modify: modules/testctl-cli/src/main/java/io/testassurance/cli/Commands.java
  • Modify: modules/testctl-cli/src/test/java/io/testassurance/cli/TestctlMainTest.java
  • Modify: tools/validate_package.py
  • Modify: docs/04-test-type-contracts.md
  • Modify: README.md
  • Create: readiness/unit-typescript-vitest.json
  • Create: readiness/component-react-vitest.json
  • Create: readiness/integration-http-msw.json
  • Create: readiness/architecture-typescript.json
  • Create: readiness/coverage-v8.json
  • Modify: readiness/e2e-playwright-chromium.json
  • Create: readiness/e2e-playwright-firefox.json
  • Create: readiness/e2e-playwright-webkit.json
  • Create: readiness/accessibility-web.json
  • Create: readiness/visual-regression-web.json

Catalog entries: The catalog grows from 16 to 25 capabilities: nine new entries plus the upgraded existing Chromium entry form the ten-capability frontend family. Every capability declares contract revision v3, execution phase, required raw artifact IDs, case/suite timeout bounds, isolation/resource claims, maximum evidence age, false-green rule IDs, and minimum readiness. Toolchain entries pin version plus observed SHA-256 for Node.js 24.14.0, pnpm 11.17.0, TypeScript 7.0.2, Vitest 4.1.10, Playwright 1.62.0, and independent Chromium/Firefox/WebKit payload IDs.

FrontendToolchainInspector reads .nvmrc, package.json.packageManager, exact devDependency versions, and the root importer in pnpm-lock.yaml. It rejects semver ranges, workspace/host fallbacks, missing artifact digests, and lock resolution drift with TA-LOCK-002/TA-LOCK-003. Commands.lock() uses this inspector for any selected frontend capability.

Capability Phase Required raw artifacts
unit-typescript-vitest SOURCE JUnit XML, Vitest discovery JSON
component-react-vitest SOURCE JUnit XML, Vitest discovery JSON
integration-http-msw SOURCE JUnit XML, Vitest discovery JSON, typed HTTP scenario receipts
architecture-typescript SOURCE TypeScript architecture graph/violations JSON
coverage-v8 SOURCE V8 coverage summary, production-module inventory
e2e-playwright-chromium ARTIFACT Playwright report, provider identity, trace/screenshot/console/network indexes, mutation receipts
e2e-playwright-firefox ARTIFACT Playwright report, provider identity, trace/screenshot/console/network indexes, mutation receipts
e2e-playwright-webkit ARTIFACT Playwright report, provider identity, trace/screenshot/console/network indexes, mutation receipts
accessibility-web ARTIFACT Playwright report, provider identity, accessibility findings, manual-review records
visual-regression-web ARTIFACT Playwright report, provider identity, baseline identity, image-diff results
  • Add tests that require exactly 25 catalog capabilities and one readiness card per capability, and fail on a missing digest, Node 22 fallback, semver range, pnpm lock drift, capability aliasing, a combined browser readiness card, or missing Firefox/WebKit resource profile.
  • Run:
./gradlew :modules:assurance-catalog:test :modules:assurance-plan:test
python3 tools/validate_package.py

Expected: RED because the capabilities and toolchains are absent.

  • Add independent catalog/resource entries and keep all nine new cards plus Chromium at R0 with explicit nonGuarantees. R1 is set only in Task 10 after real adapter and conformance evidence exists.
  • Run ./gradlew :modules:assurance-catalog:test :modules:assurance-plan:test and python3 tools/validate_package.py; expected PASS.
  • Commit:
git add machine readiness gradle/libs.versions.toml modules/assurance-catalog modules/assurance-plan modules/testctl-cli tools/validate_package.py docs/04-test-type-contracts.md README.md
git commit -m "feat(catalog): register frontend assurance capabilities"

Task 4: Replace hard-coded JUnit normalization with a strict registry

Files:

  • Modify: modules/assurance-normalizer/src/main/java/io/testassurance/normalizer/SuiteResultNormalizer.java
  • Modify: modules/assurance-normalizer/src/main/java/io/testassurance/normalizer/NormalizationContext.java
  • Create: modules/assurance-normalizer/src/main/java/io/testassurance/normalizer/NormalizerRegistry.java
  • Create: modules/assurance-normalizer/src/main/java/io/testassurance/normalizer/NormalizationArtifactReader.java
  • Create: modules/assurance-normalizer/src/test/java/io/testassurance/normalizer/NormalizerRegistryTest.java
  • Create: modules/assurance-normalizer/src/test/java/io/testassurance/normalizer/NormalizationArtifactReaderTest.java
  • Create: modules/assurance-schema/src/main/java/io/testassurance/schema/ReportSchemaRegistry.java
  • Create: modules/assurance-schema/src/test/java/io/testassurance/schema/ReportSchemaRegistryTest.java
  • Create report schemas under schemas/reports/: vitest-discovery-report.schema.json, http-scenario-receipt-set.schema.json, v8-coverage-summary.schema.json, production-module-inventory.schema.json, typescript-architecture-report.schema.json, playwright-suite-report.schema.json, browser-provider-report.schema.json, browser-mutation-receipt-set.schema.json, accessibility-report.schema.json, accessibility-manual-review.schema.json, visual-regression-report.schema.json
  • Modify: modules/assurance-schema/build.gradle.kts
  • Modify: modules/testctl-cli/src/main/java/io/testassurance/cli/Commands.java
  • Modify: modules/testctl-cli/src/test/java/io/testassurance/cli/TestctlMainTest.java
  • Modify: modules/testctl-cli/build.gradle.kts

Interfaces:

public interface SuiteResultNormalizer {
    String adapterId();
    Set<String> capabilityIds();
    NormalizedSuiteResult normalize(NormalizationContext context);
}

public final class NormalizerRegistry {
    public NormalizerRegistry(Collection<SuiteResultNormalizer> normalizers);
    public SuiteResultNormalizer requireFor(String capabilityId);
}

public final class NormalizationArtifactReader {
    public NormalizationArtifactReader(ReportSchemaRegistry schemas);
    public byte[] readRequired(Path outputRoot, String repositoryRelativePath, long maximumBytes);
    public JsonNode readRequiredJson(Path outputRoot, String repositoryRelativePath, long maximumBytes, ReportSchemaRegistry.ReportKind reportKind);
    public ArtifactReference reference(Path outputRoot, String repositoryRelativePath, String mediaType, long maximumBytes);
}

The registry rejects duplicate adapter IDs, duplicate capability ownership, and zero matches. Commands.normalize() asks the registry for exactly one adapter and no longer constructs JunitXmlNormalizer directly. The common reader uses NOFOLLOW_LINKS, checks a regular file before reading, bounds bytes, rejects empty or malformed UTF-8, canonicalizes repository-relative paths, and validates JSON against ReportSchemaRegistry before adapters inspect fields.

  • Add RED tests for no match, two owners, stable registration order, adapter exception mapping, preservation of v3 input artifacts, path escape, symlink, empty file, oversized file, malformed UTF-8, unknown report schema, and additionalProperties rejection.
  • Run:
./gradlew :modules:assurance-normalizer:test :modules:testctl-cli:test

Expected: RED because dispatch is hard-coded.

  • Implement registry wiring with explicit constructors in testctl-cli; do not use classpath scanning.
  • Run ./gradlew :modules:assurance-normalizer:test :modules:testctl-cli:test; expected PASS.
  • Commit:
git add modules/assurance-normalizer modules/assurance-schema modules/testctl-cli schemas/reports
git commit -m "refactor(normalizer): dispatch by capability contract"

Task 5: Normalize Vitest results and V8 production coverage

Files:

  • Create: adapters/vitest/build.gradle.kts
  • Create: adapters/vitest/src/main/java/io/testassurance/adapter/vitest/VitestJunitNormalizer.java
  • Create: adapters/vitest/src/main/java/io/testassurance/adapter/vitest/VitestDiscoveryReport.java
  • Create: adapters/vitest/src/test/java/io/testassurance/adapter/vitest/VitestJunitNormalizerTest.java
  • Create fixture directories: adapters/vitest/src/test/resources/valid/, adapters/vitest/src/test/resources/zero-tests/, adapters/vitest/src/test/resources/all-skipped/, adapters/vitest/src/test/resources/retry-green/, adapters/vitest/src/test/resources/missing-discovery/, adapters/vitest/src/test/resources/malformed/, adapters/vitest/src/test/resources/oversized/, adapters/vitest/src/test/resources/symlink/
  • Create: adapters/v8-coverage/build.gradle.kts
  • Create: adapters/v8-coverage/src/main/java/io/testassurance/adapter/coverage/V8CoverageNormalizer.java
  • Create: adapters/v8-coverage/src/main/java/io/testassurance/adapter/coverage/ProductionModuleInventory.java
  • Create: adapters/v8-coverage/src/test/java/io/testassurance/adapter/coverage/V8CoverageNormalizerTest.java
  • Create fixture directories: adapters/v8-coverage/src/test/resources/valid/, adapters/v8-coverage/src/test/resources/missing-module/, adapters/v8-coverage/src/test/resources/zero-universe/, adapters/v8-coverage/src/test/resources/path-mismatch/, adapters/v8-coverage/src/test/resources/malformed/, adapters/v8-coverage/src/test/resources/oversized/, adapters/v8-coverage/src/test/resources/symlink/
  • Modify: settings.gradle.kts
  • Modify: build.gradle.kts
  • Modify: modules/testctl-cli/build.gradle.kts
  • Modify: modules/testctl-cli/src/main/java/io/testassurance/cli/Commands.java

Module dependencies: Both modules expose the model/normalizer APIs, implement report schema support, and are added to root javaModules. Vitest additionally depends on :adapters:junit-gradle to reuse secure XML parsing.

Rules: VitestJunitNormalizer owns only unit-typescript-vitest and component-react-vitest; it exports VitestReportReader for HTTP composition. V8CoverageNormalizer alone owns coverage-v8. Missing production module is INCOMPLETE_DELETED_TEST; non-empty inventory with zero counter-bearing modules is INCOMPLETE_ZERO_TESTS; summary/inventory count or digest mismatch is INVALID_RESULT; configured threshold miss is FAIL_PRODUCT.

  • Write all adapter tests before implementation and confirm RED:
./gradlew :adapters:vitest:test :adapters:v8-coverage:test
  • Implement bounded regular-file reads, fatal UTF-8, secure XML, canonical repository-relative paths, cross-file count reconciliation, and no symlink following.
  • Register the three capability owners in the CLI registry.
  • Re-run focused tests plus :modules:testctl-cli:test; expected PASS.
  • Commit:
git add adapters/vitest adapters/v8-coverage settings.gradle.kts build.gradle.kts modules/testctl-cli
git commit -m "feat(normalizers): add vitest and v8 coverage evidence"

Task 6: Normalize HTTP scenarios and TypeScript architecture

Files:

  • Create: adapters/http-scenario/build.gradle.kts
  • Create: adapters/http-scenario/src/main/java/io/testassurance/adapter/http/HttpScenarioNormalizer.java
  • Create: adapters/http-scenario/src/test/java/io/testassurance/adapter/http/HttpScenarioNormalizerTest.java
  • Create fixture directories: adapters/http-scenario/src/test/resources/valid/, adapters/http-scenario/src/test/resources/missing-receipt/, adapters/http-scenario/src/test/resources/duplicate-receipt/, adapters/http-scenario/src/test/resources/unknown-scenario/, adapters/http-scenario/src/test/resources/status-mismatch/, adapters/http-scenario/src/test/resources/malformed/
  • Create: adapters/typescript-architecture/build.gradle.kts
  • Create: adapters/typescript-architecture/src/main/java/io/testassurance/adapter/architecture/TypeScriptArchitectureNormalizer.java
  • Create: adapters/typescript-architecture/src/test/java/io/testassurance/adapter/architecture/TypeScriptArchitectureNormalizerTest.java
  • Create fixture directories: adapters/typescript-architecture/src/test/resources/valid/, adapters/typescript-architecture/src/test/resources/empty-graph/, adapters/typescript-architecture/src/test/resources/unresolved-import/, adapters/typescript-architecture/src/test/resources/cycle/, adapters/typescript-architecture/src/test/resources/violation/, adapters/typescript-architecture/src/test/resources/path-escape/, adapters/typescript-architecture/src/test/resources/malformed/
  • Modify: settings.gradle.kts
  • Modify: build.gradle.kts
  • Modify: modules/testctl-cli/build.gradle.kts
  • Modify: modules/testctl-cli/src/main/java/io/testassurance/cli/Commands.java

Module dependencies: Both modules are added to root javaModules; HTTP depends on :adapters:vitest, and both depend on model, normalizer, and schema modules.

Typed inputs: HttpScenarioNormalizer is the sole owner of integration-http-msw and composes VitestReportReader. It compares exact declared/executed IDs; a missing receipt is INCOMPLETE_DELETED_TEST, duplicate/unknown receipt or catalog mismatch is INVALID_RESULT, and unmatched/catch-all behavior is FAIL_TEST. Architecture requires non-empty modules/dependencies; empty graph is INCOMPLETE_ZERO_TESTS, malformed/parser mismatch is ERROR_TOOLING or INVALID_RESULT, and unresolved import/cycle/rule violation is FAIL_PRODUCT.

  • Add fixtures/tests and run focused Gradle tests; expected RED.
  • Implement bounded ReportSchemaRegistry parsing, exact scenario/graph cross-field reconciliation, and sole capability ownership for both adapters.
  • Re-run focused tests and CLI tests; expected PASS.
  • Commit:
git add adapters/http-scenario adapters/typescript-architecture settings.gradle.kts build.gradle.kts modules/testctl-cli
git commit -m "feat(normalizers): add http and architecture evidence"

Task 7: Upgrade Playwright into three independent browser capabilities

Files:

  • Create: adapters/playwright/src/main/java/io/testassurance/adapter/playwright/PlaywrightResultNormalizer.java
  • Create: adapters/playwright/src/main/java/io/testassurance/adapter/playwright/BrowserArtifactIndex.java
  • Create: adapters/playwright/src/main/java/io/testassurance/adapter/playwright/MutationReceipt.java
  • Modify: adapters/playwright/src/main/java/io/testassurance/adapter/playwright/BrowserRunPolicy.java
  • Create: adapters/playwright/src/test/java/io/testassurance/adapter/playwright/PlaywrightResultNormalizerTest.java
  • Create fixture directories: adapters/playwright/src/test/resources/chromium/, adapters/playwright/src/test/resources/firefox/, adapters/playwright/src/test/resources/webkit/, adapters/playwright/src/test/resources/missing-project/, adapters/playwright/src/test/resources/retry-green/, adapters/playwright/src/test/resources/zero-tests/, adapters/playwright/src/test/resources/missing-trace/, adapters/playwright/src/test/resources/console-error/, adapters/playwright/src/test/resources/network-error/, adapters/playwright/src/test/resources/valid-mutation/, adapters/playwright/src/test/resources/missing-mutation-read/, adapters/playwright/src/test/resources/malformed/
  • Modify: modules/testctl-cli/src/main/java/io/testassurance/cli/Commands.java

Rules: The selected browser capability must match the report project exactly. Required trace/screenshot/console/network indexes bind repository-relative files by digest. Retry-only pass is FLAKY, not PASS. Write journeys require observed response, mutation read, and reload read receipts bound to the same scenario/candidate.

  • Add RED tests for each browser and every false-green case.
  • Run:
./gradlew :adapters:playwright:test

Expected: RED because only policy helpers exist.

  • Implement one normalizer parameterized by three independently registered capability IDs; do not aggregate readiness or results.
  • Run focused and CLI tests; expected PASS.
  • Commit:
git add adapters/playwright modules/testctl-cli
git commit -m "feat(playwright): normalize independent browser evidence"

Task 8: Add accessibility and visual-regression evidence

Files:

  • Create: adapters/accessibility-web/build.gradle.kts
  • Create: adapters/accessibility-web/src/main/java/io/testassurance/adapter/accessibility/AccessibilityNormalizer.java
  • Create: adapters/accessibility-web/src/test/java/io/testassurance/adapter/accessibility/AccessibilityNormalizerTest.java
  • Create fixture directories: adapters/accessibility-web/src/test/resources/valid/, adapters/accessibility-web/src/test/resources/violation/, adapters/accessibility-web/src/test/resources/missing-provider/, adapters/accessibility-web/src/test/resources/missing-manual-review/, adapters/accessibility-web/src/test/resources/expired-review/, adapters/accessibility-web/src/test/resources/malformed/
  • Create: adapters/visual-regression-web/build.gradle.kts
  • Create: adapters/visual-regression-web/src/main/java/io/testassurance/adapter/visual/VisualRegressionNormalizer.java
  • Create: adapters/visual-regression-web/src/test/java/io/testassurance/adapter/visual/VisualRegressionNormalizerTest.java
  • Create fixture directories: adapters/visual-regression-web/src/test/resources/valid/, adapters/visual-regression-web/src/test/resources/diff/, adapters/visual-regression-web/src/test/resources/missing-baseline/, adapters/visual-regression-web/src/test/resources/wrong-baseline-digest/, adapters/visual-regression-web/src/test/resources/missing-provider/, adapters/visual-regression-web/src/test/resources/malformed/
  • Modify: settings.gradle.kts
  • Modify: build.gradle.kts
  • Modify: modules/testctl-cli/build.gradle.kts
  • Modify: modules/testctl-cli/src/main/java/io/testassurance/cli/Commands.java

Module dependencies: Both modules are added to root javaModules and depend on model, normalizer, schema, and Playwright evidence types.

Rules: Accessibility evidence carries scanner/provider identity, rule-set digest, findings, and required manual-review records. Manual records require reviewedAt, expiresAt, sourceRevision, candidateSha256, reviewerId, and rulesetDigest; expiry is evaluated against the raw result's finishedAt, never wall-clock time. Visual evidence carries baseline ID/digest, provider identity, current image digest, diff digest, and threshold outcome. Absent identities can never PASS.

  • Add RED tests rejectsAccessibilityWithoutProvider, rejectsExpiredManualReviewAtFinishedAt, reportsUnwaivedAccessibilityViolation, rejectsVisualWithoutBaseline, rejectsWrongVisualProvider, and reportsVisualDiffOverThreshold, each backed by the named fixture directory above.
  • Implement and register both adapters.
  • Run adapter and CLI tests; expected PASS.
  • Commit:
git add adapters/accessibility-web adapters/visual-regression-web settings.gradle.kts build.gradle.kts modules/testctl-cli
git commit -m "feat(normalizers): add accessibility and visual evidence"

Task 9: Enforce artifact materialization in bounded execution

Files:

  • Modify: modules/assurance-executor/src/main/java/io/testassurance/executor/WorkItemExecutor.java
  • Modify: modules/assurance-executor/src/main/java/io/testassurance/executor/LocalProcessExecutor.java
  • Create: modules/assurance-executor/src/main/java/io/testassurance/executor/InputArtifactVerifier.java
  • Modify: modules/assurance-executor/src/test/java/io/testassurance/executor/LocalProcessExecutorTest.java
  • Modify: modules/testctl-cli/src/main/java/io/testassurance/cli/Commands.java
  • Modify: modules/testctl-cli/src/test/java/io/testassurance/cli/TestctlMainTest.java
  • Modify: docs/06-testctl-external-executor-contract.md

External contract: testctl execute-one retains --plan, --work-item-id, and --output, and v3 adds --artifact-map. The artifact map resolves artifact IDs to local regular archive files, while expected media type and SHA-256 come only from the signed v3 work item. Verification occurs before the product command starts.

The compatible public syntax is:

testctl execute-one --plan <plan.json> --work-item-id <sha256> --artifact-map <input-artifact-map.json> --output <work-dir>
testctl execute-plan --plan <plan.json> --artifact-map <input-artifact-map.json> --output <run-dir>

SOURCE plans reject --artifact-map; ARTIFACT plans require it. V3 maps only regular archive files—directories must already be represented by a canonical archive subject. InputArtifactVerifier copies each file into a private staging directory while hashing it, then exports only the verified copy path to the child.

  • Add RED tests for missing artifact, wrong digest, wrong media type, symlink, path escape, directory input, mutation between verification and use, SOURCE receiving an artifact map, ARTIFACT missing a map, unknown work-item ID, and work item not belonging to the plan digest.
  • Run executor and CLI tests; expected RED.
  • Implement verification with opened handles or verified private copies, bounded hashing, and exact environment projection; never trust a caller-supplied digest.
  • Re-run tests; expected PASS.
  • Commit:
git add modules/assurance-executor modules/testctl-cli docs/06-testctl-external-executor-contract.md
git commit -m "feat(executor): verify artifact-bound work inputs"

Task 10: Prove the v3 chain, frontend fixture, and independent R1 readiness

Files:

  • Modify: modules/assurance-normalizer/src/main/java/io/testassurance/normalizer/FalseGreenRules.java
  • Modify: modules/assurance-evidence/src/main/java/io/testassurance/evidence/EvidenceBundler.java
  • Modify: modules/assurance-evidence/src/test/java/io/testassurance/evidence/EvidenceBundlerTest.java
  • Modify: modules/assurance-assessor/src/main/java/io/testassurance/assessor/Assessor.java
  • Modify: modules/assurance-assessor/src/test/java/io/testassurance/assessor/AssessorTest.java
  • Create: modules/assurance-catalog/src/main/java/io/testassurance/catalog/ReadinessCatalog.java
  • Create: modules/assurance-catalog/src/test/java/io/testassurance/catalog/ReadinessCatalogTest.java
  • Modify: modules/assurance-catalog/build.gradle.kts
  • Modify: modules/testctl-cli/src/main/java/io/testassurance/cli/ConformanceRunner.java
  • Modify: modules/testctl-cli/src/main/java/io/testassurance/cli/Commands.java
  • Modify: modules/testctl-cli/src/test/java/io/testassurance/cli/TestctlMainTest.java
  • Create: conformance/golden-v3/source/case.json
  • Create: conformance/golden-v3/source/01-execution-request.json
  • Create: conformance/golden-v3/source/02-execution-plan.json
  • Create: conformance/golden-v3/source/03-raw-result-set.json
  • Create: conformance/golden-v3/source/normalized.json
  • Create: conformance/golden-v3/source/04-evidence-bundle.json
  • Create: conformance/golden-v3/source/05-assessment.json
  • Create: conformance/golden-v3/artifact/case.json
  • Create: conformance/golden-v3/artifact/01-execution-request.json
  • Create: conformance/golden-v3/artifact/02-execution-plan.json
  • Create: conformance/golden-v3/artifact/03-raw-result-set.json
  • Create: conformance/golden-v3/artifact/normalized.json
  • Create: conformance/golden-v3/artifact/04-evidence-bundle.json
  • Create: conformance/golden-v3/artifact/05-assessment.json
  • Create: conformance/adversarial/v3-wrong-candidate-normalized-result.json
  • Create: conformance/adversarial/v3-missing-browser-result.json
  • Create: fixtures/frontend-reference/test-assurance.yaml
  • Create: fixtures/frontend-reference/test-assurance.lock.json
  • Create: fixtures/frontend-reference/.nvmrc
  • Create: fixtures/frontend-reference/package.json
  • Create: fixtures/frontend-reference/pnpm-lock.yaml
  • Create: fixtures/frontend-reference/config/test-assurance/risks/frontend.json
  • Create: fixtures/frontend-reference/config/test-assurance/obligations/frontend.json
  • Create: fixtures/frontend-reference/config/test-assurance/suites/source.json
  • Create: fixtures/frontend-reference/config/test-assurance/suites/artifact-templates.json
  • Create: fixtures/frontend-reference/config/test-assurance/change-surfaces.json
  • Create: fixtures/frontend-reference/artifacts/frontend-static.tar
  • Create: fixtures/frontend-reference/results/source-valid/raw-result-set.json
  • Create: fixtures/frontend-reference/results/source-valid/normalized.json
  • Create: fixtures/frontend-reference/results/artifact-valid/raw-result-set.json
  • Create: fixtures/frontend-reference/results/artifact-valid/normalized.json
  • Create: fixtures/frontend-reference/results/adversarial-zero-discovery/raw-result-set.json
  • Create: fixtures/frontend-reference/results/adversarial-missing-browser/raw-result-set.json
  • Create: fixtures/frontend-reference/results/adversarial-wrong-candidate/raw-result-set.json
  • Modify: all ten frontend files under readiness/
  • Modify: docs/03-manifest-schema-contracts.md
  • Modify: docs/04-test-type-contracts.md
  • Modify: docs/06-testctl-external-executor-contract.md
  • Modify: docs/07-evidence-flaky-contract.md
  • Modify: docs/09-implementation-handoff-contract.md
  • Modify: README.md
  • Modify: validation.json

Identity rule: Bundling requires exact equality of API major, request digest, plan digest, execution phase, ordered input artifacts, work-item IDs, attempt identities, and raw artifact digests. Assessment emits the same tuple plus readinessDigest and never converts an integrity error into an unsatisfied product obligation. ReadinessCatalog packages the independent cards into the distribution; Commands.assess() supplies it to AssessmentInputs, and the assessor applies the higher of capability-contract minimum and obligation minimum readiness.

  • Add RED chain tests for v2/v3 mixing, source/artifact mixing, wrong candidate, missing work item, duplicate attempt, retry-only green, absent terminal result, empty readiness map, readiness digest drift, and an aggregate frontend readiness card.
  • Run:
./gradlew :modules:assurance-evidence:test :modules:assurance-assessor:test :modules:assurance-catalog:test :modules:testctl-cli:test

Expected: RED until the entire tuple is checked.

  • Implement fail-closed chain verification, readiness binding, and a ConformanceRunner case format that validates the five canonical stages plus listed normalizedFiles.
  • Build the frontend fixture with all ten capabilities. Compute the artifact SHA from the committed frontend-static.tar bytes and use it consistently in artifact request, materialized suites, plan, raw results, normalized results, bundle, and assessment; no repeated-character or hand-entered digest is allowed.
  • Run focused tests; expected PASS.
  • Install the CLI and run both golden cases:
./gradlew :modules:testctl-cli:installDist
modules/testctl-cli/build/install/testctl-cli/bin/testctl conformance --case conformance/golden-v3/source/case.json
modules/testctl-cli/build/install/testctl-cli/bin/testctl conformance --case conformance/golden-v3/artifact/case.json
  • Run the fixture twice, once for SOURCE and once for ARTIFACT, through validate → lock → compile → select → plan → normalize → bundle → assess; assert five selected source capability IDs and five selected artifact capability IDs with separate plan/evidence/assessment digests.
  • Update each frontend card from R0 to R1 only when its own adapter fixture and chain evidence digest exists. Keep missing evidence at R0 with explicit nonGuarantees.
  • Add documentation checks naming every capability, exact toolchain pin, v3 CLI argument, phase/template/materialization rule, false-green rule, readiness digest, and v2 compatibility limit.
  • Run full verification:
./gradlew clean build
python3 tools/validate_package.py
git diff --check

Expected: PASS. If a toolchain artifact cannot be observed locally, keep affected cards at R0; do not fabricate a digest.

  • Commit:
git add modules/assurance-normalizer modules/assurance-evidence modules/assurance-assessor modules/assurance-catalog modules/testctl-cli conformance fixtures/frontend-reference readiness docs README.md validation.json
git commit -m "test(conformance): publish frontend v3 readiness"

Handoff to CI/CD

The consuming CI/CD plan may begin only from the immutable commit produced by Task 11. It imports the v3 schemas and CLI distribution by digest, invokes only documented argument-array commands, treats normalized/evidence/assessment documents as opaque test semantics, and carries both source and artifact plan/evidence/assessment digests into release identity.

Reference in New Issue View Git Blame Copy Permalink