feat(drill): provider 증거로 로컬 drill이 못 보는 항목을 닫는다
문제: provider 확인이 필요한 drill(FE-RB-002~005)은 입력이 runbook ID와 `--release-root`뿐이라 어떤 입력으로도 PASS가 될 수 없었다. 문서는 "provider 증거가 없으면 production 단계는 닫히지 않는다"고 했지만 증거를 넘길 길이 없었다. 고친 것: `--provider-evidence <파일>`을 받는다. 파일 꼴은 `providerDrillEvidenceSchema`(runbook ID, release ID, 관찰 시각, provider run ID, 항목마다 PASS/FAIL과 근거)다. 같은 runbook·같은 release이고, 관찰이 7일 안이며 5분 넘게 앞서지 않고, 로컬 drill이 UNVERIFIED로 남긴 항목만 답한 증거만 받는다. 받은 증거는 그 항목의 UNVERIFIED만 바꾸고 로컬의 PASS·FAIL은 그대로 둔다. `provider verification` 항목도 증거가 답해야 한다. 받지 않은 증거는 이유와 함께 record의 `providerEvidence`에 남고, record schema는 그 경우 PASS를 거절한다. 버린 대안: production·field 단계를 "provider 연결 전 기본 꺼짐"으로 문서화하는 (b)는 job 그래프(`config/ci/**`)를 바꾸는 일이라 하지 않았다. 위험·한계: 템플릿의 production 게이트 명령은 아직 이 인자를 넘기지 않는다. 증거 파일을 만들고 넘기는 자리는 provider를 연결한 제품 몫이다. 유효 기간 7일은 정한 기본값이다. 검증: desktop `node scripts/run-vitest.ts run tests/unit/runbook-drill.test.ts` 통과(유효 증거 PASS, FAIL 증거, 빠진 항목, 다른 release·runbook, 만료, 미래 시각, 로컬이 본 항목, 다른 꼴 거절, PASS를 주장하는 거절 record). Refs: 0929/11-02
This commit is contained in:
@@ -452,7 +452,18 @@ provider 확인이 필요하다고 표시한 것은 모두 `UNVERIFIED`다. `FAI
|
||||
drill은 `FAIL`(exit 1)이다. 그렇지 않고 `UNVERIFIED`가 하나라도 있으면
|
||||
`UNVERIFIED`(exit 3)이고, 게이트는 `PASS`가 아니다. 그래서 provider 증거가 없으면
|
||||
production 단계는 닫히지 않는다. 의도한 결과다. 로컬 drill은 승격한 release를 rollback할
|
||||
수 있다는 것을 보여 줄 수 없다. FE-GATE-016(`rollback-drill`)은 FE-GATE-025와 같은
|
||||
수 있다는 것을 보여 줄 수 없다.
|
||||
|
||||
provider 증거는 `--provider-evidence <파일>`로 넘긴다(0929/11-02). 파일은
|
||||
`providerDrillEvidenceSchema`(`scripts/contracts/release-artifacts.ts`) 꼴이다: runbook ID,
|
||||
release ID, 관찰 시각, provider 쪽 run ID, 그리고 항목마다 `PASS`/`FAIL`과 근거다. 항목
|
||||
이름은 `UNVERIFIED` 줄이 적는 이름(`trigger`, recovery assertion 이름, `provider
|
||||
verification` 등)이다. drill은 runbook과 release가 같고, 관찰이 7일 안이고, 로컬 drill이
|
||||
보지 못한 항목만 답한 증거만 받는다. 받은 증거는 그 항목의 `UNVERIFIED`만 바꾸고, 로컬에서
|
||||
본 `PASS`·`FAIL`은 그대로 둔다. 모든 항목이 답을 얻고 `FAIL`이 없을 때만 drill이 `PASS`다.
|
||||
받지 않은 증거는 이유와 함께 record의 `providerEvidence`에 남고 drill은 `UNVERIFIED`다.
|
||||
템플릿의 production 게이트 명령은 아직 이 인자를 넘기지 않는다. provider를 연결한 제품이
|
||||
증거 파일을 만들고 drill 명령에 넘겨야 그 단계가 닫힌다. FE-GATE-016(`rollback-drill`)은 FE-GATE-025와 같은
|
||||
FE-RB-005 drill을 돌렸으므로 폐기했다(`scripts/contracts/ci-gates.ts`의
|
||||
`RETIRED_GATE_IDS`).
|
||||
|
||||
|
||||
@@ -746,16 +746,52 @@ const drillVerdict = z.enum(["PASS", "FAIL", "UNVERIFIED"]);
|
||||
|
||||
export type DrillVerdict = z.infer<typeof drillVerdict>;
|
||||
|
||||
/** FAIL beats UNVERIFIED beats PASS; provider scope alone keeps a drill UNVERIFIED. */
|
||||
/**
|
||||
* FAIL beats UNVERIFIED beats PASS. A drill with provider scope stays
|
||||
* UNVERIFIED until accepted provider evidence says the provider side passed
|
||||
* (0929/11-02).
|
||||
*/
|
||||
export function combineDrillVerdicts(
|
||||
verdicts: readonly DrillVerdict[],
|
||||
providerVerificationRequired: boolean,
|
||||
providerVerification: DrillVerdict = "UNVERIFIED",
|
||||
): DrillVerdict {
|
||||
if (verdicts.includes("FAIL")) return "FAIL";
|
||||
if (verdicts.includes("UNVERIFIED") || providerVerificationRequired) return "UNVERIFIED";
|
||||
const all = providerVerificationRequired ? [...verdicts, providerVerification] : verdicts;
|
||||
if (all.includes("FAIL")) return "FAIL";
|
||||
if (all.includes("UNVERIFIED")) return "UNVERIFIED";
|
||||
return "PASS";
|
||||
}
|
||||
|
||||
/**
|
||||
* 0929/11-02. What the hosting provider observed for one drill of one
|
||||
* release: the provider-side items a local drill cannot see (a smoke after the
|
||||
* switch, live traffic, the trigger), named as the drill's UNVERIFIED line
|
||||
* names them, plus `provider verification` itself.
|
||||
*/
|
||||
export const providerDrillEvidenceSchema = z
|
||||
.object({
|
||||
schemaVersion: z.literal(1),
|
||||
artifactType: z.literal("provider-drill-evidence"),
|
||||
runbookId: z.string().regex(/^FE-RB-00[1-5]$/u),
|
||||
releaseId: nonEmptyString,
|
||||
observedAt: timestamp,
|
||||
runId: nonEmptyString,
|
||||
observations: z
|
||||
.array(
|
||||
z
|
||||
.object({ item: nonEmptyString, result: z.enum(["PASS", "FAIL"]), evidence: nonEmptyString })
|
||||
.strict(),
|
||||
)
|
||||
.min(1)
|
||||
.max(32)
|
||||
.refine((items) => new Set(items.map(({ item }) => item)).size === items.length, {
|
||||
message: "must name each item once",
|
||||
}),
|
||||
})
|
||||
.strict();
|
||||
|
||||
export type ProviderDrillEvidence = z.infer<typeof providerDrillEvidenceSchema>;
|
||||
|
||||
export const runbookRecordArtifactSchema = z
|
||||
.object({
|
||||
schemaVersion: z.literal(2),
|
||||
@@ -778,6 +814,19 @@ export const runbookRecordArtifactSchema = z
|
||||
negativeFixtureFailedAsExpected: drillVerdict,
|
||||
windowObservedBucket: nonEmptyString,
|
||||
providerVerificationRequired: z.boolean(),
|
||||
providerEvidence: z
|
||||
.object({
|
||||
accepted: z.boolean(),
|
||||
reason: nonEmptyString.optional(),
|
||||
runId: nonEmptyString.optional(),
|
||||
observedAt: timestamp.optional(),
|
||||
providerVerification: drillVerdict,
|
||||
})
|
||||
.strict()
|
||||
.refine((evidence) => evidence.accepted || (evidence.reason !== undefined && evidence.providerVerification === "UNVERIFIED"), {
|
||||
message: "rejected provider evidence must say why and verify nothing",
|
||||
})
|
||||
.optional(),
|
||||
result: drillVerdict,
|
||||
})
|
||||
.strict()
|
||||
@@ -791,6 +840,7 @@ export const runbookRecordArtifactSchema = z
|
||||
record.negativeFixtureFailedAsExpected,
|
||||
],
|
||||
record.providerVerificationRequired,
|
||||
record.providerEvidence?.providerVerification,
|
||||
);
|
||||
if (record.result !== expected) {
|
||||
context.addIssue({
|
||||
|
||||
+142
-10
@@ -22,8 +22,10 @@ import { projectTelemetryEvent } from "../src/contracts/telemetry.ts";
|
||||
import { EXPECTED_CONTRACT_SET_PACKAGES } from "../src/features/installed-contract-contributions.ts";
|
||||
import {
|
||||
combineDrillVerdicts,
|
||||
providerDrillEvidenceSchema,
|
||||
runbookRecordArtifactSchema,
|
||||
type DrillVerdict,
|
||||
type ProviderDrillEvidence,
|
||||
} from "./contracts/release-artifacts.ts";
|
||||
import { verifyReleaseRuntimeCoherence } from "./lib/release-runtime-coherence.ts";
|
||||
import { writeValidatedJsonArtifact } from "./lib/validated-json-artifact.ts";
|
||||
@@ -87,8 +89,25 @@ export type DrillOptions = Readonly<{
|
||||
*/
|
||||
releaseRoot?: string;
|
||||
contractPackages?: readonly InstalledContractPackageIdentity[];
|
||||
/**
|
||||
* 0929/11-02. The provider's record for this drill (`--provider-evidence`),
|
||||
* parsed JSON. Without it a drill with provider scope cannot pass.
|
||||
*/
|
||||
providerEvidence?: unknown;
|
||||
/** The clock evidence age is judged against; a test passes a fixed one. */
|
||||
now?: Date;
|
||||
}>;
|
||||
|
||||
/** Provider evidence older than this no longer describes the provider as it is. */
|
||||
export const PROVIDER_EVIDENCE_MAX_AGE_MS = 7 * 24 * 60 * 60 * 1_000;
|
||||
/** How far ahead of this clock a provider's timestamp may be. */
|
||||
const PROVIDER_EVIDENCE_CLOCK_SKEW_MS = 5 * 60 * 1_000;
|
||||
/** The item names a drill's UNVERIFIED line uses, and provider evidence answers. */
|
||||
const ITEM_TRIGGER = "trigger";
|
||||
const ITEM_CONTAINMENT = "containment";
|
||||
const ITEM_NEGATIVE_FIXTURE = "negative fixture";
|
||||
export const ITEM_PROVIDER_VERIFICATION = "provider verification";
|
||||
|
||||
type DrillContext = Readonly<{
|
||||
root: string;
|
||||
contractPackages: readonly InstalledContractPackageIdentity[];
|
||||
@@ -490,14 +509,29 @@ export async function runRunbookDrill(runbookId: string, options: DrillOptions =
|
||||
const release = parseReleaseArtifact(
|
||||
await rootedReader(context.root)(DEFAULT_ROLLBACK_PATHS.release),
|
||||
);
|
||||
const result = await drill(context);
|
||||
const local = await drill(context);
|
||||
const now = options.now ?? new Date();
|
||||
const judged = options.providerEvidence === undefined
|
||||
? undefined
|
||||
: judgeProviderEvidence(options.providerEvidence, runbookId, release.releaseId, openItems(local), now);
|
||||
const result = judged?.accepted ? applyProviderEvidence(local, judged.evidence) : local;
|
||||
const providerEvidence = judged === undefined
|
||||
? undefined
|
||||
: judged.accepted
|
||||
? {
|
||||
accepted: true as const,
|
||||
runId: judged.evidence.runId,
|
||||
observedAt: judged.evidence.observedAt,
|
||||
providerVerification: observed(judged.evidence, ITEM_PROVIDER_VERIFICATION) ?? "UNVERIFIED",
|
||||
}
|
||||
: { accepted: false as const, reason: judged.reason, providerVerification: "UNVERIFIED" as const };
|
||||
const escalationPathAsserted = verdict(specification.escalation.length >= 2);
|
||||
return {
|
||||
schemaVersion: 2 as const,
|
||||
runbookId,
|
||||
gateId: specification.gateId,
|
||||
releaseId: release.releaseId,
|
||||
drillTimestamp: new Date().toISOString(),
|
||||
drillTimestamp: now.toISOString(),
|
||||
triggerInjected: specification.triggerKinds[0] ?? "",
|
||||
triggerAsserted: result.triggerAsserted,
|
||||
containmentAsserted: result.containmentAsserted,
|
||||
@@ -506,6 +540,7 @@ export async function runRunbookDrill(runbookId: string, options: DrillOptions =
|
||||
negativeFixtureFailedAsExpected: result.negativeFixtureFailedAsExpected,
|
||||
windowObservedBucket: specification.window,
|
||||
providerVerificationRequired: result.providerVerificationRequired,
|
||||
...(providerEvidence === undefined ? {} : { providerEvidence }),
|
||||
result: combineDrillVerdicts(
|
||||
[
|
||||
result.triggerAsserted,
|
||||
@@ -515,24 +550,117 @@ export async function runRunbookDrill(runbookId: string, options: DrillOptions =
|
||||
result.negativeFixtureFailedAsExpected,
|
||||
],
|
||||
result.providerVerificationRequired,
|
||||
providerEvidence?.providerVerification,
|
||||
),
|
||||
};
|
||||
}
|
||||
|
||||
/** The items the local drill left UNVERIFIED, by the names provider evidence uses. */
|
||||
function openItems(result: DrillResult): string[] {
|
||||
return [
|
||||
...(result.triggerAsserted === "UNVERIFIED" ? [ITEM_TRIGGER] : []),
|
||||
...(result.containmentAsserted === "UNVERIFIED" ? [ITEM_CONTAINMENT] : []),
|
||||
...result.recoveryAssertions.filter((item) => item.result === "UNVERIFIED").map((item) => item.assertion),
|
||||
...(result.negativeFixtureFailedAsExpected === "UNVERIFIED" ? [ITEM_NEGATIVE_FIXTURE] : []),
|
||||
...(result.providerVerificationRequired ? [ITEM_PROVIDER_VERIFICATION] : []),
|
||||
];
|
||||
}
|
||||
|
||||
type JudgedProviderEvidence =
|
||||
| Readonly<{ accepted: true; evidence: ProviderDrillEvidence }>
|
||||
| Readonly<{ accepted: false; reason: string }>;
|
||||
|
||||
/**
|
||||
* 0929/11-02. Provider evidence counts only for this runbook, this release,
|
||||
* a recent observation, and the items the local drill could not observe.
|
||||
*/
|
||||
export function judgeProviderEvidence(
|
||||
value: unknown,
|
||||
runbookId: string,
|
||||
releaseId: string,
|
||||
open: readonly string[],
|
||||
now: Date,
|
||||
): JudgedProviderEvidence {
|
||||
const parsed = providerDrillEvidenceSchema.safeParse(value);
|
||||
if (!parsed.success) {
|
||||
const issue = parsed.error.issues[0];
|
||||
return { accepted: false, reason: `not provider drill evidence (${issue?.path.join(".") || "root"}: ${issue?.message ?? "invalid"})` };
|
||||
}
|
||||
const evidence = parsed.data;
|
||||
if (evidence.runbookId !== runbookId) {
|
||||
return { accepted: false, reason: `evidence is for ${evidence.runbookId}, not ${runbookId}` };
|
||||
}
|
||||
if (evidence.releaseId !== releaseId) {
|
||||
return { accepted: false, reason: `evidence is for release ${evidence.releaseId}, not ${releaseId}` };
|
||||
}
|
||||
const age = now.getTime() - Date.parse(evidence.observedAt);
|
||||
if (age > PROVIDER_EVIDENCE_MAX_AGE_MS) {
|
||||
return { accepted: false, reason: `evidence observed at ${evidence.observedAt} is older than ${PROVIDER_EVIDENCE_MAX_AGE_MS / 86_400_000} days` };
|
||||
}
|
||||
if (age < -PROVIDER_EVIDENCE_CLOCK_SKEW_MS) {
|
||||
return { accepted: false, reason: `evidence observed at ${evidence.observedAt} is in the future` };
|
||||
}
|
||||
const extra = evidence.observations.map(({ item }) => item).filter((item) => !open.includes(item));
|
||||
if (extra.length > 0) {
|
||||
return { accepted: false, reason: `evidence names items the local drill observed or does not have: ${extra.join(", ")}` };
|
||||
}
|
||||
return { accepted: true, evidence };
|
||||
}
|
||||
|
||||
function observed(evidence: ProviderDrillEvidence, item: string): DrillVerdict | undefined {
|
||||
return evidence.observations.find((observation) => observation.item === item)?.result;
|
||||
}
|
||||
|
||||
/** Replaces each UNVERIFIED item the evidence answers; local PASS and FAIL stay. */
|
||||
function applyProviderEvidence(result: DrillResult, evidence: ProviderDrillEvidence): DrillResult {
|
||||
const answer = (item: string, local: DrillVerdict): DrillVerdict =>
|
||||
local === "UNVERIFIED" ? observed(evidence, item) ?? local : local;
|
||||
const source = (item: string) =>
|
||||
`provider evidence ${evidence.runId} at ${evidence.observedAt}: ${
|
||||
evidence.observations.find((observation) => observation.item === item)?.evidence ?? ""
|
||||
}`;
|
||||
return {
|
||||
...result,
|
||||
triggerAsserted: answer(ITEM_TRIGGER, result.triggerAsserted),
|
||||
containmentAsserted: answer(ITEM_CONTAINMENT, result.containmentAsserted),
|
||||
recoveryAssertions: result.recoveryAssertions.map((item) =>
|
||||
item.result === "UNVERIFIED" && observed(evidence, item.assertion) !== undefined
|
||||
? { ...item, evidence: source(item.assertion), result: observed(evidence, item.assertion)! }
|
||||
: item,
|
||||
),
|
||||
negativeFixtureFailedAsExpected: answer(ITEM_NEGATIVE_FIXTURE, result.negativeFixtureFailedAsExpected),
|
||||
};
|
||||
}
|
||||
|
||||
async function main(): Promise<void> {
|
||||
const runbookId = process.argv
|
||||
.slice(2)
|
||||
.find((argument) => /^FE-RB-00[1-5]$/.test(argument));
|
||||
const releaseRootFlag = process.argv.indexOf("--release-root");
|
||||
const releaseRoot = releaseRootFlag >= 0 ? process.argv[releaseRootFlag + 1] : undefined;
|
||||
if (!runbookId || (releaseRootFlag >= 0 && !releaseRoot)) {
|
||||
const evidenceFlag = process.argv.indexOf("--provider-evidence");
|
||||
const evidencePath = evidenceFlag >= 0 ? process.argv[evidenceFlag + 1] : undefined;
|
||||
if (!runbookId || (releaseRootFlag >= 0 && !releaseRoot) || (evidenceFlag >= 0 && !evidencePath)) {
|
||||
process.stderr.write(
|
||||
"Usage: drill:runbook -- FE-RB-001..FE-RB-005 [--release-root <extracted release archive>]\n",
|
||||
"Usage: drill:runbook -- FE-RB-001..FE-RB-005 [--release-root <extracted release archive>]" +
|
||||
" [--provider-evidence <provider drill evidence JSON>]\n",
|
||||
);
|
||||
process.exit(2);
|
||||
}
|
||||
let providerEvidence: unknown;
|
||||
if (evidencePath !== undefined) {
|
||||
try {
|
||||
providerEvidence = JSON.parse(await readFile(evidencePath, "utf8"));
|
||||
} catch (error) {
|
||||
process.stderr.write(`${runbookId} drill: cannot read provider evidence ${evidencePath}: ${String(error)}\n`);
|
||||
process.exit(2);
|
||||
}
|
||||
}
|
||||
|
||||
const { gateId, ...record } = await runRunbookDrill(runbookId, releaseRoot ? { releaseRoot } : {});
|
||||
const { gateId, ...record } = await runRunbookDrill(runbookId, {
|
||||
...(releaseRoot ? { releaseRoot } : {}),
|
||||
...(evidencePath !== undefined ? { providerEvidence } : {}),
|
||||
});
|
||||
const artifactDirectory = `artifacts/runbooks/${runbookId}`;
|
||||
await mkdir(artifactDirectory, { recursive: true });
|
||||
await writeValidatedJsonArtifact({
|
||||
@@ -547,14 +675,18 @@ async function main(): Promise<void> {
|
||||
if (record.result === "UNVERIFIED") {
|
||||
// Not a pass: the gate stays red until provider evidence exists (D-14).
|
||||
const open = [
|
||||
["trigger", record.triggerAsserted],
|
||||
["containment", record.containmentAsserted],
|
||||
[ITEM_TRIGGER, record.triggerAsserted],
|
||||
[ITEM_CONTAINMENT, record.containmentAsserted],
|
||||
...record.recoveryAssertions.map((item) => [item.assertion, item.result] as const),
|
||||
["negative fixture", record.negativeFixtureFailedAsExpected],
|
||||
["provider verification", record.providerVerificationRequired ? "UNVERIFIED" : "PASS"],
|
||||
[ITEM_NEGATIVE_FIXTURE, record.negativeFixtureFailedAsExpected],
|
||||
[
|
||||
ITEM_PROVIDER_VERIFICATION,
|
||||
record.providerVerificationRequired ? record.providerEvidence?.providerVerification ?? "UNVERIFIED" : "PASS",
|
||||
],
|
||||
].filter(([, result]) => result === "UNVERIFIED").map(([name]) => name);
|
||||
const rejected = record.providerEvidence?.accepted === false ? `; provider evidence rejected: ${record.providerEvidence.reason}` : "";
|
||||
process.stderr.write(
|
||||
`${runbookId} drill: UNVERIFIED (${gateId}; no local check failed; not observed locally: ${open.join(", ")})\n`,
|
||||
`${runbookId} drill: UNVERIFIED (${gateId}; no local check failed; not observed locally: ${open.join(", ")}${rejected})\n`,
|
||||
);
|
||||
process.exit(3);
|
||||
}
|
||||
|
||||
@@ -4,7 +4,8 @@ import path from "node:path";
|
||||
|
||||
import { afterEach, describe, expect, it } from "vitest";
|
||||
|
||||
import { runRunbookDrill } from "../../scripts/drill-runbook.ts";
|
||||
import { runbookRecordArtifactSchema } from "../../scripts/contracts/release-artifacts.ts";
|
||||
import { PROVIDER_EVIDENCE_MAX_AGE_MS, runRunbookDrill } from "../../scripts/drill-runbook.ts";
|
||||
import { computeContractSetDigest } from "../../src/contracts/contract-set-canonical.ts";
|
||||
import type { InstalledContractPackageIdentity } from "../../src/contracts/external-contract-runtime.ts";
|
||||
|
||||
@@ -145,3 +146,106 @@ describe("runbook drill verdicts", () => {
|
||||
expect(source).not.toMatch(/verdict\(\s*true\s*\)/u);
|
||||
});
|
||||
});
|
||||
|
||||
// 0929/11-02. The drills with provider scope took only a runbook ID and a
|
||||
// release root, so no input could ever make them PASS and the production
|
||||
// stage could not close. Provider evidence now answers the items the local
|
||||
// drill cannot observe, for this runbook and release only, and while recent.
|
||||
describe("runbook drill with provider evidence (0929/11-02)", () => {
|
||||
const NOW = new Date("2026-11-13T09:00:00.000Z");
|
||||
const evidence = (overrides: Record<string, unknown> = {}) => ({
|
||||
schemaVersion: 1,
|
||||
artifactType: "provider-drill-evidence",
|
||||
runbookId: "FE-RB-005",
|
||||
releaseId: "release-a",
|
||||
observedAt: "2026-11-13T08:30:00.000Z",
|
||||
runId: "provider-run-41",
|
||||
observations: [
|
||||
{ item: "trigger", result: "PASS", evidence: "incident INC-7 declared the release-blocking defect" },
|
||||
{ item: "핵심 smoke 시험", result: "PASS", evidence: "smoke against the prior release after the switch" },
|
||||
{ item: "provider verification", result: "PASS", evidence: "provider serves release-a again" },
|
||||
],
|
||||
...overrides,
|
||||
});
|
||||
const drill = async (providerEvidence: unknown) =>
|
||||
runRunbookDrill("FE-RB-005", { releaseRoot: await builtDist(), contractPackages, providerEvidence, now: NOW });
|
||||
|
||||
it("passes FE-RB-005 when the provider observed every item the local drill could not", async () => {
|
||||
const record = await drill(evidence());
|
||||
|
||||
expect(record.result).toBe("PASS");
|
||||
expect(record.triggerAsserted).toBe("PASS");
|
||||
expect(record.recoveryAssertions.find(({ assertion }) => assertion === "핵심 smoke 시험")).toEqual({
|
||||
assertion: "핵심 smoke 시험",
|
||||
evidence: "provider evidence provider-run-41 at 2026-11-13T08:30:00.000Z: smoke against the prior release after the switch",
|
||||
result: "PASS",
|
||||
});
|
||||
expect(record.providerEvidence).toEqual({
|
||||
accepted: true,
|
||||
runId: "provider-run-41",
|
||||
observedAt: "2026-11-13T08:30:00.000Z",
|
||||
providerVerification: "PASS",
|
||||
});
|
||||
const { gateId: _gateId, ...artifact } = record;
|
||||
expect(runbookRecordArtifactSchema.safeParse(artifact).success).toBe(true);
|
||||
});
|
||||
|
||||
it("fails when the provider observed a failure", async () => {
|
||||
const observations = evidence().observations.map((entry) =>
|
||||
entry.item === "핵심 smoke 시험" ? { ...entry, result: "FAIL" } : entry);
|
||||
expect((await drill(evidence({ observations }))).result).toBe("FAIL");
|
||||
});
|
||||
|
||||
it("stays UNVERIFIED while an item the local drill could not observe has no answer", async () => {
|
||||
const record = await drill(evidence({ observations: evidence().observations.slice(0, 2) }));
|
||||
expect(record.result).toBe("UNVERIFIED");
|
||||
expect(record.providerEvidence?.providerVerification).toBe("UNVERIFIED");
|
||||
});
|
||||
|
||||
it.each([
|
||||
["another release", { releaseId: "release-b" }, "evidence is for release release-b, not release-a"],
|
||||
["another runbook", { runbookId: "FE-RB-002" }, "evidence is for FE-RB-002, not FE-RB-005"],
|
||||
[
|
||||
"an expired observation",
|
||||
{ observedAt: new Date(NOW.getTime() - PROVIDER_EVIDENCE_MAX_AGE_MS - 1_000).toISOString() },
|
||||
"is older than 7 days",
|
||||
],
|
||||
["a future observation", { observedAt: "2026-11-13T10:00:00.000Z" }, "is in the future"],
|
||||
[
|
||||
"an answer for a check the local drill made",
|
||||
{ observations: [...evidence().observations, { item: "호환성 gate", result: "PASS", evidence: "x" }] },
|
||||
"evidence names items the local drill observed or does not have: 호환성 gate",
|
||||
],
|
||||
["a record that is not provider evidence", { artifactType: "runbook-record" }, "not provider drill evidence"],
|
||||
])("does not pass on evidence for %s", async (_case, overrides, reason) => {
|
||||
const record = await drill(evidence(overrides));
|
||||
|
||||
expect(record.result).toBe("UNVERIFIED");
|
||||
expect(record.triggerAsserted).toBe("UNVERIFIED");
|
||||
expect(record.providerEvidence?.accepted).toBe(false);
|
||||
expect(record.providerEvidence?.reason).toContain(reason);
|
||||
const { gateId: _gateId, ...artifact } = record;
|
||||
expect(runbookRecordArtifactSchema.safeParse(artifact).success).toBe(true);
|
||||
});
|
||||
|
||||
it("rejects a record that claims PASS without accepted provider evidence", () => {
|
||||
const record = {
|
||||
schemaVersion: 2,
|
||||
runbookId: "FE-RB-005",
|
||||
releaseId: "release-a",
|
||||
drillTimestamp: NOW.toISOString(),
|
||||
triggerInjected: "RELEASE_BLOCKING_DEFECT",
|
||||
triggerAsserted: "PASS",
|
||||
containmentAsserted: "PASS",
|
||||
escalationPathAsserted: "PASS",
|
||||
recoveryAssertions: [],
|
||||
negativeFixtureFailedAsExpected: "PASS",
|
||||
windowObservedBucket: "TBD",
|
||||
providerVerificationRequired: true,
|
||||
providerEvidence: { accepted: false, reason: "expired", providerVerification: "UNVERIFIED" },
|
||||
result: "PASS",
|
||||
};
|
||||
expect(runbookRecordArtifactSchema.safeParse(record).success).toBe(false);
|
||||
expect(runbookRecordArtifactSchema.safeParse({ ...record, result: "UNVERIFIED" }).success).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
Reference in New Issue
Block a user