init: document-haness 하네스 설계
This commit is contained in:
@@ -1,36 +0,0 @@
|
||||
# Core runtime test contract
|
||||
|
||||
이 디렉터리는 LLM 문장 문자열을 정답으로 고정하지 않는다. Python 3.11+ 표준 라이브러리 런타임이 보장해야 할 불변식을 세 층으로 검사한다.
|
||||
|
||||
- **unit**: route 판정, 상태 전이, Markdown·용어·무손실 규칙과 입력 오류 처리를 검사한다.
|
||||
- **golden**: `fixtures/good`은 finding 없이 통과하고 `fixtures/bad`은 `expected-rule-ids.json`의 실패 모드를 모두 재현해야 한다.
|
||||
- **offline E2E**: root 호환 CLI로 run을 만들고 산출물을 채운 뒤 lint와 verifier를 실행해 `verified` 상태까지 간다.
|
||||
- **live**: 명시적으로 켰을 때만 Claude CLI에 이 저장소의 실제 플러그인을 로드하고, 새 run이 verifier까지 통과하는지 확인한다.
|
||||
|
||||
고정 계약:
|
||||
|
||||
- runtime은 Python 표준 라이브러리만 사용한다.
|
||||
- `config/quality-rules.json`은 설치되는 canonical skill 내부 파일을 뜻한다.
|
||||
- write의 auto route는 최소 standard다. light는 기존 draft가 있고 모든 light 임계 안일 때만 자동 추천한다.
|
||||
- `00_run.json`과 `01_sources.json`의 input hash는 초기화 때 읽은 동일 byte snapshot을 기준으로 하며, run/lint의 `rules_sha256`은 실제 quality-rules byte를 고정한다.
|
||||
- lint exit은 pass=0, gate fail=1, input/schema error=2다.
|
||||
- review mode는 `final.md` 없이 두 독립 review, lint, final report로 완료할 수 있다.
|
||||
- 상태 checkpoint는 단계별 일반·비-symlink 산출물과 schema를 검사하고, review/lint의 현재 hash 및 mode verdict, final/draft byte 동일성을 확인한 뒤에만 전이한다.
|
||||
- `hold_for_review`, `failed`, `incomplete`는 terminal이며 서로 또는 진행 상태로 다시 전이할 수 없다.
|
||||
- verifier가 만든 canonical `09_final_report.json`만 최종 verdict의 기준이며, 일반 상태 CLI는 `verified`를 만들 수 없다.
|
||||
- lint/verifier output은 입력 alias와 다른 도구 소유 파일을 덮어쓰지 않는다.
|
||||
- verifier는 상태 전이 전후 snapshot과 stale lint/review provenance를 재검사한다.
|
||||
|
||||
실행:
|
||||
|
||||
```bash
|
||||
python3 -m pytest tests -q
|
||||
```
|
||||
|
||||
기본 테스트는 외부 모델을 호출하지 않는다. 실제 스킬 회귀는 Claude CLI 인증과 비용·시간 사용을 이해한 상태에서 다음처럼 별도로 실행한다.
|
||||
|
||||
```bash
|
||||
TECH_DOC_FLOW_RUN_LIVE=1 python3 -m pytest tests/test_live_skill.py -q -m live
|
||||
```
|
||||
|
||||
Claude CLI가 없으면 live test는 skip한다. 문자열 전체를 정답으로 고정하지 않고, 실행 산출물·최종 verdict·보호 토큰·용어 first-use 계약을 검사한다. `TECH_DOC_FLOW_LIVE_TIMEOUT`으로 기본 300초 timeout을 바꿀 수 있다.
|
||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Vendored
-16
@@ -1,16 +0,0 @@
|
||||
# 실패 문서
|
||||
|
||||
# 또 다른 제목
|
||||
|
||||
중복 방지는 먼저 쓰였고 TTL도 설명 없이 등장한다. 멱등성, 유효 기간, 캐시 관통을 한 문장에 몰아넣는다. RetryStorm과 HTTPX도 장부에 없다. `cacheStore` 역시 등록되지 않았다. TODO 내용을 보강해야 한다. [없는 절](#missing-section)
|
||||
|
||||
### 원리
|
||||
|
||||
이 절은 순서가 뒤집혔다.
|
||||
|
||||
## 문제
|
||||
|
||||
첫 문장이다. 둘째 문장이다. 셋째 문장이다. 넷째 문장이다. 다섯째 문장이다. 여섯째 문장이다. 일곱째 문장이다. 여덟째 문장이다. 이 문단은 독자가 한 번에 따라가기 어렵도록 같은 설명을 매우 길게 반복한다. 같은 설명을 매우 길게 반복한다. 같은 설명을 매우 길게 반복한다. 같은 설명을 매우 길게 반복한다. 같은 설명을 매우 길게 반복한다. 같은 설명을 매우 길게 반복한다. 같은 설명을 매우 길게 반복한다. 같은 설명을 매우 길게 반복한다. 같은 설명을 매우 길게 반복한다. 같은 설명을 매우 길게 반복한다. 같은 설명을 매우 길게 반복한다. 같은 설명을 매우 길게 반복한다. 같은 설명을 매우 길게 반복한다. 같은 설명을 매우 길게 반복한다. 같은 설명을 매우 길게 반복한다. 같은 설명을 매우 길게 반복한다. 같은 설명을 매우 길게 반복한다. 같은 설명을 매우 길게 반복한다. 같은 설명을 매우 길게 반복한다. 같은 설명을 매우 길게 반복한다. 같은 설명을 매우 길게 반복한다. 같은 설명을 매우 길게 반복한다. 같은 설명을 매우 길게 반복한다. 같은 설명을 매우 길게 반복한다. 같은 설명을 매우 길게 반복한다. 같은 설명을 매우 길게 반복한다. 같은 설명을 매우 길게 반복한다. 같은 설명을 매우 길게 반복한다. 같은 설명을 매우 길게 반복한다. 같은 설명을 매우 길게 반복한다. 같은 설명을 매우 길게 반복한다. 같은 설명을 매우 길게 반복한다. 같은 설명을 매우 길게 반복한다. 같은 설명을 매우 길게 반복한다. 같은 설명을 매우 길게 반복한다. 같은 설명을 매우 길게 반복한다. 같은 설명을 매우 길게 반복한다. 같은 설명을 매우 길게 반복한다. 같은 설명을 매우 길게 반복한다. 같은 설명을 매우 길게 반복한다. 같은 설명을 매우 길게 반복한다. 같은 설명을 매우 길게 반복한다. 같은 설명을 매우 길게 반복한다. 같은 설명을 매우 길게 반복한다.
|
||||
|
||||
```python
|
||||
print("닫히지 않은 fence")
|
||||
-19
@@ -1,19 +0,0 @@
|
||||
[
|
||||
"DOC-H001",
|
||||
"DOC-H002",
|
||||
"DOC-M001",
|
||||
"DOC-M002",
|
||||
"DOC-M003",
|
||||
"DOC-L001",
|
||||
"DOC-L002",
|
||||
"DOC-L003",
|
||||
"DOC-T001",
|
||||
"DOC-T002",
|
||||
"DOC-T003",
|
||||
"DOC-T004",
|
||||
"DOC-T005",
|
||||
"DOC-T007",
|
||||
"DOC-T009",
|
||||
"DOC-P001",
|
||||
"DOC-P002"
|
||||
]
|
||||
Vendored
-35
@@ -1,35 +0,0 @@
|
||||
{
|
||||
"schema_version": "1.0",
|
||||
"title": "실패 문서",
|
||||
"document_kind": "explanation",
|
||||
"core_claim": "이 핵심 주장은 문서에 존재하지 않는다.",
|
||||
"sections": [
|
||||
{
|
||||
"id": "problem",
|
||||
"heading": "문제",
|
||||
"role": "problem",
|
||||
"depends_on": [],
|
||||
"reader_state_before": "모른다.",
|
||||
"question": "문제는 무엇인가?",
|
||||
"answer_plain": "문제가 있다.",
|
||||
"claim_ids": ["C1"],
|
||||
"new_terms": [],
|
||||
"transition_to": "principle",
|
||||
"reader_state_after": "문제를 안다."
|
||||
},
|
||||
{
|
||||
"id": "principle",
|
||||
"heading": "원리",
|
||||
"role": "mechanism",
|
||||
"depends_on": ["problem"],
|
||||
"reader_state_before": "원리를 모른다.",
|
||||
"question": "원리는 무엇인가?",
|
||||
"answer_plain": "원리가 있다.",
|
||||
"claim_ids": ["C2"],
|
||||
"new_terms": ["t1", "t2", "t3"],
|
||||
"transition_to": null,
|
||||
"reader_state_after": "원리를 안다."
|
||||
}
|
||||
],
|
||||
"closure": "실패를 확인한다."
|
||||
}
|
||||
-12
@@ -1,12 +0,0 @@
|
||||
{
|
||||
"schema_version": "1.0",
|
||||
"document_kind": "explanation",
|
||||
"primary_audience": "초급 개발자",
|
||||
"purpose": "실패 fixture",
|
||||
"reader_question": "무엇이 잘못됐는가?",
|
||||
"reader_outcome": "실패를 확인한다.",
|
||||
"prerequisites": [],
|
||||
"assumed_known": [],
|
||||
"must_explain": ["멱등성", "유효 기간", "캐시 관통"],
|
||||
"non_goals": []
|
||||
}
|
||||
Vendored
-40
@@ -1,40 +0,0 @@
|
||||
{
|
||||
"schema_version": "1.0",
|
||||
"assumed_known": ["HTTP"],
|
||||
"budgets": {
|
||||
"per_sentence": 2,
|
||||
"per_paragraph": 2,
|
||||
"per_section": 7
|
||||
},
|
||||
"terms": [
|
||||
{
|
||||
"id": "t1",
|
||||
"canonical": "멱등성",
|
||||
"plain_definition": "여러 번 처리해도 결과가 같은 성질",
|
||||
"why_needed": "재시도를 설명한다.",
|
||||
"aliases": ["중복 방지"],
|
||||
"first_section": "principle",
|
||||
"first_use": "여러 번 처리해도 결과가 같은 성질을 멱등성이라고 한다."
|
||||
},
|
||||
{
|
||||
"id": "t2",
|
||||
"canonical": "유효 기간",
|
||||
"english": "time to live",
|
||||
"abbreviation": "TTL",
|
||||
"plain_definition": "데이터가 남는 시간",
|
||||
"why_needed": "만료를 설명한다.",
|
||||
"aliases": [],
|
||||
"first_section": "principle",
|
||||
"first_use": "데이터가 남는 시간인 유효 기간(time to live, TTL)을 정한다."
|
||||
},
|
||||
{
|
||||
"id": "t3",
|
||||
"canonical": "캐시 관통",
|
||||
"plain_definition": "없는 키 요청이 저장소까지 계속 내려가는 현상",
|
||||
"why_needed": "실패를 설명한다.",
|
||||
"aliases": [],
|
||||
"first_section": "principle",
|
||||
"first_use": "없는 키 요청이 저장소까지 계속 내려가는 현상을 캐시 관통이라고 한다."
|
||||
}
|
||||
]
|
||||
}
|
||||
Vendored
-19
@@ -1,19 +0,0 @@
|
||||
# 캐시 실패를 줄이는 설명
|
||||
|
||||
캐시 장애를 줄이려면 재시도 규칙과 만료 시간을 함께 설계하고 각각의 한계를 검증해야 한다.
|
||||
|
||||
## 문제
|
||||
|
||||
요청이 한꺼번에 몰릴 때 모든 호출자가 같은 데이터를 다시 계산하면 원래 장애보다 더 큰 부하가 생긴다. 이 절에서는 그 실패 장면을 먼저 확인한다. <!-- claim:C1 -->
|
||||
|
||||
다음 [원리](#원리)는 중복 계산과 오래된 데이터라는 두 문제를 나눠 다룬다.
|
||||
|
||||
## 원리
|
||||
|
||||
같은 요청을 여러 번 처리해도 결과가 한 번 처리한 것과 같게 되는 성질을 멱등성이라고 한다. 이 성질이 있으면 안전한 재시도 범위를 설명할 수 있다. <!-- claim:C1 -->
|
||||
|
||||
데이터가 자동으로 사라지기까지의 시간인 유효 기간(time to live, TTL)을 함께 정하면 오래된 값이 남는 시간을 제한할 수 있다.
|
||||
|
||||
## 검증과 한계
|
||||
|
||||
재시도 횟수와 TTL을 바꿔 부하와 오래된 값의 비율을 각각 측정한다. 이 검증은 두 설정의 상호작용을 보여 주지만 실제 장애가 모두 사라진다는 사실까지 증명하지는 않는다. <!-- claim:C2 -->
|
||||
Vendored
-50
@@ -1,50 +0,0 @@
|
||||
{
|
||||
"schema_version": "1.0",
|
||||
"title": "캐시 실패를 줄이는 설명",
|
||||
"document_kind": "explanation",
|
||||
"core_claim": "캐시 장애를 줄이려면 재시도 규칙과 만료 시간을 함께 설계하고 각각의 한계를 검증해야 한다.",
|
||||
"sections": [
|
||||
{
|
||||
"id": "problem",
|
||||
"heading": "문제",
|
||||
"role": "problem",
|
||||
"depends_on": [],
|
||||
"reader_state_before": "캐시 장애가 왜 커지는지 모른다.",
|
||||
"question": "어떤 실패가 부하를 키우는가?",
|
||||
"answer_plain": "동시에 같은 데이터를 다시 계산하는 요청이 부하를 키운다.",
|
||||
"claim_ids": ["C1"],
|
||||
"new_terms": [],
|
||||
"transition_to": "principle",
|
||||
"reader_state_after": "중복 계산이 문제임을 안다."
|
||||
},
|
||||
{
|
||||
"id": "principle",
|
||||
"heading": "원리",
|
||||
"role": "mechanism",
|
||||
"depends_on": ["problem"],
|
||||
"reader_state_before": "문제는 알지만 설정의 역할을 구분하지 못한다.",
|
||||
"question": "재시도와 만료 시간은 각각 무엇을 제한하는가?",
|
||||
"answer_plain": "재시도는 중복 실행을, 만료 시간은 오래된 값의 수명을 제한한다.",
|
||||
"claim_ids": ["C1"],
|
||||
"new_terms": ["term-idempotency", "term-ttl"],
|
||||
"transition_to": "limits",
|
||||
"reader_state_after": "두 설정의 역할을 구분한다."
|
||||
},
|
||||
{
|
||||
"id": "limits",
|
||||
"heading": "검증과 한계",
|
||||
"role": "verification",
|
||||
"depends_on": ["principle"],
|
||||
"reader_state_before": "설정은 알지만 검증 범위는 모른다.",
|
||||
"question": "무엇을 측정하고 무엇은 단정하지 않아야 하는가?",
|
||||
"answer_plain": "부하와 오래된 값은 측정하되 모든 장애 제거를 단정하지 않는다.",
|
||||
"claim_ids": ["C2"],
|
||||
"new_terms": [],
|
||||
"transition_to": null,
|
||||
"reader_state_after": "검증 범위와 한계를 설명할 수 있다.",
|
||||
"proves": ["설정 변화와 측정값의 관계"],
|
||||
"does_not_prove": ["모든 운영 장애의 제거"]
|
||||
}
|
||||
],
|
||||
"closure": "처음의 장애 문제를 두 설정과 검증 한계로 다시 연결한다."
|
||||
}
|
||||
-12
@@ -1,12 +0,0 @@
|
||||
{
|
||||
"schema_version": "1.0",
|
||||
"document_kind": "explanation",
|
||||
"primary_audience": "캐시 운영 경험이 적은 백엔드 개발자",
|
||||
"purpose": "캐시 재시도와 만료 시간의 관계를 설명한다.",
|
||||
"reader_question": "캐시 장애가 커지는 이유와 줄이는 방법은 무엇인가?",
|
||||
"reader_outcome": "재시도와 만료 시간을 함께 검증할 수 있다.",
|
||||
"prerequisites": ["웹 요청의 기본 흐름을 안다."],
|
||||
"assumed_known": ["HTTP"],
|
||||
"must_explain": ["멱등성", "유효 기간"],
|
||||
"non_goals": ["특정 캐시 제품의 설치법"]
|
||||
}
|
||||
Vendored
-32
@@ -1,32 +0,0 @@
|
||||
{
|
||||
"schema_version": "1.0",
|
||||
"assumed_known": ["HTTP"],
|
||||
"budgets": {
|
||||
"per_sentence": 2,
|
||||
"per_paragraph": 2,
|
||||
"per_section": 7
|
||||
},
|
||||
"terms": [
|
||||
{
|
||||
"id": "term-idempotency",
|
||||
"canonical": "멱등성",
|
||||
"english": "idempotency",
|
||||
"plain_definition": "같은 요청을 여러 번 처리해도 결과가 한 번 처리한 것과 같게 되는 성질",
|
||||
"why_needed": "안전한 재시도 범위를 설명하기 위해 필요하다.",
|
||||
"aliases": [],
|
||||
"first_section": "principle",
|
||||
"first_use": "같은 요청을 여러 번 처리해도 결과가 한 번 처리한 것과 같게 되는 성질을 멱등성이라고 한다."
|
||||
},
|
||||
{
|
||||
"id": "term-ttl",
|
||||
"canonical": "유효 기간",
|
||||
"english": "time to live",
|
||||
"abbreviation": "TTL",
|
||||
"plain_definition": "데이터가 자동으로 사라지기까지의 시간",
|
||||
"why_needed": "오래된 값이 남는 시간을 설명하기 위해 필요하다.",
|
||||
"aliases": [],
|
||||
"first_section": "principle",
|
||||
"first_use": "데이터가 자동으로 사라지기까지의 시간인 유효 기간(time to live, TTL)을 함께 정하면 오래된 값이 남는 시간을 제한할 수 있다."
|
||||
}
|
||||
]
|
||||
}
|
||||
+50
-173
@@ -1,179 +1,56 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import hashlib
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
from claridoc.models import Brief, SourcePack
|
||||
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
FIXTURES = ROOT / "tests" / "fixtures"
|
||||
|
||||
|
||||
def run_cli(script: str, *args: object, cwd: Path | None = None) -> subprocess.CompletedProcess[str]:
|
||||
return subprocess.run(
|
||||
[sys.executable, str(ROOT / "scripts" / script), *(str(arg) for arg in args)],
|
||||
cwd=cwd or ROOT,
|
||||
text=True,
|
||||
capture_output=True,
|
||||
check=False,
|
||||
)
|
||||
|
||||
|
||||
def read_json(path: Path) -> dict[str, Any]:
|
||||
return json.loads(path.read_text(encoding="utf-8"))
|
||||
|
||||
|
||||
def write_json(path: Path, value: Any) -> None:
|
||||
path.write_text(json.dumps(value, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
|
||||
|
||||
|
||||
def init_run(
|
||||
tmp_path: Path,
|
||||
*,
|
||||
route: str = "light",
|
||||
mode: str = "write",
|
||||
with_draft: bool = False,
|
||||
audience: str | None = None,
|
||||
) -> Path:
|
||||
brief = tmp_path / "request.md"
|
||||
brief.write_text("캐시 실패를 설명하는 문서를 작성해 주세요.\n", encoding="utf-8")
|
||||
command: list[object] = [
|
||||
"--brief",
|
||||
brief,
|
||||
"--kind",
|
||||
"explanation",
|
||||
"--route",
|
||||
route,
|
||||
"--mode",
|
||||
mode,
|
||||
"--workspace",
|
||||
tmp_path / "workspace",
|
||||
"--date",
|
||||
"2026-07-23",
|
||||
]
|
||||
if audience is not None:
|
||||
command.extend(["--audience", audience])
|
||||
if with_draft or mode in {"revise", "review"}:
|
||||
draft = tmp_path / "original.md"
|
||||
shutil.copyfile(FIXTURES / "good" / "document.md", draft)
|
||||
command.extend(["--draft", draft])
|
||||
result = run_cli("init_run.py", *command)
|
||||
assert result.returncode == 0, result.stderr
|
||||
return Path(result.stdout.split("\t", 1)[0])
|
||||
|
||||
|
||||
def install_good_contracts(run_dir: Path) -> None:
|
||||
mapping = {
|
||||
"reader-contract.json": "02_reader_contract.json",
|
||||
"logic-map.json": "04_logic_map.json",
|
||||
"term-ledger.json": "05_term_ledger.json",
|
||||
"document.md": "07_draft.md",
|
||||
}
|
||||
for source, target in mapping.items():
|
||||
shutil.copyfile(FIXTURES / "good" / source, run_dir / target)
|
||||
manifest = read_json(run_dir / "00_run.json")
|
||||
omissions = []
|
||||
if manifest["route_hint"] == "light":
|
||||
omissions.append(
|
||||
{
|
||||
"artifact": "03_evidence_map.json",
|
||||
"reason": "light fixture는 별도 evidence curation을 생략한다.",
|
||||
}
|
||||
)
|
||||
if manifest["mode"] != "review":
|
||||
omissions.extend(
|
||||
[
|
||||
{
|
||||
"artifact": "08_logic_review.json",
|
||||
"reason": "light write/revise fixture는 별도 logic review를 생략한다.",
|
||||
},
|
||||
{
|
||||
"artifact": "08_reader_review.json",
|
||||
"reason": "light write/revise fixture는 별도 reader review를 생략한다.",
|
||||
},
|
||||
]
|
||||
)
|
||||
if manifest["mode"] == "review":
|
||||
omissions.append(
|
||||
{
|
||||
"artifact": "final.md",
|
||||
"reason": "review mode는 publishable final 문서를 만들지 않는다.",
|
||||
}
|
||||
)
|
||||
manifest["omissions"] = omissions
|
||||
write_json(run_dir / "00_run.json", manifest)
|
||||
|
||||
|
||||
def write_reviews(run_dir: Path) -> None:
|
||||
draft_path = run_dir / "07_draft.md"
|
||||
draft_hash = hashlib.sha256(draft_path.read_bytes()).hexdigest()
|
||||
review_inputs = {
|
||||
"input_sha256": hashlib.sha256((run_dir / "01_input.md").read_bytes()).hexdigest(),
|
||||
"sources_sha256": hashlib.sha256((run_dir / "01_sources.json").read_bytes()).hexdigest(),
|
||||
"reader_contract_sha256": hashlib.sha256(
|
||||
(run_dir / "02_reader_contract.json").read_bytes()
|
||||
).hexdigest(),
|
||||
"evidence_map_sha256": (
|
||||
hashlib.sha256((run_dir / "03_evidence_map.json").read_bytes()).hexdigest()
|
||||
if (run_dir / "03_evidence_map.json").is_file()
|
||||
else None
|
||||
),
|
||||
"logic_map_sha256": hashlib.sha256(
|
||||
(run_dir / "04_logic_map.json").read_bytes()
|
||||
).hexdigest(),
|
||||
"term_ledger_sha256": hashlib.sha256(
|
||||
(run_dir / "05_term_ledger.json").read_bytes()
|
||||
).hexdigest(),
|
||||
}
|
||||
for review_type, filename in (
|
||||
("logic", "08_logic_review.json"),
|
||||
("reader", "08_reader_review.json"),
|
||||
):
|
||||
write_json(
|
||||
run_dir / filename,
|
||||
{
|
||||
"schema_version": "1.0",
|
||||
"review_type": review_type,
|
||||
"document": {"path": str(draft_path), "sha256": draft_hash},
|
||||
"inputs": review_inputs,
|
||||
"verdict": "pass",
|
||||
"findings": [],
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def write_evidence(run_dir: Path) -> None:
|
||||
write_json(
|
||||
run_dir / "03_evidence_map.json",
|
||||
{
|
||||
"schema_version": "1.0",
|
||||
"claims": [
|
||||
{
|
||||
"id": "C1",
|
||||
"statement": "동시 재계산이 부하를 키운다.",
|
||||
"status": "source_backed",
|
||||
"source_ids": ["brief"],
|
||||
"source_locations": [
|
||||
{"source_id": "brief", "locator": "line 1"}
|
||||
],
|
||||
"does_not_support": ["모든 장애 제거"],
|
||||
"load_bearing": True,
|
||||
},
|
||||
{
|
||||
"id": "C2",
|
||||
"statement": "검증에는 한계가 있다.",
|
||||
"status": "assumption",
|
||||
"source_ids": [],
|
||||
"source_locations": [],
|
||||
"does_not_support": ["운영 효과"],
|
||||
"load_bearing": True,
|
||||
"label": "가정",
|
||||
},
|
||||
],
|
||||
def brief_dict(document_type: str = "technical_blog") -> dict:
|
||||
return {
|
||||
"title": "A precise technical document",
|
||||
"document_type": document_type,
|
||||
"language": "en-US",
|
||||
"audience": {
|
||||
"roles": ["software engineers"],
|
||||
"prior_knowledge": ["basic programming"],
|
||||
"needs": ["a decision-ready explanation"],
|
||||
},
|
||||
)
|
||||
"reader_goal": "choose a safe implementation approach",
|
||||
"core_message": "Structure claims around reader questions and verify every important step.",
|
||||
"scope": ["one bounded implementation decision"],
|
||||
"non_scope": ["vendor-specific defaults"],
|
||||
"prerequisites": ["a test environment"],
|
||||
"required_topics": ["mechanism", "evidence", "trade-offs"],
|
||||
"constraints": {
|
||||
"target_words": 700,
|
||||
"tone": "direct and professional",
|
||||
"version_context": "checked 2026-07-23",
|
||||
"max_heading_depth": 3,
|
||||
"require_citations": True,
|
||||
"allow_external_knowledge": False,
|
||||
},
|
||||
"forbidden_claims": ["always safe"],
|
||||
"metadata": {},
|
||||
}
|
||||
|
||||
|
||||
def source_dict() -> dict:
|
||||
return {
|
||||
"sources": [
|
||||
{
|
||||
"id": "S1",
|
||||
"title": "Authoritative source",
|
||||
"url": "https://example.com/source",
|
||||
"publisher": "Example",
|
||||
"accessed": "2026-07-23",
|
||||
"facts": ["The bounded mechanism has an observable result."],
|
||||
"notes": "fixture",
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
|
||||
def make_brief(document_type: str = "technical_blog") -> Brief:
|
||||
return Brief.from_dict(brief_dict(document_type))
|
||||
|
||||
|
||||
def make_sources() -> SourcePack:
|
||||
return SourcePack.from_dict(source_dict())
|
||||
|
||||
@@ -1,89 +0,0 @@
|
||||
"""Optional live runner for the installed technical-doc-flow Claude plugin."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from helpers import ROOT
|
||||
|
||||
|
||||
CLAUDE_BIN = shutil.which("claude")
|
||||
|
||||
|
||||
class LiveSkillError(RuntimeError):
|
||||
"""The external skill run could not produce one verified run."""
|
||||
|
||||
|
||||
def run_live_case(case_dir: Path, *, timeout: int = 300) -> tuple[Path, dict[str, Any]]:
|
||||
if CLAUDE_BIN is None:
|
||||
raise LiveSkillError("claude CLI를 찾을 수 없습니다.")
|
||||
case_dir.mkdir(parents=True, exist_ok=True)
|
||||
brief = case_dir / "brief.md"
|
||||
draft = case_dir / "draft.md"
|
||||
workspace = case_dir / "_workspace"
|
||||
brief.write_text(
|
||||
"이 초안을 HTTP 요청과 응답만 아는 주니어 백엔드 개발자가 따라갈 수 있게 수정하세요. "
|
||||
"논리의 원인→요구→선택→검증→한계 흐름을 만들고, 전문용어는 쉬운 설명 뒤에 소개하세요. "
|
||||
"light revise 경로를 사용하세요.\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
draft.write_text(
|
||||
"# 캐시 복구\n\n"
|
||||
"42ms TTL 만료 뒤 cache stampede가 생기므로 `cacheKey`에 distributed mutex, "
|
||||
"request coalescing, jitter를 적용한다. 자세한 배경은 https://example.test/cache 이다.\n\n"
|
||||
"## 검증\n\nload test가 correctness와 resilience를 증명한다.\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
prompt = (
|
||||
"이 세션에 로드된 technical-doc-flow 스킬을 반드시 활성화해 전체 절차와 결정적 스크립트를 실행하세요. "
|
||||
f"brief={brief}, draft={draft}, workspace={workspace}. "
|
||||
"문서 종류는 explanation이고, 사용자가 light revise를 명시했습니다. "
|
||||
"09_final_report.json verdict가 pass가 아니면 성공이라고 말하지 마세요."
|
||||
)
|
||||
command = [
|
||||
CLAUDE_BIN,
|
||||
"--print",
|
||||
"--no-session-persistence",
|
||||
"--plugin-dir",
|
||||
str(ROOT),
|
||||
"--permission-mode",
|
||||
"acceptEdits",
|
||||
"--allowedTools",
|
||||
"Read,Write,Edit,Bash,Task",
|
||||
prompt,
|
||||
]
|
||||
environment = os.environ.copy()
|
||||
try:
|
||||
completed = subprocess.run(
|
||||
command,
|
||||
cwd=case_dir,
|
||||
stdin=subprocess.DEVNULL,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=timeout,
|
||||
env=environment,
|
||||
check=False,
|
||||
)
|
||||
except subprocess.TimeoutExpired as exc:
|
||||
raise LiveSkillError(f"live 스킬 실행이 {timeout}초를 넘었습니다.") from exc
|
||||
if completed.returncode != 0:
|
||||
diagnostic = (completed.stderr or completed.stdout)[-1000:]
|
||||
raise LiveSkillError(f"claude CLI exit={completed.returncode}: {diagnostic}")
|
||||
|
||||
reports = sorted(workspace.glob("*/09_final_report.json"))
|
||||
if len(reports) != 1:
|
||||
raise LiveSkillError(
|
||||
f"09_final_report.json이 정확히 하나여야 합니다: {[str(path) for path in reports]}"
|
||||
)
|
||||
try:
|
||||
report = json.loads(reports[0].read_text(encoding="utf-8"))
|
||||
except (OSError, UnicodeError, json.JSONDecodeError) as exc:
|
||||
raise LiveSkillError(f"final report를 읽을 수 없습니다: {exc}") from exc
|
||||
if not isinstance(report, dict):
|
||||
raise LiveSkillError("final report 최상위 값이 객체가 아닙니다.")
|
||||
return reports[0].parent, report
|
||||
@@ -0,0 +1,30 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import tempfile
|
||||
import unittest
|
||||
from contextlib import redirect_stdout
|
||||
from io import StringIO
|
||||
from pathlib import Path
|
||||
|
||||
from claridoc.cli import main
|
||||
|
||||
|
||||
class CliTests(unittest.TestCase):
|
||||
def test_init_and_validate(self) -> None:
|
||||
with tempfile.TemporaryDirectory() as temp:
|
||||
workspace = Path(temp) / "workspace"
|
||||
with redirect_stdout(StringIO()):
|
||||
self.assertEqual(main(["init", str(workspace)]), 0)
|
||||
code = main([
|
||||
"validate",
|
||||
"--brief", str(workspace / "brief.json"),
|
||||
"--sources", str(workspace / "sources.json"),
|
||||
])
|
||||
self.assertEqual(code, 0)
|
||||
self.assertTrue((workspace / "pipeline.mock.json").is_file())
|
||||
self.assertIsInstance(json.loads((workspace / "brief.json").read_text(encoding="utf-8")), dict)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -1,16 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
SKILL = ROOT / "skills" / "technical-doc-flow"
|
||||
sys.path.insert(0, str(SKILL / "scripts"))
|
||||
from harness_common import load_rules # noqa: E402
|
||||
|
||||
|
||||
def test_jargon_introduction_rules_are_default_hard_gates() -> None:
|
||||
rules = {item["id"]: item for item in load_rules()["rules"]}
|
||||
for rule_id in ("DOC-T002", "DOC-T004", "DOC-T005", "DOC-T006", "DOC-T007"):
|
||||
assert rules[rule_id]["severity"] == "error"
|
||||
@@ -1,165 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import shutil
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
from helpers import ROOT, run_cli
|
||||
|
||||
sys.path.insert(0, str(ROOT / "scripts"))
|
||||
from check_release_sync import SEMVER_RE # noqa: E402
|
||||
|
||||
|
||||
def test_canonical_quick_rules_are_current() -> None:
|
||||
result = run_cli("build_quick_rules.py", "--check")
|
||||
assert result.returncode == 0, result.stderr
|
||||
|
||||
|
||||
def test_quick_rules_check_detects_drift(tmp_path: Path) -> None:
|
||||
output = tmp_path / "quick-rules.md"
|
||||
generated = run_cli("build_quick_rules.py", "--output", output)
|
||||
assert generated.returncode == 0, generated.stderr
|
||||
current = run_cli("build_quick_rules.py", "--output", output, "--check")
|
||||
assert current.returncode == 0, current.stderr
|
||||
output.write_text(output.read_text(encoding="utf-8") + "drift\n", encoding="utf-8")
|
||||
stale = run_cli("build_quick_rules.py", "--output", output, "--check")
|
||||
assert stale.returncode == 1
|
||||
assert "out of date" in stale.stderr
|
||||
|
||||
|
||||
def test_quick_rules_render_every_route_and_threshold_switches(tmp_path: Path) -> None:
|
||||
rules = json.loads(
|
||||
(ROOT / "skills/technical-doc-flow/config/quality-rules.json").read_text(encoding="utf-8")
|
||||
)
|
||||
rules["thresholds"]["route"]["light"]["requires_existing_draft"] = False
|
||||
rules["thresholds"]["logic"]["core_claim_max_reader_paragraphs"] = 3
|
||||
rules["thresholds"]["split"]["default_max_chars"] = 4321
|
||||
rules["thresholds"]["split"]["minimum_h2_fill_ratio"] = 0.5
|
||||
custom_rules = tmp_path / "rules.json"
|
||||
custom_rules.write_text(json.dumps(rules), encoding="utf-8")
|
||||
output = tmp_path / "quick.md"
|
||||
result = run_cli(
|
||||
"build_quick_rules.py", "--rules", custom_rules, "--output", output
|
||||
)
|
||||
assert result.returncode == 0, result.stderr
|
||||
rendered = output.read_text(encoding="utf-8")
|
||||
assert "기존 초안 불필요" in rendered
|
||||
assert "독자용 앞 3개 문단" in rendered
|
||||
assert "`backpressure`" in rendered
|
||||
assert "`UTF-8`" in rendered
|
||||
assert "4321자" in rendered
|
||||
assert "50% (raw `0.5`)" in rendered
|
||||
|
||||
first = rendered
|
||||
rules["thresholds"]["logic"]["core_claim_max_reader_paragraphs"] = 4
|
||||
rules["thresholds"]["split"]["minimum_h2_fill_ratio"] = 0.501
|
||||
rules["thresholds"]["finalization"]["max_change_rate"] = 0.151
|
||||
custom_rules.write_text(json.dumps(rules), encoding="utf-8")
|
||||
assert run_cli(
|
||||
"build_quick_rules.py", "--rules", custom_rules, "--output", output
|
||||
).returncode == 0
|
||||
second = output.read_text(encoding="utf-8")
|
||||
assert first != second
|
||||
assert "독자용 앞 4개 문단" in second
|
||||
assert "raw `0.501`" in second and "raw `0.151`" in second
|
||||
|
||||
|
||||
def test_quick_rules_validates_custom_runtime_contract(tmp_path: Path) -> None:
|
||||
contract = json.loads(
|
||||
(ROOT / "skills/technical-doc-flow/config/runtime-contract.json").read_text(
|
||||
encoding="utf-8"
|
||||
)
|
||||
)
|
||||
contract["name"] = "wrong-name"
|
||||
path = tmp_path / "contract.json"
|
||||
path.write_text(json.dumps(contract), encoding="utf-8")
|
||||
result = run_cli(
|
||||
"build_quick_rules.py",
|
||||
"--contract",
|
||||
path,
|
||||
"--output",
|
||||
tmp_path / "quick.md",
|
||||
)
|
||||
assert result.returncode == 2
|
||||
assert "schema 위반" in result.stderr
|
||||
|
||||
|
||||
def test_release_and_path_contracts_are_current() -> None:
|
||||
result = run_cli("check_release_sync.py", "--root", ROOT)
|
||||
assert result.returncode == 0, result.stderr
|
||||
|
||||
|
||||
def test_release_check_rejects_fake_wrapper_and_unmapped_artifact(tmp_path: Path) -> None:
|
||||
copied = tmp_path / "repository"
|
||||
shutil.copytree(ROOT, copied, ignore=shutil.ignore_patterns("__pycache__", ".pytest_cache"))
|
||||
wrapper = copied / "scripts" / "split_document.py"
|
||||
wrapper.write_text(
|
||||
"from _runtime_entry import execute\nif False:\n execute('split_document.py')\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
relative_wrapper = copied / "scripts" / "lint_document.py"
|
||||
relative_wrapper.write_text(
|
||||
"from ._runtime_entry import execute\nexecute('lint_document.py')\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
(copied / "scripts" / "_runtime_entry.py").unlink()
|
||||
contract = copied / "skills" / "technical-doc-flow" / "config" / "runtime-contract.json"
|
||||
value = json.loads(contract.read_text(encoding="utf-8"))
|
||||
value["artifacts"]["always"].append("10_unmapped.json")
|
||||
contract.write_text(json.dumps(value), encoding="utf-8")
|
||||
verifier = copied / "skills" / "technical-doc-flow" / "scripts" / "verify_run.py"
|
||||
verifier.write_text(
|
||||
verifier.read_text(encoding="utf-8").replace(
|
||||
'"08_lint.json": "lint-report.schema.json"',
|
||||
'"08_lint.json": "review.schema.json"',
|
||||
),
|
||||
encoding="utf-8",
|
||||
)
|
||||
result = run_cli("check_release_sync.py", "--root", copied)
|
||||
assert result.returncode == 1
|
||||
assert "exact canonical execute template" in result.stderr
|
||||
assert "root runtime entry helper 누락" in result.stderr
|
||||
assert "wrapper --help smoke 실패" in result.stderr
|
||||
assert "매핑이 없는 runtime artifact" in result.stderr
|
||||
assert "artifact schema mapping 불일치" in result.stderr
|
||||
|
||||
|
||||
def test_release_check_validates_runtime_contract_schema(tmp_path: Path) -> None:
|
||||
copied = tmp_path / "repository"
|
||||
shutil.copytree(ROOT, copied, ignore=shutil.ignore_patterns("__pycache__", ".pytest_cache"))
|
||||
contract_path = copied / "skills" / "technical-doc-flow" / "config" / "runtime-contract.json"
|
||||
contract = json.loads(contract_path.read_text(encoding="utf-8"))
|
||||
contract["supported_document_kinds"] = ["garbage"]
|
||||
contract["artifacts"]["unexpected"] = []
|
||||
contract_path.write_text(json.dumps(contract), encoding="utf-8")
|
||||
result = run_cli("check_release_sync.py", "--root", copied)
|
||||
assert result.returncode == 1
|
||||
assert "runtime-contract.schema.json 검증 실패" in result.stderr
|
||||
|
||||
|
||||
def test_semver_is_strict_and_marketplace_matches_by_name(tmp_path: Path) -> None:
|
||||
for valid in ("0.1.0", "1.2.3-alpha.1", "1.2.3+build.01", "1.0.0-0"):
|
||||
assert SEMVER_RE.fullmatch(valid), valid
|
||||
for invalid in (
|
||||
"1.0.0-01",
|
||||
"1.0.0-..",
|
||||
"1.0.0-alpha..1",
|
||||
"1.0.0+..",
|
||||
"1١.0.0",
|
||||
"1.0.1٢",
|
||||
"1.0.0-1١",
|
||||
):
|
||||
assert not SEMVER_RE.fullmatch(invalid), invalid
|
||||
|
||||
copied = tmp_path / "repository"
|
||||
shutil.copytree(ROOT, copied, ignore=shutil.ignore_patterns("__pycache__", ".pytest_cache"))
|
||||
marketplace_path = copied / ".claude-plugin" / "marketplace.json"
|
||||
marketplace = json.loads(marketplace_path.read_text(encoding="utf-8"))
|
||||
marketplace["plugins"].insert(0, {"name": "unrelated", "version": "0.1.0"})
|
||||
target = next(item for item in marketplace["plugins"] if item["name"] == "technical-doc-flow")
|
||||
target["version"] = "9.9.9"
|
||||
marketplace_path.write_text(json.dumps(marketplace), encoding="utf-8")
|
||||
result = run_cli("check_release_sync.py", "--root", copied)
|
||||
assert result.returncode == 1
|
||||
assert "technical-doc-flow plugin='9.9.9'" in result.stderr
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,353 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import importlib.util
|
||||
import json
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
from helpers import FIXTURES, init_run, read_json, run_cli
|
||||
|
||||
|
||||
def test_write_auto_is_at_least_standard_and_sequences_are_unique(tmp_path: Path) -> None:
|
||||
brief = tmp_path / "brief.md"
|
||||
brief.write_text("짧은 새 문서 요청\n", encoding="utf-8")
|
||||
workspace = tmp_path / "runs"
|
||||
command = (
|
||||
"--brief",
|
||||
brief,
|
||||
"--kind",
|
||||
"explanation",
|
||||
"--route",
|
||||
"auto",
|
||||
"--workspace",
|
||||
workspace,
|
||||
"--date",
|
||||
"2026-07-23",
|
||||
)
|
||||
first = run_cli("init_run.py", *command)
|
||||
second = run_cli("init_run.py", *command)
|
||||
assert first.returncode == second.returncode == 0
|
||||
first_dir = Path(first.stdout.split("\t", 1)[0])
|
||||
second_dir = Path(second.stdout.split("\t", 1)[0])
|
||||
assert first_dir.name == "2026-07-23-001"
|
||||
assert second_dir.name == "2026-07-23-002"
|
||||
first_manifest = read_json(first_dir / "00_run.json")
|
||||
assert first_manifest["route_hint"] == "standard"
|
||||
assert first_manifest["omissions"] == []
|
||||
assert first_manifest["error"] is None
|
||||
assert first_manifest["history"][0]["error"] is None
|
||||
rules_path = (
|
||||
Path(__file__).resolve().parents[1]
|
||||
/ "skills"
|
||||
/ "technical-doc-flow"
|
||||
/ "config"
|
||||
/ "quality-rules.json"
|
||||
)
|
||||
assert first_manifest["rules_sha256"] == hashlib.sha256(
|
||||
rules_path.read_bytes()
|
||||
).hexdigest()
|
||||
|
||||
|
||||
def test_dangling_run_symlink_is_occupied_and_preserved(tmp_path: Path) -> None:
|
||||
brief = tmp_path / "brief.md"
|
||||
brief.write_text("짧은 새 문서 요청\n", encoding="utf-8")
|
||||
workspace = tmp_path / "runs"
|
||||
workspace.mkdir()
|
||||
missing_target = tmp_path / "missing-run-target"
|
||||
occupied = workspace / "2026-07-23-001"
|
||||
occupied.symlink_to(missing_target, target_is_directory=True)
|
||||
|
||||
result = run_cli(
|
||||
"init_run.py",
|
||||
"--brief",
|
||||
brief,
|
||||
"--kind",
|
||||
"explanation",
|
||||
"--route",
|
||||
"auto",
|
||||
"--workspace",
|
||||
workspace,
|
||||
"--date",
|
||||
"2026-07-23",
|
||||
)
|
||||
|
||||
assert result.returncode == 0, result.stderr
|
||||
run_dir = Path(result.stdout.split("\t", 1)[0])
|
||||
assert run_dir.name == "2026-07-23-002"
|
||||
assert occupied.is_symlink()
|
||||
assert occupied.readlink() == missing_target
|
||||
assert not occupied.exists()
|
||||
|
||||
|
||||
def test_revise_preserves_brief_and_draft_snapshots_and_auto_light(tmp_path: Path) -> None:
|
||||
brief = tmp_path / "brief.md"
|
||||
draft = tmp_path / "draft.md"
|
||||
source = tmp_path / "source.txt"
|
||||
brief.write_text("이 초안의 용어를 쉽게 풀어 주세요.\n", encoding="utf-8")
|
||||
draft.write_text("# 기존 문서\n\n짧은 본문입니다.\n", encoding="utf-8")
|
||||
source.write_text("근거 snapshot\n", encoding="utf-8")
|
||||
result = run_cli(
|
||||
"init_run.py",
|
||||
"--brief",
|
||||
brief,
|
||||
"--draft",
|
||||
draft,
|
||||
"--source",
|
||||
source,
|
||||
"--kind",
|
||||
"explanation",
|
||||
"--route",
|
||||
"auto",
|
||||
"--workspace",
|
||||
tmp_path / "runs",
|
||||
"--date",
|
||||
"2026-07-23",
|
||||
)
|
||||
assert result.returncode == 0, result.stderr
|
||||
run_dir = Path(result.stdout.split("\t", 1)[0])
|
||||
manifest = read_json(run_dir / "00_run.json")
|
||||
sources = read_json(run_dir / "01_sources.json")
|
||||
combined = (run_dir / "01_input.md").read_bytes()
|
||||
assert manifest["route_hint"] == "light"
|
||||
assert b"technical-doc-flow:brief:start" in combined
|
||||
assert b"technical-doc-flow:draft:start" in combined
|
||||
assert manifest["inputs"]["brief_sha256"] == hashlib.sha256(brief.read_bytes()).hexdigest()
|
||||
assert manifest["inputs"]["draft_sha256"] == hashlib.sha256(draft.read_bytes()).hexdigest()
|
||||
assert manifest["inputs"]["input_sha256"] == hashlib.sha256(combined).hexdigest()
|
||||
serialized_sources = json.dumps(sources, ensure_ascii=False, indent=2).encode() + b"\n"
|
||||
assert manifest["inputs"]["sources_manifest_sha256"] == hashlib.sha256(serialized_sources).hexdigest()
|
||||
|
||||
|
||||
def test_auto_deep_for_many_sources(tmp_path: Path) -> None:
|
||||
brief = tmp_path / "brief.md"
|
||||
draft = tmp_path / "draft.md"
|
||||
brief.write_text("정밀 검토\n", encoding="utf-8")
|
||||
draft.write_text("# 초안\n", encoding="utf-8")
|
||||
sources = []
|
||||
for index in range(9):
|
||||
path = tmp_path / f"source-{index}.txt"
|
||||
path.write_text(f"source {index}\n", encoding="utf-8")
|
||||
sources.append(path)
|
||||
result = run_cli(
|
||||
"init_run.py",
|
||||
"--brief",
|
||||
brief,
|
||||
"--draft",
|
||||
draft,
|
||||
"--source",
|
||||
*sources,
|
||||
"--kind",
|
||||
"explanation",
|
||||
"--route",
|
||||
"auto",
|
||||
"--workspace",
|
||||
tmp_path / "runs",
|
||||
"--date",
|
||||
"2026-07-23",
|
||||
)
|
||||
assert result.returncode == 0, result.stderr
|
||||
run_dir = Path(result.stdout.split("\t", 1)[0])
|
||||
assert read_json(run_dir / "00_run.json")["route_hint"] == "deep"
|
||||
|
||||
|
||||
def test_auto_route_counts_setext_headings_with_the_lint_parser(tmp_path: Path) -> None:
|
||||
brief = tmp_path / "brief.md"
|
||||
draft = tmp_path / "draft.md"
|
||||
brief.write_text("제목이 많은 기존 문서를 다듬어 주세요.\n", encoding="utf-8")
|
||||
sections = ["문서\n====\n"]
|
||||
sections.extend(
|
||||
f"절 {index}\n----\n짧은 설명입니다.\n" for index in range(1, 25)
|
||||
)
|
||||
draft.write_text("\n".join(sections), encoding="utf-8")
|
||||
result = run_cli(
|
||||
"init_run.py",
|
||||
"--brief",
|
||||
brief,
|
||||
"--draft",
|
||||
draft,
|
||||
"--kind",
|
||||
"explanation",
|
||||
"--route",
|
||||
"auto",
|
||||
"--workspace",
|
||||
tmp_path / "runs",
|
||||
"--date",
|
||||
"2026-07-23",
|
||||
)
|
||||
assert result.returncode == 0, result.stderr
|
||||
run_dir = Path(result.stdout.split("\t", 1)[0])
|
||||
manifest = read_json(run_dir / "00_run.json")
|
||||
assert manifest["route_metrics"]["total_headings"] == 25
|
||||
assert manifest["route_hint"] == "deep"
|
||||
|
||||
|
||||
def test_missing_input_is_exit_two_and_does_not_publish_run(tmp_path: Path) -> None:
|
||||
workspace = tmp_path / "runs"
|
||||
result = run_cli(
|
||||
"init_run.py",
|
||||
"--brief",
|
||||
tmp_path / "missing.md",
|
||||
"--kind",
|
||||
"explanation",
|
||||
"--workspace",
|
||||
workspace,
|
||||
)
|
||||
assert result.returncode == 2
|
||||
assert not workspace.exists()
|
||||
|
||||
|
||||
def test_review_requires_draft(tmp_path: Path) -> None:
|
||||
brief = tmp_path / "brief.md"
|
||||
brief.write_text("검토해 주세요.\n", encoding="utf-8")
|
||||
result = run_cli(
|
||||
"init_run.py",
|
||||
"--brief",
|
||||
brief,
|
||||
"--mode",
|
||||
"review",
|
||||
"--kind",
|
||||
"explanation",
|
||||
"--workspace",
|
||||
tmp_path / "runs",
|
||||
)
|
||||
assert result.returncode == 2
|
||||
|
||||
|
||||
def test_review_initializes_exact_immutable_draft_target(tmp_path: Path) -> None:
|
||||
brief = tmp_path / "brief.md"
|
||||
draft = tmp_path / "draft.md"
|
||||
brief.write_text("바이트를 바꾸지 말고 검토해 주세요.\n", encoding="utf-8")
|
||||
draft_bytes = "# 원본\r\n\r\n검토 대상 `CacheKey`입니다.\r\n".encode("utf-8")
|
||||
draft.write_bytes(draft_bytes)
|
||||
result = run_cli(
|
||||
"init_run.py",
|
||||
"--brief",
|
||||
brief,
|
||||
"--draft",
|
||||
draft,
|
||||
"--mode",
|
||||
"review",
|
||||
"--kind",
|
||||
"explanation",
|
||||
"--workspace",
|
||||
tmp_path / "runs",
|
||||
"--date",
|
||||
"2026-07-23",
|
||||
)
|
||||
assert result.returncode == 0, result.stderr
|
||||
run_dir = Path(result.stdout.split("\t", 1)[0])
|
||||
assert (run_dir / "07_draft.md").read_bytes() == draft_bytes
|
||||
assert read_json(run_dir / "00_run.json")["inputs"]["draft_sha256"] == hashlib.sha256(
|
||||
draft_bytes
|
||||
).hexdigest()
|
||||
|
||||
|
||||
def test_invalid_calendar_date_is_rejected(tmp_path: Path) -> None:
|
||||
brief = tmp_path / "brief.md"
|
||||
brief.write_text("요청\n", encoding="utf-8")
|
||||
result = run_cli(
|
||||
"init_run.py",
|
||||
"--brief",
|
||||
brief,
|
||||
"--kind",
|
||||
"explanation",
|
||||
"--workspace",
|
||||
tmp_path / "runs",
|
||||
"--date",
|
||||
"2026-02-30",
|
||||
)
|
||||
assert result.returncode == 2
|
||||
|
||||
|
||||
def test_stage_creation_failure_releases_sequence_reservation(tmp_path: Path, monkeypatch) -> None:
|
||||
canonical = (
|
||||
Path(__file__).resolve().parents[1]
|
||||
/ "skills"
|
||||
/ "technical-doc-flow"
|
||||
/ "scripts"
|
||||
/ "init_run.py"
|
||||
)
|
||||
sys.path.insert(0, str(canonical.parent))
|
||||
spec = importlib.util.spec_from_file_location("canonical_init_run", canonical)
|
||||
assert spec and spec.loader
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(module)
|
||||
brief = tmp_path / "brief.md"
|
||||
brief.write_text("요청\n", encoding="utf-8")
|
||||
workspace = tmp_path / "runs"
|
||||
args = module.parser().parse_args(
|
||||
[
|
||||
"--brief",
|
||||
str(brief),
|
||||
"--kind",
|
||||
"explanation",
|
||||
"--workspace",
|
||||
str(workspace),
|
||||
"--date",
|
||||
"2026-07-23",
|
||||
]
|
||||
)
|
||||
|
||||
def fail_mkdtemp(*_args, **_kwargs):
|
||||
raise OSError("simulated stage failure")
|
||||
|
||||
monkeypatch.setattr(module.tempfile, "mkdtemp", fail_mkdtemp)
|
||||
try:
|
||||
module.create_run(args)
|
||||
except OSError:
|
||||
pass
|
||||
else:
|
||||
raise AssertionError("stage failure must propagate")
|
||||
assert not list(workspace.glob("*.reserve"))
|
||||
|
||||
|
||||
def test_publish_race_preserves_empty_target_and_retries(tmp_path: Path, monkeypatch) -> None:
|
||||
canonical = (
|
||||
Path(__file__).resolve().parents[1]
|
||||
/ "skills"
|
||||
/ "technical-doc-flow"
|
||||
/ "scripts"
|
||||
/ "init_run.py"
|
||||
)
|
||||
sys.path.insert(0, str(canonical.parent))
|
||||
spec = importlib.util.spec_from_file_location("canonical_init_run_publish_race", canonical)
|
||||
assert spec and spec.loader
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(module)
|
||||
brief = tmp_path / "brief.md"
|
||||
brief.write_text("요청\n", encoding="utf-8")
|
||||
workspace = tmp_path / "runs"
|
||||
args = module.parser().parse_args(
|
||||
[
|
||||
"--brief",
|
||||
str(brief),
|
||||
"--kind",
|
||||
"explanation",
|
||||
"--workspace",
|
||||
str(workspace),
|
||||
"--date",
|
||||
"2026-07-23",
|
||||
]
|
||||
)
|
||||
original_publish = module.publish_directory_noreplace
|
||||
raced_target = workspace / "2026-07-23-001"
|
||||
raced_inode: int | None = None
|
||||
|
||||
def create_target_during_publish(source: Path, destination: Path) -> None:
|
||||
nonlocal raced_inode
|
||||
if destination == raced_target:
|
||||
destination.mkdir()
|
||||
raced_inode = destination.stat().st_ino
|
||||
original_publish(source, destination)
|
||||
|
||||
monkeypatch.setattr(module, "publish_directory_noreplace", create_target_during_publish)
|
||||
run_dir, _, _ = module.create_run(args)
|
||||
|
||||
assert run_dir.name == "2026-07-23-002"
|
||||
assert raced_inode is not None
|
||||
assert raced_target.is_dir()
|
||||
assert raced_target.stat().st_ino == raced_inode
|
||||
assert list(raced_target.iterdir()) == []
|
||||
assert not list(workspace.glob(".*.reserve"))
|
||||
assert not list(workspace.glob(".2026-07-23-001-*"))
|
||||
@@ -0,0 +1,94 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
from claridoc.lint import lint_document
|
||||
from claridoc.models import ProviderSpec, Severity, SourcePack
|
||||
from claridoc.prompts import drafting_prompt
|
||||
from claridoc.providers.base import ProviderRequest
|
||||
from claridoc.providers.mock import MockProvider
|
||||
from claridoc.structures import create_outline
|
||||
from tests.helpers import make_brief, make_sources
|
||||
|
||||
|
||||
class LintTests(unittest.TestCase):
|
||||
def test_mock_document_meets_structural_gate(self) -> None:
|
||||
brief = make_brief()
|
||||
sources = make_sources()
|
||||
outline = create_outline(brief, sources)
|
||||
provider = MockProvider(ProviderSpec(provider="mock"))
|
||||
response = provider.generate(ProviderRequest("draft", drafting_prompt(brief, outline, sources), Path.cwd()))
|
||||
report = lint_document(response.text, brief, outline, sources)
|
||||
material = [issue for issue in report.issues if issue.severity in {Severity.BLOCKER, Severity.ERROR}]
|
||||
self.assertEqual(material, [])
|
||||
self.assertGreaterEqual(report.score, 75)
|
||||
|
||||
def test_unclosed_fence_and_destructive_command_are_blockers(self) -> None:
|
||||
brief = make_brief()
|
||||
sources = make_sources()
|
||||
outline = create_outline(brief, sources)
|
||||
text = "# Wrong\n\n```bash\nrm -rf /tmp/example\n"
|
||||
report = lint_document(text, brief, outline, sources)
|
||||
codes = {issue.code for issue in report.issues if issue.severity == Severity.BLOCKER}
|
||||
self.assertIn("MD001", codes)
|
||||
self.assertIn("SAFE001", codes)
|
||||
|
||||
def test_unknown_source_marker_is_error(self) -> None:
|
||||
brief = make_brief()
|
||||
sources = make_sources()
|
||||
outline = create_outline(brief, sources)
|
||||
provider = MockProvider(ProviderSpec(provider="mock"))
|
||||
text = provider.generate(ProviderRequest("draft", drafting_prompt(brief, outline, sources), Path.cwd())).text
|
||||
report = lint_document(text.replace("[S1]", "[S404]"), brief, outline, sources)
|
||||
self.assertIn("EVD001", {issue.code for issue in report.issues})
|
||||
|
||||
def test_non_s_prefixed_source_id_is_recognized(self) -> None:
|
||||
brief = make_brief()
|
||||
sources = SourcePack.from_dict({
|
||||
"sources": [{
|
||||
"id": "RFC9110",
|
||||
"title": "HTTP Semantics",
|
||||
"url": "https://example.com/rfc9110",
|
||||
"facts": ["The example fact is bounded."],
|
||||
}]
|
||||
})
|
||||
outline = create_outline(brief, sources)
|
||||
provider = MockProvider(ProviderSpec(provider="mock"))
|
||||
text = provider.generate(ProviderRequest("draft", drafting_prompt(brief, outline, sources), Path.cwd())).text
|
||||
report = lint_document(text, brief, outline, sources)
|
||||
codes = {issue.code for issue in report.issues}
|
||||
self.assertNotIn("EVD001", codes)
|
||||
self.assertNotIn("EVD003", codes)
|
||||
|
||||
def test_required_h2_must_appear_exactly_once(self) -> None:
|
||||
brief = make_brief()
|
||||
sources = make_sources()
|
||||
outline = create_outline(brief, sources)
|
||||
provider = MockProvider(ProviderSpec(provider="mock"))
|
||||
text = provider.generate(ProviderRequest("draft", drafting_prompt(brief, outline, sources), Path.cwd())).text
|
||||
text += f"\n## {outline.sections[0].title}\n\nDuplicate section.\n"
|
||||
report = lint_document(text, brief, outline, sources)
|
||||
self.assertIn("STR009", {issue.code for issue in report.issues if issue.severity == Severity.ERROR})
|
||||
|
||||
def test_destructive_command_requires_all_safety_controls(self) -> None:
|
||||
brief = make_brief()
|
||||
sources = make_sources()
|
||||
outline = create_outline(brief, sources)
|
||||
warning_only = "# A precise technical document\n\nWarning: this is destructive.\n\n```bash\nrm -rf /tmp/example\n```\n"
|
||||
report = lint_document(warning_only, brief, outline, sources)
|
||||
self.assertIn("SAFE001", {issue.code for issue in report.issues})
|
||||
|
||||
controlled = (
|
||||
"# A precise technical document\n\n"
|
||||
"Warning: this removes the test directory. Create a backup checkpoint first. "
|
||||
"Expected result: the directory is absent; verify with a read-only listing. "
|
||||
"Rollback by restoring the backup.\n\n"
|
||||
"```bash\nrm -rf /tmp/example\n```\n"
|
||||
)
|
||||
controlled_report = lint_document(controlled, brief, outline, sources)
|
||||
self.assertNotIn("SAFE001", {issue.code for issue in controlled_report.issues})
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -1,796 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from helpers import FIXTURES, read_json, run_cli, write_json
|
||||
|
||||
|
||||
CANONICAL_SCRIPTS = (
|
||||
Path(__file__).resolve().parents[1] / "skills" / "technical-doc-flow" / "scripts"
|
||||
)
|
||||
sys.path.insert(0, str(CANONICAL_SCRIPTS))
|
||||
import lint_document as lint_document_module # noqa: E402
|
||||
from lint_document import ( # noqa: E402
|
||||
bounded_change_rate,
|
||||
github_slug,
|
||||
markdown_inline_links,
|
||||
mask_markdown_metadata,
|
||||
parse_headings,
|
||||
protected_items,
|
||||
)
|
||||
|
||||
|
||||
def lint_fixture(tmp_path: Path, name: str, *extra: object):
|
||||
base = FIXTURES / name
|
||||
output = tmp_path / f"{name}-lint.json"
|
||||
result = run_cli(
|
||||
"lint_document.py",
|
||||
"--document",
|
||||
base / "document.md",
|
||||
"--logic-map",
|
||||
base / "logic-map.json",
|
||||
"--term-ledger",
|
||||
base / "term-ledger.json",
|
||||
"--reader-contract",
|
||||
base / "reader-contract.json",
|
||||
"--output",
|
||||
output,
|
||||
*extra,
|
||||
)
|
||||
return result, read_json(output)
|
||||
|
||||
|
||||
def lint_main_args(output: Path, *, document: Path | None = None) -> list[str]:
|
||||
base = FIXTURES / "good"
|
||||
return [
|
||||
"--document",
|
||||
str(document or base / "document.md"),
|
||||
"--logic-map",
|
||||
str(base / "logic-map.json"),
|
||||
"--term-ledger",
|
||||
str(base / "term-ledger.json"),
|
||||
"--reader-contract",
|
||||
str(base / "reader-contract.json"),
|
||||
"--output",
|
||||
str(output),
|
||||
]
|
||||
|
||||
|
||||
def test_good_golden_passes_without_findings(tmp_path: Path) -> None:
|
||||
result, report = lint_fixture(tmp_path, "good")
|
||||
assert result.returncode == 0, result.stderr
|
||||
assert report["verdict"] == "pass"
|
||||
assert report["findings"] == []
|
||||
rules_path = CANONICAL_SCRIPTS.parent / "config" / "quality-rules.json"
|
||||
assert report["rules_sha256"] == hashlib.sha256(
|
||||
rules_path.read_bytes()
|
||||
).hexdigest()
|
||||
|
||||
|
||||
def test_bad_golden_covers_all_declared_failure_modes(tmp_path: Path) -> None:
|
||||
result, report = lint_fixture(tmp_path, "bad")
|
||||
expected = set(json.loads((FIXTURES / "bad" / "expected-rule-ids.json").read_text()))
|
||||
actual = {finding["rule_id"] for finding in report["findings"]}
|
||||
assert result.returncode == 1
|
||||
assert expected <= actual
|
||||
assert len(expected) >= 8
|
||||
|
||||
|
||||
def test_malformed_alias_returns_input_error_report(tmp_path: Path) -> None:
|
||||
ledger = read_json(FIXTURES / "good" / "term-ledger.json")
|
||||
ledger["terms"][0]["aliases"] = [123]
|
||||
ledger_path = tmp_path / "ledger.json"
|
||||
output = tmp_path / "lint.json"
|
||||
write_json(ledger_path, ledger)
|
||||
result = run_cli(
|
||||
"lint_document.py",
|
||||
"--document",
|
||||
FIXTURES / "good" / "document.md",
|
||||
"--logic-map",
|
||||
FIXTURES / "good" / "logic-map.json",
|
||||
"--term-ledger",
|
||||
ledger_path,
|
||||
"--reader-contract",
|
||||
FIXTURES / "good" / "reader-contract.json",
|
||||
"--output",
|
||||
output,
|
||||
)
|
||||
assert result.returncode == 2
|
||||
assert read_json(output)["verdict"] == "input_error"
|
||||
|
||||
|
||||
def test_schema_invalid_contract_cannot_receive_a_pass_report(tmp_path: Path) -> None:
|
||||
reader = read_json(FIXTURES / "good" / "reader-contract.json")
|
||||
reader["document_kind"] = "garbage"
|
||||
reader_path = tmp_path / "reader-contract.json"
|
||||
output = tmp_path / "lint.json"
|
||||
write_json(reader_path, reader)
|
||||
result = run_cli(
|
||||
"lint_document.py",
|
||||
"--document",
|
||||
FIXTURES / "good" / "document.md",
|
||||
"--logic-map",
|
||||
FIXTURES / "good" / "logic-map.json",
|
||||
"--term-ledger",
|
||||
FIXTURES / "good" / "term-ledger.json",
|
||||
"--reader-contract",
|
||||
reader_path,
|
||||
"--output",
|
||||
output,
|
||||
)
|
||||
assert result.returncode == 2
|
||||
assert read_json(output)["verdict"] == "input_error"
|
||||
assert "schema 위반" in read_json(output)["findings"][0]["message"]
|
||||
|
||||
|
||||
def test_fence_with_trailing_text_is_not_a_closer(tmp_path: Path) -> None:
|
||||
document = (FIXTURES / "good" / "document.md").read_text(encoding="utf-8")
|
||||
insertion = "```text\n```anything\n# 코드 안 가짜 제목\n```\n\n"
|
||||
document = document.replace("## 문제", insertion + "## 문제")
|
||||
path = tmp_path / "document.md"
|
||||
path.write_text(document, encoding="utf-8")
|
||||
output = tmp_path / "lint.json"
|
||||
result = run_cli(
|
||||
"lint_document.py",
|
||||
"--document",
|
||||
path,
|
||||
"--logic-map",
|
||||
FIXTURES / "good" / "logic-map.json",
|
||||
"--term-ledger",
|
||||
FIXTURES / "good" / "term-ledger.json",
|
||||
"--reader-contract",
|
||||
FIXTURES / "good" / "reader-contract.json",
|
||||
"--output",
|
||||
output,
|
||||
)
|
||||
assert result.returncode == 0, read_json(output)["findings"]
|
||||
|
||||
|
||||
def test_setext_heading_is_supported_and_comment_heading_is_ignored(tmp_path: Path) -> None:
|
||||
document = (FIXTURES / "good" / "document.md").read_text(encoding="utf-8")
|
||||
document = document.replace(
|
||||
"# 캐시 실패를 줄이는 설명",
|
||||
"캐시 실패를 줄이는 설명\n========================",
|
||||
)
|
||||
document = document.replace("## 문제", "<!--\n# 주석 속 가짜 제목\n-->\n\n## 문제")
|
||||
path = tmp_path / "document.md"
|
||||
path.write_text(document, encoding="utf-8")
|
||||
output = tmp_path / "lint.json"
|
||||
result = run_cli(
|
||||
"lint_document.py",
|
||||
"--document",
|
||||
path,
|
||||
"--logic-map",
|
||||
FIXTURES / "good" / "logic-map.json",
|
||||
"--term-ledger",
|
||||
FIXTURES / "good" / "term-ledger.json",
|
||||
"--reader-contract",
|
||||
FIXTURES / "good" / "reader-contract.json",
|
||||
"--output",
|
||||
output,
|
||||
)
|
||||
assert result.returncode == 0, read_json(output)["findings"]
|
||||
|
||||
|
||||
def test_list_items_are_not_counted_as_one_oversized_sentence_paragraph(
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
document = (FIXTURES / "good" / "document.md").read_text(encoding="utf-8")
|
||||
document += "\n" + "\n".join(
|
||||
f"{index}. 독자가 확인할 항목입니다." for index in range(1, 9)
|
||||
)
|
||||
path = tmp_path / "document.md"
|
||||
path.write_text(document, encoding="utf-8")
|
||||
output = tmp_path / "lint.json"
|
||||
result = run_cli(
|
||||
"lint_document.py",
|
||||
"--document",
|
||||
path,
|
||||
"--logic-map",
|
||||
FIXTURES / "good" / "logic-map.json",
|
||||
"--term-ledger",
|
||||
FIXTURES / "good" / "term-ledger.json",
|
||||
"--reader-contract",
|
||||
FIXTURES / "good" / "reader-contract.json",
|
||||
"--output",
|
||||
output,
|
||||
)
|
||||
assert result.returncode == 0
|
||||
assert "DOC-P002" not in {item["rule_id"] for item in read_json(output)["findings"]}
|
||||
|
||||
|
||||
def test_fidelity_uses_typed_exact_multisets(tmp_path: Path) -> None:
|
||||
baseline = tmp_path / "baseline.md"
|
||||
baseline.write_text(
|
||||
'# 기준\n\n```text\nalpha\n```\n\n`foo` https://example.com/api 500ms "정확한 인용"\n',
|
||||
encoding="utf-8",
|
||||
)
|
||||
document = (FIXTURES / "good" / "document.md").read_text(encoding="utf-8")
|
||||
document += '\n`foobar` https://example.com/api-v2 5000ms "정확한 인용문"\n'
|
||||
document_path = tmp_path / "document.md"
|
||||
document_path.write_text(document, encoding="utf-8")
|
||||
output = tmp_path / "lint.json"
|
||||
result = run_cli(
|
||||
"lint_document.py",
|
||||
"--document",
|
||||
document_path,
|
||||
"--logic-map",
|
||||
FIXTURES / "good" / "logic-map.json",
|
||||
"--term-ledger",
|
||||
FIXTURES / "good" / "term-ledger.json",
|
||||
"--reader-contract",
|
||||
FIXTURES / "good" / "reader-contract.json",
|
||||
"--baseline",
|
||||
baseline,
|
||||
"--output",
|
||||
output,
|
||||
)
|
||||
report = read_json(output)
|
||||
rule_ids = {item["rule_id"] for item in report["findings"]}
|
||||
assert result.returncode == 1
|
||||
assert {"DOC-F001", "DOC-F002", "DOC-F003", "DOC-F004", "DOC-F005"} <= rule_ids
|
||||
assert report["fidelity"]["missing"] == 5
|
||||
|
||||
|
||||
def test_fidelity_binds_values_to_nearby_meaning_instead_of_only_multisets(
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
baseline = tmp_path / "baseline.md"
|
||||
baseline.write_text(
|
||||
"# 기준\n\n"
|
||||
"A 코드 `alpha`, B 코드 `beta`.\n"
|
||||
"A 링크 https://example.com/a, B 링크 https://example.com/b.\n"
|
||||
"A 지연 10ms, B 지연 20ms.\n"
|
||||
'A 문구 "첫 번째", B 문구 "두 번째".\n',
|
||||
encoding="utf-8",
|
||||
)
|
||||
document = (FIXTURES / "good" / "document.md").read_text(encoding="utf-8")
|
||||
document += (
|
||||
"\nA 코드 `beta`, B 코드 `alpha`.\n"
|
||||
"A 링크 https://example.com/b, B 링크 https://example.com/a.\n"
|
||||
"A 지연 20ms, B 지연 10ms.\n"
|
||||
'A 문구 "두 번째", B 문구 "첫 번째".\n'
|
||||
)
|
||||
document_path = tmp_path / "document.md"
|
||||
document_path.write_text(document, encoding="utf-8")
|
||||
output = tmp_path / "lint.json"
|
||||
result = run_cli(
|
||||
"lint_document.py",
|
||||
"--document",
|
||||
document_path,
|
||||
"--logic-map",
|
||||
FIXTURES / "good" / "logic-map.json",
|
||||
"--term-ledger",
|
||||
FIXTURES / "good" / "term-ledger.json",
|
||||
"--reader-contract",
|
||||
FIXTURES / "good" / "reader-contract.json",
|
||||
"--baseline",
|
||||
baseline,
|
||||
"--output",
|
||||
output,
|
||||
)
|
||||
report = read_json(output)
|
||||
rule_ids = {item["rule_id"] for item in report["findings"]}
|
||||
assert result.returncode == 1
|
||||
assert {"DOC-F002", "DOC-F003", "DOC-F004", "DOC-F005"} <= rule_ids
|
||||
|
||||
|
||||
def test_number_units_are_protected_when_joined_to_korean_particles() -> None:
|
||||
inventory = protected_items("약10ms를 기다리고 20초동안 관찰하며 30msfoo는 식별자로 둔다.")
|
||||
assert inventory["number_unit"] == ["약10ms", "20초"]
|
||||
|
||||
|
||||
def test_both_endpoint_unit_ranges_are_single_ordered_protected_values() -> None:
|
||||
inventory = protected_items("지연 5ms-10ms, 비율 5%-10%, 비호환 5ms-10s.")
|
||||
assert inventory["number_range"] == ["5ms-10ms", "5%-10%"]
|
||||
assert inventory["number_unit"] == ["5ms", "10s"]
|
||||
|
||||
|
||||
def test_reader_visible_multiline_and_entity_quotes_exclude_metadata_and_code() -> None:
|
||||
inventory = protected_items(
|
||||
'[링크](dest "metadata") `"inline"`\n'
|
||||
'<pre>\n"raw code"\n</pre>\n'
|
||||
'보이는 "첫 줄\n둘째 줄"과 '
|
||||
'“굽은 인용”이다.\n'
|
||||
)
|
||||
assert inventory["quote"] == ['"첫 줄\n둘째 줄"', '“굽은 인용”']
|
||||
|
||||
|
||||
@pytest.mark.parametrize("thematic_break", ["_ _ _", "* * *"])
|
||||
def test_spaced_thematic_break_terminates_lazy_blockquote_inventory(
|
||||
thematic_break: str,
|
||||
) -> None:
|
||||
inventory = protected_items(
|
||||
f"> 첫 줄\n게으른 이어쓰기\n{thematic_break}\n분리된 본문\n"
|
||||
)
|
||||
assert inventory["quote"] == ["첫 줄\n게으른 이어쓰기"]
|
||||
assert protected_items(
|
||||
f'"첫 줄\n{thematic_break}\n분리된 둘째 줄"'
|
||||
)["quote"] == []
|
||||
|
||||
|
||||
def test_common_absolute_uri_schemes_are_protected() -> None:
|
||||
uris = [
|
||||
"http://ApiHost.test/HttpToken",
|
||||
"https://ApiHost.test/HttpsToken",
|
||||
"ftp://FileHost.test/FtpToken",
|
||||
"ftps://FileHost.test/FtpsToken",
|
||||
"file:///tmp/FileToken",
|
||||
"mailto:UserToken@example.com",
|
||||
"ssh://GitHost.test/SshToken",
|
||||
"git://GitHost.test/GitToken",
|
||||
]
|
||||
assert protected_items("\n".join(uris))["url"] == uris
|
||||
|
||||
|
||||
def test_inline_link_parser_always_progresses_and_supports_multiline_title() -> None:
|
||||
assert markdown_inline_links("[x](dest") == []
|
||||
links = markdown_inline_links('[go](#missing "long\n title")')
|
||||
assert len(links) == 1
|
||||
assert links[0][2:4] == (5, 13)
|
||||
assert mask_markdown_metadata("[x\n\ny](TODO)") == "[x\n\ny](TODO)"
|
||||
|
||||
|
||||
def test_heading_entities_require_semicolon_and_slugs_are_globally_unique() -> None:
|
||||
assert github_slug("A © B") == "a-copy-b"
|
||||
assert github_slug("A © B") == "a-b"
|
||||
assert [item.slug for item in parse_headings("## foo\n## foo-1\n## foo\n")] == [
|
||||
"foo",
|
||||
"foo-1",
|
||||
"foo-2",
|
||||
]
|
||||
|
||||
|
||||
def test_multiline_inline_code_allows_non_interrupting_ordered_marker() -> None:
|
||||
inventory = protected_items("text `a\n2. TODO\nc` tail")
|
||||
assert inventory["inline_code"] == ["a 2. TODO c"]
|
||||
|
||||
|
||||
def test_finalizer_change_rate_detects_nonlocal_reorder_and_is_linear_enough() -> None:
|
||||
paragraphs = [f"문단 {index}에는 고유한 설명과 값 {index}가 있다." for index in range(5000)]
|
||||
before = "\n\n".join(paragraphs)
|
||||
after = "\n\n".join(reversed(paragraphs))
|
||||
started = time.monotonic()
|
||||
rate = bounded_change_rate(before, after)
|
||||
elapsed = time.monotonic() - started
|
||||
assert rate > 0.15
|
||||
assert elapsed < 5.0
|
||||
|
||||
|
||||
def test_finalizer_change_rate_allows_one_local_word_edit() -> None:
|
||||
before = "이 문서는 캐시 장애의 원인과 검증 범위를 독자에게 차례로 설명한다."
|
||||
after = "이 문서는 캐시 장애의 원인과 검증 한계를 독자에게 차례로 설명한다."
|
||||
assert bounded_change_rate(before, after) <= 0.15
|
||||
|
||||
|
||||
def test_finalizer_change_rate_detects_reorder_even_when_every_paragraph_is_edited() -> None:
|
||||
paragraphs = [
|
||||
f"문단 {index}은 캐시 동작 {index}와 고유한 검증 범위를 설명한다."
|
||||
for index in range(120)
|
||||
]
|
||||
before = "\n\n".join(paragraphs)
|
||||
after = "\n\n".join(f"{item} 보충" for item in reversed(paragraphs))
|
||||
assert bounded_change_rate(before, after) > 0.15
|
||||
|
||||
|
||||
def test_draft_baseline_hash_and_change_rate_are_reported(tmp_path: Path) -> None:
|
||||
draft = tmp_path / "draft.md"
|
||||
draft.write_text("# 완전히 다른 초안\n\n" + "다른 문장. " * 100, encoding="utf-8")
|
||||
result, report = lint_fixture(tmp_path, "good", "--draft-baseline", draft)
|
||||
assert result.returncode == 1
|
||||
assert "FNL-001" in {item["rule_id"] for item in report["findings"]}
|
||||
assert report["fidelity"]["draft_baseline"]["sha256"]
|
||||
assert report["fidelity"]["finalization_change_rate"] > 0.15
|
||||
|
||||
|
||||
def test_unregistered_candidate_is_a_default_hard_gate(tmp_path: Path) -> None:
|
||||
document = (FIXTURES / "good" / "document.md").read_text(encoding="utf-8")
|
||||
document += "\n등록하지 않은 NewCacheMode 후보가 있다.\n"
|
||||
path = tmp_path / "document.md"
|
||||
path.write_text(document, encoding="utf-8")
|
||||
output = tmp_path / "lint.json"
|
||||
common = (
|
||||
"--document",
|
||||
path,
|
||||
"--logic-map",
|
||||
FIXTURES / "good" / "logic-map.json",
|
||||
"--term-ledger",
|
||||
FIXTURES / "good" / "term-ledger.json",
|
||||
"--reader-contract",
|
||||
FIXTURES / "good" / "reader-contract.json",
|
||||
"--output",
|
||||
output,
|
||||
)
|
||||
assert run_cli("lint_document.py", *common).returncode == 1
|
||||
assert "DOC-T007" in {
|
||||
item["rule_id"] for item in read_json(output)["findings"]
|
||||
}
|
||||
|
||||
|
||||
def test_fail_on_warning_changes_exit_code(tmp_path: Path) -> None:
|
||||
document = (FIXTURES / "good" / "document.md").read_text(encoding="utf-8")
|
||||
document += "\n\n" + ("긴" * 901) + "\n"
|
||||
path = tmp_path / "document.md"
|
||||
path.write_text(document, encoding="utf-8")
|
||||
output = tmp_path / "lint.json"
|
||||
common = (
|
||||
"--document",
|
||||
path,
|
||||
"--logic-map",
|
||||
FIXTURES / "good" / "logic-map.json",
|
||||
"--term-ledger",
|
||||
FIXTURES / "good" / "term-ledger.json",
|
||||
"--reader-contract",
|
||||
FIXTURES / "good" / "reader-contract.json",
|
||||
"--output",
|
||||
output,
|
||||
)
|
||||
assert run_cli("lint_document.py", *common).returncode == 0
|
||||
assert "DOC-P001" in {
|
||||
item["rule_id"] for item in read_json(output)["findings"]
|
||||
}
|
||||
assert run_cli("lint_document.py", *common, "--fail-on", "warning").returncode == 1
|
||||
|
||||
|
||||
def test_definition_window_threshold_is_actually_enforced(tmp_path: Path) -> None:
|
||||
rules_path = (
|
||||
Path(__file__).resolve().parents[1]
|
||||
/ "skills"
|
||||
/ "technical-doc-flow"
|
||||
/ "config"
|
||||
/ "quality-rules.json"
|
||||
)
|
||||
rules = read_json(rules_path)
|
||||
rules["thresholds"]["term"]["definition_window_chars"] = 10
|
||||
strict_rules = tmp_path / "quality-rules.json"
|
||||
write_json(strict_rules, rules)
|
||||
result, report = lint_fixture(tmp_path, "good", "--rules", strict_rules)
|
||||
assert result.returncode == 1
|
||||
assert "DOC-T001" in {item["rule_id"] for item in report["findings"]}
|
||||
|
||||
|
||||
def test_definition_window_also_accepts_plain_definition_after_first_name(
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
document = (FIXTURES / "good" / "document.md").read_text(encoding="utf-8")
|
||||
document = document.replace(
|
||||
"같은 요청을 여러 번 처리해도 결과가 한 번 처리한 것과 같게 되는 성질을 멱등성이라고 한다.",
|
||||
"멱등성이라는 이름을 쓴다. 같은 요청을 여러 번 처리해도 결과가 한 번 처리한 것과 같게 되는 성질을 뜻한다.",
|
||||
)
|
||||
ledger = read_json(FIXTURES / "good" / "term-ledger.json")
|
||||
ledger["terms"][0]["first_use"] = "멱등성이라는 이름을 쓴다."
|
||||
document_path = tmp_path / "document.md"
|
||||
ledger_path = tmp_path / "term-ledger.json"
|
||||
output = tmp_path / "lint.json"
|
||||
document_path.write_text(document, encoding="utf-8")
|
||||
write_json(ledger_path, ledger)
|
||||
result = run_cli(
|
||||
"lint_document.py",
|
||||
"--document",
|
||||
document_path,
|
||||
"--logic-map",
|
||||
FIXTURES / "good" / "logic-map.json",
|
||||
"--term-ledger",
|
||||
ledger_path,
|
||||
"--reader-contract",
|
||||
FIXTURES / "good" / "reader-contract.json",
|
||||
"--output",
|
||||
output,
|
||||
)
|
||||
assert result.returncode == 0, read_json(output)["findings"]
|
||||
|
||||
|
||||
def test_hidden_html_comment_cannot_satisfy_term_first_use(tmp_path: Path) -> None:
|
||||
document = (FIXTURES / "good" / "document.md").read_text(encoding="utf-8")
|
||||
first_use = (
|
||||
"같은 요청을 여러 번 처리해도 결과가 한 번 처리한 것과 같게 되는 성질을 "
|
||||
"멱등성이라고 한다."
|
||||
)
|
||||
document = document.replace(
|
||||
first_use,
|
||||
f"<!-- {first_use} -->\n\n멱등성을 사용한다.",
|
||||
)
|
||||
path = tmp_path / "document.md"
|
||||
path.write_text(document, encoding="utf-8")
|
||||
output = tmp_path / "lint.json"
|
||||
result = run_cli(
|
||||
"lint_document.py",
|
||||
"--document",
|
||||
path,
|
||||
"--logic-map",
|
||||
FIXTURES / "good" / "logic-map.json",
|
||||
"--term-ledger",
|
||||
FIXTURES / "good" / "term-ledger.json",
|
||||
"--reader-contract",
|
||||
FIXTURES / "good" / "reader-contract.json",
|
||||
"--output",
|
||||
output,
|
||||
)
|
||||
assert result.returncode == 1
|
||||
assert "DOC-T001" in {
|
||||
item["rule_id"] for item in read_json(output)["findings"]
|
||||
}
|
||||
|
||||
|
||||
def test_ledger_term_must_actually_appear_and_first_use_must_name_it(
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
document = (FIXTURES / "good" / "document.md").read_text(encoding="utf-8")
|
||||
original = (
|
||||
"같은 요청을 여러 번 처리해도 결과가 한 번 처리한 것과 같게 되는 성질을 "
|
||||
"멱등성이라고 한다."
|
||||
)
|
||||
generic = "같은 요청을 여러 번 처리해도 결과가 한 번 처리한 것과 같게 되는 성질을 설명한다."
|
||||
document = document.replace(original, generic).replace("멱등성", "같은 결과 성질")
|
||||
ledger = read_json(FIXTURES / "good" / "term-ledger.json")
|
||||
ledger["terms"][0]["first_use"] = generic
|
||||
document_path = tmp_path / "document.md"
|
||||
ledger_path = tmp_path / "term-ledger.json"
|
||||
output = tmp_path / "lint.json"
|
||||
document_path.write_text(document, encoding="utf-8")
|
||||
write_json(ledger_path, ledger)
|
||||
result = run_cli(
|
||||
"lint_document.py",
|
||||
"--document",
|
||||
document_path,
|
||||
"--logic-map",
|
||||
FIXTURES / "good" / "logic-map.json",
|
||||
"--term-ledger",
|
||||
ledger_path,
|
||||
"--reader-contract",
|
||||
FIXTURES / "good" / "reader-contract.json",
|
||||
"--output",
|
||||
output,
|
||||
)
|
||||
assert result.returncode == 1
|
||||
messages = [
|
||||
item["message"]
|
||||
for item in read_json(output)["findings"]
|
||||
if item["rule_id"] == "DOC-T001"
|
||||
]
|
||||
assert any("실제로 등장하지" in message for message in messages)
|
||||
assert any("first_use에 정식 용어" in message for message in messages)
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"protected_name",
|
||||
["document.md", "logic-map.json", "term-ledger.json", "reader-contract.json"],
|
||||
)
|
||||
def test_lint_output_cannot_overwrite_inputs(tmp_path: Path, protected_name: str) -> None:
|
||||
base = FIXTURES / "good"
|
||||
protected = tmp_path / protected_name
|
||||
shutil.copyfile(base / protected_name, protected)
|
||||
before = protected.read_bytes()
|
||||
paths = {
|
||||
"document.md": protected if protected_name == "document.md" else base / "document.md",
|
||||
"logic-map.json": protected if protected_name == "logic-map.json" else base / "logic-map.json",
|
||||
"term-ledger.json": protected if protected_name == "term-ledger.json" else base / "term-ledger.json",
|
||||
"reader-contract.json": protected if protected_name == "reader-contract.json" else base / "reader-contract.json",
|
||||
}
|
||||
result = run_cli(
|
||||
"lint_document.py",
|
||||
"--document",
|
||||
paths["document.md"],
|
||||
"--logic-map",
|
||||
paths["logic-map.json"],
|
||||
"--term-ledger",
|
||||
paths["term-ledger.json"],
|
||||
"--reader-contract",
|
||||
paths["reader-contract.json"],
|
||||
"--output",
|
||||
protected,
|
||||
)
|
||||
assert result.returncode == 2
|
||||
assert protected.read_bytes() == before
|
||||
|
||||
|
||||
def test_lint_existing_output_requires_same_tool_ownership(tmp_path: Path) -> None:
|
||||
output = tmp_path / "report.json"
|
||||
output.write_text('{"tool":"verify_run","sentinel":true}\n', encoding="utf-8")
|
||||
before = output.read_bytes()
|
||||
result = run_cli(
|
||||
"lint_document.py",
|
||||
"--document",
|
||||
FIXTURES / "good" / "document.md",
|
||||
"--logic-map",
|
||||
FIXTURES / "good" / "logic-map.json",
|
||||
"--term-ledger",
|
||||
FIXTURES / "good" / "term-ledger.json",
|
||||
"--reader-contract",
|
||||
FIXTURES / "good" / "reader-contract.json",
|
||||
"--output",
|
||||
output,
|
||||
)
|
||||
assert result.returncode == 2
|
||||
assert output.read_bytes() == before
|
||||
|
||||
|
||||
def test_lint_malformed_same_tool_output_is_not_overwritten(tmp_path: Path) -> None:
|
||||
output = tmp_path / "report.json"
|
||||
output.write_text('{"tool":"lint_document","sentinel":true}\n', encoding="utf-8")
|
||||
before = output.read_bytes()
|
||||
result = run_cli(
|
||||
"lint_document.py",
|
||||
"--document",
|
||||
FIXTURES / "good" / "document.md",
|
||||
"--logic-map",
|
||||
FIXTURES / "good" / "logic-map.json",
|
||||
"--term-ledger",
|
||||
FIXTURES / "good" / "term-ledger.json",
|
||||
"--reader-contract",
|
||||
FIXTURES / "good" / "reader-contract.json",
|
||||
"--output",
|
||||
output,
|
||||
)
|
||||
assert result.returncode == 2
|
||||
assert output.read_bytes() == before
|
||||
|
||||
|
||||
@pytest.mark.parametrize("raced_entry", ["file", "symlink", "hardlink"])
|
||||
def test_lint_publish_rejects_entry_created_after_preflight(
|
||||
tmp_path: Path, monkeypatch: pytest.MonkeyPatch, raced_entry: str
|
||||
) -> None:
|
||||
output = tmp_path / "report.json"
|
||||
protected = tmp_path / "document.md"
|
||||
shutil.copyfile(FIXTURES / "good" / "document.md", protected)
|
||||
protected_before = protected.read_bytes()
|
||||
original_publish = lint_document_module.publish_report_json
|
||||
|
||||
def publish_after_race(prepared, value):
|
||||
if raced_entry == "file":
|
||||
prepared.path.write_bytes(b"unrelated sentinel\n")
|
||||
elif raced_entry == "symlink":
|
||||
prepared.path.symlink_to(protected)
|
||||
else:
|
||||
os.link(protected, prepared.path)
|
||||
return original_publish(prepared, value)
|
||||
|
||||
monkeypatch.setattr(
|
||||
lint_document_module,
|
||||
"publish_report_json",
|
||||
publish_after_race,
|
||||
)
|
||||
|
||||
result = lint_document_module.main(lint_main_args(output, document=protected))
|
||||
|
||||
assert result == 2
|
||||
assert protected.read_bytes() == protected_before
|
||||
if raced_entry == "file":
|
||||
assert output.read_bytes() == b"unrelated sentinel\n"
|
||||
elif raced_entry == "symlink":
|
||||
assert output.is_symlink()
|
||||
assert output.resolve() == protected.resolve()
|
||||
else:
|
||||
assert not output.is_symlink()
|
||||
assert os.path.samefile(output, protected)
|
||||
|
||||
|
||||
def test_lint_publish_rejects_owned_report_changed_after_preflight(
|
||||
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
output = tmp_path / "report.json"
|
||||
assert lint_document_module.main(lint_main_args(output)) == 0
|
||||
original_publish = lint_document_module.publish_report_json
|
||||
changed = output.read_bytes() + b"\n"
|
||||
original_inode = output.stat().st_ino
|
||||
|
||||
def publish_after_race(prepared, value):
|
||||
prepared.path.write_bytes(changed)
|
||||
assert prepared.path.stat().st_ino == original_inode
|
||||
return original_publish(prepared, value)
|
||||
|
||||
monkeypatch.setattr(
|
||||
lint_document_module,
|
||||
"publish_report_json",
|
||||
publish_after_race,
|
||||
)
|
||||
|
||||
result = lint_document_module.main(lint_main_args(output))
|
||||
|
||||
assert result == 2
|
||||
assert output.stat().st_ino == original_inode
|
||||
assert output.read_bytes() == changed
|
||||
|
||||
|
||||
def test_lint_can_conditionally_refresh_owned_report(tmp_path: Path) -> None:
|
||||
output = tmp_path / "report.json"
|
||||
|
||||
assert lint_document_module.main(lint_main_args(output)) == 0
|
||||
assert lint_document_module.main(lint_main_args(output)) == 0
|
||||
|
||||
assert read_json(output)["tool"] == "lint_document"
|
||||
|
||||
|
||||
def test_fidelity_does_not_count_values_hidden_in_html_comments(
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
baseline = tmp_path / "baseline.md"
|
||||
baseline.write_text(
|
||||
'# 기준\n\n`SecretToken987` https://example.com/unique-cache 731ms "고유 인용문"\n',
|
||||
encoding="utf-8",
|
||||
)
|
||||
document = (FIXTURES / "good" / "document.md").read_text(encoding="utf-8")
|
||||
document += (
|
||||
'\n<!-- `SecretToken987` https://example.com/unique-cache 731ms "고유 인용문" -->\n'
|
||||
)
|
||||
document_path = tmp_path / "document.md"
|
||||
output = tmp_path / "lint.json"
|
||||
document_path.write_text(document, encoding="utf-8")
|
||||
result = run_cli(
|
||||
"lint_document.py",
|
||||
"--document",
|
||||
document_path,
|
||||
"--logic-map",
|
||||
FIXTURES / "good" / "logic-map.json",
|
||||
"--term-ledger",
|
||||
FIXTURES / "good" / "term-ledger.json",
|
||||
"--reader-contract",
|
||||
FIXTURES / "good" / "reader-contract.json",
|
||||
"--baseline",
|
||||
baseline,
|
||||
"--output",
|
||||
output,
|
||||
)
|
||||
report = read_json(output)
|
||||
assert result.returncode == 1
|
||||
assert {"DOC-F002", "DOC-F003", "DOC-F004", "DOC-F005"} <= {
|
||||
item["rule_id"] for item in report["findings"]
|
||||
}
|
||||
assert report["fidelity"]["missing"] == 4
|
||||
|
||||
|
||||
def test_registered_multiword_term_suppresses_its_internal_candidate_only(
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
document = (FIXTURES / "good" / "document.md").read_text(encoding="utf-8")
|
||||
document += "\n중복 실행을 묶는 single-flight coordinator를 사용한다.\n"
|
||||
ledger = read_json(FIXTURES / "good" / "term-ledger.json")
|
||||
reader = read_json(FIXTURES / "good" / "reader-contract.json")
|
||||
ledger["assumed_known"].append("single-flight coordinator")
|
||||
reader["assumed_known"].append("single-flight coordinator")
|
||||
document_path = tmp_path / "document.md"
|
||||
ledger_path = tmp_path / "term-ledger.json"
|
||||
reader_path = tmp_path / "reader-contract.json"
|
||||
output = tmp_path / "lint.json"
|
||||
document_path.write_text(document, encoding="utf-8")
|
||||
write_json(ledger_path, ledger)
|
||||
write_json(reader_path, reader)
|
||||
result = run_cli(
|
||||
"lint_document.py",
|
||||
"--document",
|
||||
document_path,
|
||||
"--logic-map",
|
||||
FIXTURES / "good" / "logic-map.json",
|
||||
"--term-ledger",
|
||||
ledger_path,
|
||||
"--reader-contract",
|
||||
reader_path,
|
||||
"--output",
|
||||
output,
|
||||
)
|
||||
candidates = {
|
||||
item["context"]
|
||||
for item in read_json(output)["findings"]
|
||||
if item["rule_id"] == "DOC-T007"
|
||||
}
|
||||
assert result.returncode == 0
|
||||
assert "single-flight" not in candidates
|
||||
|
||||
|
||||
def test_registered_prefix_does_not_hide_longer_unregistered_term(tmp_path: Path) -> None:
|
||||
result, report = lint_fixture(tmp_path, "bad")
|
||||
assert result.returncode == 1
|
||||
candidates = {
|
||||
item["context"]
|
||||
for item in report["findings"]
|
||||
if item["rule_id"] == "DOC-T007"
|
||||
}
|
||||
assert "HTTPX" in candidates
|
||||
@@ -1,27 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from live_runner import CLAUDE_BIN, run_live_case
|
||||
|
||||
|
||||
LIVE_ENABLED = os.environ.get("TECH_DOC_FLOW_RUN_LIVE") == "1"
|
||||
|
||||
|
||||
@pytest.mark.live
|
||||
@pytest.mark.skipif(not LIVE_ENABLED, reason="TECH_DOC_FLOW_RUN_LIVE=1일 때만 실제 LLM을 호출")
|
||||
@pytest.mark.skipif(CLAUDE_BIN is None, reason="claude CLI가 없음")
|
||||
def test_live_skill_produces_verified_readable_revision(tmp_path: Path) -> None:
|
||||
timeout = int(os.environ.get("TECH_DOC_FLOW_LIVE_TIMEOUT", "300"))
|
||||
run_dir, report = run_live_case(tmp_path / "live-case", timeout=timeout)
|
||||
assert report["verdict"] == "pass"
|
||||
final = (run_dir / "final.md").read_text(encoding="utf-8")
|
||||
for protected in ("42ms", "cacheKey", "https://example.test/cache"):
|
||||
assert protected in final
|
||||
ledger = json.loads((run_dir / "05_term_ledger.json").read_text(encoding="utf-8"))
|
||||
assert all(term["plain_definition"].strip() for term in ledger["terms"])
|
||||
assert all(term["first_use"] in final for term in ledger["terms"])
|
||||
@@ -1,251 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from helpers import FIXTURES, read_json, run_cli, write_json
|
||||
|
||||
|
||||
CANONICAL_SCRIPTS = (
|
||||
Path(__file__).resolve().parents[1]
|
||||
/ "skills"
|
||||
/ "technical-doc-flow"
|
||||
/ "scripts"
|
||||
)
|
||||
sys.path.insert(0, str(CANONICAL_SCRIPTS))
|
||||
|
||||
from lint_document import ( # noqa: E402
|
||||
markdown_inline_links,
|
||||
markdown_reference_links,
|
||||
parse_headings,
|
||||
reference_definitions,
|
||||
)
|
||||
from markdown_structure import inline_code_spans # noqa: E402
|
||||
|
||||
|
||||
def lint_custom(
|
||||
tmp_path: Path,
|
||||
document: str,
|
||||
*,
|
||||
logic: dict | None = None,
|
||||
ledger: dict | None = None,
|
||||
):
|
||||
document_path = tmp_path / "document.md"
|
||||
logic_path = tmp_path / "logic.json"
|
||||
ledger_path = tmp_path / "ledger.json"
|
||||
output = tmp_path / "lint.json"
|
||||
document_path.write_text(document, encoding="utf-8")
|
||||
write_json(logic_path, logic or read_json(FIXTURES / "good" / "logic-map.json"))
|
||||
write_json(
|
||||
ledger_path,
|
||||
ledger or read_json(FIXTURES / "good" / "term-ledger.json"),
|
||||
)
|
||||
result = run_cli(
|
||||
"lint_document.py",
|
||||
"--document",
|
||||
document_path,
|
||||
"--logic-map",
|
||||
logic_path,
|
||||
"--term-ledger",
|
||||
ledger_path,
|
||||
"--reader-contract",
|
||||
FIXTURES / "good" / "reader-contract.json",
|
||||
"--output",
|
||||
output,
|
||||
)
|
||||
return result, read_json(output)
|
||||
|
||||
|
||||
def test_multiline_labels_stop_at_markdown_block_boundaries() -> None:
|
||||
assert markdown_inline_links("[x\n## y](TODO)") == []
|
||||
text = "[id]: #missing\n\n[x\n## y][id]\n"
|
||||
definitions = reference_definitions(text)
|
||||
assert definitions
|
||||
links = markdown_reference_links(text, definitions)
|
||||
assert all(link.start != text.index("[x") for link in links)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("marker", ["01.", "001)"])
|
||||
def test_leading_zero_one_list_marker_ends_multiline_inline_code(marker: str) -> None:
|
||||
assert inline_code_spans(f"text `a\n{marker} TODO\nc` tail") == []
|
||||
|
||||
|
||||
def test_html_comment_block_ends_multiline_inline_code() -> None:
|
||||
assert inline_code_spans("text `a\n<!-- TODO -->\nb` tail") == []
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"definition",
|
||||
[
|
||||
'[id]: #missing "long\n title"',
|
||||
'[id]: #missing\n "long\n title"',
|
||||
'[id]:\n #missing "long\n title"',
|
||||
],
|
||||
)
|
||||
def test_multiline_reference_titles_are_fully_parsed(definition: str) -> None:
|
||||
text = f"{definition}\n\n[go][id]\n"
|
||||
definitions = reference_definitions(text)
|
||||
assert len(definitions) == 1
|
||||
assert definitions[0].destination == "#missing"
|
||||
assert text[definitions[0].start : definitions[0].end].rstrip() == definition
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"definition",
|
||||
['[id]: <#missing>"title"', "[id]: #missing (a(b)"],
|
||||
)
|
||||
def test_invalid_reference_title_grammar_does_not_create_definition(
|
||||
definition: str,
|
||||
) -> None:
|
||||
assert reference_definitions(f"{definition}\n\n[go][id]\n") == []
|
||||
|
||||
|
||||
def test_noninterrupting_ordered_marker_cannot_create_reference_definition() -> None:
|
||||
text = "paragraph\n2. [id]: #missing\n\n[go][id]\n"
|
||||
assert reference_definitions(text) == []
|
||||
|
||||
|
||||
def test_reference_definition_inside_raw_html_does_not_resolve_link() -> None:
|
||||
text = "<div>\n[id]: #ghost\n</div>\n\n[go][id]\n"
|
||||
definitions = reference_definitions(text)
|
||||
assert definitions == []
|
||||
assert markdown_reference_links(text, definitions) == []
|
||||
|
||||
|
||||
def test_multiline_setext_heading_cannot_impersonate_logic_heading(
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
document = (FIXTURES / "good" / "document.md").read_text(encoding="utf-8")
|
||||
document = document.replace(
|
||||
"## 문제",
|
||||
"독자가 읽어야 할 흐름이다.\n문제\n---",
|
||||
)
|
||||
result, report = lint_custom(tmp_path, document)
|
||||
assert result.returncode == 1
|
||||
assert "DOC-L001" in {item["rule_id"] for item in report["findings"]}
|
||||
|
||||
|
||||
def test_noninterrupting_ordered_line_remains_part_of_setext_heading() -> None:
|
||||
headings = parse_headings("paragraph\n2. text\n---\n")
|
||||
assert [(heading.level, heading.text) for heading in headings] == [
|
||||
(2, "paragraph\n2. text")
|
||||
]
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"wrapper",
|
||||
["**{value}**", "[{value}](docs/x)", "`{value}`"],
|
||||
)
|
||||
def test_core_claim_search_uses_reader_visible_formatting(
|
||||
tmp_path: Path,
|
||||
wrapper: str,
|
||||
) -> None:
|
||||
document = (FIXTURES / "good" / "document.md").read_text(encoding="utf-8")
|
||||
logic = read_json(FIXTURES / "good" / "logic-map.json")
|
||||
core = logic["core_claim"]
|
||||
document = document.replace(
|
||||
core,
|
||||
core.replace("장애를", wrapper.format(value="장애를")),
|
||||
1,
|
||||
)
|
||||
result, report = lint_custom(tmp_path, document, logic=logic)
|
||||
assert result.returncode == 0, report["findings"]
|
||||
|
||||
|
||||
def test_core_claim_wholly_inside_inline_code_is_not_explanatory_prose(
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
document = (FIXTURES / "good" / "document.md").read_text(encoding="utf-8")
|
||||
logic = read_json(FIXTURES / "good" / "logic-map.json")
|
||||
core = logic["core_claim"]
|
||||
document = document.replace(core, f"`{core}`", 1)
|
||||
result, report = lint_custom(tmp_path, document, logic=logic)
|
||||
assert result.returncode == 1
|
||||
assert "DOC-L002" in {item["rule_id"] for item in report["findings"]}
|
||||
|
||||
|
||||
@pytest.mark.parametrize("wrapper", ["**{value}**", "[{value}](docs/x)"])
|
||||
def test_term_first_use_search_uses_reader_visible_formatting(
|
||||
tmp_path: Path,
|
||||
wrapper: str,
|
||||
) -> None:
|
||||
document = (FIXTURES / "good" / "document.md").read_text(encoding="utf-8")
|
||||
ledger = read_json(FIXTURES / "good" / "term-ledger.json")
|
||||
first_use = ledger["terms"][0]["first_use"]
|
||||
document = document.replace(
|
||||
first_use,
|
||||
first_use.replace("한 번", wrapper.format(value="한 번")),
|
||||
1,
|
||||
)
|
||||
result, report = lint_custom(tmp_path, document, ledger=ledger)
|
||||
assert result.returncode == 0, report["findings"]
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"raw_marker",
|
||||
["[근거:C1]", "<!-- claim:C1 -->"],
|
||||
)
|
||||
def test_raw_code_html_cannot_supply_claim_marker(
|
||||
tmp_path: Path,
|
||||
raw_marker: str,
|
||||
) -> None:
|
||||
document = (FIXTURES / "good" / "document.md").read_text(encoding="utf-8")
|
||||
document = document.replace("<!-- claim:C1 -->", "")
|
||||
document += f"\n<pre>\n{raw_marker}\n</pre>\n"
|
||||
result, report = lint_custom(tmp_path, document)
|
||||
assert result.returncode == 1
|
||||
assert "DOC-L003" in {item["rule_id"] for item in report["findings"]}
|
||||
|
||||
|
||||
def test_inline_markdown_inside_raw_div_remains_literal_reader_text(
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
document = (FIXTURES / "good" / "document.md").read_text(encoding="utf-8")
|
||||
document += "\n<div>\n`TODO`\n</div>\n"
|
||||
result, report = lint_custom(tmp_path, document)
|
||||
assert result.returncode == 1
|
||||
assert "DOC-M001" in {item["rule_id"] for item in report["findings"]}
|
||||
|
||||
|
||||
def test_tag_shaped_script_data_cannot_supply_explicit_anchor(tmp_path: Path) -> None:
|
||||
document = (FIXTURES / "good" / "document.md").read_text(encoding="utf-8")
|
||||
document += (
|
||||
"\n<script>\nconst x = '<a id=\"ghost\">';\n</script>\n"
|
||||
"\n[이동](#ghost)\n"
|
||||
)
|
||||
result, report = lint_custom(tmp_path, document)
|
||||
assert result.returncode == 1
|
||||
assert any(
|
||||
item["rule_id"] == "DOC-M003" and item["context"] == "ghost"
|
||||
for item in report["findings"]
|
||||
)
|
||||
|
||||
|
||||
def test_heading_slugs_preserve_escaped_underscores_and_resolve_references() -> None:
|
||||
headings = parse_headings(
|
||||
"## \\_foo\\_\n## foo\\_bar\n## [visible][id]\n\n[id]: /url\n"
|
||||
)
|
||||
assert [heading.slug for heading in headings] == ["_foo_", "foo_bar", "visible"]
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"table",
|
||||
[
|
||||
"> 항목 | 설명\n> --- | ---\n> 하나 | {cell}\n",
|
||||
"- 항목 | 설명\n --- | ---\n 하나 | {cell}\n",
|
||||
],
|
||||
)
|
||||
def test_nested_gfm_tables_do_not_consume_prose_budgets(
|
||||
tmp_path: Path,
|
||||
table: str,
|
||||
) -> None:
|
||||
document = (FIXTURES / "good" / "document.md").read_text(encoding="utf-8")
|
||||
document += "\n" + table.format(cell="긴 설명이다. " * 120)
|
||||
result, report = lint_custom(tmp_path, document)
|
||||
assert result.returncode == 0, report["findings"]
|
||||
assert not {
|
||||
item["rule_id"] for item in report["findings"]
|
||||
} & {"DOC-P001", "DOC-P002"}
|
||||
|
||||
@@ -0,0 +1,92 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import math
|
||||
import unittest
|
||||
|
||||
from claridoc.models import (
|
||||
REVIEW_DIMENSIONS,
|
||||
Brief,
|
||||
DocumentType,
|
||||
ModelReview,
|
||||
PipelineConfig,
|
||||
QualityGate,
|
||||
SourcePack,
|
||||
ValidationError,
|
||||
)
|
||||
from claridoc.templates import mock_pipeline_config
|
||||
from tests.helpers import brief_dict, source_dict
|
||||
|
||||
|
||||
class ModelTests(unittest.TestCase):
|
||||
def test_valid_brief_round_trip(self) -> None:
|
||||
brief = Brief.from_dict(brief_dict())
|
||||
self.assertEqual(brief.document_type, DocumentType.TECHNICAL_BLOG)
|
||||
self.assertEqual(Brief.from_dict(brief.to_dict()).title, brief.title)
|
||||
|
||||
def test_invalid_document_type_is_rejected(self) -> None:
|
||||
data = brief_dict()
|
||||
data["document_type"] = "essay"
|
||||
with self.assertRaises(ValidationError):
|
||||
Brief.from_dict(data)
|
||||
|
||||
def test_duplicate_source_ids_are_rejected(self) -> None:
|
||||
data = source_dict()
|
||||
data["sources"].append(dict(data["sources"][0]))
|
||||
with self.assertRaises(ValidationError):
|
||||
SourcePack.from_dict(data)
|
||||
|
||||
def test_target_words_range_is_enforced(self) -> None:
|
||||
data = brief_dict()
|
||||
data["constraints"]["target_words"] = 50
|
||||
with self.assertRaises(ValidationError):
|
||||
Brief.from_dict(data)
|
||||
|
||||
def test_quality_gate_rejects_non_finite_weights(self) -> None:
|
||||
with self.assertRaises(ValidationError):
|
||||
QualityGate.from_dict({"deterministic_weight": math.nan, "model_weight": math.nan})
|
||||
|
||||
def test_pipeline_requires_explicit_reviewers(self) -> None:
|
||||
data = mock_pipeline_config()
|
||||
data["reviewers"] = []
|
||||
with self.assertRaises(ValidationError):
|
||||
PipelineConfig.from_dict(data)
|
||||
|
||||
def test_pipeline_rejects_duplicate_reviewer_roles(self) -> None:
|
||||
data = mock_pipeline_config()
|
||||
data["reviewers"].append({"role": "logic", "provider": "mock"})
|
||||
with self.assertRaises(ValidationError):
|
||||
PipelineConfig.from_dict(data)
|
||||
|
||||
def test_review_requires_every_scoring_dimension(self) -> None:
|
||||
review = self._valid_review()
|
||||
del review["dimension_scores"][REVIEW_DIMENSIONS[0]]
|
||||
with self.assertRaises(ValidationError):
|
||||
ModelReview.from_dict(review, role="logic", provider="mock")
|
||||
|
||||
def test_review_rejects_unknown_issue_severity(self) -> None:
|
||||
review = self._valid_review()
|
||||
review["issues"] = [
|
||||
{
|
||||
"section": "Mechanism",
|
||||
"problem": "A causal step is missing.",
|
||||
"why_it_matters": "The conclusion cannot be reproduced.",
|
||||
"fix": "Add the missing state transition.",
|
||||
"severity": "critical",
|
||||
}
|
||||
]
|
||||
with self.assertRaises(ValidationError):
|
||||
ModelReview.from_dict(review, role="logic", provider="mock")
|
||||
|
||||
@staticmethod
|
||||
def _valid_review() -> dict:
|
||||
return {
|
||||
"score": 90,
|
||||
"dimension_scores": {name: 90 for name in REVIEW_DIMENSIONS},
|
||||
"issues": [],
|
||||
"strengths": ["The structure is explicit."],
|
||||
"questions": [],
|
||||
}
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,86 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import tempfile
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
from claridoc.models import PipelineConfig
|
||||
from claridoc.pipeline import _mock_provider_warning, run_pipeline
|
||||
from claridoc.templates import mock_pipeline_config
|
||||
from tests.helpers import make_brief, make_sources
|
||||
|
||||
|
||||
class PipelineTests(unittest.TestCase):
|
||||
def test_mixed_mock_pipeline_is_explicitly_flagged(self) -> None:
|
||||
config_data = mock_pipeline_config()
|
||||
config_data["writer"] = {"provider": "claude"}
|
||||
warning = _mock_provider_warning(PipelineConfig.from_dict(config_data))
|
||||
self.assertIn("mixes external providers", warning)
|
||||
self.assertIn("synthetic", warning)
|
||||
|
||||
def test_end_to_end_mock_run_creates_auditable_artifacts(self) -> None:
|
||||
with tempfile.TemporaryDirectory() as temp:
|
||||
output = Path(temp) / "run"
|
||||
result = run_pipeline(
|
||||
make_brief(),
|
||||
make_sources(),
|
||||
PipelineConfig.from_dict(mock_pipeline_config()),
|
||||
output,
|
||||
)
|
||||
self.assertTrue(result.passed)
|
||||
self.assertTrue(result.final_path.is_file())
|
||||
self.assertTrue(result.report_path.is_file())
|
||||
self.assertTrue(result.manifest_path.is_file())
|
||||
run_data = json.loads((output / "run.json").read_text(encoding="utf-8"))
|
||||
self.assertTrue(run_data["passed"])
|
||||
self.assertTrue(any("deterministic mocks" in warning for warning in result.warnings))
|
||||
report_text = result.report_path.read_text(encoding="utf-8")
|
||||
self.assertIn("Provider topology", report_text)
|
||||
self.assertIn("synthetic", report_text)
|
||||
manifest = json.loads(result.manifest_path.read_text(encoding="utf-8"))
|
||||
paths = {item["path"] for item in manifest["files"]}
|
||||
self.assertIn("final/document.md", paths)
|
||||
self.assertIn("provider-events.jsonl", paths)
|
||||
self.assertNotIn("manifest.json", paths)
|
||||
for item in manifest["files"]:
|
||||
artifact = output / item["path"]
|
||||
self.assertEqual(artifact.stat().st_size, item["bytes"])
|
||||
self.assertEqual(hashlib.sha256(artifact.read_bytes()).hexdigest(), item["sha256"])
|
||||
|
||||
def test_revision_limit_is_enforced_when_gate_cannot_pass(self) -> None:
|
||||
with tempfile.TemporaryDirectory() as temp:
|
||||
output = Path(temp) / "run"
|
||||
config_data = mock_pipeline_config()
|
||||
config_data["quality_gate"]["minimum_score"] = 99
|
||||
config_data["quality_gate"]["max_revisions"] = 1
|
||||
result = run_pipeline(
|
||||
make_brief(),
|
||||
make_sources(),
|
||||
PipelineConfig.from_dict(config_data),
|
||||
output,
|
||||
)
|
||||
self.assertFalse(result.passed)
|
||||
self.assertEqual(len(result.rounds), 2)
|
||||
self.assertTrue((output / "rounds" / "round-01" / "revision.raw.txt").is_file())
|
||||
|
||||
def test_reviewer_role_cannot_escape_artifact_directory(self) -> None:
|
||||
with tempfile.TemporaryDirectory() as temp:
|
||||
root = Path(temp)
|
||||
output = root / "run"
|
||||
config_data = mock_pipeline_config()
|
||||
config_data["reviewers"] = [{"role": "../../logic reviewer", "provider": "mock"}]
|
||||
run_pipeline(
|
||||
make_brief(),
|
||||
make_sources(),
|
||||
PipelineConfig.from_dict(config_data),
|
||||
output,
|
||||
)
|
||||
review_files = list((output / "rounds" / "round-01").glob("review-*"))
|
||||
self.assertTrue(any(path.name.endswith("logic-reviewer.json") for path in review_files))
|
||||
self.assertEqual(list(root.glob("logic*")), [])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,113 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import sys
|
||||
import tempfile
|
||||
import textwrap
|
||||
import types
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
from unittest.mock import patch
|
||||
|
||||
from claridoc.models import ProviderSpec
|
||||
from claridoc.providers.antigravity import AntigravityProvider
|
||||
from claridoc.providers.base import ProviderRequest
|
||||
from claridoc.providers.claude import ClaudeProvider
|
||||
from claridoc.providers.codex import CodexProvider
|
||||
|
||||
|
||||
class ProviderAdapterTests(unittest.TestCase):
|
||||
def _script(self, directory: Path, name: str, body: str) -> Path:
|
||||
path = directory / name
|
||||
path.write_text("#!/usr/bin/env python3\n" + textwrap.dedent(body), encoding="utf-8")
|
||||
path.chmod(0o755)
|
||||
return path
|
||||
|
||||
def test_codex_adapter_reads_output_last_message(self) -> None:
|
||||
with tempfile.TemporaryDirectory() as temp:
|
||||
directory = Path(temp)
|
||||
script = self._script(directory, "fake_codex.py", r'''
|
||||
import json, pathlib, sys
|
||||
prompt = sys.stdin.read()
|
||||
index = sys.argv.index("--output-last-message")
|
||||
pathlib.Path(sys.argv[index + 1]).write_text(json.dumps({"ok": True, "prompt": prompt}), encoding="utf-8")
|
||||
''')
|
||||
provider = CodexProvider(ProviderSpec(provider="codex", options={"binary": str(script)}))
|
||||
response = provider.generate(ProviderRequest("plan", "hello", directory))
|
||||
self.assertIn('"ok": true', response.text)
|
||||
self.assertIn("hello", response.text)
|
||||
self.assertIn("--sandbox", response.command)
|
||||
self.assertIn("read-only", response.command)
|
||||
|
||||
def test_claude_adapter_pipes_prompt(self) -> None:
|
||||
with tempfile.TemporaryDirectory() as temp:
|
||||
directory = Path(temp)
|
||||
script = self._script(directory, "fake_claude.py", r'''
|
||||
import sys
|
||||
data = sys.stdin.read()
|
||||
print("received:" + data)
|
||||
''')
|
||||
provider = ClaudeProvider(ProviderSpec(provider="claude", options={"binary": str(script)}))
|
||||
response = provider.generate(ProviderRequest("draft", "payload", directory))
|
||||
self.assertEqual(response.text, "received:payload")
|
||||
self.assertEqual(response.command[1:4], ["-p", "--output-format", "text"])
|
||||
|
||||
def test_antigravity_adapter_uses_sdk_contract_and_isolates_cwd(self) -> None:
|
||||
state: dict[str, object] = {}
|
||||
fake_google = types.ModuleType("google")
|
||||
fake_google.__path__ = [] # type: ignore[attr-defined]
|
||||
fake_sdk = types.ModuleType("google.antigravity")
|
||||
|
||||
class FakeLocalAgentConfig:
|
||||
def __init__(self, **kwargs: object) -> None:
|
||||
state["config"] = kwargs
|
||||
|
||||
class FakeResponse:
|
||||
async def text(self) -> str:
|
||||
return "sdk-response"
|
||||
|
||||
class FakeAgent:
|
||||
def __init__(self, config: FakeLocalAgentConfig) -> None:
|
||||
state["agent_config"] = config
|
||||
|
||||
async def __aenter__(self) -> "FakeAgent":
|
||||
state["cwd"] = str(Path.cwd())
|
||||
return self
|
||||
|
||||
async def __aexit__(self, exc_type: object, exc: object, traceback: object) -> None:
|
||||
return None
|
||||
|
||||
async def chat(self, prompt: str) -> FakeResponse:
|
||||
state["prompt"] = prompt
|
||||
return FakeResponse()
|
||||
|
||||
fake_sdk.Agent = FakeAgent # type: ignore[attr-defined]
|
||||
fake_sdk.LocalAgentConfig = FakeLocalAgentConfig # type: ignore[attr-defined]
|
||||
|
||||
original_cwd = Path.cwd()
|
||||
with tempfile.TemporaryDirectory() as temp:
|
||||
directory = Path(temp).resolve()
|
||||
with patch.dict(sys.modules, {"google": fake_google, "google.antigravity": fake_sdk}):
|
||||
provider = AntigravityProvider(
|
||||
ProviderSpec(
|
||||
provider="antigravity",
|
||||
model="model-under-test",
|
||||
options={"config": {"temperature": 0.2}},
|
||||
)
|
||||
)
|
||||
response = provider.generate(ProviderRequest("review", "inspect this", directory))
|
||||
|
||||
self.assertEqual(response.text, "sdk-response")
|
||||
self.assertEqual(state["prompt"], "inspect this")
|
||||
self.assertEqual(state["cwd"], str(directory))
|
||||
self.assertEqual(state["config"], {"temperature": 0.2, "model": "model-under-test"})
|
||||
self.assertEqual(Path.cwd(), original_cwd)
|
||||
|
||||
def test_antigravity_doctor_handles_missing_sdk(self) -> None:
|
||||
provider = AntigravityProvider(ProviderSpec(provider="antigravity"))
|
||||
with patch("claridoc.providers.antigravity.importlib.util.find_spec", side_effect=ModuleNotFoundError):
|
||||
result = provider.check()
|
||||
self.assertFalse(result["available"])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
+50
-58
@@ -1,70 +1,62 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import sys
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
from claridoc.models import Brief, Outline, PipelineConfig, SourcePack
|
||||
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
SKILL = ROOT / "skills" / "technical-doc-flow"
|
||||
sys.path.insert(0, str(SKILL / "scripts"))
|
||||
from harness_common import InputError, load_rules, validate_with_schema # noqa: E402
|
||||
|
||||
|
||||
def test_all_bundled_json_and_schemas_parse() -> None:
|
||||
paths = [*SKILL.rglob("*.json"), ROOT / "harness.json"]
|
||||
assert paths
|
||||
for path in paths:
|
||||
value = json.loads(path.read_text(encoding="utf-8"))
|
||||
assert isinstance(value, dict), path
|
||||
class SchemaTests(unittest.TestCase):
|
||||
def test_all_schema_documents_are_well_formed_draft_2020_12(self) -> None:
|
||||
schema_paths = sorted((ROOT / "schemas").glob("*.schema.json"))
|
||||
self.assertEqual(len(schema_paths), 5)
|
||||
for path in schema_paths:
|
||||
with self.subTest(path=path.name):
|
||||
data = json.loads(path.read_text(encoding="utf-8"))
|
||||
self.assertEqual(data["$schema"], "https://json-schema.org/draft/2020-12/schema")
|
||||
self.assertEqual(data["type"], "object")
|
||||
|
||||
def test_examples_match_runtime_contracts(self) -> None:
|
||||
brief_data = json.loads(
|
||||
(ROOT / "examples" / "briefs" / "retry-policy-blog.json").read_text(encoding="utf-8")
|
||||
)
|
||||
sources_data = json.loads(
|
||||
(ROOT / "examples" / "sources" / "retry-policy-sources.json").read_text(encoding="utf-8")
|
||||
)
|
||||
mock_config_data = json.loads((ROOT / "config" / "pipeline.mock.json").read_text(encoding="utf-8"))
|
||||
multi_config_data = json.loads(
|
||||
(ROOT / "config" / "pipeline.multi-agent.example.json").read_text(encoding="utf-8")
|
||||
)
|
||||
|
||||
self.assertTrue(Brief.from_dict(brief_data).title)
|
||||
self.assertGreaterEqual(len(SourcePack.from_dict(sources_data).sources), 1)
|
||||
self.assertEqual(PipelineConfig.from_dict(mock_config_data).writer.provider, "mock")
|
||||
self.assertEqual(PipelineConfig.from_dict(multi_config_data).writer.provider, "claude")
|
||||
|
||||
def test_outline_schema_covers_runtime_outline_shape(self) -> None:
|
||||
sample = {
|
||||
"title": "Example",
|
||||
"document_type": "technical_blog",
|
||||
"sections": [
|
||||
{
|
||||
"id": "01-promise",
|
||||
"intent": "promise",
|
||||
"title": "Decision",
|
||||
"reader_question": "What should I do?",
|
||||
"purpose": "State the useful conclusion.",
|
||||
"must_include": ["scope"],
|
||||
"evidence_ids": ["S1"],
|
||||
"transition_to_next": "Explain why.",
|
||||
}
|
||||
],
|
||||
}
|
||||
outline = Outline.from_dict(sample)
|
||||
self.assertEqual(outline.sections[0].intent, "promise")
|
||||
|
||||
|
||||
def test_quality_rules_have_unique_ids_and_every_runtime_threshold() -> None:
|
||||
rules = load_rules()
|
||||
ids = [item["id"] for item in rules["rules"]]
|
||||
assert len(ids) == len(set(ids))
|
||||
assert {
|
||||
"DOC-H001",
|
||||
"DOC-M002",
|
||||
"DOC-L003",
|
||||
"DOC-T009",
|
||||
"DOC-F005",
|
||||
"DOC-M004",
|
||||
"FNL-001",
|
||||
} <= set(ids)
|
||||
assert rules["thresholds"]["route"]["light"]["requires_existing_draft"] is True
|
||||
assert rules["thresholds"]["finalization"]["max_change_rate"] == 0.0
|
||||
|
||||
|
||||
def test_artifact_contract_schema_names_exist() -> None:
|
||||
expected = {
|
||||
"run.schema.json",
|
||||
"sources.schema.json",
|
||||
"reader-contract.schema.json",
|
||||
"evidence-map.schema.json",
|
||||
"logic-map.schema.json",
|
||||
"term-ledger.schema.json",
|
||||
"review.schema.json",
|
||||
"lint-report.schema.json",
|
||||
"final-report.schema.json",
|
||||
"chunk-manifest.schema.json",
|
||||
"quality-rules.schema.json",
|
||||
"runtime-contract.schema.json",
|
||||
}
|
||||
actual = {path.name for path in (SKILL / "schemas").glob("*.json")}
|
||||
assert expected <= actual
|
||||
|
||||
|
||||
def test_reader_contract_rejects_whitespace_values_and_undeclared_fields() -> None:
|
||||
fixture = ROOT / "tests" / "fixtures" / "good" / "reader-contract.json"
|
||||
reader = json.loads(fixture.read_text(encoding="utf-8"))
|
||||
reader["non_goals"] = [" "]
|
||||
with pytest.raises(InputError):
|
||||
validate_with_schema(reader, "reader-contract.schema.json", "reader")
|
||||
|
||||
reader = json.loads(fixture.read_text(encoding="utf-8"))
|
||||
reader["undeclared"] = True
|
||||
with pytest.raises(InputError):
|
||||
validate_with_schema(reader, "reader-contract.schema.json", "reader")
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
||||
@@ -1,500 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import shutil
|
||||
from pathlib import Path
|
||||
|
||||
from helpers import read_json, run_cli
|
||||
|
||||
|
||||
def sample_bytes() -> bytes:
|
||||
return (
|
||||
"# 문서\r\n"
|
||||
"\r\n"
|
||||
"첫 문단은 분할 전후에 그대로 남아야 합니다.\r\n"
|
||||
"\r\n"
|
||||
"## 코드\r\n"
|
||||
"\r\n"
|
||||
"```python\r\n"
|
||||
"## 이것은 제목이 아니다\r\n"
|
||||
"```not-a-closing-fence\r\n"
|
||||
"print('계속 같은 fence 안')\r\n"
|
||||
"```\r\n"
|
||||
"\r\n"
|
||||
"## 끝\r\n"
|
||||
"\r\n"
|
||||
"마지막 문단입니다.\r\n"
|
||||
).encode("utf-8")
|
||||
|
||||
|
||||
def test_split_and_input_reassembly_are_byte_identical(tmp_path: Path) -> None:
|
||||
document = tmp_path / "source.md"
|
||||
document.write_bytes(sample_bytes())
|
||||
chunks = tmp_path / "chunks"
|
||||
split = run_cli(
|
||||
"split_document.py",
|
||||
"--document",
|
||||
document,
|
||||
"--output-dir",
|
||||
chunks,
|
||||
"--max-chars",
|
||||
55,
|
||||
)
|
||||
assert split.returncode == 0, split.stderr
|
||||
manifest = read_json(chunks / "manifest.json")
|
||||
assert manifest["self_check"] is True
|
||||
assert manifest["source"]["sha256"] == hashlib.sha256(sample_bytes()).hexdigest()
|
||||
assert [item["index"] for item in manifest["chunks"]] == list(
|
||||
range(1, len(manifest["chunks"]) + 1)
|
||||
)
|
||||
|
||||
fenced_piece = [
|
||||
(chunks / item["input_file"]).read_text(encoding="utf-8")
|
||||
for item in manifest["chunks"]
|
||||
if "```not-a-closing-fence" in (chunks / item["input_file"]).read_text(encoding="utf-8")
|
||||
]
|
||||
assert len(fenced_piece) == 1
|
||||
assert "print('계속 같은 fence 안')" in fenced_piece[0]
|
||||
assert fenced_piece[0].rstrip().endswith("```")
|
||||
|
||||
output = tmp_path / "round-trip.md"
|
||||
assembled = run_cli(
|
||||
"reassemble_document.py",
|
||||
"--manifest",
|
||||
chunks / "manifest.json",
|
||||
"--output",
|
||||
output,
|
||||
)
|
||||
assert assembled.returncode == 0, assembled.stderr
|
||||
assert output.read_bytes() == sample_bytes()
|
||||
|
||||
|
||||
def test_reassemble_rewritten_chunks_and_reject_tampered_input(tmp_path: Path) -> None:
|
||||
document = tmp_path / "source.md"
|
||||
document.write_text("# 제목\n\n첫 문단\n\n## 다음\n\n둘째 문단\n", encoding="utf-8")
|
||||
chunks = tmp_path / "chunks"
|
||||
split = run_cli(
|
||||
"split_document.py",
|
||||
"--document",
|
||||
document,
|
||||
"--output-dir",
|
||||
chunks,
|
||||
"--max-chars",
|
||||
20,
|
||||
)
|
||||
assert split.returncode == 0, split.stderr
|
||||
manifest = read_json(chunks / "manifest.json")
|
||||
expected: list[str] = []
|
||||
for item in manifest["chunks"]:
|
||||
source = (chunks / item["input_file"]).read_text(encoding="utf-8")
|
||||
rewritten = f"[청크 {item['index']}]\n{source}"
|
||||
(chunks / item["rewritten_file"]).write_text(rewritten, encoding="utf-8")
|
||||
expected.append(rewritten)
|
||||
|
||||
output = tmp_path / "rewritten.md"
|
||||
assembled = run_cli(
|
||||
"reassemble_document.py",
|
||||
"--manifest",
|
||||
chunks / "manifest.json",
|
||||
"--output",
|
||||
output,
|
||||
"--source",
|
||||
"rewritten",
|
||||
)
|
||||
assert assembled.returncode == 0, assembled.stderr
|
||||
assert output.read_text(encoding="utf-8") == "".join(expected)
|
||||
|
||||
first = chunks / manifest["chunks"][0]["input_file"]
|
||||
first.write_text(first.read_text(encoding="utf-8") + "tampered", encoding="utf-8")
|
||||
rejected_output = tmp_path / "must-not-exist.md"
|
||||
rejected = run_cli(
|
||||
"reassemble_document.py",
|
||||
"--manifest",
|
||||
chunks / "manifest.json",
|
||||
"--output",
|
||||
rejected_output,
|
||||
"--source",
|
||||
"rewritten",
|
||||
)
|
||||
assert rejected.returncode == 2
|
||||
assert not rejected_output.exists()
|
||||
|
||||
|
||||
def test_reassemble_rejects_manifest_path_escape(tmp_path: Path) -> None:
|
||||
document = tmp_path / "source.md"
|
||||
document.write_text("# 제목\n\n본문\n", encoding="utf-8")
|
||||
chunks = tmp_path / "chunks"
|
||||
assert run_cli(
|
||||
"split_document.py",
|
||||
"--document",
|
||||
document,
|
||||
"--output-dir",
|
||||
chunks,
|
||||
"--max-chars",
|
||||
50,
|
||||
).returncode == 0
|
||||
manifest_path = chunks / "manifest.json"
|
||||
manifest = read_json(manifest_path)
|
||||
outside = tmp_path / "outside.md"
|
||||
shutil.copyfile(chunks / manifest["chunks"][0]["input_file"], outside)
|
||||
manifest["chunks"][0]["input_file"] = "../outside.md"
|
||||
manifest_path.write_text(json.dumps(manifest), encoding="utf-8")
|
||||
result = run_cli(
|
||||
"reassemble_document.py",
|
||||
"--manifest",
|
||||
manifest_path,
|
||||
"--output",
|
||||
tmp_path / "output.md",
|
||||
)
|
||||
assert result.returncode == 2
|
||||
assert "벗어납니다" in result.stderr
|
||||
|
||||
|
||||
def test_reassemble_applies_chunk_manifest_schema(tmp_path: Path) -> None:
|
||||
document = tmp_path / "source.md"
|
||||
document.write_text("# 제목\n\n본문\n", encoding="utf-8")
|
||||
chunks = tmp_path / "chunks"
|
||||
assert run_cli(
|
||||
"split_document.py",
|
||||
"--document",
|
||||
document,
|
||||
"--output-dir",
|
||||
chunks,
|
||||
).returncode == 0
|
||||
manifest_path = chunks / "manifest.json"
|
||||
manifest = read_json(manifest_path)
|
||||
manifest["chunks"][0]["boundary_reason"] = "INVALID"
|
||||
manifest_path.write_text(json.dumps(manifest), encoding="utf-8")
|
||||
result = run_cli(
|
||||
"reassemble_document.py",
|
||||
"--manifest",
|
||||
manifest_path,
|
||||
"--output",
|
||||
tmp_path / "output.md",
|
||||
)
|
||||
assert result.returncode == 2
|
||||
assert "schema 위반" in result.stderr
|
||||
|
||||
|
||||
def test_split_keeps_indented_list_fence_atomic(tmp_path: Path) -> None:
|
||||
document = tmp_path / "nested.md"
|
||||
nested = (
|
||||
"# 제목\n\n- 항목\n\n"
|
||||
" ```python\n"
|
||||
" first = 1\n"
|
||||
"\n"
|
||||
" second = 2\n"
|
||||
" ```\n\n"
|
||||
"## 다음\n\n본문\n"
|
||||
)
|
||||
document.write_text(nested, encoding="utf-8")
|
||||
chunks = tmp_path / "nested-chunks"
|
||||
result = run_cli(
|
||||
"split_document.py",
|
||||
"--document",
|
||||
document,
|
||||
"--output-dir",
|
||||
chunks,
|
||||
"--max-chars",
|
||||
40,
|
||||
)
|
||||
assert result.returncode == 0, result.stderr
|
||||
manifest = read_json(chunks / "manifest.json")
|
||||
pieces = [
|
||||
(chunks / item["input_file"]).read_text(encoding="utf-8")
|
||||
for item in manifest["chunks"]
|
||||
]
|
||||
containing_code = [piece for piece in pieces if "first = 1" in piece or "second = 2" in piece]
|
||||
assert len(containing_code) == 1
|
||||
assert "first = 1" in containing_code[0] and "second = 2" in containing_code[0]
|
||||
|
||||
|
||||
def test_indented_backticks_inside_top_level_fence_do_not_close_it(tmp_path: Path) -> None:
|
||||
document = tmp_path / "top-level-fence.md"
|
||||
text = (
|
||||
"# 제목\n\n"
|
||||
"```text\n"
|
||||
"line\n"
|
||||
" ```\n"
|
||||
"\n"
|
||||
"## 이것도 코드 내용\n"
|
||||
"secret\n"
|
||||
"```\n\n"
|
||||
"## 실제 다음 절\n\n본문\n"
|
||||
)
|
||||
document.write_text(text, encoding="utf-8")
|
||||
chunks = tmp_path / "top-level-chunks"
|
||||
result = run_cli(
|
||||
"split_document.py",
|
||||
"--document",
|
||||
document,
|
||||
"--output-dir",
|
||||
chunks,
|
||||
"--max-chars",
|
||||
38,
|
||||
)
|
||||
assert result.returncode == 0, result.stderr
|
||||
manifest = read_json(chunks / "manifest.json")
|
||||
pieces = [
|
||||
(chunks / item["input_file"]).read_text(encoding="utf-8")
|
||||
for item in manifest["chunks"]
|
||||
]
|
||||
code_pieces = [piece for piece in pieces if "secret" in piece or "이것도 코드 내용" in piece]
|
||||
assert len(code_pieces) == 1
|
||||
assert " ```" in code_pieces[0] and "secret" in code_pieces[0]
|
||||
|
||||
|
||||
def test_same_line_list_and_blockquote_fences_stay_atomic(tmp_path: Path) -> None:
|
||||
document = tmp_path / "container-fences.md"
|
||||
text = (
|
||||
"# 제목\n\n"
|
||||
"- ```python\n"
|
||||
" first = 1\n\n"
|
||||
" ## 목록 안 코드 제목\n"
|
||||
" second = 2\n"
|
||||
" ```\n\n"
|
||||
"> ```text\n"
|
||||
"> quote line\n"
|
||||
">\n"
|
||||
"> ## 인용 안 코드 제목\n"
|
||||
"> ```\n\n"
|
||||
"## 실제 다음 절\n\n본문\n"
|
||||
)
|
||||
document.write_text(text, encoding="utf-8")
|
||||
chunks = tmp_path / "container-chunks"
|
||||
result = run_cli(
|
||||
"split_document.py",
|
||||
"--document",
|
||||
document,
|
||||
"--output-dir",
|
||||
chunks,
|
||||
"--max-chars",
|
||||
35,
|
||||
)
|
||||
assert result.returncode == 0, result.stderr
|
||||
manifest = read_json(chunks / "manifest.json")
|
||||
pieces = [
|
||||
(chunks / item["input_file"]).read_text(encoding="utf-8")
|
||||
for item in manifest["chunks"]
|
||||
]
|
||||
assert len([piece for piece in pieces if "first = 1" in piece or "second = 2" in piece]) == 1
|
||||
assert len([piece for piece in pieces if "quote line" in piece or "인용 안 코드 제목" in piece]) == 1
|
||||
|
||||
|
||||
def test_nested_list_heading_is_not_used_as_a_top_level_split_boundary(
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
document = tmp_path / "nested-list-heading.md"
|
||||
text = (
|
||||
"# 제목\n\n"
|
||||
"도입 설명은 청크를 채우기 위해 충분한 길이로 이어진다.\n"
|
||||
"- 목록 항목은 다음 설명과 소제목을 소유한다.\n"
|
||||
" ## 목록 내부 소제목\n"
|
||||
" 이 내용은 목록 항목에서 분리되면 의미가 달라진다.\n"
|
||||
"## 실제 절\n"
|
||||
"본문이다.\n"
|
||||
)
|
||||
document.write_text(text, encoding="utf-8")
|
||||
chunks = tmp_path / "nested-list-heading-chunks"
|
||||
result = run_cli(
|
||||
"split_document.py",
|
||||
"--document",
|
||||
document,
|
||||
"--output-dir",
|
||||
chunks,
|
||||
"--max-chars",
|
||||
95,
|
||||
)
|
||||
assert result.returncode == 0, result.stderr
|
||||
manifest = read_json(chunks / "manifest.json")
|
||||
pieces = [
|
||||
(chunks / item["input_file"]).read_text(encoding="utf-8")
|
||||
for item in manifest["chunks"]
|
||||
]
|
||||
nested_piece = next(piece for piece in pieces if "목록 내부 소제목" in piece)
|
||||
assert "- 목록 항목" in nested_piece
|
||||
assert not any(piece.startswith(" ## 목록 내부 소제목") for piece in pieces)
|
||||
|
||||
|
||||
def test_comment_token_inside_html_attribute_does_not_make_document_atomic(
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
document = tmp_path / "inline-html-attribute.md"
|
||||
document.write_text(
|
||||
'<span title="<!--">보이는 설명</span>\n\n## 다음\n본문이다.\n',
|
||||
encoding="utf-8",
|
||||
)
|
||||
chunks = tmp_path / "inline-html-attribute-chunks"
|
||||
result = run_cli(
|
||||
"split_document.py",
|
||||
"--document",
|
||||
document,
|
||||
"--output-dir",
|
||||
chunks,
|
||||
"--max-chars",
|
||||
40,
|
||||
)
|
||||
assert result.returncode == 0, result.stderr
|
||||
manifest = read_json(chunks / "manifest.json")
|
||||
assert len(manifest["chunks"]) >= 2
|
||||
assert all(item["boundary_reason"] != "oversize_atomic_block" for item in manifest["chunks"])
|
||||
|
||||
|
||||
def test_multiline_html_comment_stays_atomic(tmp_path: Path) -> None:
|
||||
document = tmp_path / "comment.md"
|
||||
text = (
|
||||
"# 제목\n\n<!--\nhidden start\n\n## hidden heading\nhidden end\n-->\n\n"
|
||||
"## 실제 절\n\n본문\n"
|
||||
)
|
||||
document.write_text(text, encoding="utf-8")
|
||||
chunks = tmp_path / "comment-chunks"
|
||||
result = run_cli(
|
||||
"split_document.py",
|
||||
"--document",
|
||||
document,
|
||||
"--output-dir",
|
||||
chunks,
|
||||
"--max-chars",
|
||||
35,
|
||||
)
|
||||
assert result.returncode == 0, result.stderr
|
||||
manifest = read_json(chunks / "manifest.json")
|
||||
pieces = [
|
||||
(chunks / item["input_file"]).read_text(encoding="utf-8")
|
||||
for item in manifest["chunks"]
|
||||
]
|
||||
comment_pieces = [piece for piece in pieces if "hidden start" in piece or "hidden end" in piece]
|
||||
assert len(comment_pieces) == 1
|
||||
|
||||
|
||||
def test_html_comment_marker_in_fence_info_does_not_hide_fence(tmp_path: Path) -> None:
|
||||
document = tmp_path / "fence-info-comment.md"
|
||||
text = (
|
||||
"# 제목\n\n"
|
||||
"```html <!--\n"
|
||||
"<section>\n\n"
|
||||
"## 여전히 코드인 제목\n"
|
||||
"</section>\n"
|
||||
"-->\n"
|
||||
"```\n\n"
|
||||
"## 실제 절\n\n본문\n"
|
||||
)
|
||||
document.write_text(text, encoding="utf-8")
|
||||
chunks = tmp_path / "fence-info-comment-chunks"
|
||||
result = run_cli(
|
||||
"split_document.py",
|
||||
"--document",
|
||||
document,
|
||||
"--output-dir",
|
||||
chunks,
|
||||
"--max-chars",
|
||||
38,
|
||||
)
|
||||
assert result.returncode == 0, result.stderr
|
||||
manifest = read_json(chunks / "manifest.json")
|
||||
pieces = [
|
||||
(chunks / item["input_file"]).read_text(encoding="utf-8")
|
||||
for item in manifest["chunks"]
|
||||
]
|
||||
code_pieces = [
|
||||
piece
|
||||
for piece in pieces
|
||||
if "<section>" in piece or "여전히 코드인 제목" in piece
|
||||
]
|
||||
assert len(code_pieces) == 1
|
||||
assert code_pieces[0].startswith("```html <!--")
|
||||
assert "-->\n```" in code_pieces[0]
|
||||
|
||||
|
||||
def test_output_dangling_symlinks_are_not_followed_without_force(tmp_path: Path) -> None:
|
||||
document = tmp_path / "source.md"
|
||||
document.write_text("# 제목\n\n본문\n", encoding="utf-8")
|
||||
outside_dir = tmp_path / "outside-dir"
|
||||
split_link = tmp_path / "split-link"
|
||||
split_link.symlink_to(outside_dir, target_is_directory=True)
|
||||
split = run_cli(
|
||||
"split_document.py",
|
||||
"--document",
|
||||
document,
|
||||
"--output-dir",
|
||||
split_link,
|
||||
)
|
||||
assert split.returncode == 2
|
||||
assert split_link.is_symlink()
|
||||
assert not outside_dir.exists()
|
||||
|
||||
chunks = tmp_path / "chunks"
|
||||
assert run_cli(
|
||||
"split_document.py",
|
||||
"--document",
|
||||
document,
|
||||
"--output-dir",
|
||||
chunks,
|
||||
).returncode == 0
|
||||
outside_file = tmp_path / "outside.md"
|
||||
output_link = tmp_path / "output-link.md"
|
||||
output_link.symlink_to(outside_file)
|
||||
assembled = run_cli(
|
||||
"reassemble_document.py",
|
||||
"--manifest",
|
||||
chunks / "manifest.json",
|
||||
"--output",
|
||||
output_link,
|
||||
)
|
||||
assert assembled.returncode == 2
|
||||
assert output_link.is_symlink()
|
||||
assert not outside_file.exists()
|
||||
|
||||
|
||||
def test_force_reassemble_cannot_overwrite_manifest_or_chunks(tmp_path: Path) -> None:
|
||||
document = tmp_path / "source.md"
|
||||
document.write_text("# 제목\n\n본문\n", encoding="utf-8")
|
||||
chunks = tmp_path / "chunks"
|
||||
assert run_cli(
|
||||
"split_document.py",
|
||||
"--document",
|
||||
document,
|
||||
"--output-dir",
|
||||
chunks,
|
||||
).returncode == 0
|
||||
manifest_path = chunks / "manifest.json"
|
||||
manifest_before = manifest_path.read_bytes()
|
||||
manifest = read_json(manifest_path)
|
||||
input_path = chunks / manifest["chunks"][0]["input_file"]
|
||||
input_before = input_path.read_bytes()
|
||||
for protected in (manifest_path, input_path, document):
|
||||
result = run_cli(
|
||||
"reassemble_document.py",
|
||||
"--manifest",
|
||||
manifest_path,
|
||||
"--output",
|
||||
protected,
|
||||
"--force",
|
||||
)
|
||||
assert result.returncode == 2
|
||||
assert "덮을 수 없습니다" in result.stderr
|
||||
assert manifest_path.read_bytes() == manifest_before
|
||||
assert input_path.read_bytes() == input_before
|
||||
|
||||
|
||||
def test_thematic_break_is_a_safe_split_boundary(tmp_path: Path) -> None:
|
||||
document = tmp_path / "thematic.md"
|
||||
document.write_text("가" * 30 + "\n* * *\n" + "나" * 30 + "\n", encoding="utf-8")
|
||||
chunks = tmp_path / "thematic-chunks"
|
||||
result = run_cli(
|
||||
"split_document.py",
|
||||
"--document",
|
||||
document,
|
||||
"--output-dir",
|
||||
chunks,
|
||||
"--max-chars",
|
||||
40,
|
||||
)
|
||||
assert result.returncode == 0, result.stderr
|
||||
manifest = read_json(chunks / "manifest.json")
|
||||
assert len(manifest["chunks"]) == 2
|
||||
assert all(
|
||||
item["boundary_reason"] != "oversize_atomic_block"
|
||||
for item in manifest["chunks"]
|
||||
)
|
||||
@@ -0,0 +1,50 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import copy
|
||||
import unittest
|
||||
|
||||
from claridoc.models import Brief, DocumentType, Outline, ValidationError
|
||||
from claridoc.structures import create_outline, reconcile_outline
|
||||
from tests.helpers import brief_dict, make_sources
|
||||
|
||||
|
||||
class StructureTests(unittest.TestCase):
|
||||
def test_every_document_type_has_unique_required_intents(self) -> None:
|
||||
for document_type in DocumentType:
|
||||
brief = Brief.from_dict(brief_dict(document_type.value))
|
||||
outline = create_outline(brief, make_sources())
|
||||
intents = [section.intent for section in outline.sections]
|
||||
self.assertEqual(len(intents), len(set(intents)), document_type.value)
|
||||
self.assertGreaterEqual(len(intents), 7, document_type.value)
|
||||
|
||||
def test_reconcile_preserves_contract_order(self) -> None:
|
||||
brief = Brief.from_dict(brief_dict())
|
||||
sources = make_sources()
|
||||
base = create_outline(brief, sources)
|
||||
candidate = Outline.from_dict(copy.deepcopy(base.to_dict()))
|
||||
candidate.sections[0].title = "A sharper promise"
|
||||
merged = reconcile_outline(base, candidate, sources)
|
||||
self.assertEqual(merged.sections[0].title, "A sharper promise")
|
||||
self.assertEqual([s.intent for s in merged.sections], [s.intent for s in base.sections])
|
||||
|
||||
def test_reconcile_rejects_removed_required_intent(self) -> None:
|
||||
brief = Brief.from_dict(brief_dict())
|
||||
sources = make_sources()
|
||||
base = create_outline(brief, sources)
|
||||
data = base.to_dict()
|
||||
data["sections"] = data["sections"][1:]
|
||||
with self.assertRaises(ValidationError):
|
||||
reconcile_outline(base, Outline.from_dict(data), sources)
|
||||
|
||||
def test_reconcile_rejects_unknown_source(self) -> None:
|
||||
brief = Brief.from_dict(brief_dict())
|
||||
sources = make_sources()
|
||||
base = create_outline(brief, sources)
|
||||
data = base.to_dict()
|
||||
data["sections"][0]["evidence_ids"] = ["S999"]
|
||||
with self.assertRaises(ValidationError):
|
||||
reconcile_outline(base, Outline.from_dict(data), sources)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -1,951 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import fcntl
|
||||
import os
|
||||
import shutil
|
||||
from pathlib import Path
|
||||
|
||||
from helpers import (
|
||||
FIXTURES,
|
||||
init_run,
|
||||
install_good_contracts,
|
||||
read_json,
|
||||
run_cli,
|
||||
write_evidence,
|
||||
write_json,
|
||||
write_reviews,
|
||||
)
|
||||
|
||||
|
||||
def transition(run_dir: Path, status: str):
|
||||
return run_cli("update_run.py", "--run-dir", run_dir, "--status", status, "--reason", "test")
|
||||
|
||||
|
||||
def install_plan_contracts(run_dir: Path) -> None:
|
||||
for source, target in (
|
||||
("reader-contract.json", "02_reader_contract.json"),
|
||||
("logic-map.json", "04_logic_map.json"),
|
||||
("term-ledger.json", "05_term_ledger.json"),
|
||||
):
|
||||
shutil.copyfile(FIXTURES / "good" / source, run_dir / target)
|
||||
|
||||
|
||||
def lint_checkpoint(run_dir: Path, *, review: bool = False) -> None:
|
||||
manifest = read_json(run_dir / "00_run.json")
|
||||
document = run_dir / ("07_draft.md" if review else "final.md")
|
||||
arguments: list[object] = [
|
||||
"lint_document.py",
|
||||
"--document",
|
||||
document,
|
||||
"--logic-map",
|
||||
run_dir / "04_logic_map.json",
|
||||
"--term-ledger",
|
||||
run_dir / "05_term_ledger.json",
|
||||
"--reader-contract",
|
||||
run_dir / "02_reader_contract.json",
|
||||
"--output",
|
||||
run_dir / "08_lint.json",
|
||||
]
|
||||
if not review:
|
||||
arguments.extend(["--draft-baseline", run_dir / "07_draft.md"])
|
||||
if manifest["mode"] == "revise":
|
||||
arguments.extend(["--baseline", manifest["inputs"]["draft"]["resolved_path"]])
|
||||
if manifest["route_hint"] == "deep":
|
||||
arguments.extend(["--fail-on", "warning"])
|
||||
result = run_cli(*arguments)
|
||||
expected = {0, 1} if review else {0}
|
||||
assert result.returncode in expected, result.stderr
|
||||
|
||||
|
||||
def prepare_light_publish(run_dir: Path, *, with_reviews: bool = False) -> None:
|
||||
install_good_contracts(run_dir)
|
||||
if with_reviews:
|
||||
manifest_path = run_dir / "00_run.json"
|
||||
manifest = read_json(manifest_path)
|
||||
manifest["omissions"] = [
|
||||
item
|
||||
for item in manifest["omissions"]
|
||||
if item["artifact"]
|
||||
not in {"08_logic_review.json", "08_reader_review.json"}
|
||||
]
|
||||
write_json(manifest_path, manifest)
|
||||
write_reviews(run_dir)
|
||||
shutil.copyfile(FIXTURES / "good" / "document.md", run_dir / "final.md")
|
||||
lint_checkpoint(run_dir)
|
||||
|
||||
|
||||
def prepare_review_run(run_dir: Path, *, with_evidence: bool = False) -> None:
|
||||
install_good_contracts(run_dir)
|
||||
if with_evidence:
|
||||
write_evidence(run_dir)
|
||||
write_reviews(run_dir)
|
||||
lint_checkpoint(run_dir, review=True)
|
||||
|
||||
|
||||
def test_only_verifier_can_record_verified_status(tmp_path: Path) -> None:
|
||||
run_dir = init_run(tmp_path, route="light")
|
||||
prepare_light_publish(run_dir)
|
||||
for status in ("planned", "drafted", "finalized"):
|
||||
result = transition(run_dir, status)
|
||||
assert result.returncode == 0, result.stderr
|
||||
rejected = transition(run_dir, "verified")
|
||||
assert rejected.returncode == 2
|
||||
assert "verify_run.py" in rejected.stderr
|
||||
manifest = read_json(run_dir / "00_run.json")
|
||||
assert manifest["status"] == "finalized"
|
||||
assert [item["to"] for item in manifest["history"]] == [
|
||||
"initialized",
|
||||
"planned",
|
||||
"drafted",
|
||||
"finalized",
|
||||
]
|
||||
|
||||
|
||||
def test_light_write_can_record_optional_independent_review_stage(tmp_path: Path) -> None:
|
||||
run_dir = init_run(tmp_path, route="light")
|
||||
prepare_light_publish(run_dir, with_reviews=True)
|
||||
for status in ("planned", "drafted", "reviewed", "finalized"):
|
||||
result = transition(run_dir, status)
|
||||
assert result.returncode == 0, result.stderr
|
||||
assert read_json(run_dir / "00_run.json")["status"] == "finalized"
|
||||
|
||||
|
||||
def test_standard_requires_evidence_and_review_stages(tmp_path: Path) -> None:
|
||||
run_dir = init_run(tmp_path, route="standard")
|
||||
install_good_contracts(run_dir)
|
||||
write_evidence(run_dir)
|
||||
write_reviews(run_dir)
|
||||
shutil.copyfile(FIXTURES / "good" / "document.md", run_dir / "final.md")
|
||||
lint_checkpoint(run_dir)
|
||||
illegal = transition(run_dir, "planned")
|
||||
assert illegal.returncode == 2
|
||||
assert read_json(run_dir / "00_run.json")["status"] == "initialized"
|
||||
for status in ("evidence_ready", "planned", "drafted", "reviewed", "finalized"):
|
||||
assert transition(run_dir, status).returncode == 0
|
||||
|
||||
|
||||
def test_illegal_backward_transition_is_rejected(tmp_path: Path) -> None:
|
||||
run_dir = init_run(tmp_path, route="light")
|
||||
install_plan_contracts(run_dir)
|
||||
assert transition(run_dir, "planned").returncode == 0
|
||||
assert transition(run_dir, "initialized").returncode == 2
|
||||
assert read_json(run_dir / "00_run.json")["status"] == "planned"
|
||||
|
||||
|
||||
def test_any_active_stage_can_hold_for_review(tmp_path: Path) -> None:
|
||||
run_dir = init_run(tmp_path, route="light")
|
||||
result = transition(run_dir, "hold_for_review")
|
||||
assert result.returncode == 0
|
||||
manifest = read_json(run_dir / "00_run.json")
|
||||
assert manifest["status"] == "hold_for_review"
|
||||
assert manifest["error"] == {
|
||||
"stage": "initialized",
|
||||
"code": "TERMINAL_HOLD_FOR_REVIEW",
|
||||
"message": "test",
|
||||
"affected_artifact": None,
|
||||
"retryable": False,
|
||||
"safe_next_action": (
|
||||
"기록된 오류와 run artifact를 확인하고 blocker를 해결한 뒤 재개 여부를 판단한다."
|
||||
),
|
||||
}
|
||||
assert manifest["history"][-1]["error"] == manifest["error"]
|
||||
|
||||
|
||||
def test_terminal_transition_records_explicit_structured_error(tmp_path: Path) -> None:
|
||||
run_dir = init_run(tmp_path, route="light")
|
||||
result = run_cli(
|
||||
"update_run.py",
|
||||
"--run-dir",
|
||||
run_dir,
|
||||
"--status",
|
||||
"hold_for_review",
|
||||
"--reason",
|
||||
"required source is unavailable",
|
||||
"--error-stage",
|
||||
"evidence",
|
||||
"--error-code",
|
||||
"SOURCE_UNAVAILABLE",
|
||||
"--error-message",
|
||||
"필수 source를 읽을 수 없습니다.",
|
||||
"--error-affected-artifact",
|
||||
"03_evidence_map.json",
|
||||
"--error-retryable",
|
||||
"--error-safe-next-action",
|
||||
"source 접근을 복구한 뒤 evidence 단계부터 재실행한다.",
|
||||
)
|
||||
assert result.returncode == 0, result.stderr
|
||||
manifest = read_json(run_dir / "00_run.json")
|
||||
assert manifest["error"] == {
|
||||
"stage": "evidence",
|
||||
"code": "SOURCE_UNAVAILABLE",
|
||||
"message": "필수 source를 읽을 수 없습니다.",
|
||||
"affected_artifact": "03_evidence_map.json",
|
||||
"retryable": True,
|
||||
"safe_next_action": "source 접근을 복구한 뒤 evidence 단계부터 재실행한다.",
|
||||
}
|
||||
assert manifest["history"][-1] == {
|
||||
"at": manifest["updated_at"],
|
||||
"from": "initialized",
|
||||
"to": "hold_for_review",
|
||||
"reason": "required source is unavailable",
|
||||
"error": manifest["error"],
|
||||
}
|
||||
|
||||
|
||||
def test_run_schema_requires_error_shape_to_match_status(tmp_path: Path) -> None:
|
||||
nonterminal_case = tmp_path / "nonterminal"
|
||||
nonterminal_case.mkdir()
|
||||
nonterminal_dir = init_run(nonterminal_case, route="light")
|
||||
nonterminal_path = nonterminal_dir / "00_run.json"
|
||||
nonterminal = read_json(nonterminal_path)
|
||||
nonterminal["error"] = {
|
||||
"stage": "initialized",
|
||||
"code": "SHOULD_BE_NULL",
|
||||
"message": "non-terminal error",
|
||||
"affected_artifact": None,
|
||||
"retryable": False,
|
||||
"safe_next_action": "오류를 제거한다.",
|
||||
}
|
||||
write_json(nonterminal_path, nonterminal)
|
||||
rejected_nonterminal = transition(nonterminal_dir, "planned")
|
||||
assert rejected_nonterminal.returncode == 2
|
||||
assert "schema 위반" in rejected_nonterminal.stderr
|
||||
|
||||
terminal_case = tmp_path / "terminal"
|
||||
terminal_case.mkdir()
|
||||
terminal_dir = init_run(terminal_case, route="light")
|
||||
assert transition(terminal_dir, "failed").returncode == 0
|
||||
terminal_path = terminal_dir / "00_run.json"
|
||||
terminal = read_json(terminal_path)
|
||||
terminal["error"] = None
|
||||
terminal["history"][-1]["error"] = None
|
||||
write_json(terminal_path, terminal)
|
||||
rejected_terminal = transition(terminal_dir, "failed")
|
||||
assert rejected_terminal.returncode == 2
|
||||
assert "schema 위반" in rejected_terminal.stderr
|
||||
|
||||
|
||||
def test_run_history_error_snapshot_must_match_top_level_error(tmp_path: Path) -> None:
|
||||
run_dir = init_run(tmp_path, route="light")
|
||||
assert transition(run_dir, "incomplete").returncode == 0
|
||||
manifest_path = run_dir / "00_run.json"
|
||||
manifest = read_json(manifest_path)
|
||||
manifest["error"]["message"] = "history와 다른 message"
|
||||
write_json(manifest_path, manifest)
|
||||
|
||||
result = transition(run_dir, "incomplete")
|
||||
|
||||
assert result.returncode == 2
|
||||
assert "마지막 history error snapshot" in result.stderr
|
||||
|
||||
|
||||
def test_review_finishes_at_reviewed_without_finalized(tmp_path: Path) -> None:
|
||||
run_dir = init_run(tmp_path, route="light", mode="review", with_draft=True)
|
||||
prepare_review_run(run_dir)
|
||||
assert transition(run_dir, "planned").returncode == 0
|
||||
assert transition(run_dir, "reviewed").returncode == 0
|
||||
assert transition(run_dir, "finalized").returncode == 2
|
||||
|
||||
|
||||
def test_standard_review_requires_evidence_before_plan(tmp_path: Path) -> None:
|
||||
run_dir = init_run(tmp_path, route="standard", mode="review", with_draft=True)
|
||||
prepare_review_run(run_dir, with_evidence=True)
|
||||
assert transition(run_dir, "planned").returncode == 2
|
||||
for status in ("evidence_ready", "planned", "reviewed"):
|
||||
result = transition(run_dir, status)
|
||||
assert result.returncode == 0, result.stderr
|
||||
assert read_json(run_dir / "00_run.json")["status"] == "reviewed"
|
||||
|
||||
|
||||
def test_light_omission_only_records_multiple_reasons(tmp_path: Path) -> None:
|
||||
run_dir = init_run(tmp_path, route="light")
|
||||
result = run_cli(
|
||||
"update_run.py",
|
||||
"--run-dir",
|
||||
run_dir,
|
||||
"--omit",
|
||||
"03_evidence_map.json",
|
||||
"light 경로에서 별도 근거 큐레이션을 생략한다.",
|
||||
"--omit",
|
||||
"08_logic_review.json",
|
||||
"light 경로에서 독립 논리 리뷰를 생략한다.",
|
||||
"--omit",
|
||||
"08_reader_review.json",
|
||||
"light 경로에서 독립 독자 리뷰를 생략한다.",
|
||||
)
|
||||
assert result.returncode == 0, result.stderr
|
||||
manifest = read_json(run_dir / "00_run.json")
|
||||
assert manifest["status"] == "initialized"
|
||||
assert [item["artifact"] for item in manifest["omissions"]] == [
|
||||
"03_evidence_map.json",
|
||||
"08_logic_review.json",
|
||||
"08_reader_review.json",
|
||||
]
|
||||
assert [item["to"] for item in manifest["history"]] == ["initialized"]
|
||||
|
||||
|
||||
def test_transition_and_omission_are_one_atomic_update(tmp_path: Path) -> None:
|
||||
run_dir = init_run(tmp_path, route="light")
|
||||
install_plan_contracts(run_dir)
|
||||
result = run_cli(
|
||||
"update_run.py",
|
||||
"--run-dir",
|
||||
run_dir,
|
||||
"--status",
|
||||
"planned",
|
||||
"--reason",
|
||||
"planning contracts validated",
|
||||
"--omit",
|
||||
"03_evidence_map.json",
|
||||
"light 경로에서 evidence map을 생략한다.",
|
||||
)
|
||||
assert result.returncode == 0, result.stderr
|
||||
manifest = read_json(run_dir / "00_run.json")
|
||||
assert manifest["status"] == "planned"
|
||||
assert manifest["omissions"] == [
|
||||
{
|
||||
"artifact": "03_evidence_map.json",
|
||||
"reason": "light 경로에서 evidence map을 생략한다.",
|
||||
}
|
||||
]
|
||||
assert manifest["history"][-1]["to"] == "planned"
|
||||
|
||||
|
||||
def test_duplicate_or_unknown_omission_is_rejected_without_partial_write(
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
run_dir = init_run(tmp_path, route="light")
|
||||
duplicate = run_cli(
|
||||
"update_run.py",
|
||||
"--run-dir",
|
||||
run_dir,
|
||||
"--status",
|
||||
"planned",
|
||||
"--omit",
|
||||
"03_evidence_map.json",
|
||||
"첫 번째 이유",
|
||||
"--omit",
|
||||
"03_evidence_map.json",
|
||||
"중복 이유",
|
||||
)
|
||||
assert duplicate.returncode == 2
|
||||
assert "duplicate=" in duplicate.stderr
|
||||
manifest = read_json(run_dir / "00_run.json")
|
||||
assert manifest["status"] == "initialized"
|
||||
assert manifest["omissions"] == []
|
||||
|
||||
unknown = run_cli(
|
||||
"update_run.py",
|
||||
"--run-dir",
|
||||
run_dir,
|
||||
"--omit",
|
||||
"08_combined_review.json",
|
||||
"계약에 없는 이름",
|
||||
)
|
||||
assert unknown.returncode == 2
|
||||
assert "unknown=" in unknown.stderr
|
||||
assert read_json(run_dir / "00_run.json")["omissions"] == []
|
||||
|
||||
|
||||
def test_present_optional_artifact_cannot_be_declared_omitted(tmp_path: Path) -> None:
|
||||
run_dir = init_run(tmp_path, route="light")
|
||||
(run_dir / "03_evidence_map.json").write_text("{}\n", encoding="utf-8")
|
||||
result = run_cli(
|
||||
"update_run.py",
|
||||
"--run-dir",
|
||||
run_dir,
|
||||
"--omit",
|
||||
"03_evidence_map.json",
|
||||
"이미 있는데 생략했다고 선언",
|
||||
)
|
||||
assert result.returncode == 2
|
||||
assert "present=" in result.stderr
|
||||
assert read_json(run_dir / "00_run.json")["omissions"] == []
|
||||
|
||||
|
||||
def test_route_and_mode_required_artifacts_cannot_be_omitted(tmp_path: Path) -> None:
|
||||
cases = (
|
||||
("light", "write", "07_draft.md"),
|
||||
("standard", "write", "03_evidence_map.json"),
|
||||
("light", "review", "08_logic_review.json"),
|
||||
)
|
||||
for index, (route, mode, artifact) in enumerate(cases):
|
||||
case_dir = tmp_path / f"case-{index}"
|
||||
case_dir.mkdir()
|
||||
run_dir = init_run(case_dir, route=route, mode=mode, with_draft=mode == "review")
|
||||
result = run_cli(
|
||||
"update_run.py",
|
||||
"--run-dir",
|
||||
run_dir,
|
||||
"--omit",
|
||||
artifact,
|
||||
"필수 산출물을 잘못 생략",
|
||||
)
|
||||
assert result.returncode == 2
|
||||
assert "required=" in result.stderr
|
||||
assert read_json(run_dir / "00_run.json")["omissions"] == []
|
||||
|
||||
|
||||
def test_update_rejects_invalid_run_schema_before_writing(tmp_path: Path) -> None:
|
||||
run_dir = init_run(tmp_path, route="light")
|
||||
manifest_path = run_dir / "00_run.json"
|
||||
manifest = read_json(manifest_path)
|
||||
manifest["kind_reason"] = ""
|
||||
write_json(manifest_path, manifest)
|
||||
|
||||
result = transition(run_dir, "planned")
|
||||
assert result.returncode == 2
|
||||
assert "schema 위반" in result.stderr
|
||||
assert read_json(manifest_path)["status"] == "initialized"
|
||||
|
||||
|
||||
def test_unomit_then_create_optional_artifact(tmp_path: Path) -> None:
|
||||
run_dir = init_run(tmp_path, route="light")
|
||||
install_plan_contracts(run_dir)
|
||||
omitted = run_cli(
|
||||
"update_run.py",
|
||||
"--run-dir",
|
||||
run_dir,
|
||||
"--omit",
|
||||
"08_logic_review.json",
|
||||
"처음에는 별도 논리 리뷰가 불필요했다.",
|
||||
)
|
||||
assert omitted.returncode == 0, omitted.stderr
|
||||
|
||||
restored = run_cli(
|
||||
"update_run.py",
|
||||
"--run-dir",
|
||||
run_dir,
|
||||
"--unomit",
|
||||
"08_logic_review.json",
|
||||
)
|
||||
assert restored.returncode == 0, restored.stderr
|
||||
assert read_json(run_dir / "00_run.json")["omissions"] == []
|
||||
|
||||
(run_dir / "08_logic_review.json").write_text("{}\n", encoding="utf-8")
|
||||
advanced = transition(run_dir, "planned")
|
||||
assert advanced.returncode == 0, advanced.stderr
|
||||
|
||||
|
||||
def test_unomit_and_new_omission_can_be_one_atomic_transition(tmp_path: Path) -> None:
|
||||
run_dir = init_run(tmp_path, route="light")
|
||||
install_plan_contracts(run_dir)
|
||||
initial = run_cli(
|
||||
"update_run.py",
|
||||
"--run-dir",
|
||||
run_dir,
|
||||
"--omit",
|
||||
"03_evidence_map.json",
|
||||
"처음 생략",
|
||||
)
|
||||
assert initial.returncode == 0, initial.stderr
|
||||
|
||||
changed = run_cli(
|
||||
"update_run.py",
|
||||
"--run-dir",
|
||||
run_dir,
|
||||
"--status",
|
||||
"planned",
|
||||
"--unomit",
|
||||
"03_evidence_map.json",
|
||||
"--omit",
|
||||
"08_logic_review.json",
|
||||
"대신 논리 리뷰를 생략",
|
||||
)
|
||||
assert changed.returncode == 0, changed.stderr
|
||||
manifest = read_json(run_dir / "00_run.json")
|
||||
assert manifest["status"] == "planned"
|
||||
assert [item["artifact"] for item in manifest["omissions"]] == [
|
||||
"08_logic_review.json"
|
||||
]
|
||||
|
||||
|
||||
def test_unknown_or_undeclared_unomit_is_rejected(tmp_path: Path) -> None:
|
||||
run_dir = init_run(tmp_path, route="light")
|
||||
undeclared = run_cli(
|
||||
"update_run.py",
|
||||
"--run-dir",
|
||||
run_dir,
|
||||
"--unomit",
|
||||
"08_logic_review.json",
|
||||
)
|
||||
assert undeclared.returncode == 2
|
||||
assert "not_declared=" in undeclared.stderr
|
||||
|
||||
unknown = run_cli(
|
||||
"update_run.py",
|
||||
"--run-dir",
|
||||
run_dir,
|
||||
"--unomit",
|
||||
"08_combined_review.json",
|
||||
)
|
||||
assert unknown.returncode == 2
|
||||
assert "unknown_unomit=" in unknown.stderr
|
||||
assert read_json(run_dir / "00_run.json")["omissions"] == []
|
||||
|
||||
|
||||
def test_status_only_terminal_transition_ignores_business_invalid_omission(
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
run_dir = init_run(tmp_path, route="light")
|
||||
omitted = run_cli(
|
||||
"update_run.py",
|
||||
"--run-dir",
|
||||
run_dir,
|
||||
"--omit",
|
||||
"03_evidence_map.json",
|
||||
"현재는 생략한다.",
|
||||
)
|
||||
assert omitted.returncode == 0, omitted.stderr
|
||||
(run_dir / "03_evidence_map.json").write_text("{}\n", encoding="utf-8")
|
||||
|
||||
held = transition(run_dir, "hold_for_review")
|
||||
assert held.returncode == 0, held.stderr
|
||||
manifest = read_json(run_dir / "00_run.json")
|
||||
assert manifest["status"] == "hold_for_review"
|
||||
assert manifest["omissions"][0]["artifact"] == "03_evidence_map.json"
|
||||
|
||||
|
||||
def test_normal_status_transition_revalidates_existing_omissions(tmp_path: Path) -> None:
|
||||
run_dir = init_run(tmp_path, route="light")
|
||||
omitted = run_cli(
|
||||
"update_run.py",
|
||||
"--run-dir",
|
||||
run_dir,
|
||||
"--omit",
|
||||
"03_evidence_map.json",
|
||||
"현재는 생략한다.",
|
||||
)
|
||||
assert omitted.returncode == 0, omitted.stderr
|
||||
(run_dir / "03_evidence_map.json").write_text("{}\n", encoding="utf-8")
|
||||
|
||||
normal = transition(run_dir, "planned")
|
||||
assert normal.returncode == 2
|
||||
assert "present=" in normal.stderr
|
||||
assert read_json(run_dir / "00_run.json")["status"] == "initialized"
|
||||
|
||||
|
||||
def test_whitespace_only_omission_reason_is_rejected(tmp_path: Path) -> None:
|
||||
run_dir = init_run(tmp_path, route="light")
|
||||
result = run_cli(
|
||||
"update_run.py",
|
||||
"--run-dir",
|
||||
run_dir,
|
||||
"--omit",
|
||||
"03_evidence_map.json",
|
||||
" ",
|
||||
)
|
||||
assert result.returncode == 2
|
||||
assert "reason" in result.stderr
|
||||
assert read_json(run_dir / "00_run.json")["omissions"] == []
|
||||
|
||||
|
||||
def test_stale_lock_file_does_not_block_recovery(tmp_path: Path) -> None:
|
||||
run_dir = init_run(tmp_path, route="light")
|
||||
install_plan_contracts(run_dir)
|
||||
(run_dir / ".00_run.lock").write_text("99999999", encoding="ascii")
|
||||
result = transition(run_dir, "planned")
|
||||
assert result.returncode == 0, result.stderr
|
||||
assert read_json(run_dir / "00_run.json")["status"] == "planned"
|
||||
|
||||
|
||||
def test_live_run_lock_rejects_concurrent_update_without_mutation(tmp_path: Path) -> None:
|
||||
run_dir = init_run(tmp_path, route="light")
|
||||
descriptor = os.open(run_dir / ".00_run.lock", os.O_CREAT | os.O_RDWR, 0o600)
|
||||
try:
|
||||
fcntl.flock(descriptor, fcntl.LOCK_EX | fcntl.LOCK_NB)
|
||||
result = transition(run_dir, "planned")
|
||||
assert result.returncode == 2
|
||||
assert "진행 중" in result.stderr
|
||||
assert read_json(run_dir / "00_run.json")["status"] == "initialized"
|
||||
finally:
|
||||
fcntl.flock(descriptor, fcntl.LOCK_UN)
|
||||
os.close(descriptor)
|
||||
|
||||
|
||||
def test_run_lock_rejects_hard_link_without_mutating_the_other_name(
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
run_dir = init_run(tmp_path, route="light")
|
||||
outside = tmp_path / "sensitive.txt"
|
||||
outside.write_text("SENSITIVE-CONTENT", encoding="utf-8")
|
||||
os.link(outside, run_dir / ".00_run.lock")
|
||||
result = transition(run_dir, "planned")
|
||||
assert result.returncode == 2
|
||||
assert "hard link" in result.stderr
|
||||
assert outside.read_text(encoding="utf-8") == "SENSITIVE-CONTENT"
|
||||
assert read_json(run_dir / "00_run.json")["status"] == "initialized"
|
||||
|
||||
|
||||
def test_checkpoint_transitions_require_current_stage_artifacts(tmp_path: Path) -> None:
|
||||
light_root = tmp_path / "light"
|
||||
light_root.mkdir()
|
||||
light_dir = init_run(light_root, route="light")
|
||||
missing_plan = transition(light_dir, "planned")
|
||||
assert missing_plan.returncode == 2
|
||||
assert "02_reader_contract.json" in missing_plan.stderr
|
||||
assert read_json(light_dir / "00_run.json")["status"] == "initialized"
|
||||
|
||||
standard_root = tmp_path / "standard"
|
||||
standard_root.mkdir()
|
||||
standard_dir = init_run(standard_root, route="standard")
|
||||
missing_evidence = transition(standard_dir, "evidence_ready")
|
||||
assert missing_evidence.returncode == 2
|
||||
assert "03_evidence_map.json" in missing_evidence.stderr
|
||||
assert read_json(standard_dir / "00_run.json")["status"] == "initialized"
|
||||
|
||||
install_plan_contracts(light_dir)
|
||||
assert transition(light_dir, "planned").returncode == 0
|
||||
missing_draft = transition(light_dir, "drafted")
|
||||
assert missing_draft.returncode == 2
|
||||
assert "07_draft.md" in missing_draft.stderr
|
||||
assert read_json(light_dir / "00_run.json")["status"] == "planned"
|
||||
|
||||
|
||||
def test_drafted_checkpoint_rejects_non_utf8_markdown(tmp_path: Path) -> None:
|
||||
run_dir = init_run(tmp_path, route="light")
|
||||
install_plan_contracts(run_dir)
|
||||
assert transition(run_dir, "planned").returncode == 0
|
||||
(run_dir / "07_draft.md").write_bytes(b"\xff\xfe\x00")
|
||||
|
||||
rejected = transition(run_dir, "drafted")
|
||||
|
||||
assert rejected.returncode == 2
|
||||
assert "UTF-8" in rejected.stderr
|
||||
assert read_json(run_dir / "00_run.json")["status"] == "planned"
|
||||
|
||||
|
||||
def test_drafted_checkpoint_rejects_structurally_invalid_markdown(
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
run_dir = init_run(tmp_path, route="light")
|
||||
install_plan_contracts(run_dir)
|
||||
assert transition(run_dir, "planned").returncode == 0
|
||||
(run_dir / "07_draft.md").write_text(
|
||||
"# 캐시 실패를 줄이는 설명\n\n```text\n닫히지 않은 코드\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
rejected = transition(run_dir, "drafted")
|
||||
|
||||
assert rejected.returncode == 2
|
||||
assert "structural lint" in rejected.stderr
|
||||
assert "DOC-M002" in rejected.stderr
|
||||
assert read_json(run_dir / "00_run.json")["status"] == "planned"
|
||||
|
||||
|
||||
def test_evidence_checkpoint_binds_registry_and_external_source_hashes(
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
registry_root = tmp_path / "registry"
|
||||
registry_root.mkdir()
|
||||
registry_dir = init_run(registry_root, route="standard")
|
||||
write_evidence(registry_dir)
|
||||
sources_path = registry_dir / "01_sources.json"
|
||||
sources = read_json(sources_path)
|
||||
sources["brief"]["sha256"] = "0" * 64
|
||||
write_json(sources_path, sources)
|
||||
|
||||
tampered_registry = transition(registry_dir, "evidence_ready")
|
||||
|
||||
assert tampered_registry.returncode == 2
|
||||
assert "sources_manifest_sha256" in tampered_registry.stderr
|
||||
assert read_json(registry_dir / "00_run.json")["status"] == "initialized"
|
||||
|
||||
source_root = tmp_path / "source"
|
||||
source_root.mkdir()
|
||||
source_dir = init_run(source_root, route="standard")
|
||||
write_evidence(source_dir)
|
||||
manifest = read_json(source_dir / "00_run.json")
|
||||
brief_path = Path(manifest["inputs"]["brief"]["resolved_path"])
|
||||
brief_path.write_text("drifted source\n", encoding="utf-8")
|
||||
|
||||
drifted_source = transition(source_dir, "evidence_ready")
|
||||
|
||||
assert drifted_source.returncode == 2
|
||||
assert "run 초기화 snapshot" in drifted_source.stderr
|
||||
assert read_json(source_dir / "00_run.json")["status"] == "initialized"
|
||||
|
||||
|
||||
def test_evidence_checkpoint_rejects_duplicate_source_locator(tmp_path: Path) -> None:
|
||||
run_dir = init_run(tmp_path, route="standard")
|
||||
write_evidence(run_dir)
|
||||
evidence_path = run_dir / "03_evidence_map.json"
|
||||
evidence = read_json(evidence_path)
|
||||
evidence["claims"][0]["source_locations"].append(
|
||||
dict(evidence["claims"][0]["source_locations"][0])
|
||||
)
|
||||
write_json(evidence_path, evidence)
|
||||
|
||||
rejected = transition(run_dir, "evidence_ready")
|
||||
|
||||
assert rejected.returncode == 2
|
||||
assert "source locator가 중복" in rejected.stderr
|
||||
assert read_json(run_dir / "00_run.json")["status"] == "initialized"
|
||||
|
||||
|
||||
def test_light_checkpoint_validates_present_optional_evidence(tmp_path: Path) -> None:
|
||||
run_dir = init_run(tmp_path, route="light")
|
||||
install_good_contracts(run_dir)
|
||||
manifest_path = run_dir / "00_run.json"
|
||||
manifest = read_json(manifest_path)
|
||||
manifest["omissions"] = [
|
||||
item
|
||||
for item in manifest["omissions"]
|
||||
if item["artifact"] != "03_evidence_map.json"
|
||||
]
|
||||
write_json(manifest_path, manifest)
|
||||
write_evidence(run_dir)
|
||||
evidence_path = run_dir / "03_evidence_map.json"
|
||||
evidence = read_json(evidence_path)
|
||||
evidence["claims"][0]["source_locations"].append(
|
||||
dict(evidence["claims"][0]["source_locations"][0])
|
||||
)
|
||||
write_json(evidence_path, evidence)
|
||||
|
||||
rejected = transition(run_dir, "planned")
|
||||
|
||||
assert rejected.returncode == 2
|
||||
assert "source locator가 중복" in rejected.stderr
|
||||
assert read_json(run_dir / "00_run.json")["status"] == "initialized"
|
||||
|
||||
|
||||
def test_plan_checkpoint_rejects_invalid_schema_and_symlink(tmp_path: Path) -> None:
|
||||
invalid_root = tmp_path / "invalid"
|
||||
invalid_root.mkdir()
|
||||
invalid_dir = init_run(invalid_root, route="light")
|
||||
install_plan_contracts(invalid_dir)
|
||||
write_json(invalid_dir / "02_reader_contract.json", {})
|
||||
invalid = transition(invalid_dir, "planned")
|
||||
assert invalid.returncode == 2
|
||||
assert "schema 위반" in invalid.stderr
|
||||
assert read_json(invalid_dir / "00_run.json")["status"] == "initialized"
|
||||
|
||||
symlink_root = tmp_path / "symlink"
|
||||
symlink_root.mkdir()
|
||||
symlink_dir = init_run(symlink_root, route="light")
|
||||
install_plan_contracts(symlink_dir)
|
||||
reader = symlink_dir / "02_reader_contract.json"
|
||||
external = tmp_path / "reader.json"
|
||||
shutil.copyfile(reader, external)
|
||||
reader.unlink()
|
||||
reader.symlink_to(external)
|
||||
linked = transition(symlink_dir, "planned")
|
||||
assert linked.returncode == 2
|
||||
assert "symbolic link" in linked.stderr
|
||||
assert read_json(symlink_dir / "00_run.json")["status"] == "initialized"
|
||||
|
||||
|
||||
def test_plan_checkpoint_rejects_normalized_term_name_collision(
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
run_dir = init_run(tmp_path, route="light")
|
||||
install_plan_contracts(run_dir)
|
||||
ledger_path = run_dir / "05_term_ledger.json"
|
||||
ledger = read_json(ledger_path)
|
||||
ledger["terms"][1]["aliases"].append(" 멱등성 ")
|
||||
write_json(ledger_path, ledger)
|
||||
|
||||
rejected = transition(run_dir, "planned")
|
||||
|
||||
assert rejected.returncode == 2
|
||||
assert "둘 이상에 배정" in rejected.stderr
|
||||
assert read_json(run_dir / "00_run.json")["status"] == "initialized"
|
||||
|
||||
|
||||
def test_plan_checkpoint_binds_term_budgets_to_quality_rules(tmp_path: Path) -> None:
|
||||
run_dir = init_run(tmp_path, route="light")
|
||||
install_plan_contracts(run_dir)
|
||||
ledger_path = run_dir / "05_term_ledger.json"
|
||||
ledger = read_json(ledger_path)
|
||||
ledger["budgets"]["per_sentence"] = 999
|
||||
write_json(ledger_path, ledger)
|
||||
|
||||
rejected = transition(run_dir, "planned")
|
||||
|
||||
assert rejected.returncode == 2
|
||||
assert "budgets가 quality rules와 다릅니다" in rejected.stderr
|
||||
assert read_json(run_dir / "00_run.json")["status"] == "initialized"
|
||||
|
||||
|
||||
def test_plan_checkpoint_rejects_blank_reader_list_item(tmp_path: Path) -> None:
|
||||
run_dir = init_run(tmp_path, route="light")
|
||||
install_plan_contracts(run_dir)
|
||||
reader_path = run_dir / "02_reader_contract.json"
|
||||
reader = read_json(reader_path)
|
||||
reader["non_goals"] = [""]
|
||||
write_json(reader_path, reader)
|
||||
|
||||
rejected = transition(run_dir, "planned")
|
||||
|
||||
assert rejected.returncode == 2
|
||||
assert "schema 위반" in rejected.stderr
|
||||
assert "non_goals" in rejected.stderr
|
||||
assert read_json(run_dir / "00_run.json")["status"] == "initialized"
|
||||
|
||||
|
||||
def test_review_checkpoint_rejects_stale_draft_and_upstream_hashes(
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
run_dir = init_run(tmp_path, route="light")
|
||||
prepare_light_publish(run_dir, with_reviews=True)
|
||||
assert transition(run_dir, "planned").returncode == 0
|
||||
assert transition(run_dir, "drafted").returncode == 0
|
||||
draft_path = run_dir / "07_draft.md"
|
||||
draft_path.write_text(
|
||||
draft_path.read_text(encoding="utf-8") + "\n현재 draft 변경\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
rejected = transition(run_dir, "reviewed")
|
||||
|
||||
assert rejected.returncode == 2
|
||||
assert "현재 07_draft.md" in rejected.stderr
|
||||
assert read_json(run_dir / "00_run.json")["status"] == "drafted"
|
||||
|
||||
|
||||
def test_review_mode_checkpoint_requires_current_lint_report(tmp_path: Path) -> None:
|
||||
run_dir = init_run(
|
||||
tmp_path,
|
||||
route="light",
|
||||
mode="review",
|
||||
with_draft=True,
|
||||
)
|
||||
prepare_review_run(run_dir)
|
||||
assert transition(run_dir, "planned").returncode == 0
|
||||
lint_report = read_json(run_dir / "08_lint.json")
|
||||
lint_report["limitations"].append("forged limitation")
|
||||
write_json(run_dir / "08_lint.json", lint_report)
|
||||
|
||||
rejected = transition(run_dir, "reviewed")
|
||||
|
||||
assert rejected.returncode == 2
|
||||
assert "canonical lint" in rejected.stderr
|
||||
assert read_json(run_dir / "00_run.json")["status"] == "planned"
|
||||
|
||||
|
||||
def test_review_mode_planned_checkpoint_rejects_mutated_immutable_draft(
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
run_dir = init_run(
|
||||
tmp_path,
|
||||
route="light",
|
||||
mode="review",
|
||||
with_draft=True,
|
||||
)
|
||||
install_good_contracts(run_dir)
|
||||
draft_path = run_dir / "07_draft.md"
|
||||
draft_path.write_text(
|
||||
draft_path.read_text(encoding="utf-8") + "\n초기화 뒤 변경된 문장입니다.\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
rejected = transition(run_dir, "planned")
|
||||
|
||||
assert rejected.returncode == 2
|
||||
assert "byte-identical" in rejected.stderr
|
||||
assert read_json(run_dir / "00_run.json")["status"] == "initialized"
|
||||
|
||||
|
||||
def test_verifier_independently_rejects_mutated_review_mode_draft(
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
run_dir = init_run(
|
||||
tmp_path,
|
||||
route="light",
|
||||
mode="review",
|
||||
with_draft=True,
|
||||
)
|
||||
prepare_review_run(run_dir)
|
||||
assert transition(run_dir, "planned").returncode == 0
|
||||
assert transition(run_dir, "reviewed").returncode == 0
|
||||
draft_path = run_dir / "07_draft.md"
|
||||
draft_path.write_text(
|
||||
draft_path.read_text(encoding="utf-8") + "\n초기화 뒤 변경된 문장입니다.\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
write_reviews(run_dir)
|
||||
lint_checkpoint(run_dir, review=True)
|
||||
|
||||
result = run_cli("verify_run.py", "--run-dir", run_dir)
|
||||
|
||||
assert result.returncode == 1
|
||||
report = read_json(run_dir / "09_final_report.json")
|
||||
immutable_check = next(
|
||||
item for item in report["checks"] if item["id"] == "review-draft-immutable"
|
||||
)
|
||||
assert immutable_check["status"] == "fail"
|
||||
assert read_json(run_dir / "00_run.json")["status"] == "hold_for_review"
|
||||
|
||||
|
||||
def test_final_checkpoint_requires_exact_copy_and_canonical_pass_lint(
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
mismatch_root = tmp_path / "mismatch"
|
||||
mismatch_root.mkdir()
|
||||
mismatch_dir = init_run(mismatch_root, route="light")
|
||||
prepare_light_publish(mismatch_dir)
|
||||
assert transition(mismatch_dir, "planned").returncode == 0
|
||||
assert transition(mismatch_dir, "drafted").returncode == 0
|
||||
(mismatch_dir / "final.md").write_text("different\n", encoding="utf-8")
|
||||
mismatch = transition(mismatch_dir, "finalized")
|
||||
assert mismatch.returncode == 2
|
||||
assert "byte-identical" in mismatch.stderr
|
||||
assert read_json(mismatch_dir / "00_run.json")["status"] == "drafted"
|
||||
|
||||
forged_root = tmp_path / "forged"
|
||||
forged_root.mkdir()
|
||||
forged_dir = init_run(forged_root, route="light")
|
||||
prepare_light_publish(forged_dir)
|
||||
assert transition(forged_dir, "planned").returncode == 0
|
||||
assert transition(forged_dir, "drafted").returncode == 0
|
||||
lint_report = read_json(forged_dir / "08_lint.json")
|
||||
lint_report["limitations"].append("forged limitation")
|
||||
write_json(forged_dir / "08_lint.json", lint_report)
|
||||
forged = transition(forged_dir, "finalized")
|
||||
assert forged.returncode == 2
|
||||
assert "canonical lint" in forged.stderr
|
||||
assert read_json(forged_dir / "00_run.json")["status"] == "drafted"
|
||||
|
||||
|
||||
def test_revise_final_checkpoint_rejects_drifted_original_draft(
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
run_dir = init_run(
|
||||
tmp_path,
|
||||
route="light",
|
||||
mode="revise",
|
||||
with_draft=True,
|
||||
)
|
||||
prepare_light_publish(run_dir)
|
||||
assert transition(run_dir, "planned").returncode == 0
|
||||
assert transition(run_dir, "drafted").returncode == 0
|
||||
manifest = read_json(run_dir / "00_run.json")
|
||||
original_path = Path(manifest["inputs"]["draft"]["resolved_path"])
|
||||
original_path.write_text(
|
||||
original_path.read_text(encoding="utf-8") + "\n원본이 나중에 바뀜\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
rejected = transition(run_dir, "finalized")
|
||||
|
||||
assert rejected.returncode == 2
|
||||
assert "run 초기화" in rejected.stderr
|
||||
assert "SHA-256" in rejected.stderr
|
||||
assert read_json(run_dir / "00_run.json")["status"] == "drafted"
|
||||
|
||||
|
||||
def test_failure_statuses_are_terminal(tmp_path: Path) -> None:
|
||||
run_dir = init_run(tmp_path, route="light")
|
||||
assert transition(run_dir, "failed").returncode == 0
|
||||
|
||||
rejected = transition(run_dir, "hold_for_review")
|
||||
|
||||
assert rejected.returncode == 2
|
||||
assert "허용되지 않은 status 전이" in rejected.stderr
|
||||
manifest = read_json(run_dir / "00_run.json")
|
||||
assert manifest["status"] == "failed"
|
||||
assert [entry["to"] for entry in manifest["history"]] == [
|
||||
"initialized",
|
||||
"failed",
|
||||
]
|
||||
@@ -1,863 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import importlib.util
|
||||
import shutil
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
from helpers import (
|
||||
FIXTURES,
|
||||
init_run,
|
||||
install_good_contracts,
|
||||
read_json,
|
||||
run_cli,
|
||||
write_evidence,
|
||||
write_json,
|
||||
write_reviews,
|
||||
)
|
||||
|
||||
|
||||
def transition(run_dir: Path, status: str) -> None:
|
||||
result = run_cli("update_run.py", "--run-dir", run_dir, "--status", status, "--reason", "test")
|
||||
assert result.returncode == 0, result.stderr
|
||||
|
||||
|
||||
def lint_final(run_dir: Path, document_name: str = "final.md") -> None:
|
||||
arguments: list[object] = [
|
||||
"lint_document.py",
|
||||
"--document",
|
||||
run_dir / document_name,
|
||||
"--logic-map",
|
||||
run_dir / "04_logic_map.json",
|
||||
"--term-ledger",
|
||||
run_dir / "05_term_ledger.json",
|
||||
"--reader-contract",
|
||||
run_dir / "02_reader_contract.json",
|
||||
"--output",
|
||||
run_dir / "08_lint.json",
|
||||
]
|
||||
manifest = read_json(run_dir / "00_run.json")
|
||||
if manifest["mode"] in {"write", "revise"}:
|
||||
arguments.extend(["--draft-baseline", run_dir / "07_draft.md"])
|
||||
if manifest["mode"] == "revise":
|
||||
arguments.extend(["--baseline", manifest["inputs"]["draft"]["resolved_path"]])
|
||||
if manifest["route_hint"] == "deep":
|
||||
arguments.extend(["--fail-on", "warning"])
|
||||
result = run_cli(*arguments)
|
||||
assert result.returncode == 0, result.stderr
|
||||
|
||||
|
||||
def blocking_finding(finding_id: str = "R-BLOCK") -> dict[str, str]:
|
||||
return {
|
||||
"id": finding_id,
|
||||
"severity": "high",
|
||||
"location": "원리 절 1문단",
|
||||
"reader_impact": "핵심 인과를 잘못 이해할 수 있다.",
|
||||
"suggestion": "현재 계약 안에서 원인과 결과의 연결을 보완한다.",
|
||||
"evidence": "원인 설명 없이 해결책이 먼저 제시된다.",
|
||||
"violated_rule": "quality-rubric.md#hard-gate",
|
||||
"owner": "doc-drafter",
|
||||
}
|
||||
|
||||
|
||||
def make_light_complete(tmp_path: Path) -> Path:
|
||||
run_dir = init_run(tmp_path, route="light")
|
||||
install_good_contracts(run_dir)
|
||||
shutil.copyfile(FIXTURES / "good" / "document.md", run_dir / "final.md")
|
||||
lint_final(run_dir)
|
||||
for status in ("planned", "drafted", "finalized"):
|
||||
transition(run_dir, status)
|
||||
return run_dir
|
||||
|
||||
|
||||
def test_offline_e2e_light_reaches_verified(tmp_path: Path) -> None:
|
||||
run_dir = make_light_complete(tmp_path)
|
||||
result = run_cli("verify_run.py", "--run-dir", run_dir)
|
||||
assert result.returncode == 0, result.stderr
|
||||
report = read_json(run_dir / "09_final_report.json")
|
||||
manifest = read_json(run_dir / "00_run.json")
|
||||
assert report["verdict"] == "pass"
|
||||
assert report["document_verdict"] == "pass"
|
||||
assert manifest["status"] == "verified"
|
||||
assert report["summary"]["omissions"] == manifest["omissions"]
|
||||
lint_report = read_json(run_dir / "08_lint.json")
|
||||
assert report["summary"]["lint"]["finding_counts"] == lint_report["summary"]
|
||||
assert report["summary"]["lint"]["finding_rule_ids"] == list(
|
||||
dict.fromkeys(item["rule_id"] for item in lint_report["findings"])
|
||||
)
|
||||
assert report["summary"]["lint"]["fidelity"] == lint_report["fidelity"]
|
||||
assert report["summary"]["lint"]["limitations"] == lint_report["limitations"]
|
||||
assert report["summary"]["lint"]["rules_sha256"] == lint_report["rules_sha256"]
|
||||
|
||||
|
||||
def test_verifier_rejects_invented_lint_findings_even_with_matching_aggregates(
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
run_dir = make_light_complete(tmp_path)
|
||||
lint_report = read_json(run_dir / "08_lint.json")
|
||||
duplicate_rule = "DOC-P001"
|
||||
for line in (2, 3):
|
||||
lint_report["findings"].append(
|
||||
{
|
||||
"rule_id": duplicate_rule,
|
||||
"severity": "warning",
|
||||
"message": f"중복 rule 집계 검증 {line}",
|
||||
"path": str(run_dir / "final.md"),
|
||||
"line": line,
|
||||
"column": 1,
|
||||
"section_id": None,
|
||||
"context": None,
|
||||
}
|
||||
)
|
||||
lint_report["summary"]["warnings"] += 2
|
||||
lint_report["summary"]["total"] += 2
|
||||
write_json(run_dir / "08_lint.json", lint_report)
|
||||
|
||||
result = run_cli("verify_run.py", "--run-dir", run_dir)
|
||||
assert result.returncode == 1
|
||||
report = read_json(run_dir / "09_final_report.json")
|
||||
assert report["summary"]["lint"] is None
|
||||
assert any(
|
||||
item["id"] == "lint-canonical-replay" and item["status"] == "fail"
|
||||
for item in report["checks"]
|
||||
)
|
||||
|
||||
|
||||
def test_forged_empty_pass_lint_cannot_hide_current_document_errors(
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
run_dir = make_light_complete(tmp_path)
|
||||
bad_document = "# 첫 제목\n\n# 둘째 제목\n\nTODO: 설명을 완성한다.\n"
|
||||
(run_dir / "07_draft.md").write_text(bad_document, encoding="utf-8")
|
||||
(run_dir / "final.md").write_text(bad_document, encoding="utf-8")
|
||||
digest = hashlib.sha256(bad_document.encode("utf-8")).hexdigest()
|
||||
lint_report = read_json(run_dir / "08_lint.json")
|
||||
lint_report["document"]["sha256"] = digest
|
||||
lint_report["fidelity"]["draft_baseline"]["sha256"] = digest
|
||||
lint_report["fidelity"]["finalization_change_rate"] = 0.0
|
||||
lint_report["findings"] = []
|
||||
lint_report["summary"] = {"errors": 0, "warnings": 0, "info": 0, "total": 0}
|
||||
lint_report["verdict"] = "pass"
|
||||
write_json(run_dir / "08_lint.json", lint_report)
|
||||
|
||||
result = run_cli("verify_run.py", "--run-dir", run_dir)
|
||||
assert result.returncode == 1
|
||||
report = read_json(run_dir / "09_final_report.json")
|
||||
replay = next(
|
||||
item for item in report["checks"] if item["id"] == "lint-canonical-replay"
|
||||
)
|
||||
assert replay["status"] == "fail"
|
||||
assert "DOC-H002" in replay["message"]
|
||||
assert "DOC-M001" in replay["message"]
|
||||
|
||||
|
||||
def test_lint_summary_and_verdict_are_recomputed_from_findings(tmp_path: Path) -> None:
|
||||
run_dir = make_light_complete(tmp_path)
|
||||
lint_report = read_json(run_dir / "08_lint.json")
|
||||
lint_report["summary"]["errors"] = 1
|
||||
lint_report["summary"]["total"] = 1
|
||||
write_json(run_dir / "08_lint.json", lint_report)
|
||||
|
||||
result = run_cli("verify_run.py", "--run-dir", run_dir)
|
||||
assert result.returncode == 1
|
||||
report = read_json(run_dir / "09_final_report.json")
|
||||
assert report["document_verdict"] == "not_evaluated"
|
||||
assert report["summary"]["lint"] is None
|
||||
assert any(
|
||||
item["id"] == "lint-semantics" and item["status"] == "fail"
|
||||
for item in report["checks"]
|
||||
)
|
||||
|
||||
|
||||
def test_missing_required_artifact_fails_and_holds(tmp_path: Path) -> None:
|
||||
run_dir = make_light_complete(tmp_path)
|
||||
(run_dir / "05_term_ledger.json").unlink()
|
||||
result = run_cli("verify_run.py", "--run-dir", run_dir)
|
||||
assert result.returncode == 1
|
||||
report = read_json(run_dir / "09_final_report.json")
|
||||
assert report["verdict"] == "fail"
|
||||
assert read_json(run_dir / "00_run.json")["status"] == "hold_for_review"
|
||||
|
||||
|
||||
def test_omissions_reject_duplicates_required_or_present_artifacts(tmp_path: Path) -> None:
|
||||
run_dir = make_light_complete(tmp_path)
|
||||
manifest = read_json(run_dir / "00_run.json")
|
||||
manifest["omissions"].append(dict(manifest["omissions"][0]))
|
||||
manifest["omissions"].append(
|
||||
{"artifact": "07_draft.md", "reason": "필수 draft를 생략했다고 잘못 선언"}
|
||||
)
|
||||
write_json(run_dir / "00_run.json", manifest)
|
||||
result = run_cli("verify_run.py", "--run-dir", run_dir)
|
||||
assert result.returncode == 1
|
||||
report = read_json(run_dir / "09_final_report.json")
|
||||
failure = next(item for item in report["checks"] if item["id"] == "omissions-valid")
|
||||
assert failure["status"] == "fail"
|
||||
assert "duplicate=" in failure["message"]
|
||||
assert "required=" in failure["message"]
|
||||
assert "present=" in failure["message"]
|
||||
assert report["summary"]["omissions"] == manifest["omissions"]
|
||||
|
||||
|
||||
def test_every_missing_optional_artifact_needs_an_omission_reason(tmp_path: Path) -> None:
|
||||
run_dir = make_light_complete(tmp_path)
|
||||
manifest = read_json(run_dir / "00_run.json")
|
||||
manifest["omissions"] = [
|
||||
item
|
||||
for item in manifest["omissions"]
|
||||
if item["artifact"] != "08_logic_review.json"
|
||||
]
|
||||
write_json(run_dir / "00_run.json", manifest)
|
||||
result = run_cli("verify_run.py", "--run-dir", run_dir)
|
||||
assert result.returncode == 1
|
||||
failure = next(
|
||||
item
|
||||
for item in read_json(run_dir / "09_final_report.json")["checks"]
|
||||
if item["id"] == "omissions-complete"
|
||||
)
|
||||
assert failure["status"] == "fail"
|
||||
assert "08_logic_review.json" in failure["message"]
|
||||
|
||||
|
||||
def test_unknown_omission_artifact_is_a_schema_input_error(tmp_path: Path) -> None:
|
||||
run_dir = make_light_complete(tmp_path)
|
||||
manifest = read_json(run_dir / "00_run.json")
|
||||
manifest["omissions"].append(
|
||||
{"artifact": "08_combined_review.json", "reason": "알 수 없는 합성 파일"}
|
||||
)
|
||||
write_json(run_dir / "00_run.json", manifest)
|
||||
result = run_cli("verify_run.py", "--run-dir", run_dir)
|
||||
assert result.returncode == 2
|
||||
assert read_json(run_dir / "09_final_report.json")["verdict"] == "input_error"
|
||||
|
||||
|
||||
def test_review_mode_does_not_require_final_document(tmp_path: Path) -> None:
|
||||
run_dir = init_run(tmp_path, route="light", mode="review", with_draft=True)
|
||||
install_good_contracts(run_dir)
|
||||
write_reviews(run_dir)
|
||||
lint_final(run_dir, "07_draft.md")
|
||||
transition(run_dir, "planned")
|
||||
transition(run_dir, "reviewed")
|
||||
result = run_cli("verify_run.py", "--run-dir", run_dir)
|
||||
assert result.returncode == 0, result.stderr
|
||||
assert not (run_dir / "final.md").exists()
|
||||
report = read_json(run_dir / "09_final_report.json")
|
||||
assert report["document_verdict"] == "pass"
|
||||
assert read_json(run_dir / "00_run.json")["status"] == "reviewed"
|
||||
|
||||
|
||||
def test_standard_review_keeps_route_evidence_requirement(tmp_path: Path) -> None:
|
||||
run_dir = init_run(tmp_path, route="standard", mode="review", with_draft=True)
|
||||
install_good_contracts(run_dir)
|
||||
write_evidence(run_dir)
|
||||
write_reviews(run_dir)
|
||||
lint_final(run_dir, "07_draft.md")
|
||||
for status in ("evidence_ready", "planned", "reviewed"):
|
||||
transition(run_dir, status)
|
||||
(run_dir / "03_evidence_map.json").unlink()
|
||||
result = run_cli("verify_run.py", "--run-dir", run_dir)
|
||||
assert result.returncode == 1
|
||||
assert any(
|
||||
item["id"] == "artifact-03_evidence_map.json" and item["status"] == "fail"
|
||||
for item in read_json(run_dir / "09_final_report.json")["checks"]
|
||||
)
|
||||
|
||||
|
||||
def test_review_must_record_exact_current_draft_path_hash_and_filename_type(
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
run_dir = init_run(tmp_path, route="light", mode="review", with_draft=True)
|
||||
install_good_contracts(run_dir)
|
||||
write_reviews(run_dir)
|
||||
lint_final(run_dir, "07_draft.md")
|
||||
for status in ("planned", "reviewed"):
|
||||
transition(run_dir, status)
|
||||
logic_review = read_json(run_dir / "08_logic_review.json")
|
||||
logic_review["review_type"] = "reader"
|
||||
logic_review["document"]["path"] = "different.md"
|
||||
logic_review["document"]["sha256"] = "0" * 64
|
||||
write_json(run_dir / "08_logic_review.json", logic_review)
|
||||
result = run_cli("verify_run.py", "--run-dir", run_dir)
|
||||
assert result.returncode == 1
|
||||
failed = {
|
||||
item["id"]
|
||||
for item in read_json(run_dir / "09_final_report.json")["checks"]
|
||||
if item["status"] == "fail"
|
||||
}
|
||||
assert "review-type-08_logic_review.json" in failed
|
||||
assert "review-document-08_logic_review.json" in failed
|
||||
assert read_json(run_dir / "09_final_report.json")["document_verdict"] == "not_evaluated"
|
||||
assert all(
|
||||
item["artifact"] != "08_logic_review.json"
|
||||
for item in read_json(run_dir / "09_final_report.json")["summary"]["reviews"]
|
||||
)
|
||||
|
||||
|
||||
def test_review_nested_field_types_are_checked_by_schema(tmp_path: Path) -> None:
|
||||
run_dir = init_run(tmp_path, route="light", mode="review", with_draft=True)
|
||||
install_good_contracts(run_dir)
|
||||
write_reviews(run_dir)
|
||||
lint_final(run_dir, "07_draft.md")
|
||||
for status in ("planned", "reviewed"):
|
||||
transition(run_dir, status)
|
||||
logic_review = read_json(run_dir / "08_logic_review.json")
|
||||
logic_review["findings"] = [
|
||||
{
|
||||
"id": "R1",
|
||||
"severity": "high",
|
||||
"location": 12,
|
||||
"reader_impact": 34,
|
||||
"suggestion": 56,
|
||||
}
|
||||
]
|
||||
write_json(run_dir / "08_logic_review.json", logic_review)
|
||||
result = run_cli("verify_run.py", "--run-dir", run_dir)
|
||||
assert result.returncode == 1
|
||||
assert any(
|
||||
item["id"] == "schema-08_logic_review.json" and item["status"] == "fail"
|
||||
for item in read_json(run_dir / "09_final_report.json")["checks"]
|
||||
)
|
||||
|
||||
|
||||
def test_review_optional_finding_fields_and_final_summary_use_exact_contract(
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
run_dir = init_run(tmp_path, route="standard")
|
||||
install_good_contracts(run_dir)
|
||||
write_evidence(run_dir)
|
||||
write_reviews(run_dir)
|
||||
logic_review = read_json(run_dir / "08_logic_review.json")
|
||||
logic_review["findings"] = [
|
||||
{
|
||||
"id": "R-LOW",
|
||||
"severity": "low",
|
||||
"location": "검증과 한계 절",
|
||||
"reader_impact": "빠른 독자가 한 번 더 읽게 된다.",
|
||||
"suggestion": "짧은 연결 문장을 보완한다.",
|
||||
"evidence": "두 문장의 관계가 암시적으로만 드러난다.",
|
||||
"violated_rule": "quality-rubric.md#reader_fit",
|
||||
"owner": "doc-drafter",
|
||||
}
|
||||
]
|
||||
write_json(run_dir / "08_logic_review.json", logic_review)
|
||||
shutil.copyfile(FIXTURES / "good" / "document.md", run_dir / "final.md")
|
||||
lint_final(run_dir)
|
||||
for status in ("evidence_ready", "planned", "drafted", "reviewed", "finalized"):
|
||||
transition(run_dir, status)
|
||||
result = run_cli("verify_run.py", "--run-dir", run_dir)
|
||||
assert result.returncode == 0, result.stderr
|
||||
report = read_json(run_dir / "09_final_report.json")
|
||||
logic_summary = next(
|
||||
item
|
||||
for item in report["summary"]["reviews"]
|
||||
if item["review_type"] == "logic"
|
||||
)
|
||||
assert logic_summary["finding_counts"] == {
|
||||
"critical": 0,
|
||||
"high": 0,
|
||||
"medium": 0,
|
||||
"low": 1,
|
||||
"total": 1,
|
||||
}
|
||||
assert logic_summary["finding_ids"] == ["R-LOW"]
|
||||
|
||||
|
||||
def test_review_finding_rejects_finalizer_as_owner(tmp_path: Path) -> None:
|
||||
run_dir = init_run(tmp_path, route="light", mode="review", with_draft=True)
|
||||
install_good_contracts(run_dir)
|
||||
write_reviews(run_dir)
|
||||
lint_final(run_dir, "07_draft.md")
|
||||
for status in ("planned", "reviewed"):
|
||||
transition(run_dir, status)
|
||||
logic_review = read_json(run_dir / "08_logic_review.json")
|
||||
logic_review["findings"] = [
|
||||
{
|
||||
"id": "R-OWNER",
|
||||
"severity": "low",
|
||||
"location": "검증과 한계 절",
|
||||
"reader_impact": "수정 책임이 final 단계로 잘못 전달된다.",
|
||||
"suggestion": "Phase 3의 draft owner에게 반환한다.",
|
||||
"owner": "doc-finalizer",
|
||||
}
|
||||
]
|
||||
write_json(run_dir / "08_logic_review.json", logic_review)
|
||||
|
||||
result = run_cli("verify_run.py", "--run-dir", run_dir)
|
||||
|
||||
assert result.returncode == 1
|
||||
assert any(
|
||||
item["id"] == "schema-08_logic_review.json" and item["status"] == "fail"
|
||||
for item in read_json(run_dir / "09_final_report.json")["checks"]
|
||||
)
|
||||
|
||||
|
||||
def test_review_rejects_undeclared_optional_field_alias(tmp_path: Path) -> None:
|
||||
run_dir = init_run(tmp_path, route="light", mode="review", with_draft=True)
|
||||
install_good_contracts(run_dir)
|
||||
write_reviews(run_dir)
|
||||
lint_final(run_dir, "07_draft.md")
|
||||
for status in ("planned", "reviewed"):
|
||||
transition(run_dir, status)
|
||||
logic_review = read_json(run_dir / "08_logic_review.json")
|
||||
logic_review["findings"] = [
|
||||
{
|
||||
"id": "R-ALIAS",
|
||||
"severity": "low",
|
||||
"location": "문제 절",
|
||||
"reader_impact": "표현이 모호하다.",
|
||||
"suggestion": "연결을 명시한다.",
|
||||
"observed_evidence": "schema에 없는 별칭",
|
||||
}
|
||||
]
|
||||
write_json(run_dir / "08_logic_review.json", logic_review)
|
||||
result = run_cli("verify_run.py", "--run-dir", run_dir)
|
||||
assert result.returncode == 1
|
||||
assert any(
|
||||
item["id"] == "schema-08_logic_review.json" and item["status"] == "fail"
|
||||
for item in read_json(run_dir / "09_final_report.json")["checks"]
|
||||
)
|
||||
|
||||
|
||||
def test_review_pass_rejects_blocking_finding(tmp_path: Path) -> None:
|
||||
run_dir = init_run(tmp_path, route="light", mode="review", with_draft=True)
|
||||
install_good_contracts(run_dir)
|
||||
write_reviews(run_dir)
|
||||
lint_final(run_dir, "07_draft.md")
|
||||
for status in ("planned", "reviewed"):
|
||||
transition(run_dir, status)
|
||||
logic_review = read_json(run_dir / "08_logic_review.json")
|
||||
logic_review["findings"] = [blocking_finding("R-PASS-BLOCK")]
|
||||
write_json(run_dir / "08_logic_review.json", logic_review)
|
||||
result = run_cli("verify_run.py", "--run-dir", run_dir)
|
||||
assert result.returncode == 1
|
||||
report = read_json(run_dir / "09_final_report.json")
|
||||
assert report["document_verdict"] == "not_evaluated"
|
||||
assert any(
|
||||
item["id"] == "schema-08_logic_review.json" and item["status"] == "fail"
|
||||
for item in report["checks"]
|
||||
)
|
||||
|
||||
|
||||
def test_review_mode_reports_revise_without_failing_successful_diagnostics(
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
brief_path = tmp_path / "request.md"
|
||||
brief_path.write_text("기존 문서의 문제를 검토해 주세요.\n", encoding="utf-8")
|
||||
original_path = tmp_path / "original.md"
|
||||
original_path.write_text(
|
||||
(FIXTURES / "good" / "document.md").read_text(encoding="utf-8")
|
||||
+ "\nTODO: 독자 연결을 보완한다.\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
initialized = run_cli(
|
||||
"init_run.py",
|
||||
"--brief",
|
||||
brief_path,
|
||||
"--draft",
|
||||
original_path,
|
||||
"--kind",
|
||||
"explanation",
|
||||
"--route",
|
||||
"light",
|
||||
"--mode",
|
||||
"review",
|
||||
"--workspace",
|
||||
tmp_path / "workspace",
|
||||
"--date",
|
||||
"2026-07-23",
|
||||
)
|
||||
assert initialized.returncode == 0, initialized.stderr
|
||||
run_dir = Path(initialized.stdout.split("\t", 1)[0])
|
||||
install_good_contracts(run_dir)
|
||||
draft_path = run_dir / "07_draft.md"
|
||||
shutil.copyfile(original_path, draft_path)
|
||||
write_reviews(run_dir)
|
||||
logic_review = read_json(run_dir / "08_logic_review.json")
|
||||
logic_review["verdict"] = "revise"
|
||||
logic_review["findings"] = [blocking_finding()]
|
||||
write_json(run_dir / "08_logic_review.json", logic_review)
|
||||
lint_result = run_cli(
|
||||
"lint_document.py",
|
||||
"--document",
|
||||
draft_path,
|
||||
"--logic-map",
|
||||
run_dir / "04_logic_map.json",
|
||||
"--term-ledger",
|
||||
run_dir / "05_term_ledger.json",
|
||||
"--reader-contract",
|
||||
run_dir / "02_reader_contract.json",
|
||||
"--output",
|
||||
run_dir / "08_lint.json",
|
||||
)
|
||||
assert lint_result.returncode == 1
|
||||
for status in ("planned", "reviewed"):
|
||||
transition(run_dir, status)
|
||||
result = run_cli("verify_run.py", "--run-dir", run_dir)
|
||||
assert result.returncode == 0, result.stderr
|
||||
report = read_json(run_dir / "09_final_report.json")
|
||||
assert report["verdict"] == "pass"
|
||||
assert report["document_verdict"] == "revise"
|
||||
assert report["summary"]["lint"]["verdict"] == "fail"
|
||||
lint_report = read_json(run_dir / "08_lint.json")
|
||||
assert report["summary"]["lint"]["finding_counts"] == lint_report["summary"]
|
||||
assert report["summary"]["lint"]["finding_rule_ids"] == list(
|
||||
dict.fromkeys(item["rule_id"] for item in lint_report["findings"])
|
||||
)
|
||||
assert read_json(run_dir / "00_run.json")["status"] == "reviewed"
|
||||
assert all(
|
||||
item["status"] == "pass"
|
||||
for item in report["checks"]
|
||||
if item["id"] in {"lint-verdict", "review-verdict-08_logic_review.json"}
|
||||
)
|
||||
|
||||
|
||||
def test_review_mode_hold_is_execution_failure_and_not_evaluated(
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
run_dir = init_run(tmp_path, route="light", mode="review", with_draft=True)
|
||||
install_good_contracts(run_dir)
|
||||
write_reviews(run_dir)
|
||||
lint_final(run_dir, "07_draft.md")
|
||||
for status in ("planned", "reviewed"):
|
||||
transition(run_dir, status)
|
||||
logic_review = read_json(run_dir / "08_logic_review.json")
|
||||
logic_review["verdict"] = "hold_for_review"
|
||||
finding = blocking_finding("R-HOLD")
|
||||
finding["evidence"] = "필수 source 파일에 접근할 수 없다."
|
||||
finding["owner"] = "doc-evidence-curator"
|
||||
logic_review["findings"] = [finding]
|
||||
write_json(run_dir / "08_logic_review.json", logic_review)
|
||||
result = run_cli("verify_run.py", "--run-dir", run_dir)
|
||||
assert result.returncode == 1
|
||||
report = read_json(run_dir / "09_final_report.json")
|
||||
assert report["verdict"] == "fail"
|
||||
assert report["document_verdict"] == "not_evaluated"
|
||||
assert read_json(run_dir / "00_run.json")["status"] == "hold_for_review"
|
||||
|
||||
|
||||
def test_standard_evidence_contract_rejects_unbound_load_bearing_claim(tmp_path: Path) -> None:
|
||||
run_dir = init_run(tmp_path, route="standard")
|
||||
install_good_contracts(run_dir)
|
||||
write_evidence(run_dir)
|
||||
write_reviews(run_dir)
|
||||
shutil.copyfile(FIXTURES / "good" / "document.md", run_dir / "final.md")
|
||||
lint_final(run_dir)
|
||||
for status in ("evidence_ready", "planned", "drafted", "reviewed", "finalized"):
|
||||
transition(run_dir, status)
|
||||
evidence = read_json(run_dir / "03_evidence_map.json")
|
||||
evidence["claims"][0]["source_ids"] = []
|
||||
write_json(run_dir / "03_evidence_map.json", evidence)
|
||||
result = run_cli("verify_run.py", "--run-dir", run_dir)
|
||||
assert result.returncode == 1
|
||||
report = read_json(run_dir / "09_final_report.json")
|
||||
assert any(
|
||||
item["id"] in {"schema-03_evidence_map.json", "evidence-contract"}
|
||||
and item["status"] == "fail"
|
||||
for item in report["checks"]
|
||||
)
|
||||
|
||||
|
||||
def test_source_hash_drift_is_rejected(tmp_path: Path) -> None:
|
||||
run_dir = make_light_complete(tmp_path)
|
||||
manifest = read_json(run_dir / "00_run.json")
|
||||
Path(manifest["inputs"]["brief"]["resolved_path"]).write_text("변조됨\n", encoding="utf-8")
|
||||
result = run_cli("verify_run.py", "--run-dir", run_dir)
|
||||
assert result.returncode == 1
|
||||
assert any(
|
||||
item["id"] == "hash-sources" and item["status"] == "fail"
|
||||
for item in read_json(run_dir / "09_final_report.json")["checks"]
|
||||
)
|
||||
|
||||
|
||||
def test_run_input_summary_must_match_sources_inventory(tmp_path: Path) -> None:
|
||||
run_dir = make_light_complete(tmp_path)
|
||||
manifest = read_json(run_dir / "00_run.json")
|
||||
manifest["inputs"]["source_count"] = 1
|
||||
manifest["inputs"]["brief_sha256"] = "0" * 64
|
||||
write_json(run_dir / "00_run.json", manifest)
|
||||
result = run_cli("verify_run.py", "--run-dir", run_dir)
|
||||
assert result.returncode == 1
|
||||
assert any(
|
||||
item["id"] == "hash-input-inventory" and item["status"] == "fail"
|
||||
for item in read_json(run_dir / "09_final_report.json")["checks"]
|
||||
)
|
||||
|
||||
|
||||
def test_nested_lint_types_are_checked_by_schema(tmp_path: Path) -> None:
|
||||
run_dir = make_light_complete(tmp_path)
|
||||
lint_report = read_json(run_dir / "08_lint.json")
|
||||
lint_report["summary"]["errors"] = "0"
|
||||
write_json(run_dir / "08_lint.json", lint_report)
|
||||
result = run_cli("verify_run.py", "--run-dir", run_dir)
|
||||
assert result.returncode == 1
|
||||
failed = {
|
||||
item["id"]
|
||||
for item in read_json(run_dir / "09_final_report.json")["checks"]
|
||||
if item["status"] == "fail"
|
||||
}
|
||||
assert "schema-08_lint.json" in failed
|
||||
|
||||
|
||||
def test_lint_contract_input_hash_drift_is_rejected(tmp_path: Path) -> None:
|
||||
run_dir = make_light_complete(tmp_path)
|
||||
logic = read_json(run_dir / "04_logic_map.json")
|
||||
logic["closure"] += " 추가 문장"
|
||||
write_json(run_dir / "04_logic_map.json", logic)
|
||||
result = run_cli("verify_run.py", "--run-dir", run_dir)
|
||||
assert result.returncode == 1
|
||||
assert any(
|
||||
item["id"] == "lint-input-hash-04_logic_map.json"
|
||||
and item["status"] == "fail"
|
||||
for item in read_json(run_dir / "09_final_report.json")["checks"]
|
||||
)
|
||||
|
||||
|
||||
def test_stale_lint_rules_fingerprint_is_not_summarized(tmp_path: Path) -> None:
|
||||
run_dir = make_light_complete(tmp_path)
|
||||
lint_report = read_json(run_dir / "08_lint.json")
|
||||
lint_report["rules_sha256"] = "0" * 64
|
||||
write_json(run_dir / "08_lint.json", lint_report)
|
||||
result = run_cli("verify_run.py", "--run-dir", run_dir)
|
||||
assert result.returncode == 1
|
||||
report = read_json(run_dir / "09_final_report.json")
|
||||
assert report["summary"]["lint"] is None
|
||||
assert report["document_verdict"] == "not_evaluated"
|
||||
assert any(
|
||||
item["id"] == "lint-rules-sha256" and item["status"] == "fail"
|
||||
for item in report["checks"]
|
||||
)
|
||||
|
||||
|
||||
def test_verify_output_is_canonical_and_cannot_overwrite_run_inputs(
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
run_dir = make_light_complete(tmp_path)
|
||||
manifest_path = run_dir / "00_run.json"
|
||||
before = manifest_path.read_bytes()
|
||||
result = run_cli(
|
||||
"verify_run.py",
|
||||
"--run-dir",
|
||||
run_dir,
|
||||
"--output",
|
||||
manifest_path,
|
||||
)
|
||||
assert result.returncode == 2
|
||||
assert manifest_path.read_bytes() == before
|
||||
assert read_json(manifest_path)["status"] == "finalized"
|
||||
|
||||
custom = tmp_path / "custom-report.json"
|
||||
result = run_cli(
|
||||
"verify_run.py",
|
||||
"--run-dir",
|
||||
run_dir,
|
||||
"--output",
|
||||
custom,
|
||||
)
|
||||
assert result.returncode == 2
|
||||
assert not custom.exists()
|
||||
assert read_json(manifest_path)["status"] == "finalized"
|
||||
|
||||
|
||||
def test_verify_existing_canonical_output_requires_tool_ownership(
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
run_dir = make_light_complete(tmp_path)
|
||||
output = run_dir / "09_final_report.json"
|
||||
output.write_text('{"tool":"lint_document","sentinel":true}\n', encoding="utf-8")
|
||||
before = output.read_bytes()
|
||||
result = run_cli("verify_run.py", "--run-dir", run_dir)
|
||||
assert result.returncode == 2
|
||||
assert output.read_bytes() == before
|
||||
assert read_json(run_dir / "00_run.json")["status"] == "finalized"
|
||||
|
||||
|
||||
def test_verify_malformed_same_tool_output_is_not_overwritten(tmp_path: Path) -> None:
|
||||
run_dir = make_light_complete(tmp_path)
|
||||
output = run_dir / "09_final_report.json"
|
||||
output.write_text('{"tool":"verify_run","sentinel":true}\n', encoding="utf-8")
|
||||
before = output.read_bytes()
|
||||
result = run_cli("verify_run.py", "--run-dir", run_dir)
|
||||
assert result.returncode == 2
|
||||
assert output.read_bytes() == before
|
||||
assert read_json(run_dir / "00_run.json")["status"] == "finalized"
|
||||
|
||||
|
||||
def test_verify_publish_rejects_entry_created_after_preflight(
|
||||
tmp_path: Path, monkeypatch
|
||||
) -> None:
|
||||
run_dir = make_light_complete(tmp_path)
|
||||
first = run_cli("verify_run.py", "--run-dir", run_dir)
|
||||
assert first.returncode == 0, first.stderr
|
||||
output = run_dir / "09_final_report.json"
|
||||
output.unlink()
|
||||
|
||||
canonical = (
|
||||
Path(__file__).resolve().parents[1]
|
||||
/ "skills"
|
||||
/ "technical-doc-flow"
|
||||
/ "scripts"
|
||||
/ "verify_run.py"
|
||||
)
|
||||
sys.path.insert(0, str(canonical.parent))
|
||||
spec = importlib.util.spec_from_file_location("conditional_verify_run", canonical)
|
||||
assert spec and spec.loader
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(module)
|
||||
original_publish = module.publish_report_json
|
||||
|
||||
def publish_after_race(prepared, value):
|
||||
prepared.path.write_bytes(b"unrelated verifier sentinel\n")
|
||||
return original_publish(prepared, value)
|
||||
|
||||
monkeypatch.setattr(module, "publish_report_json", publish_after_race)
|
||||
|
||||
result = module.main(["--run-dir", str(run_dir)])
|
||||
|
||||
assert result == 2
|
||||
assert output.read_bytes() == b"unrelated verifier sentinel\n"
|
||||
assert read_json(run_dir / "00_run.json")["status"] == "verified"
|
||||
|
||||
|
||||
def test_verify_can_conditionally_refresh_owned_report(tmp_path: Path) -> None:
|
||||
run_dir = make_light_complete(tmp_path)
|
||||
|
||||
first = run_cli("verify_run.py", "--run-dir", run_dir)
|
||||
second = run_cli("verify_run.py", "--run-dir", run_dir)
|
||||
|
||||
assert first.returncode == 0, first.stderr
|
||||
assert second.returncode == 0, second.stderr
|
||||
assert read_json(run_dir / "09_final_report.json")["tool"] == "verify_run"
|
||||
assert read_json(run_dir / "00_run.json")["status"] == "verified"
|
||||
|
||||
|
||||
def test_verify_rejects_symlinked_manifest_without_writing_external_target(
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
run_dir = make_light_complete(tmp_path)
|
||||
manifest_path = run_dir / "00_run.json"
|
||||
external = tmp_path / "outside-manifest.json"
|
||||
manifest_path.replace(external)
|
||||
manifest_path.symlink_to(external)
|
||||
before = external.read_bytes()
|
||||
|
||||
result = run_cli("verify_run.py", "--run-dir", run_dir)
|
||||
assert result.returncode == 2
|
||||
assert external.read_bytes() == before
|
||||
assert read_json(external)["status"] == "finalized"
|
||||
|
||||
|
||||
def test_verify_detects_document_change_during_verified_transition(
|
||||
tmp_path: Path, monkeypatch
|
||||
) -> None:
|
||||
run_dir = make_light_complete(tmp_path)
|
||||
canonical = (
|
||||
Path(__file__).resolve().parents[1]
|
||||
/ "skills"
|
||||
/ "technical-doc-flow"
|
||||
/ "scripts"
|
||||
/ "verify_run.py"
|
||||
)
|
||||
sys.path.insert(0, str(canonical.parent))
|
||||
spec = importlib.util.spec_from_file_location("canonical_verify_run", canonical)
|
||||
assert spec and spec.loader
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(module)
|
||||
original_transition = module.transition_manifest
|
||||
|
||||
def mutate_then_transition(manifest_path, target, *, reason, allow_same=False):
|
||||
if target == "verified":
|
||||
final_path = run_dir / "final.md"
|
||||
final_path.write_text(
|
||||
final_path.read_text(encoding="utf-8") + "\n동시 변경\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
return original_transition(
|
||||
manifest_path,
|
||||
target,
|
||||
reason=reason,
|
||||
allow_same=allow_same,
|
||||
)
|
||||
|
||||
monkeypatch.setattr(module, "transition_manifest", mutate_then_transition)
|
||||
result = module.main(["--run-dir", str(run_dir)])
|
||||
assert result == 1
|
||||
report = read_json(run_dir / "09_final_report.json")
|
||||
assert report["verdict"] == "fail"
|
||||
assert read_json(run_dir / "00_run.json")["status"] == "hold_for_review"
|
||||
assert any(
|
||||
item["id"] == "verification-snapshot-post-transition"
|
||||
and item["status"] == "fail"
|
||||
for item in report["checks"]
|
||||
)
|
||||
|
||||
|
||||
def test_optional_review_is_validated_and_gates_when_present(tmp_path: Path) -> None:
|
||||
run_dir = make_light_complete(tmp_path)
|
||||
write_reviews(run_dir)
|
||||
manifest = read_json(run_dir / "00_run.json")
|
||||
manifest["omissions"] = [
|
||||
item
|
||||
for item in manifest["omissions"]
|
||||
if item["artifact"] not in {"08_logic_review.json", "08_reader_review.json"}
|
||||
]
|
||||
write_json(run_dir / "00_run.json", manifest)
|
||||
logic_review = read_json(run_dir / "08_logic_review.json")
|
||||
logic_review["verdict"] = "revise"
|
||||
logic_review["findings"] = [blocking_finding("R-OPTIONAL")]
|
||||
write_json(run_dir / "08_logic_review.json", logic_review)
|
||||
result = run_cli("verify_run.py", "--run-dir", run_dir)
|
||||
assert result.returncode == 1
|
||||
assert any(
|
||||
item["id"] == "review-verdict-08_logic_review.json"
|
||||
and item["status"] == "fail"
|
||||
for item in read_json(run_dir / "09_final_report.json")["checks"]
|
||||
)
|
||||
|
||||
|
||||
def test_runtime_contract_override_is_schema_validated(tmp_path: Path) -> None:
|
||||
run_dir = make_light_complete(tmp_path)
|
||||
contract_path = (
|
||||
Path(__file__).resolve().parents[1]
|
||||
/ "skills"
|
||||
/ "technical-doc-flow"
|
||||
/ "config"
|
||||
/ "runtime-contract.json"
|
||||
)
|
||||
contract = read_json(contract_path)
|
||||
contract["agents"] = [123]
|
||||
invalid_contract = tmp_path / "runtime-contract.json"
|
||||
write_json(invalid_contract, contract)
|
||||
result = run_cli(
|
||||
"verify_run.py", "--run-dir", run_dir, "--contract", invalid_contract
|
||||
)
|
||||
assert result.returncode == 2
|
||||
assert read_json(run_dir / "09_final_report.json")["verdict"] == "input_error"
|
||||
|
||||
|
||||
def test_non_pass_independent_review_blocks_standard_run(tmp_path: Path) -> None:
|
||||
run_dir = init_run(tmp_path, route="standard")
|
||||
install_good_contracts(run_dir)
|
||||
write_evidence(run_dir)
|
||||
write_reviews(run_dir)
|
||||
shutil.copyfile(FIXTURES / "good" / "document.md", run_dir / "final.md")
|
||||
lint_final(run_dir)
|
||||
for status in ("evidence_ready", "planned", "drafted", "reviewed", "finalized"):
|
||||
transition(run_dir, status)
|
||||
logic_review = read_json(run_dir / "08_logic_review.json")
|
||||
logic_review["verdict"] = "revise"
|
||||
logic_review["findings"] = [blocking_finding("R-REVISE")]
|
||||
write_json(run_dir / "08_logic_review.json", logic_review)
|
||||
result = run_cli("verify_run.py", "--run-dir", run_dir)
|
||||
assert result.returncode == 1
|
||||
report = read_json(run_dir / "09_final_report.json")
|
||||
assert any(
|
||||
item["id"] == "review-verdict-08_logic_review.json" and item["status"] == "fail"
|
||||
for item in report["checks"]
|
||||
)
|
||||
assert "Phase 3" in next(
|
||||
item["message"]
|
||||
for item in report["checks"]
|
||||
if item["id"] == "review-verdict-08_logic_review.json"
|
||||
)
|
||||
assert report["document_verdict"] == "revise"
|
||||
assert read_json(run_dir / "00_run.json")["status"] == "hold_for_review"
|
||||
Reference in New Issue
Block a user