622 lines
26 KiB
Markdown
622 lines
26 KiB
Markdown
# Spec B — Judge Verdict Schema & Adversarial Quorum Implementation Plan
|
||
|
||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||
> **Commits excluded** per user instruction — no `git commit` steps; each task ends with a test-green checkpoint.
|
||
|
||
**Goal:** Make judge-agent verdicts machine-validated at the SubagentStop boundary and make the adversarial-review kill-decision a deterministic quorum tally — without the Workflow tool (Claude subagents lack tool-layer schema enforcement).
|
||
|
||
**Architecture:** Extend the shared `wiki_rules.py` with `validate_verdict_block` (P1 schema + verdict↔count consistency) and `tally_quorum` (≥2 REJECT = KILL, default-refute, abstain≠pass). `wiki_claim_gate.subagent_stop_gate` validates any subagent output carrying a `wiki-verdict` marker (block on schema error; non-judge output untouched). New `wiki_quorum.py` CLI tallies N adversarial-review outputs. Five judge `.md` files emit the machine block; `wiki-adversarial-reviewer` also flips to default-refute. Deterministic core is unit-tested; agent prose is mirrored across 3 platforms by hand (the generator is absent).
|
||
|
||
**Tech Stack:** Python 3 stdlib (`re`, `json`, `argparse`, `unittest`, `subprocess`). Claude Code hooks. Markdown agent specs.
|
||
|
||
**Spec:** `docs/superpowers/specs/2026-06-06-spec-b-judge-verdict-schema-and-quorum-design.md`
|
||
|
||
---
|
||
|
||
## File Structure
|
||
|
||
- **Modify** `.claude/hooks/wiki_rules.py` — add `parse_verdict_block`, `validate_verdict_block`, `tally_quorum`, constants.
|
||
- **Modify** `.claude/hooks/test_wiki_rules.py` — add `TestVerdictBlock`, `TestTallyQuorum`.
|
||
- **Create** `.claude/hooks/wiki_quorum.py` — CLI over `tally_quorum`.
|
||
- **Create** `.claude/hooks/test_wiki_quorum.py` — CLI integration tests.
|
||
- **Modify** `.claude/hooks/wiki_claim_gate.py` — `subagent_stop_gate` calls `validate_verdict_block`.
|
||
- **Modify** `.claude/hooks/test_wiki_claim_gate.py` — add `TestSubagentStopVerdict` (subprocess).
|
||
- **Modify** `.claude/agents/wiki-adversarial-reviewer.md` — per-finding block + default-refute + quorum doc.
|
||
- **Modify** `.claude/agents/{branch-depth-auditor,coverage-auditor,project-readiness-auditor,wiki-diagram-reviewer}.md` — standard verdict block.
|
||
- **Mirror (manual)** the 4 shared judges into `.agents/plugins/wiki-superpowers/agents/<name>.md`, `.codex/agents/<name>.md`, `.codex/agents/<name>.toml`. (`project-readiness-auditor` is Claude-only — no mirror.)
|
||
|
||
---
|
||
|
||
## Task 1: `wiki_rules` — verdict block parse + validate
|
||
|
||
**Files:**
|
||
- Modify: `.claude/hooks/wiki_rules.py` (append functions + constants)
|
||
- Test: `.claude/hooks/test_wiki_rules.py` (append `TestVerdictBlock`)
|
||
|
||
- [ ] **Step 1: Write the failing test**
|
||
|
||
Append to `.claude/hooks/test_wiki_rules.py` before the `if __name__` line:
|
||
```python
|
||
STD_OK = "리포트...\n```wiki-verdict\nagent: branch-depth-auditor\nverdict: not-ready\nblocking: 2\nshould_fix: 1\nadvisory: 0\n```\n끝"
|
||
STD_CONTRADICT = "```wiki-verdict\nagent: branch-depth-auditor\nverdict: ready\nblocking: 2\nshould_fix: 0\nadvisory: 0\n```"
|
||
STD_BADVERDICT = "```wiki-verdict\nagent: x\nverdict: foo\nblocking: 0\nshould_fix: 0\nadvisory: 0\n```"
|
||
ADV_OK = "```wiki-verdict\nagent: wiki-adversarial-reviewer\nfinding: 4.1.1 action: KEEP\nfinding: 4.2.1 action: REJECT\n```"
|
||
ADV_BADACTION = "```wiki-verdict\nagent: wiki-adversarial-reviewer\nfinding: 4.1.1 action: NOPE\n```"
|
||
ADV_EMPTY = "```wiki-verdict\nagent: wiki-adversarial-reviewer\n```"
|
||
|
||
|
||
class TestVerdictBlock(unittest.TestCase):
|
||
def test_no_marker_returns_none(self):
|
||
parsed, errors = wr.validate_verdict_block("그냥 산문, 마커 없음")
|
||
self.assertIsNone(parsed)
|
||
self.assertEqual(errors, [])
|
||
|
||
def test_standard_valid(self):
|
||
parsed, errors = wr.validate_verdict_block(STD_OK)
|
||
self.assertEqual(errors, [])
|
||
self.assertEqual(parsed["agent"], "branch-depth-auditor")
|
||
self.assertEqual(parsed["kv"]["verdict"], "not-ready")
|
||
|
||
def test_standard_contradiction_flagged(self):
|
||
_, errors = wr.validate_verdict_block(STD_CONTRADICT)
|
||
self.assertTrue(any("blocking" in e for e in errors))
|
||
|
||
def test_standard_bad_verdict_flagged(self):
|
||
_, errors = wr.validate_verdict_block(STD_BADVERDICT)
|
||
self.assertTrue(any("verdict" in e for e in errors))
|
||
|
||
def test_adversarial_valid(self):
|
||
parsed, errors = wr.validate_verdict_block(ADV_OK)
|
||
self.assertEqual(errors, [])
|
||
self.assertEqual(len(parsed["findings"]), 2)
|
||
|
||
def test_adversarial_bad_action_flagged(self):
|
||
_, errors = wr.validate_verdict_block(ADV_BADACTION)
|
||
self.assertTrue(any("action" in e for e in errors))
|
||
|
||
def test_adversarial_empty_findings_flagged(self):
|
||
_, errors = wr.validate_verdict_block(ADV_EMPTY)
|
||
self.assertTrue(any("finding" in e for e in errors))
|
||
```
|
||
|
||
- [ ] **Step 2: Run test to verify it fails**
|
||
|
||
Run: `python3 .claude/hooks/test_wiki_rules.py -k Verdict`
|
||
Expected: FAIL — `AttributeError: module 'wiki_rules' has no attribute 'validate_verdict_block'`.
|
||
|
||
- [ ] **Step 3: Implement in `wiki_rules.py`**
|
||
|
||
Append to `.claude/hooks/wiki_rules.py` (the `import re` must be added to the top import block — currently `wiki_rules.py` imports only `json, shlex, sys`):
|
||
```python
|
||
import re # (add to the existing import block at top of file)
|
||
|
||
VERDICT_FENCE_RE = re.compile(r"```wiki-verdict\s*\n(.*?)\n```", re.S)
|
||
VALID_VERDICT = {"ready", "not-ready", "blocked"}
|
||
VALID_ACTION = {"KEEP", "DOWNGRADE", "REJECT"}
|
||
REFUTATIONS_REQUIRED = 2 # ≥2 REJECT → kill (deep-research 기본값)
|
||
|
||
|
||
def parse_verdict_block(text):
|
||
"""본문에서 wiki-verdict fenced 블록을 찾아 dict 로 파싱. 없으면 None."""
|
||
m = VERDICT_FENCE_RE.search(text or "")
|
||
if not m:
|
||
return None
|
||
out = {"agent": None, "kv": {}, "findings": []}
|
||
for line in m.group(1).splitlines():
|
||
line = line.strip()
|
||
if not line:
|
||
continue
|
||
fm = re.match(r"finding:\s*(\S+)\s+action:\s*(\S+)", line)
|
||
if fm:
|
||
out["findings"].append((fm.group(1), fm.group(2)))
|
||
continue
|
||
kv = re.match(r"([a-z_]+):\s*(.+)$", line)
|
||
if kv:
|
||
k, v = kv.group(1), kv.group(2).strip()
|
||
if k == "agent":
|
||
out["agent"] = v
|
||
else:
|
||
out["kv"][k] = v
|
||
return out
|
||
|
||
|
||
def validate_verdict_block(text):
|
||
"""(parsed, errors). parsed None → 마커 없음(judge 아님, caller 통과).
|
||
errors 비어있지 않으면 스키마 위반 → SubagentStop 차단."""
|
||
parsed = parse_verdict_block(text)
|
||
if parsed is None:
|
||
return None, []
|
||
errors = []
|
||
if not parsed["agent"]:
|
||
errors.append("wiki-verdict 블록에 `agent:` 누락")
|
||
if parsed["agent"] == "wiki-adversarial-reviewer":
|
||
if not parsed["findings"]:
|
||
errors.append("adversarial verdict 블록에 `finding: <id> action: <act>` 행 ≥1 필요")
|
||
for fid, act in parsed["findings"]:
|
||
if act not in VALID_ACTION:
|
||
errors.append(f"finding {fid}: action '{act}' 비허용(KEEP|DOWNGRADE|REJECT)")
|
||
else:
|
||
v = parsed["kv"].get("verdict")
|
||
if v not in VALID_VERDICT:
|
||
errors.append(f"verdict '{v}' 비허용(ready|not-ready|blocked)")
|
||
blocking = None
|
||
try:
|
||
blocking = int(parsed["kv"].get("blocking", ""))
|
||
int(parsed["kv"].get("should_fix", ""))
|
||
int(parsed["kv"].get("advisory", ""))
|
||
except ValueError:
|
||
errors.append("blocking/should_fix/advisory 는 정수여야 함")
|
||
if blocking is not None and v == "ready" and blocking != 0:
|
||
errors.append("verdict=ready 인데 blocking≠0 (모순)")
|
||
if blocking is not None and v == "not-ready" and blocking < 1:
|
||
errors.append("verdict=not-ready 인데 blocking<1 (모순)")
|
||
return parsed, errors
|
||
```
|
||
|
||
- [ ] **Step 4: Run test to verify it passes**
|
||
|
||
Run: `python3 .claude/hooks/test_wiki_rules.py -k Verdict`
|
||
Expected: PASS (7 tests).
|
||
|
||
- [ ] **Step 5: Checkpoint (no commit)** — `python3 .claude/hooks/test_wiki_rules.py` all green.
|
||
|
||
---
|
||
|
||
## Task 2: `wiki_rules` — quorum tally
|
||
|
||
**Files:**
|
||
- Modify: `.claude/hooks/wiki_rules.py` (append `tally_quorum`)
|
||
- Test: `.claude/hooks/test_wiki_rules.py` (append `TestTallyQuorum`)
|
||
|
||
- [ ] **Step 1: Write the failing test**
|
||
|
||
Append:
|
||
```python
|
||
def _adv(*pairs):
|
||
lines = "\n".join(f"finding: {fid} action: {act}" for fid, act in pairs)
|
||
return f"```wiki-verdict\nagent: wiki-adversarial-reviewer\n{lines}\n```"
|
||
|
||
|
||
class TestTallyQuorum(unittest.TestCase):
|
||
def test_two_rejects_kill(self):
|
||
blocks = [_adv(("A", "REJECT")), _adv(("A", "REJECT")), _adv(("A", "KEEP"))]
|
||
per = wr.tally_quorum(blocks)
|
||
self.assertEqual(per["A"]["decision"], "KILL")
|
||
|
||
def test_unanimous_keep(self):
|
||
blocks = [_adv(("A", "KEEP")), _adv(("A", "KEEP")), _adv(("A", "KEEP"))]
|
||
self.assertEqual(wr.tally_quorum(blocks)["A"]["decision"], "KEEP")
|
||
|
||
def test_reject_plus_downgrade_is_downgrade(self):
|
||
blocks = [_adv(("A", "REJECT")), _adv(("A", "DOWNGRADE")), _adv(("A", "KEEP"))]
|
||
self.assertEqual(wr.tally_quorum(blocks)["A"]["decision"], "DOWNGRADE")
|
||
|
||
def test_abstain_not_pass(self):
|
||
# 한 블록만 KEEP, 나머지 둘은 A 를 누락(abstain) → 정족수 미달 → UNVERIFIED
|
||
blocks = [_adv(("A", "KEEP")), _adv(("B", "KEEP")), _adv(("C", "KEEP"))]
|
||
self.assertEqual(wr.tally_quorum(blocks)["A"]["decision"], "UNVERIFIED")
|
||
```
|
||
|
||
- [ ] **Step 2: Run test to verify it fails**
|
||
|
||
Run: `python3 .claude/hooks/test_wiki_rules.py -k Tally`
|
||
Expected: FAIL — `AttributeError: ... 'tally_quorum'`.
|
||
|
||
- [ ] **Step 3: Implement in `wiki_rules.py`**
|
||
|
||
Append:
|
||
```python
|
||
def tally_quorum(block_texts, refutations_required=REFUTATIONS_REQUIRED):
|
||
"""N개 adversarial verdict 블록 → per-finding 결정론 판정.
|
||
|
||
refute = DOWNGRADE 또는 REJECT (원 severity 반박).
|
||
default-refute: 어떤 pass 가 finding 을 누락/malformed → abstain(non-KEEP).
|
||
결정: reject≥req → KILL · (reject+downgrade)≥req → DOWNGRADE ·
|
||
keep≥req → KEEP · 그 외(정족수 미달) → UNVERIFIED(통과 금지).
|
||
"""
|
||
parsed_all = [parse_verdict_block(t) for t in block_texts]
|
||
all_fids = set()
|
||
for p in parsed_all:
|
||
if p:
|
||
for fid, _ in p["findings"]:
|
||
all_fids.add(fid)
|
||
per = {}
|
||
for fid in all_fids:
|
||
keep = downgrade = reject = abstain = 0
|
||
for p in parsed_all:
|
||
act = None
|
||
if p:
|
||
for f, a in p["findings"]:
|
||
if f == fid:
|
||
act = a
|
||
break
|
||
if act == "KEEP":
|
||
keep += 1
|
||
elif act == "DOWNGRADE":
|
||
downgrade += 1
|
||
elif act == "REJECT":
|
||
reject += 1
|
||
else:
|
||
abstain += 1
|
||
if reject >= refutations_required:
|
||
decision = "KILL"
|
||
elif (reject + downgrade) >= refutations_required:
|
||
decision = "DOWNGRADE"
|
||
elif keep >= refutations_required:
|
||
decision = "KEEP"
|
||
else:
|
||
decision = "UNVERIFIED"
|
||
per[fid] = {"keep": keep, "downgrade": downgrade, "reject": reject,
|
||
"abstain": abstain, "n": len(block_texts), "decision": decision}
|
||
return per
|
||
```
|
||
|
||
- [ ] **Step 4: Run test to verify it passes**
|
||
|
||
Run: `python3 .claude/hooks/test_wiki_rules.py -k Tally`
|
||
Expected: PASS (4 tests).
|
||
|
||
- [ ] **Step 5: Checkpoint (no commit)** — `python3 .claude/hooks/test_wiki_rules.py` all green.
|
||
|
||
---
|
||
|
||
## Task 3: `wiki_quorum.py` CLI
|
||
|
||
**Files:**
|
||
- Create: `.claude/hooks/wiki_quorum.py`
|
||
- Test: `.claude/hooks/test_wiki_quorum.py`
|
||
|
||
- [ ] **Step 1: Write the failing test**
|
||
|
||
Create `.claude/hooks/test_wiki_quorum.py`:
|
||
```python
|
||
#!/usr/bin/env python3
|
||
"""wiki_quorum.py CLI 통합 테스트."""
|
||
import subprocess
|
||
import tempfile
|
||
import unittest
|
||
from pathlib import Path
|
||
|
||
CLI = str(Path(__file__).with_name("wiki_quorum.py"))
|
||
|
||
|
||
def _adv(*pairs):
|
||
lines = "\n".join(f"finding: {fid} action: {act}" for fid, act in pairs)
|
||
return f"```wiki-verdict\nagent: wiki-adversarial-reviewer\n{lines}\n```"
|
||
|
||
|
||
class TestQuorumCLI(unittest.TestCase):
|
||
def _files(self, d, *texts):
|
||
paths = []
|
||
for i, t in enumerate(texts):
|
||
p = Path(d) / f"v{i}.md"
|
||
p.write_text(t)
|
||
paths.append(str(p))
|
||
return paths
|
||
|
||
def test_kill_exits_1(self):
|
||
with tempfile.TemporaryDirectory() as d:
|
||
paths = self._files(d, _adv(("A", "REJECT")), _adv(("A", "REJECT")), _adv(("A", "KEEP")))
|
||
r = subprocess.run(["python3", CLI] + paths, capture_output=True, text=True)
|
||
self.assertEqual(r.returncode, 1)
|
||
self.assertIn("KILL", r.stdout)
|
||
|
||
def test_all_keep_exits_0(self):
|
||
with tempfile.TemporaryDirectory() as d:
|
||
paths = self._files(d, _adv(("A", "KEEP")), _adv(("A", "KEEP")), _adv(("A", "KEEP")))
|
||
r = subprocess.run(["python3", CLI] + paths, capture_output=True, text=True)
|
||
self.assertEqual(r.returncode, 0)
|
||
self.assertIn("KEEP", r.stdout)
|
||
|
||
|
||
if __name__ == "__main__":
|
||
unittest.main(verbosity=2)
|
||
```
|
||
|
||
- [ ] **Step 2: Run test to verify it fails**
|
||
|
||
Run: `python3 .claude/hooks/test_wiki_quorum.py`
|
||
Expected: FAIL — `FileNotFoundError` (wiki_quorum.py missing) → subprocess returncode nonzero / can't run.
|
||
|
||
- [ ] **Step 3: Implement `wiki_quorum.py`**
|
||
|
||
Create `.claude/hooks/wiki_quorum.py`:
|
||
```python
|
||
#!/usr/bin/env python3
|
||
"""wiki_quorum.py — N개 adversarial verdict 블록의 결정론 quorum tally CLI.
|
||
|
||
사용:
|
||
python3 wiki_quorum.py vote1.md vote2.md vote3.md
|
||
cat votes.md | python3 wiki_quorum.py --stdin # '---' 구분 멀티블록
|
||
exit: 1 if any KILL/UNVERIFIED, else 0.
|
||
"""
|
||
from __future__ import annotations
|
||
import sys
|
||
from pathlib import Path
|
||
|
||
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
||
import wiki_rules
|
||
|
||
|
||
def main():
|
||
args = sys.argv[1:]
|
||
if "--stdin" in args:
|
||
blob = sys.stdin.read()
|
||
blocks = [b for b in blob.split("\n---\n") if "wiki-verdict" in b]
|
||
else:
|
||
blocks = []
|
||
for a in args:
|
||
try:
|
||
blocks.append(Path(a).read_text(encoding="utf-8"))
|
||
except Exception as e:
|
||
print(f"파일 읽기 실패: {a} — {e}", file=sys.stderr)
|
||
if not blocks:
|
||
print("verdict 블록 입력 없음", file=sys.stderr)
|
||
sys.exit(2)
|
||
per = wiki_rules.tally_quorum(blocks)
|
||
print(f"== Quorum tally: N={len(blocks)} votes, {len(per)} findings ==")
|
||
print("| finding | keep | down | reject | abstain | decision |")
|
||
print("|---|---|---|---|---|---|")
|
||
bad = 0
|
||
for fid in sorted(per):
|
||
r = per[fid]
|
||
if r["decision"] in ("KILL", "UNVERIFIED"):
|
||
bad += 1
|
||
print(f"| {fid} | {r['keep']} | {r['downgrade']} | {r['reject']} | {r['abstain']} | {r['decision']} |")
|
||
print(f"\nKILL/UNVERIFIED: {bad} / {len(per)}")
|
||
sys.exit(1 if bad else 0)
|
||
|
||
|
||
if __name__ == "__main__":
|
||
main()
|
||
```
|
||
|
||
- [ ] **Step 4: Run test to verify it passes**
|
||
|
||
Run: `python3 .claude/hooks/test_wiki_quorum.py`
|
||
Expected: PASS (2 tests).
|
||
|
||
- [ ] **Step 5: Checkpoint (no commit)** — green.
|
||
|
||
---
|
||
|
||
## Task 4: `claim_gate.subagent_stop_gate` verdict enforcement
|
||
|
||
**Files:**
|
||
- Modify: `.claude/hooks/wiki_claim_gate.py` (`subagent_stop_gate`)
|
||
- Test: `.claude/hooks/test_wiki_claim_gate.py` (append `TestSubagentStopVerdict`, subprocess-based)
|
||
|
||
- [ ] **Step 1: Write the failing test**
|
||
|
||
Append to `.claude/hooks/test_wiki_claim_gate.py`:
|
||
```python
|
||
import json as _json
|
||
import subprocess as _sp
|
||
|
||
_GATE = str(Path(__file__).with_name("wiki_claim_gate.py"))
|
||
|
||
|
||
def _run_stop(message):
|
||
ev = {"hook_event_name": "SubagentStop", "last_assistant_message": message}
|
||
return _sp.run(["python3", _GATE], input=_json.dumps(ev), capture_output=True, text=True)
|
||
|
||
|
||
class TestSubagentStopVerdict(unittest.TestCase):
|
||
def test_malformed_verdict_blocks(self):
|
||
msg = "리뷰 끝.\n```wiki-verdict\nagent: branch-depth-auditor\nverdict: ready\nblocking: 3\nshould_fix: 0\nadvisory: 0\n```"
|
||
r = _run_stop(msg)
|
||
self.assertEqual(r.returncode, 2)
|
||
|
||
def test_valid_verdict_allows(self):
|
||
msg = "리뷰 끝.\n```wiki-verdict\nagent: branch-depth-auditor\nverdict: ready\nblocking: 0\nshould_fix: 0\nadvisory: 0\n```"
|
||
r = _run_stop(msg)
|
||
self.assertEqual(r.returncode, 0)
|
||
|
||
def test_no_marker_allows(self):
|
||
r = _run_stop("그냥 일반 subagent 출력, 마커 없음")
|
||
self.assertEqual(r.returncode, 0)
|
||
```
|
||
|
||
- [ ] **Step 2: Run test to verify it fails**
|
||
|
||
Run: `python3 .claude/hooks/test_wiki_claim_gate.py -k SubagentStopVerdict`
|
||
Expected: FAIL — `test_malformed_verdict_blocks` gets returncode 0 (gate not yet checking verdict).
|
||
|
||
- [ ] **Step 3: Implement — extend `subagent_stop_gate`**
|
||
|
||
In `.claude/hooks/wiki_claim_gate.py`, replace the `subagent_stop_gate` body's final `emit_allow()` with a verdict check before it:
|
||
```python
|
||
def subagent_stop_gate(event: dict) -> None:
|
||
message = event.get("last_assistant_message") or ""
|
||
if not isinstance(message, str):
|
||
emit_allow()
|
||
if re.search(r"Verdict:\s*COMPLETE|\bDONE\b|완료", message):
|
||
missing = []
|
||
for term in ("Claim ID", "Decision Evidence Map", "UNSUPPORTED_DECISION"):
|
||
if term not in message:
|
||
missing.append(term)
|
||
if missing and not event.get("stop_hook_active"):
|
||
emit_block(
|
||
"Subagent output claims completion but does not report claim-traceability checks: "
|
||
+ ", ".join(missing)
|
||
)
|
||
# judge 출력에 wiki-verdict 마커가 있으면 스키마 검증(없으면 judge 아님 → 통과).
|
||
parsed, verr = wiki_rules.validate_verdict_block(message)
|
||
if parsed is not None and verr and not event.get("stop_hook_active"):
|
||
emit_block("judge verdict 블록 스키마 오류:\n- " + "\n- ".join(verr))
|
||
emit_allow()
|
||
```
|
||
(`wiki_rules` is already imported in `wiki_claim_gate.py` from Spec A.)
|
||
|
||
- [ ] **Step 4: Run test to verify it passes**
|
||
|
||
Run: `python3 .claude/hooks/test_wiki_claim_gate.py -k SubagentStopVerdict`
|
||
Expected: PASS (3 tests).
|
||
|
||
- [ ] **Step 5: Checkpoint (no commit)** — `python3 .claude/hooks/test_wiki_claim_gate.py` all green (old 11 + 3).
|
||
|
||
---
|
||
|
||
## Task 5: `wiki-adversarial-reviewer.md` — block + default-refute + quorum doc
|
||
|
||
**Files:**
|
||
- Modify: `.claude/agents/wiki-adversarial-reviewer.md`
|
||
|
||
- [ ] **Step 1: Add the machine verdict block to the Output section**
|
||
|
||
In the `## Output` markdown template (after the `## Aggregate metrics` block, before the closing ```), add — as part of the report the agent must emit:
|
||
````
|
||
## Machine verdict (필수 — SubagentStop 가 검증)
|
||
|
||
```wiki-verdict
|
||
agent: wiki-adversarial-reviewer
|
||
finding: 4.1.1 action: KEEP
|
||
finding: 4.2.1 action: DOWNGRADE
|
||
```
|
||
(모든 Falsification Summary 행의 Finding ID 를 `finding: <id> action: KEEP|DOWNGRADE|REJECT` 로 1:1 반영.)
|
||
````
|
||
|
||
- [ ] **Step 2: Flip to default-refute**
|
||
|
||
In the `## Severity Adjustment` and `Shortcut Trap` sections, change the uncertainty default. Replace the current `INSUFFICIENT_CONTEXT` guidance so that **uncertainty leans toward REJECT/DOWNGRADE, not KEEP**:
|
||
- Add to `## Severity Adjustment`:
|
||
> **Default-refute (deep-research 정렬):** 세 검사 중 하나라도 확신이 안 서면 KEEP 이 아니라 최소 DOWNGRADE. `INSUFFICIENT_CONTEXT` 는 "판단 보류 후 KEEP" 이 아니라 "근거 부족 → 그 finding 의 원 severity 를 신뢰할 수 없음 → DOWNGRADE 권고" 로 처리한다. KEEP 은 세 검사가 *적극적으로* 통과할 때만.
|
||
|
||
- [ ] **Step 3: Document the N=3 quorum flow**
|
||
|
||
Add a new section `## Quorum (opt-in N=3)`:
|
||
````
|
||
고위험 검증 시 controller 가 이 에이전트를 **독립적으로 N=3 병렬 dispatch** 하고, 각 출력의 `wiki-verdict` 블록을 `wiki_quorum.py` 에 투입한다:
|
||
|
||
```
|
||
python3 .claude/hooks/wiki_quorum.py vote1.md vote2.md vote3.md
|
||
```
|
||
|
||
`wiki_quorum.py` 가 per-finding 결정(KILL/DOWNGRADE/KEEP/UNVERIFIED)을 **결정론적으로** 계산한다(≥2 REJECT=KILL, abstain≠pass). controller 는 임계값을 못 바꾼다. 기본은 N=1(단일 패스).
|
||
````
|
||
|
||
- [ ] **Step 4: Verify block present**
|
||
|
||
Run: `grep -c "wiki-verdict" .claude/agents/wiki-adversarial-reviewer.md`
|
||
Expected: ≥1.
|
||
|
||
- [ ] **Step 5: Checkpoint (no commit).**
|
||
|
||
---
|
||
|
||
## Task 6: Standard 4 judges — verdict block
|
||
|
||
**Files:**
|
||
- Modify: `.claude/agents/branch-depth-auditor.md`, `coverage-auditor.md`, `project-readiness-auditor.md`, `wiki-diagram-reviewer.md`
|
||
|
||
- [ ] **Step 1: Add the block to each judge's Output section**
|
||
|
||
In each agent's output template (`## 출력` / `## Output`), directly under the human `Verdict:` line, add the machine block. Use the agent's own name and map its verdict:
|
||
````
|
||
```wiki-verdict
|
||
agent: <이 에이전트 name>
|
||
verdict: ready|not-ready|blocked
|
||
blocking: <N>
|
||
should_fix: <M>
|
||
advisory: <K>
|
||
```
|
||
````
|
||
Per-agent mapping note to include inline:
|
||
- `branch-depth-auditor`: `verdict=ready` ⟺ Blocking 0; else `not-ready`.
|
||
- `coverage-auditor`: `verdict=ready` ⟺ missing(Blocking) 0; else `not-ready`. (`Covered`→ready, `Not-covered`→not-ready.)
|
||
- `project-readiness-auditor`: `verdict=ready` ⟺ Blocking 0 (Ready); else `not-ready`.
|
||
- `wiki-diagram-reviewer`: `verdict=ready` ⟺ 점수 ≥95 (PASS); `<95`→`not-ready`; BLOCKED→`blocked`. `blocking` = HARD-STOP 수.
|
||
|
||
- [ ] **Step 2: Verify all four have the block**
|
||
|
||
Run: `for a in branch-depth-auditor coverage-auditor project-readiness-auditor wiki-diagram-reviewer; do echo -n "$a: "; grep -c "wiki-verdict" .claude/agents/$a.md; done`
|
||
Expected: each prints ≥1.
|
||
|
||
- [ ] **Step 3: Checkpoint (no commit).**
|
||
|
||
---
|
||
|
||
## Task 7: 3-platform manual mirror (4 shared judges)
|
||
|
||
**Files (mirror the SAME block/edits made in Tasks 5-6):**
|
||
- `.agents/plugins/wiki-superpowers/agents/{branch-depth-auditor,coverage-auditor,wiki-adversarial-reviewer,wiki-diagram-reviewer}.md`
|
||
- `.codex/agents/{branch-depth-auditor,coverage-auditor,wiki-adversarial-reviewer,wiki-diagram-reviewer}.md`
|
||
- `.codex/agents/{...}.toml` (the verdict block goes inside the `developer_instructions` string)
|
||
- **NOT** `project-readiness-auditor` — Claude-only, no variants exist.
|
||
|
||
- [ ] **Step 1: Mirror the body changes**
|
||
|
||
For each of the 4 shared judges, copy the verdict-block addition (and for adversarial, the default-refute + quorum sections) from the `.claude/agents/<name>.md` into the three variant files. Keep each platform's frontmatter/format; only the body content is mirrored.
|
||
|
||
- [ ] **Step 2: Grep-verify parity**
|
||
|
||
Run:
|
||
```bash
|
||
for a in branch-depth-auditor coverage-auditor wiki-adversarial-reviewer wiki-diagram-reviewer; do
|
||
echo "== $a =="
|
||
grep -l "wiki-verdict" .claude/agents/$a.md .agents/plugins/wiki-superpowers/agents/$a.md .codex/agents/$a.md .codex/agents/$a.toml
|
||
done
|
||
```
|
||
Expected: all 4 files listed for each judge (16 total).
|
||
|
||
- [ ] **Step 3: Checkpoint (no commit).**
|
||
|
||
---
|
||
|
||
## Task 8: Full regression + acceptance smoke (spec §6)
|
||
|
||
**Files:** none (verification)
|
||
|
||
- [ ] **Step 1: Run all hook test suites**
|
||
|
||
Run:
|
||
```bash
|
||
for t in test_wiki_rules test_wiki_claim_gate test_wiki_structure_lint test_wiki_quorum; do
|
||
out=$(python3 .claude/hooks/$t.py 2>&1 | tail -1); echo "$t -> $out"
|
||
done
|
||
```
|
||
Expected: all `OK`.
|
||
|
||
- [ ] **Step 2: Acceptance §6.4 — SubagentStop live (malformed blocks, valid passes)**
|
||
|
||
Run (printf to avoid JSON mangling; no `>` redirect to avoid the claim_gate bash-gate):
|
||
```bash
|
||
printf '%s' '{"hook_event_name":"SubagentStop","last_assistant_message":"x\n```wiki-verdict\nagent: branch-depth-auditor\nverdict: ready\nblocking: 4\nshould_fix: 0\nadvisory: 0\n```"}' | python3 .claude/hooks/wiki_claim_gate.py; echo "malformed exit=$?"
|
||
printf '%s' '{"hook_event_name":"SubagentStop","last_assistant_message":"x\n```wiki-verdict\nagent: branch-depth-auditor\nverdict: ready\nblocking: 0\nshould_fix: 0\nadvisory: 0\n```"}' | python3 .claude/hooks/wiki_claim_gate.py; echo "valid exit=$?"
|
||
```
|
||
Expected: `malformed exit=2`, `valid exit=0`.
|
||
|
||
- [ ] **Step 3: Acceptance §6.5 — wiki_quorum CLI live**
|
||
|
||
Run (build 3 vote files in /tmp via a /tmp helper to avoid the bash-gate, then tally):
|
||
```bash
|
||
python3 - <<'PY'
|
||
from pathlib import Path
|
||
d = Path("/tmp/qsmoke"); d.mkdir(exist_ok=True)
|
||
def adv(fid_acts):
|
||
lines = "\n".join(f"finding: {f} action: {a}" for f,a in fid_acts)
|
||
return f"```wiki-verdict\nagent: wiki-adversarial-reviewer\n{lines}\n```"
|
||
(d/"v0.md").write_text(adv([("A","REJECT")]))
|
||
(d/"v1.md").write_text(adv([("A","REJECT")]))
|
||
(d/"v2.md").write_text(adv([("A","KEEP")]))
|
||
print("wrote", d)
|
||
PY
|
||
python3 .claude/hooks/wiki_quorum.py /tmp/qsmoke/v0.md /tmp/qsmoke/v1.md /tmp/qsmoke/v2.md; echo "quorum exit=$?"
|
||
rm -rf /tmp/qsmoke
|
||
```
|
||
Expected: table with finding `A … KILL`, `quorum exit=1`.
|
||
|
||
- [ ] **Step 4: Acceptance §6.7 — grep parity**
|
||
|
||
Run the Task 7 Step 2 grep + the Task 6 Step 2 grep. Expected: every judge file carries `wiki-verdict`.
|
||
|
||
- [ ] **Step 5: Final checkpoint (no commit)** — report all results. Leave changes in working tree.
|
||
|
||
---
|
||
|
||
## Self-Review (completed by plan author)
|
||
|
||
- **Spec coverage:** §4.1 block format → Tasks 5/6. §4.2 validate+tally → Tasks 1/2. §4.3 SubagentStop → Task 4. §4.4 wiki_quorum.py → Task 3. §4.5 agent edits + mirror → Tasks 5/6/7. §6 acceptance 1-7 → Tasks 1-4 tests + Task 8 smokes. No requirement unmapped.
|
||
- **Placeholder scan:** agent-edit tasks give the exact block text + exact mapping per agent + exact mirror file list; no "TBD"/"similar to". Deterministic-core tasks carry full runnable code.
|
||
- **Type/name consistency:** `parse_verdict_block`/`validate_verdict_block`/`tally_quorum` names match across Tasks 1-4 and the CLI; the `wiki-verdict` fence string is identical in tests, impl, agent blocks, and smokes; `REFUTATIONS_REQUIRED=2` matches the tally tests (REJECT×2 = KILL).
|