llm-wiki/rules/evidence-first-research.md

# Evidence-First Research Rule

This rule applies to every wiki research, document review, design critique, planning, audit, or task where the agent summarizes or evaluates files in this LLM Wiki repository.

**Wiki scope:** 본 rule은 `raw/`, `wiki/`, `templates/` 디렉토리의 마크다운 문서에 대한 evidence discipline을 강제한다. 코드(Java/CA) 작업의 evidence discipline은 ca-tmpl `.agents/plugins/ca-superpowers/rules/evidence-first-research.md` 가 처리한다 — 본 rule 과 핵심 원칙은 동일하나 subagent dispatch 대상이 다르다 (본 rule은 `wiki-research-lane`, ca-tmpl rule은 `ca-implementer`/`ca-architect-sentinel`).

## Prime Rule

> **A file is not "reviewed" until its body has been opened and inspected.**

Filenames, paths, titles, prior memory, and general expertise are not evidence. They produce hallucinated conclusions and dishonest reports.

## Rationalization Stop List

If the agent catches itself thinking any of the following, it must stop and either read the missing files or dispatch subagents. None of these thoughts are valid reasons to skip reading.

| If you are thinking... | The truth |
|---|---|
| "I can infer this from the filename, it is obvious." | That is `FILENAME_INFERENCE`. Not acceptable as a final finding. |
| "I remember roughly what is in this file from earlier." | That is `MEMORY_HALLUCINATION`. Memory of files is stale and not evidence. |
| "I am confident this is what the file says." | Confidence without a read is `CONFIDENCE_WITHOUT_READ`. Open the file. |
| "All these files probably follow the same pattern, I can answer for the batch." | That is `BATCH_ASSUMPTION`. Each file must be read or marked `NOT_READ`. |
| "Reading all of them will take too long, I will summarize from a few." | Split work across multiple Read calls. For document-heavy research, redirect to LLM Wiki (`wiki-research-lane`). There is no shortcut. |
| "The user only approved a few files, I will fill in the rest from training data." | Files outside the approved slice are `UNVERIFIED`. Report them as such, do not invent content. |
| "A quick high-level pass is good enough for now." | A high-level pass without evidence is not a finding, it is a guess. |
| "The user will not notice if I skip a few files." | The user always notices. Honesty about coverage is required. |

## Named Failure Modes

Use these exact labels when reporting on unread or under-read material:

- `FACT`: directly supported by file content, command output, or tool result.
- `INFERENCE`: reasoned from explicit facts. Must be marked `INFERENCE`, not stated as fact.
- `FILENAME_INFERENCE`: guessed from path or title only. Not acceptable as a final conclusion.
- `MEMORY_HALLUCINATION`: produced from prior memory of a file rather than a current read. Not acceptable.
- `CONFIDENCE_WITHOUT_READ`: stated with confidence but no read evidence. Not acceptable.
- `BATCH_ASSUMPTION`: extrapolated from a few files to a larger group. Not acceptable.
- `UNVERIFIED`: not read, not accessible, or not approved for reading. Acceptable as a status, never as a finding.

Any unread material that appears in a response must be presented as `NOT_READ` / `BLOCKED` / `UNVERIFIED`. It cannot be promoted to a conclusion.

## Approved Scope Discipline

If the user approved reading only N specific files, the agent reads exactly those N files and reports every other in-scope file as `NOT_READ`. The agent does not claim coverage of files outside the approved slice. The agent does not "fill in" content for files it could not open.

If the agent realizes that the approved slice is too narrow for the user's request, the agent surfaces this gap and asks for permission to expand the slice or to dispatch subagents. It does not proceed by guessing.

## Required Evidence Matrix

For any multi-file review, response must include:

```text
| Path | Status | Evidence | Extracted facts |
| --- | --- | --- | --- |
| raw/branch-notes/feature-keycloak-oauth2-proxy-oidc-flow.md | READ_FULL | lines 1-140 | OIDC handshake 흐름 정의, oauth2-proxy 결정 근거 |
| raw/official-docs/oidc-discovery-keycloak-official.md | READ_PARTIAL | lines 1-80, 220-280 | Discovery endpoint 명세만 정독, token-introspection 미정독 |
| raw/branch-notes/feature-keycloak-nginx-auth-request-integration.md | NOT_READ | not approved / not found | UNVERIFIED |
```

Allowed status values are exactly:

- `READ_FULL`: file body read end to end.
- `READ_PARTIAL`: only named sections or line ranges read.
- `NOT_READ`: file body not read.
- `BLOCKED`: file could not be read due to permission, path, tooling, or approval limits.

If any in-scope file is `NOT_READ` or `BLOCKED`, the response must state that whole-corpus conclusions are incomplete.

## Claim Traceability Gate

For source-backed wiki work, evidence must be traceable at claim granularity.

`raw/official-docs/` and `raw/company-tech-blogs/` documents should expose stable claim IDs in a `Claims Extracted` table:

```text
| Claim ID | Claim | Evidence quote | Strength | Applies to | Does not prove |
|---|---|---|---|---|---|
| C1 | <source-backed claim> | <quote/section> | official-vendor-doc | <condition> | <boundary> |
```

`raw/branch-notes/` documents should map decisions to those claims:

```text
| Decision ID | Decision | Supporting Claims | Evidence Strength | Open Risk |
|---|---|---|---|---|
| D1 | <decision> | raw/official-docs/<slug>.md#C1 | official-vendor-doc | <risk> |
```

Rules:

1. A source claim is only what the source directly says. Project application is not a source claim.
2. A branch decision without at least one supporting claim is `UNSUPPORTED_DECISION`.
3. A company-tech-blog claim is a case study, not a universal rule, unless corroborated by an official source.
4. A wiki concept may summarize only claim-backed knowledge as `FACT`. Anything else must be marked `INFERENCE` or `needs-confirmation`.
5. Audit reports must not say a decision is "officially supported" unless the linked claim strength is `official-standard`, `official-vendor-doc`, or `official-reference`.


## Whole-Corpus Claim Gate

The agent does not summarize, rank, approve, reject, or make recommendations about a whole corpus unless every in-scope file is either:

- `READ_FULL`, or
- `READ_PARTIAL` with the limitation explicitly carried into the conclusion.

If any in-scope file is `NOT_READ` or `BLOCKED`, the agent must say the corpus-level conclusion is incomplete and identify exactly which files remain unreviewed.

## Mandatory Subagent Dispatch

Split work across multiple Read calls (or redirect document-heavy research to LLM Wiki `wiki-research-lane`) when any of these are true:

- More than 10 files must be reviewed.
- More than 5,000 lines must be reviewed.
- The corpus contains 3 or more independent topics.
- The user asks for an exhaustive review.
- The user explicitly asks the agent to use subagents.
- The agent cannot safely keep all evidence in one context window.

Each dispatched subagent must receive:

- the exact file list for its slice,
- the required output contract,
- the requirement to produce an evidence matrix,
- a prohibition on filename-only conclusions,
- instructions to label unread files as `NOT_READ` or `BLOCKED`.

The controller merges only evidence-backed findings. Subagent reports without an evidence matrix are treated as `BLOCKED`.

## Pre-Send Output Gate

Before sending any multi-file response, the agent must run a literal text check against its own draft. The draft is `BLOCKED` and must be rewritten if any of these conditions fails.

1. The draft contains the literal string `| Path | Status | Evidence | Extracted facts |`. No matrix → `BLOCKED`.
2. Row count in the matrix equals the number of in-scope files. Count mismatch → `BLOCKED` unless the draft includes an explicit reconciliation block naming every file that is in scope but absent from the matrix, along with the reason.
3. Every row's Status column is exactly one of `READ_FULL`, `READ_PARTIAL`, `NOT_READ`, `BLOCKED`. Any other value → `BLOCKED`.
4. Every concrete factual claim in the draft (numbers, setting names, literal quotes, behavior assertions) is tied to a row whose Status is `READ_FULL` or `READ_PARTIAL`. A claim tied to a `NOT_READ` or `BLOCKED` row → remove the claim or relabel it `UNVERIFIED` before sending.
5. Any file mentioned in a priority list, "top issues" table, summary table, or recommendation block must also have a row in the evidence matrix with Status `READ_FULL` or `READ_PARTIAL`. Priority references to unread files → `BLOCKED`.
6. The user-stated count of files (if the user said "N files") matches the matrix row count, or the draft contains an explicit reconciliation paragraph naming every file that did not get its own row and why.

If the draft fails this gate, the agent does not send it. It marks the draft `BLOCKED`, identifies the missing rows or unsupported claims, dispatches the necessary subagents or reads, and produces a new draft that passes the gate.

Apology is not evidence. Polished prose is not evidence. Confidence is not evidence. Only `READ_FULL` and `READ_PARTIAL` rows are evidence.

## Failure Handling

If the agent realizes mid-response that it answered from filenames, memory, assumptions, or general expertise:

1. Stop expanding the answer.
2. State exactly which claims were unsupported, using the named labels above.
3. Provide the actual read status for each in-scope file.
4. Re-run the work with subagent dispatch and an evidence matrix before stating any new conclusions.

The agent does not paper over missing evidence with apology, confidence, or polished prose. Apologies are not evidence.