97 lines
4.8 KiB
YAML
97 lines
4.8 KiB
YAML
agent-operating-kpi:
|
|
version: 1
|
|
# fan-out 토큰 예산(권고 #1). 근거: Anthropic 멀티에이전트 ~15배 토큰, 토큰량이 성능분산 80% 설명.
|
|
# 초과 시 Orchestrator가 fan-out을 collapse(단일 종합)로 강등(execution-policy fan-out-collapse-policy).
|
|
token-budgets:
|
|
note: wave당 fan-out 누적 토큰 상한(가늠값 — 실측 token-ledger로 보정). 강제 주체는 Orchestrator, 계측·게이트는 token_ledger.py.
|
|
per-wave:
|
|
light: 150000
|
|
standard: 500000
|
|
heavy: 2000000
|
|
cost-per-1k-tokens-usd: 0.015 # cost-per-decision 추정용 대략치(모델/플랜별 상이)
|
|
enforced-by: ".claude/hooks/token_ledger.py (log/dashboard/check) + run-wave Orchestrator 강등 규칙"
|
|
ledger: state/token-ledger.jsonl (append-only)
|
|
dashboard: reports/TOKENS.md
|
|
# finding #19: 수집기. 파생 가능한 KPI는 기존 아티팩트(completion-records 시도수·acceptance-events
|
|
# 결정·token-ledger)에서 실측하고, 나머지는 수동 이벤트/미측정으로 정직히 구분한다.
|
|
collector:
|
|
tool: .claude/hooks/kpi_ledger.py
|
|
commands: "derive [--workflow WF] · log --metric M --value V · dashboard"
|
|
ledger: state/kpi-ledger.jsonl (append-only)
|
|
dashboard: reports/KPI.md
|
|
derived-now: [rework-rate, duplicate-report-rate, human-intervention-rate, release-gate-failure-rate, context-bloat-rate]
|
|
honesty: 미측정 KPI를 '측정됨'으로 위장하지 않는다(대시보드가 derived/manual/미측정 구분).
|
|
metrics:
|
|
rework-rate:
|
|
definition: Changes Requested count / submitted outputs
|
|
target: decrease-over-time
|
|
blocker-reopen-rate:
|
|
definition: reopened blockers / resolved blockers
|
|
target: below-5-percent
|
|
hallucination-rate:
|
|
definition: rejected unsupported claims / total claims reviewed
|
|
target: below-2-percent
|
|
context-bloat-rate:
|
|
definition: unused-or-duplicated-context-tokens / total-planned-context-tokens
|
|
target: below-10-percent
|
|
decision-latency:
|
|
definition: time from Decision Brief to user or CEO AI decision
|
|
target: fit-by-priority
|
|
release-gate-failure-rate:
|
|
definition: failed Release Acceptance checks / total release checks
|
|
target: decrease-over-time
|
|
human-intervention-rate:
|
|
definition: human interventions / total workflows
|
|
target: expected-for-high-risk-only
|
|
duplicate-report-rate:
|
|
definition: duplicated reports / total reports
|
|
target: below-5-percent
|
|
skipped-role-incident-rate:
|
|
definition: incidents caused by skipped roles / total incidents
|
|
target: zero-critical
|
|
time-to-market:
|
|
definition: time from approved product hypothesis to first verified delivery
|
|
target: decrease-without-quality-loss
|
|
value-stream-lead-time:
|
|
definition: time from user/business friction discovery to accepted workflow outcome
|
|
target: decrease-over-time
|
|
platform-adoption-rate:
|
|
definition: workflows using approved golden path / eligible platform workflows
|
|
target: increase-over-time
|
|
golden-path-exception-rate:
|
|
definition: approved exceptions to platform golden path / eligible platform workflows
|
|
target: below-10-percent
|
|
slo-risk-escape-rate:
|
|
definition: SLO-impacting risks found after release acceptance / SLO-impacting workflows
|
|
target: zero-critical
|
|
shift-left-detection-rate:
|
|
definition: security or quality issues found before implementation / total issues found
|
|
target: increase-over-time
|
|
capability-duplication-rate:
|
|
definition: duplicated business or platform capabilities / reviewed capabilities
|
|
target: decrease-over-time
|
|
learning-capture-rate:
|
|
definition: workflows with explicit learning and simpler-alternative reflection / total closed workflows
|
|
target: above-90-percent
|
|
pipeline-forecast-accuracy:
|
|
definition: 1 - absolute pipeline forecast variance
|
|
target: above-90-percent
|
|
PQL-to-SQL-conversion-rate:
|
|
definition: sales-qualified leads from product-qualified leads / product-qualified leads
|
|
target: increase-over-time
|
|
lead-to-cash-cycle-time:
|
|
definition: time from qualified lead or PLS trigger to closed-won or closed-lost outcome
|
|
target: decrease-without-quality-loss
|
|
net-revenue-retention:
|
|
definition: retained recurring revenue plus expansion minus contraction and churn
|
|
target: above-100-percent
|
|
pricing-governance-violation-rate:
|
|
definition: unauthorized pricing or discount exceptions / total reviewed deals
|
|
target: below-2-percent
|
|
legal-review-latency:
|
|
definition: time from contract or compliance review request to approval or changes requested
|
|
target: fit-by-priority
|
|
partner-sourced-revenue-share:
|
|
definition: partner-sourced ARR / total new ARR
|
|
target: increase-where-strategic
|