feat(ci): add a read-only CI metrics script for the desktop runner

There is no number for how long CI runs wait, how busy the one runner is,
how many runs a pull request costs or how long main stays red. Run times
were added up by hand from single runs, so whether a change helped could
not be told apart from load.

infrastructure/ci/metrics/ci_metrics.py reads the Gitea Actions API (runs,
their jobs, closed pull requests) for the repositories the runner serves
and prints the CI manual's metrics 1-5 per repository and stage: run
execution p50/p90, queue wait p50/p90, runner utilisation per KST day,
pull request runs per merged pull request, and main red time. It only
reads; credentials come from GITEA_TOKEN or `git credential fill` and are
never printed.

The arithmetic follows what our Gitea 1.27 returns, which differs from the
API reference: a run has no created_at (the earliest job created_at is
used), unstarted runs and jobs report 1970-01-01 times (treated as
absent), and a job's created_at is its run's, so queue wait is measured
per run (first job start minus run creation) instead of per job.

Verification: python3 -m unittest test_ci_metrics (11 offline tests) and
one run against the server for 2026-11-07..13 (two repositories, 30 s,
rc=0). The sensitive-source scan needs rg and ran on the desktop host.

Refs: design-docs ci/03-time-reduction-plan.md M-0
This commit is contained in:
DongHyeonka
2026-11-13 19:34:38 +09:00
parent 5d27096ed0
commit dd8e9140b7
3 changed files with 554 additions and 0 deletions
+46
View File
@@ -0,0 +1,46 @@
# CI 지표
desktop runner 하나가 받는 repo들의 CI 지표를 Gitea Actions API에서 뽑는다. 읽기만 하고 서버에 아무것도 쓰지 않는다. 지표의 정의와 목표는 design-docs `ci/manual.md` 10절이고, 주 1회 결과를 design-docs `ci/metrics.md`에 한 줄 남긴다.
## 돌리기
```bash
python3 infrastructure/ci/metrics/ci_metrics.py \
--repo clean-architecture-backend-template \
--repo clean-architecture-frontend-template \
--since 2026-11-07 --until 2026-11-13
```
- 날짜는 KST 하루 단위이고 `--until`도 그날을 포함한다(`--utc-offset-hours`로 바꾼다).
- 인증: `GITEA_TOKEN`(읽기 전용 토큰)이 있으면 그것을, 없으면 `git credential fill`의 그 호스트 자격 증명을 쓴다. 어느 쪽도 출력하지 않는다.
- `--json`은 표 대신 숫자를 JSON으로 낸다.
- 두 repo, 지난 7일이면 30초 안팎이다(run마다 jobs API를 한 번 부른다).
## 내는 것
| # | 지표 | 계산 |
| --- | --- | --- |
| 1 | 실행 시간 p50·p90 | 단계(`workflow:event`)별, 끝난 run(success·failure)의 job 실행 시간 합. cancelled run은 run 수에만 센다 |
| 2 | 대기 시간 p50·p90 | run의 첫 job 시작 − run 생성 |
| 3 | runner 사용률 | 날짜(job 시작 시각, KST)별 job 실행 시간 합 ÷ 1,440분. 준 repo를 모두 더한다(runner가 하나라서) |
| 4 | PR당 run 수 | 기간 안에 병합된 PR마다 그 PR의 `pull_request` run 수. 병합 전 7일의 run까지 센다 |
| 5 | main 빨강 시간 | workflow별 main push run이 처음 실패한 run의 생성부터 다음 성공 run의 끝까지. 기간 끝까지 빨강이면 그때까지 |
지표 6(흔들림)은 각 repo `docs/ledger/flaky-tests.jsonl`, 지표 7(캐시 적중)은 PR run 로그에서 나온다. 이 스크립트는 보지 않는다.
## 우리 서버(Gitea 1.27)가 API 명세와 다른 곳
2026-11-13 응답 한 건씩으로 확인했다. 계산은 서버를 따른다.
- run에 `created_at`이 없다. run의 job은 run과 함께 만들어지므로, job `created_at`의 가장 이른 값을 run 생성 시각으로 쓴다.
- 시작하지 않은 run·job의 `started_at`·`completed_at`은 `1970-01-01T00:00:00Z`다. 없는 값으로 본다.
- job의 `created_at`은 그 job이 실행할 수 있게 된 시각이 아니라 run의 생성 시각이다. runner 하나가 run의 job을 차례로 돌리므로 job마다 `started_at − created_at`을 대기로 세면 앞 job의 실행이 대기에 들어간다. 그래서 대기는 run 단위로 센다. `needs`가 있는 job이 늦게 풀려 대기열 뒤로 가는 시간은 이 지표에 안 보이고 실행 시간에도 안 들어간다.
- run 목록의 run에는 `head_branch`가 없다. 단계와 ref는 `path`(`<workflow>@<ref>`)에서 읽고, PR 번호는 `pull_requests`에서 읽는다.
## 시험
```bash
cd infrastructure/ci/metrics && python3 -m unittest -v test_ci_metrics
```
서버 없이 계산만 본다. 만든 run·job 기록으로 대기·실행·빨강 시간·사용률을 확인한다.
+384
View File
@@ -0,0 +1,384 @@
#!/usr/bin/env python3
"""CI metrics for the repositories one Gitea Actions runner serves.
Reads the Gitea Actions API (runs, their jobs, closed pull requests) and prints, per repository and
stage, the CI manual's metrics 1-5: run execution time, queue wait, runner utilisation, pull
request runs per merged pull request and time main stayed red. Read-only; it never writes to the
server.
What our Gitea 1.27 returns differs from the gitea.com API reference in three places, and the
arithmetic below follows the server:
* A run carries no ``created_at``. A run's jobs are all created with the run, so the earliest job
``created_at`` is the run's creation time.
* A run or job that has not started reports ``started_at`` and ``completed_at`` as
``1970-01-01T00:00:00Z``. Those are treated as absent.
* A job's ``created_at`` is the run's, not the moment it became runnable. One runner runs a run's
jobs one after another, so ``started_at - created_at`` of a later job counts the earlier jobs'
execution as waiting. Queue wait is therefore measured per run: first job start minus run
creation.
Credentials: ``GITEA_TOKEN`` (a read-only token) if set, otherwise ``git credential fill`` for the
server host. Neither is printed.
"""
from __future__ import annotations
import argparse
import base64
import json
import math
import os
import subprocess
import sys
import urllib.parse
import urllib.request
from dataclasses import dataclass, field
from datetime import date, datetime, timedelta, timezone
EPOCH_PREFIXES = ("1970-01-01", "0001-01-01")
FINISHED = {"success", "failure"}
def parse_time(value: str | None) -> datetime | None:
"""An API timestamp, or None for a missing or zero value."""
if not value or value.startswith(EPOCH_PREFIXES):
return None
return datetime.fromisoformat(value.replace("Z", "+00:00"))
def percentile(values: list[float], fraction: float) -> float | None:
"""Nearest-rank percentile; None for no values."""
if not values:
return None
ordered = sorted(values)
rank = max(1, math.ceil(fraction * len(ordered)))
return ordered[rank - 1]
@dataclass
class Job:
name: str
created: datetime | None
started: datetime | None
completed: datetime | None
conclusion: str | None
@property
def minutes(self) -> float:
if self.started is None or self.completed is None:
return 0.0
return max(0.0, (self.completed - self.started).total_seconds() / 60)
@dataclass
class Run:
repo: str
id: int
workflow: str
ref: str
event: str
status: str
conclusion: str | None
pull_request: int | None
jobs: list[Job] = field(default_factory=list)
@property
def stage(self) -> str:
return f"{self.workflow}:{self.event}"
@property
def created(self) -> datetime | None:
times = [job.created for job in self.jobs if job.created is not None]
return min(times) if times else None
@property
def first_start(self) -> datetime | None:
times = [job.started for job in self.jobs if job.started is not None]
return min(times) if times else None
@property
def completed(self) -> datetime | None:
times = [job.completed for job in self.jobs if job.completed is not None]
return max(times) if times and self.status == "completed" else None
@property
def execution_minutes(self) -> float:
return sum(job.minutes for job in self.jobs)
@property
def queue_minutes(self) -> float | None:
if self.created is None or self.first_start is None:
return None
return max(0.0, (self.first_start - self.created).total_seconds() / 60)
@property
def main_push(self) -> bool:
return self.event == "push" and self.ref == "refs/heads/main"
def run_from_api(repo: str, raw: dict, jobs: list[dict]) -> Run:
workflow, _, ref = raw.get("path", "").partition("@")
pulls = raw.get("pull_requests") or []
return Run(
repo=repo,
id=raw["id"],
workflow=workflow,
ref=ref,
event=raw.get("event", ""),
status=raw.get("status", ""),
conclusion=raw.get("conclusion"),
pull_request=pulls[0]["number"] if pulls else None,
jobs=[
Job(
name=job.get("name", ""),
created=parse_time(job.get("created_at")),
started=parse_time(job.get("started_at")),
completed=parse_time(job.get("completed_at")),
conclusion=job.get("conclusion"),
)
for job in jobs
],
)
class Gitea:
"""The few read-only API calls this script makes."""
def __init__(self, server: str):
self.server = server.rstrip("/")
self.header = self._authorization()
def _authorization(self) -> str:
token = os.environ.get("GITEA_TOKEN")
if token:
return f"token {token}"
host = urllib.parse.urlparse(self.server).netloc
answer = subprocess.run(
["git", "credential", "fill"],
input=f"protocol=https\nhost={host}\n\n",
capture_output=True,
text=True,
check=True,
).stdout
fields = dict(line.split("=", 1) for line in answer.splitlines() if "=" in line)
pair = f"{fields['username']}:{fields['password']}".encode()
return "Basic " + base64.b64encode(pair).decode()
def get(self, path: str) -> object:
request = urllib.request.Request(
f"{self.server}/api/v1{path}",
headers={"Authorization": self.header, "Accept": "application/json"},
)
with urllib.request.urlopen(request, timeout=30) as response:
return json.load(response)
def fetch_runs(api: Gitea, owner: str, repo: str, since: datetime) -> list[Run]:
"""Runs created at or after ``since``, newest first. Run ids grow with creation time."""
runs: list[Run] = []
page = 1
while True:
body = api.get(f"/repos/{owner}/{repo}/actions/runs?limit=50&page={page}")
batch = body.get("workflow_runs") or []
if not batch:
return runs
for raw in batch:
jobs = api.get(f"/repos/{owner}/{repo}/actions/runs/{raw['id']}/jobs").get("jobs") or []
run = run_from_api(repo, raw, jobs)
if run.created is not None and run.created < since:
return runs
runs.append(run)
page += 1
def fetch_merged_pulls(api: Gitea, owner: str, repo: str, since: datetime, until: datetime) -> list[int]:
numbers: list[int] = []
page = 1
while True:
batch = api.get(f"/repos/{owner}/{repo}/pulls?state=closed&sort=recentupdate&limit=50&page={page}")
if not batch:
return numbers
for pull in batch:
merged = parse_time(pull.get("merged_at"))
if pull.get("merged") and merged is not None and since <= merged < until:
numbers.append(pull["number"])
updated = [parse_time(pull.get("updated_at")) for pull in batch]
if all(time is not None and time < since for time in updated):
return numbers
page += 1
@dataclass
class RedEpisode:
workflow: str
first_failed_run: int
fixed_by_run: int | None
hours: float
def red_episodes(runs: list[Run], now: datetime) -> list[RedEpisode]:
"""Main stays red from the first failing main push run's creation to the next green one's end."""
episodes: list[RedEpisode] = []
by_workflow: dict[str, list[Run]] = {}
for run in runs:
if run.main_push and run.status == "completed" and run.conclusion in FINISHED:
by_workflow.setdefault(run.workflow, []).append(run)
for workflow, history in by_workflow.items():
history.sort(key=lambda run: run.created or now)
start: Run | None = None
for run in history:
if run.conclusion == "failure" and start is None:
start = run
elif run.conclusion == "success" and start is not None:
end = run.completed or now
episodes.append(
RedEpisode(workflow, start.id, run.id, (end - start.created).total_seconds() / 3600)
)
start = None
if start is not None:
episodes.append(RedEpisode(workflow, start.id, None, (now - start.created).total_seconds() / 3600))
return episodes
def stage_rows(runs: list[Run]) -> list[dict]:
rows = []
for stage in sorted({run.stage for run in runs}):
finished = [run for run in runs if run.stage == stage and run.conclusion in FINISHED]
waited = [run.queue_minutes for run in runs if run.stage == stage and run.queue_minutes is not None]
executed = [run.execution_minutes for run in finished]
rows.append(
{
"stage": stage,
"runs": len([run for run in runs if run.stage == stage]),
"finished": len(finished),
"red": len([run for run in finished if run.conclusion == "failure"]),
"exec_p50": percentile(executed, 0.5),
"exec_p90": percentile(executed, 0.9),
"wait_p50": percentile(waited, 0.5),
"wait_p90": percentile(waited, 0.9),
}
)
return rows
def utilisation(runs: list[Run], offset: timedelta) -> dict[date, float]:
"""Job execution minutes per local day (job start day) across every repository given."""
minutes: dict[date, float] = {}
for run in runs:
for job in run.jobs:
if job.started is not None and job.minutes > 0:
day = (job.started + offset).date()
minutes[day] = minutes.get(day, 0.0) + job.minutes
return minutes
def runs_per_merged_pull(runs: list[Run], merged: list[int]) -> list[int]:
return [len([run for run in runs if run.pull_request == number]) for number in merged]
def fmt(value: float | None, digits: int = 1) -> str:
return "-" if value is None else f"{value:.{digits}f}"
def report(window: str, per_repo: dict[str, dict], days: dict[date, float]) -> str:
lines = [f"# CI 지표 ({window})", ""]
for repo, data in per_repo.items():
lines += [
f"## {repo}",
"",
"| 단계 | run | 끝난 run | 빨강 | 실행 p50(분) | 실행 p90(분) | 대기 p50(분) | 대기 p90(분) |",
"|---|---|---|---|---|---|---|---|",
]
for row in data["stages"]:
lines.append(
f"| {row['stage']} | {row['runs']} | {row['finished']} | {row['red']} | {fmt(row['exec_p50'])} | "
f"{fmt(row['exec_p90'])} | {fmt(row['wait_p50'])} | {fmt(row['wait_p90'])} |"
)
counts = data["runs_per_pull"]
lines += [
"",
f"- 병합된 PR {len(counts)}개, PR당 run 수 p50 {fmt(percentile(counts, 0.5), 0)}, "
f"p90 {fmt(percentile(counts, 0.9), 0)}, 최대 {max(counts) if counts else '-'}",
]
episodes = data["red"]
if episodes:
longest = max(episodes, key=lambda episode: episode.hours)
lines.append(
f"- main 빨강 {len(episodes)}회, 합 {sum(e.hours for e in episodes):.1f}시간, p90 "
f"{fmt(percentile([e.hours for e in episodes], 0.9))}시간, 가장 긴 것 {longest.hours:.1f}시간"
f"({longest.workflow} run {longest.first_failed_run} → "
f"{longest.fixed_by_run if longest.fixed_by_run else '아직 빨강'})"
)
else:
lines.append("- main 빨강 0회")
lines.append("")
lines += ["## runner 사용률 (위 repo 합, job 시작 날짜 기준)", "", "| 날짜 | 실행 합(분) | 사용률 |", "|---|---|---|"]
for day in sorted(days):
lines.append(f"| {day} | {days[day]:.0f} | {days[day] / 1440:.0%} |")
if days:
busiest = max(days, key=days.get)
lines += ["", f"- 바쁜 날 {busiest}: {days[busiest] / 1440:.0%}"]
return "\n".join(lines) + "\n"
def main(argv: list[str]) -> int:
parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
parser.add_argument("--server", default="https://git.learn.hyeonworks.com")
parser.add_argument("--owner", default="donghyeon.kang")
parser.add_argument("--repo", action="append", required=True, help="repeat for each repository")
parser.add_argument("--since", required=True, type=date.fromisoformat, help="first local day, YYYY-MM-DD")
parser.add_argument("--until", required=True, type=date.fromisoformat, help="last local day, inclusive")
parser.add_argument("--utc-offset-hours", type=int, default=9, help="local day boundary (default KST)")
parser.add_argument("--json", action="store_true", help="print the numbers as JSON instead of tables")
args = parser.parse_args(argv)
offset = timedelta(hours=args.utc_offset_hours)
since = datetime.combine(args.since, datetime.min.time(), timezone.utc) - offset
until = datetime.combine(args.until + timedelta(days=1), datetime.min.time(), timezone.utc) - offset
# A pull request merged in the window may have had runs in the week before it.
lookback = since - timedelta(days=7)
now = min(datetime.now(timezone.utc), until)
api = Gitea(args.server)
per_repo: dict[str, dict] = {}
window_runs: list[Run] = []
for repo in args.repo:
fetched = fetch_runs(api, args.owner, repo, lookback)
in_window = [run for run in fetched if run.created is not None and since <= run.created < until]
window_runs += in_window
merged = fetch_merged_pulls(api, args.owner, repo, since, until)
per_repo[repo] = {
"stages": stage_rows(in_window),
"runs_per_pull": runs_per_merged_pull(fetched, merged),
"red": red_episodes(in_window, now),
}
days = utilisation(window_runs, offset)
window = f"{args.since} ~ {args.until}, UTC{args.utc_offset_hours:+d}"
if args.json:
print(
json.dumps(
{
"window": window,
"repos": {
repo: {
"stages": data["stages"],
"runs_per_pull": data["runs_per_pull"],
"red": [episode.__dict__ for episode in data["red"]],
}
for repo, data in per_repo.items()
},
"utilisation_minutes": {str(day): round(value, 1) for day, value in sorted(days.items())},
},
ensure_ascii=False,
indent=2,
)
)
else:
print(report(window, per_repo, days), end="")
return 0
if __name__ == "__main__":
sys.exit(main(sys.argv[1:]))
@@ -0,0 +1,124 @@
"""Offline tests for ci_metrics: the arithmetic on run and job records, no server."""
import unittest
from datetime import datetime, timedelta, timezone
import ci_metrics as m
T0 = datetime(2026, 11, 13, 0, 0, tzinfo=timezone.utc)
def at(minutes: float) -> str:
return (T0 + timedelta(minutes=minutes)).isoformat().replace("+00:00", "Z")
def job(name, created, started=None, completed=None, conclusion="success"):
return {
"name": name,
"created_at": at(created),
"started_at": at(started) if started is not None else "1970-01-01T00:00:00Z",
"completed_at": at(completed) if completed is not None else "1970-01-01T00:00:00Z",
"conclusion": conclusion,
}
def run(run_id, path, event, conclusion, jobs, status="completed", pull=None):
raw = {"id": run_id, "path": path, "event": event, "status": status, "conclusion": conclusion}
if pull is not None:
raw["pull_requests"] = [{"number": pull}]
return m.run_from_api("backend", raw, jobs)
class ParsingTest(unittest.TestCase):
def test_a_zero_timestamp_from_a_job_that_has_not_started_is_absent(self):
self.assertIsNone(m.parse_time("1970-01-01T00:00:00Z"))
def test_the_run_is_created_when_its_earliest_job_is(self):
r = run(1, "pr.yml@refs/pull/7/head", "pull_request", "success", [job("done", 5, 20, 32)], pull=7)
self.assertEqual((r.created, r.stage, r.pull_request), (T0 + timedelta(minutes=5), "pr.yml:pull_request", 7))
class TimingTest(unittest.TestCase):
def test_queue_wait_is_the_first_job_start_minus_run_creation_not_each_jobs_own_gap(self):
# given: one runner runs the jobs one after another, so the second job's own gap is 40 minutes
r = run(1, "main.yml@refs/heads/main", "push", "success", [job("a", 0, 10, 30), job("b", 0, 30, 40)])
# when
waited = r.queue_minutes
# then
self.assertEqual(waited, 10)
def test_execution_is_the_sum_of_job_run_times_and_a_skipped_job_adds_nothing(self):
r = run(1, "main.yml@refs/heads/main", "push", "success",
[job("a", 0, 10, 30), job("b", 0, 30, 40), job("c", 0, 40, 40, "skipped")])
self.assertEqual(r.execution_minutes, 30)
def test_a_queued_run_has_no_wait_and_no_execution_yet(self):
r = run(1, "pr.yml@refs/pull/7/head", "pull_request", None, [job("done", 0)], status="queued", pull=7)
self.assertEqual((r.queue_minutes, r.execution_minutes), (None, 0.0))
def test_percentiles_use_the_nearest_rank(self):
self.assertEqual((m.percentile([3, 1, 2, 10], 0.5), m.percentile([3, 1, 2, 10], 0.9)), (2, 10))
class StageTest(unittest.TestCase):
def test_a_cancelled_run_counts_as_a_run_but_not_in_execution_percentiles(self):
# given
runs = [
run(1, "pr.yml@refs/pull/7/head", "pull_request", "success", [job("done", 0, 1, 13)], pull=7),
run(2, "pr.yml@refs/pull/7/head", "pull_request", "cancelled", [job("done", 0, 1, 2)], pull=7),
]
# when
row = m.stage_rows(runs)[0]
# then
self.assertEqual((row["runs"], row["finished"], row["exec_p50"], row["wait_p50"]), (2, 1, 12, 1))
def test_runs_per_merged_pull_request_count_every_attempt_of_that_pull_request(self):
runs = [
run(1, "pr.yml@refs/pull/7/head", "pull_request", "cancelled", [job("done", 0)], pull=7),
run(2, "pr.yml@refs/pull/7/head", "pull_request", "success", [job("done", 1, 2, 14)], pull=7),
run(3, "pr.yml@refs/pull/8/head", "pull_request", "success", [job("done", 1, 2, 14)], pull=8),
]
self.assertEqual(m.runs_per_merged_pull(runs, [7, 8]), [2, 1])
class RedTest(unittest.TestCase):
def test_main_is_red_from_the_first_failing_runs_creation_to_the_next_green_runs_end(self):
# given: two failures in a row, then a green run that ends at minute 210
runs = [
run(1, "main.yml@refs/heads/main", "push", "failure", [job("main", 0, 5, 30, "failure")]),
run(2, "main.yml@refs/heads/main", "push", "failure", [job("main", 60, 65, 95, "failure")]),
run(3, "main.yml@refs/heads/main", "push", "success", [job("main", 120, 125, 210)]),
run(4, "pr.yml@refs/pull/7/head", "pull_request", "failure", [job("done", 0, 1, 2, "failure")], pull=7),
]
# when
episodes = m.red_episodes(runs, now=T0 + timedelta(hours=10))
# then
self.assertEqual([(e.first_failed_run, e.fixed_by_run, e.hours) for e in episodes], [(1, 3, 3.5)])
def test_main_still_red_at_the_end_of_the_window_counts_up_to_now(self):
runs = [run(1, "main.yml@refs/heads/main", "push", "failure", [job("main", 0, 5, 30, "failure")])]
episodes = m.red_episodes(runs, now=T0 + timedelta(hours=2))
self.assertEqual([(e.fixed_by_run, e.hours) for e in episodes], [(None, 2.0)])
class UtilisationTest(unittest.TestCase):
def test_job_minutes_count_on_the_local_day_the_job_started(self):
# given: 14:30 UTC is 23:30 KST on the 13th, 15:30 UTC is 00:30 KST on the 14th
r = run(1, "main.yml@refs/heads/main", "push", "success",
[job("a", 0, 14 * 60 + 30, 14 * 60 + 50), job("b", 0, 15 * 60 + 30, 15 * 60 + 40)])
# when
days = m.utilisation([r], timedelta(hours=9))
# then
self.assertEqual({str(day): value for day, value in days.items()}, {"2026-11-13": 20, "2026-11-14": 10})
if __name__ == "__main__":
unittest.main()