feat(ci): add a read-only CI metrics script for the desktop runner
There is no number for how long CI runs wait, how busy the one runner is, how many runs a pull request costs or how long main stays red. Run times were added up by hand from single runs, so whether a change helped could not be told apart from load. infrastructure/ci/metrics/ci_metrics.py reads the Gitea Actions API (runs, their jobs, closed pull requests) for the repositories the runner serves and prints the CI manual's metrics 1-5 per repository and stage: run execution p50/p90, queue wait p50/p90, runner utilisation per KST day, pull request runs per merged pull request, and main red time. It only reads; credentials come from GITEA_TOKEN or `git credential fill` and are never printed. The arithmetic follows what our Gitea 1.27 returns, which differs from the API reference: a run has no created_at (the earliest job created_at is used), unstarted runs and jobs report 1970-01-01 times (treated as absent), and a job's created_at is its run's, so queue wait is measured per run (first job start minus run creation) instead of per job. Verification: python3 -m unittest test_ci_metrics (11 offline tests) and one run against the server for 2026-11-07..13 (two repositories, 30 s, rc=0). The sensitive-source scan needs rg and ran on the desktop host. Refs: design-docs ci/03-time-reduction-plan.md M-0
This commit is contained in:
@@ -0,0 +1,46 @@
|
||||
# CI 지표
|
||||
|
||||
desktop runner 하나가 받는 repo들의 CI 지표를 Gitea Actions API에서 뽑는다. 읽기만 하고 서버에 아무것도 쓰지 않는다. 지표의 정의와 목표는 design-docs `ci/manual.md` 10절이고, 주 1회 결과를 design-docs `ci/metrics.md`에 한 줄 남긴다.
|
||||
|
||||
## 돌리기
|
||||
|
||||
```bash
|
||||
python3 infrastructure/ci/metrics/ci_metrics.py \
|
||||
--repo clean-architecture-backend-template \
|
||||
--repo clean-architecture-frontend-template \
|
||||
--since 2026-11-07 --until 2026-11-13
|
||||
```
|
||||
|
||||
- 날짜는 KST 하루 단위이고 `--until`도 그날을 포함한다(`--utc-offset-hours`로 바꾼다).
|
||||
- 인증: `GITEA_TOKEN`(읽기 전용 토큰)이 있으면 그것을, 없으면 `git credential fill`의 그 호스트 자격 증명을 쓴다. 어느 쪽도 출력하지 않는다.
|
||||
- `--json`은 표 대신 숫자를 JSON으로 낸다.
|
||||
- 두 repo, 지난 7일이면 30초 안팎이다(run마다 jobs API를 한 번 부른다).
|
||||
|
||||
## 내는 것
|
||||
|
||||
| # | 지표 | 계산 |
|
||||
| --- | --- | --- |
|
||||
| 1 | 실행 시간 p50·p90 | 단계(`workflow:event`)별, 끝난 run(success·failure)의 job 실행 시간 합. cancelled run은 run 수에만 센다 |
|
||||
| 2 | 대기 시간 p50·p90 | run의 첫 job 시작 − run 생성 |
|
||||
| 3 | runner 사용률 | 날짜(job 시작 시각, KST)별 job 실행 시간 합 ÷ 1,440분. 준 repo를 모두 더한다(runner가 하나라서) |
|
||||
| 4 | PR당 run 수 | 기간 안에 병합된 PR마다 그 PR의 `pull_request` run 수. 병합 전 7일의 run까지 센다 |
|
||||
| 5 | main 빨강 시간 | workflow별 main push run이 처음 실패한 run의 생성부터 다음 성공 run의 끝까지. 기간 끝까지 빨강이면 그때까지 |
|
||||
|
||||
지표 6(흔들림)은 각 repo `docs/ledger/flaky-tests.jsonl`, 지표 7(캐시 적중)은 PR run 로그에서 나온다. 이 스크립트는 보지 않는다.
|
||||
|
||||
## 우리 서버(Gitea 1.27)가 API 명세와 다른 곳
|
||||
|
||||
2026-11-13 응답 한 건씩으로 확인했다. 계산은 서버를 따른다.
|
||||
|
||||
- run에 `created_at`이 없다. run의 job은 run과 함께 만들어지므로, job `created_at`의 가장 이른 값을 run 생성 시각으로 쓴다.
|
||||
- 시작하지 않은 run·job의 `started_at`·`completed_at`은 `1970-01-01T00:00:00Z`다. 없는 값으로 본다.
|
||||
- job의 `created_at`은 그 job이 실행할 수 있게 된 시각이 아니라 run의 생성 시각이다. runner 하나가 run의 job을 차례로 돌리므로 job마다 `started_at − created_at`을 대기로 세면 앞 job의 실행이 대기에 들어간다. 그래서 대기는 run 단위로 센다. `needs`가 있는 job이 늦게 풀려 대기열 뒤로 가는 시간은 이 지표에 안 보이고 실행 시간에도 안 들어간다.
|
||||
- run 목록의 run에는 `head_branch`가 없다. 단계와 ref는 `path`(`<workflow>@<ref>`)에서 읽고, PR 번호는 `pull_requests`에서 읽는다.
|
||||
|
||||
## 시험
|
||||
|
||||
```bash
|
||||
cd infrastructure/ci/metrics && python3 -m unittest -v test_ci_metrics
|
||||
```
|
||||
|
||||
서버 없이 계산만 본다. 만든 run·job 기록으로 대기·실행·빨강 시간·사용률을 확인한다.
|
||||
@@ -0,0 +1,384 @@
|
||||
#!/usr/bin/env python3
|
||||
"""CI metrics for the repositories one Gitea Actions runner serves.
|
||||
|
||||
Reads the Gitea Actions API (runs, their jobs, closed pull requests) and prints, per repository and
|
||||
stage, the CI manual's metrics 1-5: run execution time, queue wait, runner utilisation, pull
|
||||
request runs per merged pull request and time main stayed red. Read-only; it never writes to the
|
||||
server.
|
||||
|
||||
What our Gitea 1.27 returns differs from the gitea.com API reference in three places, and the
|
||||
arithmetic below follows the server:
|
||||
|
||||
* A run carries no ``created_at``. A run's jobs are all created with the run, so the earliest job
|
||||
``created_at`` is the run's creation time.
|
||||
* A run or job that has not started reports ``started_at`` and ``completed_at`` as
|
||||
``1970-01-01T00:00:00Z``. Those are treated as absent.
|
||||
* A job's ``created_at`` is the run's, not the moment it became runnable. One runner runs a run's
|
||||
jobs one after another, so ``started_at - created_at`` of a later job counts the earlier jobs'
|
||||
execution as waiting. Queue wait is therefore measured per run: first job start minus run
|
||||
creation.
|
||||
|
||||
Credentials: ``GITEA_TOKEN`` (a read-only token) if set, otherwise ``git credential fill`` for the
|
||||
server host. Neither is printed.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import base64
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
import urllib.parse
|
||||
import urllib.request
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import date, datetime, timedelta, timezone
|
||||
|
||||
EPOCH_PREFIXES = ("1970-01-01", "0001-01-01")
|
||||
FINISHED = {"success", "failure"}
|
||||
|
||||
|
||||
def parse_time(value: str | None) -> datetime | None:
|
||||
"""An API timestamp, or None for a missing or zero value."""
|
||||
if not value or value.startswith(EPOCH_PREFIXES):
|
||||
return None
|
||||
return datetime.fromisoformat(value.replace("Z", "+00:00"))
|
||||
|
||||
|
||||
def percentile(values: list[float], fraction: float) -> float | None:
|
||||
"""Nearest-rank percentile; None for no values."""
|
||||
if not values:
|
||||
return None
|
||||
ordered = sorted(values)
|
||||
rank = max(1, math.ceil(fraction * len(ordered)))
|
||||
return ordered[rank - 1]
|
||||
|
||||
|
||||
@dataclass
|
||||
class Job:
|
||||
name: str
|
||||
created: datetime | None
|
||||
started: datetime | None
|
||||
completed: datetime | None
|
||||
conclusion: str | None
|
||||
|
||||
@property
|
||||
def minutes(self) -> float:
|
||||
if self.started is None or self.completed is None:
|
||||
return 0.0
|
||||
return max(0.0, (self.completed - self.started).total_seconds() / 60)
|
||||
|
||||
|
||||
@dataclass
|
||||
class Run:
|
||||
repo: str
|
||||
id: int
|
||||
workflow: str
|
||||
ref: str
|
||||
event: str
|
||||
status: str
|
||||
conclusion: str | None
|
||||
pull_request: int | None
|
||||
jobs: list[Job] = field(default_factory=list)
|
||||
|
||||
@property
|
||||
def stage(self) -> str:
|
||||
return f"{self.workflow}:{self.event}"
|
||||
|
||||
@property
|
||||
def created(self) -> datetime | None:
|
||||
times = [job.created for job in self.jobs if job.created is not None]
|
||||
return min(times) if times else None
|
||||
|
||||
@property
|
||||
def first_start(self) -> datetime | None:
|
||||
times = [job.started for job in self.jobs if job.started is not None]
|
||||
return min(times) if times else None
|
||||
|
||||
@property
|
||||
def completed(self) -> datetime | None:
|
||||
times = [job.completed for job in self.jobs if job.completed is not None]
|
||||
return max(times) if times and self.status == "completed" else None
|
||||
|
||||
@property
|
||||
def execution_minutes(self) -> float:
|
||||
return sum(job.minutes for job in self.jobs)
|
||||
|
||||
@property
|
||||
def queue_minutes(self) -> float | None:
|
||||
if self.created is None or self.first_start is None:
|
||||
return None
|
||||
return max(0.0, (self.first_start - self.created).total_seconds() / 60)
|
||||
|
||||
@property
|
||||
def main_push(self) -> bool:
|
||||
return self.event == "push" and self.ref == "refs/heads/main"
|
||||
|
||||
|
||||
def run_from_api(repo: str, raw: dict, jobs: list[dict]) -> Run:
|
||||
workflow, _, ref = raw.get("path", "").partition("@")
|
||||
pulls = raw.get("pull_requests") or []
|
||||
return Run(
|
||||
repo=repo,
|
||||
id=raw["id"],
|
||||
workflow=workflow,
|
||||
ref=ref,
|
||||
event=raw.get("event", ""),
|
||||
status=raw.get("status", ""),
|
||||
conclusion=raw.get("conclusion"),
|
||||
pull_request=pulls[0]["number"] if pulls else None,
|
||||
jobs=[
|
||||
Job(
|
||||
name=job.get("name", ""),
|
||||
created=parse_time(job.get("created_at")),
|
||||
started=parse_time(job.get("started_at")),
|
||||
completed=parse_time(job.get("completed_at")),
|
||||
conclusion=job.get("conclusion"),
|
||||
)
|
||||
for job in jobs
|
||||
],
|
||||
)
|
||||
|
||||
|
||||
class Gitea:
|
||||
"""The few read-only API calls this script makes."""
|
||||
|
||||
def __init__(self, server: str):
|
||||
self.server = server.rstrip("/")
|
||||
self.header = self._authorization()
|
||||
|
||||
def _authorization(self) -> str:
|
||||
token = os.environ.get("GITEA_TOKEN")
|
||||
if token:
|
||||
return f"token {token}"
|
||||
host = urllib.parse.urlparse(self.server).netloc
|
||||
answer = subprocess.run(
|
||||
["git", "credential", "fill"],
|
||||
input=f"protocol=https\nhost={host}\n\n",
|
||||
capture_output=True,
|
||||
text=True,
|
||||
check=True,
|
||||
).stdout
|
||||
fields = dict(line.split("=", 1) for line in answer.splitlines() if "=" in line)
|
||||
pair = f"{fields['username']}:{fields['password']}".encode()
|
||||
return "Basic " + base64.b64encode(pair).decode()
|
||||
|
||||
def get(self, path: str) -> object:
|
||||
request = urllib.request.Request(
|
||||
f"{self.server}/api/v1{path}",
|
||||
headers={"Authorization": self.header, "Accept": "application/json"},
|
||||
)
|
||||
with urllib.request.urlopen(request, timeout=30) as response:
|
||||
return json.load(response)
|
||||
|
||||
|
||||
def fetch_runs(api: Gitea, owner: str, repo: str, since: datetime) -> list[Run]:
|
||||
"""Runs created at or after ``since``, newest first. Run ids grow with creation time."""
|
||||
runs: list[Run] = []
|
||||
page = 1
|
||||
while True:
|
||||
body = api.get(f"/repos/{owner}/{repo}/actions/runs?limit=50&page={page}")
|
||||
batch = body.get("workflow_runs") or []
|
||||
if not batch:
|
||||
return runs
|
||||
for raw in batch:
|
||||
jobs = api.get(f"/repos/{owner}/{repo}/actions/runs/{raw['id']}/jobs").get("jobs") or []
|
||||
run = run_from_api(repo, raw, jobs)
|
||||
if run.created is not None and run.created < since:
|
||||
return runs
|
||||
runs.append(run)
|
||||
page += 1
|
||||
|
||||
|
||||
def fetch_merged_pulls(api: Gitea, owner: str, repo: str, since: datetime, until: datetime) -> list[int]:
|
||||
numbers: list[int] = []
|
||||
page = 1
|
||||
while True:
|
||||
batch = api.get(f"/repos/{owner}/{repo}/pulls?state=closed&sort=recentupdate&limit=50&page={page}")
|
||||
if not batch:
|
||||
return numbers
|
||||
for pull in batch:
|
||||
merged = parse_time(pull.get("merged_at"))
|
||||
if pull.get("merged") and merged is not None and since <= merged < until:
|
||||
numbers.append(pull["number"])
|
||||
updated = [parse_time(pull.get("updated_at")) for pull in batch]
|
||||
if all(time is not None and time < since for time in updated):
|
||||
return numbers
|
||||
page += 1
|
||||
|
||||
|
||||
@dataclass
|
||||
class RedEpisode:
|
||||
workflow: str
|
||||
first_failed_run: int
|
||||
fixed_by_run: int | None
|
||||
hours: float
|
||||
|
||||
|
||||
def red_episodes(runs: list[Run], now: datetime) -> list[RedEpisode]:
|
||||
"""Main stays red from the first failing main push run's creation to the next green one's end."""
|
||||
episodes: list[RedEpisode] = []
|
||||
by_workflow: dict[str, list[Run]] = {}
|
||||
for run in runs:
|
||||
if run.main_push and run.status == "completed" and run.conclusion in FINISHED:
|
||||
by_workflow.setdefault(run.workflow, []).append(run)
|
||||
for workflow, history in by_workflow.items():
|
||||
history.sort(key=lambda run: run.created or now)
|
||||
start: Run | None = None
|
||||
for run in history:
|
||||
if run.conclusion == "failure" and start is None:
|
||||
start = run
|
||||
elif run.conclusion == "success" and start is not None:
|
||||
end = run.completed or now
|
||||
episodes.append(
|
||||
RedEpisode(workflow, start.id, run.id, (end - start.created).total_seconds() / 3600)
|
||||
)
|
||||
start = None
|
||||
if start is not None:
|
||||
episodes.append(RedEpisode(workflow, start.id, None, (now - start.created).total_seconds() / 3600))
|
||||
return episodes
|
||||
|
||||
|
||||
def stage_rows(runs: list[Run]) -> list[dict]:
|
||||
rows = []
|
||||
for stage in sorted({run.stage for run in runs}):
|
||||
finished = [run for run in runs if run.stage == stage and run.conclusion in FINISHED]
|
||||
waited = [run.queue_minutes for run in runs if run.stage == stage and run.queue_minutes is not None]
|
||||
executed = [run.execution_minutes for run in finished]
|
||||
rows.append(
|
||||
{
|
||||
"stage": stage,
|
||||
"runs": len([run for run in runs if run.stage == stage]),
|
||||
"finished": len(finished),
|
||||
"red": len([run for run in finished if run.conclusion == "failure"]),
|
||||
"exec_p50": percentile(executed, 0.5),
|
||||
"exec_p90": percentile(executed, 0.9),
|
||||
"wait_p50": percentile(waited, 0.5),
|
||||
"wait_p90": percentile(waited, 0.9),
|
||||
}
|
||||
)
|
||||
return rows
|
||||
|
||||
|
||||
def utilisation(runs: list[Run], offset: timedelta) -> dict[date, float]:
|
||||
"""Job execution minutes per local day (job start day) across every repository given."""
|
||||
minutes: dict[date, float] = {}
|
||||
for run in runs:
|
||||
for job in run.jobs:
|
||||
if job.started is not None and job.minutes > 0:
|
||||
day = (job.started + offset).date()
|
||||
minutes[day] = minutes.get(day, 0.0) + job.minutes
|
||||
return minutes
|
||||
|
||||
|
||||
def runs_per_merged_pull(runs: list[Run], merged: list[int]) -> list[int]:
|
||||
return [len([run for run in runs if run.pull_request == number]) for number in merged]
|
||||
|
||||
|
||||
def fmt(value: float | None, digits: int = 1) -> str:
|
||||
return "-" if value is None else f"{value:.{digits}f}"
|
||||
|
||||
|
||||
def report(window: str, per_repo: dict[str, dict], days: dict[date, float]) -> str:
|
||||
lines = [f"# CI 지표 ({window})", ""]
|
||||
for repo, data in per_repo.items():
|
||||
lines += [
|
||||
f"## {repo}",
|
||||
"",
|
||||
"| 단계 | run | 끝난 run | 빨강 | 실행 p50(분) | 실행 p90(분) | 대기 p50(분) | 대기 p90(분) |",
|
||||
"|---|---|---|---|---|---|---|---|",
|
||||
]
|
||||
for row in data["stages"]:
|
||||
lines.append(
|
||||
f"| {row['stage']} | {row['runs']} | {row['finished']} | {row['red']} | {fmt(row['exec_p50'])} | "
|
||||
f"{fmt(row['exec_p90'])} | {fmt(row['wait_p50'])} | {fmt(row['wait_p90'])} |"
|
||||
)
|
||||
counts = data["runs_per_pull"]
|
||||
lines += [
|
||||
"",
|
||||
f"- 병합된 PR {len(counts)}개, PR당 run 수 p50 {fmt(percentile(counts, 0.5), 0)}, "
|
||||
f"p90 {fmt(percentile(counts, 0.9), 0)}, 최대 {max(counts) if counts else '-'}",
|
||||
]
|
||||
episodes = data["red"]
|
||||
if episodes:
|
||||
longest = max(episodes, key=lambda episode: episode.hours)
|
||||
lines.append(
|
||||
f"- main 빨강 {len(episodes)}회, 합 {sum(e.hours for e in episodes):.1f}시간, p90 "
|
||||
f"{fmt(percentile([e.hours for e in episodes], 0.9))}시간, 가장 긴 것 {longest.hours:.1f}시간"
|
||||
f"({longest.workflow} run {longest.first_failed_run} → "
|
||||
f"{longest.fixed_by_run if longest.fixed_by_run else '아직 빨강'})"
|
||||
)
|
||||
else:
|
||||
lines.append("- main 빨강 0회")
|
||||
lines.append("")
|
||||
lines += ["## runner 사용률 (위 repo 합, job 시작 날짜 기준)", "", "| 날짜 | 실행 합(분) | 사용률 |", "|---|---|---|"]
|
||||
for day in sorted(days):
|
||||
lines.append(f"| {day} | {days[day]:.0f} | {days[day] / 1440:.0%} |")
|
||||
if days:
|
||||
busiest = max(days, key=days.get)
|
||||
lines += ["", f"- 바쁜 날 {busiest}: {days[busiest] / 1440:.0%}"]
|
||||
return "\n".join(lines) + "\n"
|
||||
|
||||
|
||||
def main(argv: list[str]) -> int:
|
||||
parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
|
||||
parser.add_argument("--server", default="https://git.learn.hyeonworks.com")
|
||||
parser.add_argument("--owner", default="donghyeon.kang")
|
||||
parser.add_argument("--repo", action="append", required=True, help="repeat for each repository")
|
||||
parser.add_argument("--since", required=True, type=date.fromisoformat, help="first local day, YYYY-MM-DD")
|
||||
parser.add_argument("--until", required=True, type=date.fromisoformat, help="last local day, inclusive")
|
||||
parser.add_argument("--utc-offset-hours", type=int, default=9, help="local day boundary (default KST)")
|
||||
parser.add_argument("--json", action="store_true", help="print the numbers as JSON instead of tables")
|
||||
args = parser.parse_args(argv)
|
||||
|
||||
offset = timedelta(hours=args.utc_offset_hours)
|
||||
since = datetime.combine(args.since, datetime.min.time(), timezone.utc) - offset
|
||||
until = datetime.combine(args.until + timedelta(days=1), datetime.min.time(), timezone.utc) - offset
|
||||
# A pull request merged in the window may have had runs in the week before it.
|
||||
lookback = since - timedelta(days=7)
|
||||
now = min(datetime.now(timezone.utc), until)
|
||||
|
||||
api = Gitea(args.server)
|
||||
per_repo: dict[str, dict] = {}
|
||||
window_runs: list[Run] = []
|
||||
for repo in args.repo:
|
||||
fetched = fetch_runs(api, args.owner, repo, lookback)
|
||||
in_window = [run for run in fetched if run.created is not None and since <= run.created < until]
|
||||
window_runs += in_window
|
||||
merged = fetch_merged_pulls(api, args.owner, repo, since, until)
|
||||
per_repo[repo] = {
|
||||
"stages": stage_rows(in_window),
|
||||
"runs_per_pull": runs_per_merged_pull(fetched, merged),
|
||||
"red": red_episodes(in_window, now),
|
||||
}
|
||||
days = utilisation(window_runs, offset)
|
||||
|
||||
window = f"{args.since} ~ {args.until}, UTC{args.utc_offset_hours:+d}"
|
||||
if args.json:
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"window": window,
|
||||
"repos": {
|
||||
repo: {
|
||||
"stages": data["stages"],
|
||||
"runs_per_pull": data["runs_per_pull"],
|
||||
"red": [episode.__dict__ for episode in data["red"]],
|
||||
}
|
||||
for repo, data in per_repo.items()
|
||||
},
|
||||
"utilisation_minutes": {str(day): round(value, 1) for day, value in sorted(days.items())},
|
||||
},
|
||||
ensure_ascii=False,
|
||||
indent=2,
|
||||
)
|
||||
)
|
||||
else:
|
||||
print(report(window, per_repo, days), end="")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main(sys.argv[1:]))
|
||||
@@ -0,0 +1,124 @@
|
||||
"""Offline tests for ci_metrics: the arithmetic on run and job records, no server."""
|
||||
|
||||
import unittest
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
import ci_metrics as m
|
||||
|
||||
T0 = datetime(2026, 11, 13, 0, 0, tzinfo=timezone.utc)
|
||||
|
||||
|
||||
def at(minutes: float) -> str:
|
||||
return (T0 + timedelta(minutes=minutes)).isoformat().replace("+00:00", "Z")
|
||||
|
||||
|
||||
def job(name, created, started=None, completed=None, conclusion="success"):
|
||||
return {
|
||||
"name": name,
|
||||
"created_at": at(created),
|
||||
"started_at": at(started) if started is not None else "1970-01-01T00:00:00Z",
|
||||
"completed_at": at(completed) if completed is not None else "1970-01-01T00:00:00Z",
|
||||
"conclusion": conclusion,
|
||||
}
|
||||
|
||||
|
||||
def run(run_id, path, event, conclusion, jobs, status="completed", pull=None):
|
||||
raw = {"id": run_id, "path": path, "event": event, "status": status, "conclusion": conclusion}
|
||||
if pull is not None:
|
||||
raw["pull_requests"] = [{"number": pull}]
|
||||
return m.run_from_api("backend", raw, jobs)
|
||||
|
||||
|
||||
class ParsingTest(unittest.TestCase):
|
||||
def test_a_zero_timestamp_from_a_job_that_has_not_started_is_absent(self):
|
||||
self.assertIsNone(m.parse_time("1970-01-01T00:00:00Z"))
|
||||
|
||||
def test_the_run_is_created_when_its_earliest_job_is(self):
|
||||
r = run(1, "pr.yml@refs/pull/7/head", "pull_request", "success", [job("done", 5, 20, 32)], pull=7)
|
||||
self.assertEqual((r.created, r.stage, r.pull_request), (T0 + timedelta(minutes=5), "pr.yml:pull_request", 7))
|
||||
|
||||
|
||||
class TimingTest(unittest.TestCase):
|
||||
def test_queue_wait_is_the_first_job_start_minus_run_creation_not_each_jobs_own_gap(self):
|
||||
# given: one runner runs the jobs one after another, so the second job's own gap is 40 minutes
|
||||
r = run(1, "main.yml@refs/heads/main", "push", "success", [job("a", 0, 10, 30), job("b", 0, 30, 40)])
|
||||
|
||||
# when
|
||||
waited = r.queue_minutes
|
||||
|
||||
# then
|
||||
self.assertEqual(waited, 10)
|
||||
|
||||
def test_execution_is_the_sum_of_job_run_times_and_a_skipped_job_adds_nothing(self):
|
||||
r = run(1, "main.yml@refs/heads/main", "push", "success",
|
||||
[job("a", 0, 10, 30), job("b", 0, 30, 40), job("c", 0, 40, 40, "skipped")])
|
||||
self.assertEqual(r.execution_minutes, 30)
|
||||
|
||||
def test_a_queued_run_has_no_wait_and_no_execution_yet(self):
|
||||
r = run(1, "pr.yml@refs/pull/7/head", "pull_request", None, [job("done", 0)], status="queued", pull=7)
|
||||
self.assertEqual((r.queue_minutes, r.execution_minutes), (None, 0.0))
|
||||
|
||||
def test_percentiles_use_the_nearest_rank(self):
|
||||
self.assertEqual((m.percentile([3, 1, 2, 10], 0.5), m.percentile([3, 1, 2, 10], 0.9)), (2, 10))
|
||||
|
||||
|
||||
class StageTest(unittest.TestCase):
|
||||
def test_a_cancelled_run_counts_as_a_run_but_not_in_execution_percentiles(self):
|
||||
# given
|
||||
runs = [
|
||||
run(1, "pr.yml@refs/pull/7/head", "pull_request", "success", [job("done", 0, 1, 13)], pull=7),
|
||||
run(2, "pr.yml@refs/pull/7/head", "pull_request", "cancelled", [job("done", 0, 1, 2)], pull=7),
|
||||
]
|
||||
|
||||
# when
|
||||
row = m.stage_rows(runs)[0]
|
||||
|
||||
# then
|
||||
self.assertEqual((row["runs"], row["finished"], row["exec_p50"], row["wait_p50"]), (2, 1, 12, 1))
|
||||
|
||||
def test_runs_per_merged_pull_request_count_every_attempt_of_that_pull_request(self):
|
||||
runs = [
|
||||
run(1, "pr.yml@refs/pull/7/head", "pull_request", "cancelled", [job("done", 0)], pull=7),
|
||||
run(2, "pr.yml@refs/pull/7/head", "pull_request", "success", [job("done", 1, 2, 14)], pull=7),
|
||||
run(3, "pr.yml@refs/pull/8/head", "pull_request", "success", [job("done", 1, 2, 14)], pull=8),
|
||||
]
|
||||
self.assertEqual(m.runs_per_merged_pull(runs, [7, 8]), [2, 1])
|
||||
|
||||
|
||||
class RedTest(unittest.TestCase):
|
||||
def test_main_is_red_from_the_first_failing_runs_creation_to_the_next_green_runs_end(self):
|
||||
# given: two failures in a row, then a green run that ends at minute 210
|
||||
runs = [
|
||||
run(1, "main.yml@refs/heads/main", "push", "failure", [job("main", 0, 5, 30, "failure")]),
|
||||
run(2, "main.yml@refs/heads/main", "push", "failure", [job("main", 60, 65, 95, "failure")]),
|
||||
run(3, "main.yml@refs/heads/main", "push", "success", [job("main", 120, 125, 210)]),
|
||||
run(4, "pr.yml@refs/pull/7/head", "pull_request", "failure", [job("done", 0, 1, 2, "failure")], pull=7),
|
||||
]
|
||||
|
||||
# when
|
||||
episodes = m.red_episodes(runs, now=T0 + timedelta(hours=10))
|
||||
|
||||
# then
|
||||
self.assertEqual([(e.first_failed_run, e.fixed_by_run, e.hours) for e in episodes], [(1, 3, 3.5)])
|
||||
|
||||
def test_main_still_red_at_the_end_of_the_window_counts_up_to_now(self):
|
||||
runs = [run(1, "main.yml@refs/heads/main", "push", "failure", [job("main", 0, 5, 30, "failure")])]
|
||||
episodes = m.red_episodes(runs, now=T0 + timedelta(hours=2))
|
||||
self.assertEqual([(e.fixed_by_run, e.hours) for e in episodes], [(None, 2.0)])
|
||||
|
||||
|
||||
class UtilisationTest(unittest.TestCase):
|
||||
def test_job_minutes_count_on_the_local_day_the_job_started(self):
|
||||
# given: 14:30 UTC is 23:30 KST on the 13th, 15:30 UTC is 00:30 KST on the 14th
|
||||
r = run(1, "main.yml@refs/heads/main", "push", "success",
|
||||
[job("a", 0, 14 * 60 + 30, 14 * 60 + 50), job("b", 0, 15 * 60 + 30, 15 * 60 + 40)])
|
||||
|
||||
# when
|
||||
days = m.utilisation([r], timedelta(hours=9))
|
||||
|
||||
# then
|
||||
self.assertEqual({str(day): value for day, value in days.items()}, {"2026-11-13": 20, "2026-11-14": 10})
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
Reference in New Issue
Block a user