Compare commits
7
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
d0875bfe44 | ||
|
|
dd8e9140b7 | ||
|
|
5d27096ed0 | ||
|
|
ae4e256c95 | ||
|
|
27eb8c1799 | ||
|
|
149b885e35 | ||
|
|
205927f4e7 |
@@ -0,0 +1,46 @@
|
||||
# CI 지표
|
||||
|
||||
desktop runner 하나가 받는 repo들의 CI 지표를 Gitea Actions API에서 뽑는다. 읽기만 하고 서버에 아무것도 쓰지 않는다. 지표의 정의와 목표는 design-docs `ci/manual.md` 10절이고, 주 1회 결과를 design-docs `ci/metrics.md`에 한 줄 남긴다.
|
||||
|
||||
## 돌리기
|
||||
|
||||
```bash
|
||||
python3 infrastructure/ci/metrics/ci_metrics.py \
|
||||
--repo clean-architecture-backend-template \
|
||||
--repo clean-architecture-frontend-template \
|
||||
--since 2026-11-07 --until 2026-11-13
|
||||
```
|
||||
|
||||
- 날짜는 KST 하루 단위이고 `--until`도 그날을 포함한다(`--utc-offset-hours`로 바꾼다).
|
||||
- 인증: `GITEA_TOKEN`(읽기 전용 토큰)이 있으면 그것을, 없으면 `git credential fill`의 그 호스트 자격 증명을 쓴다. 어느 쪽도 출력하지 않는다.
|
||||
- `--json`은 표 대신 숫자를 JSON으로 낸다.
|
||||
- 두 repo, 지난 7일이면 30초 안팎이다(run마다 jobs API를 한 번 부른다).
|
||||
|
||||
## 내는 것
|
||||
|
||||
| # | 지표 | 계산 |
|
||||
| --- | --- | --- |
|
||||
| 1 | 실행 시간 p50·p90 | 단계(`workflow:event`)별, 끝난 run(success·failure)의 job 실행 시간 합. cancelled run은 run 수에만 센다 |
|
||||
| 2 | 대기 시간 p50·p90 | run의 첫 job 시작 − run 생성 |
|
||||
| 3 | runner 사용률 | 날짜(job 시작 시각, KST)별 job 실행 시간 합 ÷ 1,440분. 준 repo를 모두 더한다(runner가 하나라서) |
|
||||
| 4 | PR당 run 수 | 기간 안에 병합된 PR마다 그 PR의 `pull_request` run 수. 병합 전 7일의 run까지 센다 |
|
||||
| 5 | main 빨강 시간 | workflow별 main push run이 처음 실패한 run의 생성부터 다음 성공 run의 끝까지. 기간 끝까지 빨강이면 그때까지 |
|
||||
|
||||
지표 6(흔들림)은 각 repo `docs/ledger/flaky-tests.jsonl`, 지표 7(캐시 적중)은 PR run 로그에서 나온다. 이 스크립트는 보지 않는다.
|
||||
|
||||
## 우리 서버(Gitea 1.27)가 API 명세와 다른 곳
|
||||
|
||||
2026-11-13 응답 한 건씩으로 확인했다. 계산은 서버를 따른다.
|
||||
|
||||
- run에 `created_at`이 없다. run의 job은 run과 함께 만들어지므로, job `created_at`의 가장 이른 값을 run 생성 시각으로 쓴다.
|
||||
- 시작하지 않은 run·job의 `started_at`·`completed_at`은 `1970-01-01T00:00:00Z`다. 없는 값으로 본다.
|
||||
- job의 `created_at`은 그 job이 실행할 수 있게 된 시각이 아니라 run의 생성 시각이다. runner 하나가 run의 job을 차례로 돌리므로 job마다 `started_at − created_at`을 대기로 세면 앞 job의 실행이 대기에 들어간다. 그래서 대기는 run 단위로 센다. `needs`가 있는 job이 늦게 풀려 대기열 뒤로 가는 시간은 이 지표에 안 보이고 실행 시간에도 안 들어간다.
|
||||
- run 목록의 run에는 `head_branch`가 없다. 단계와 ref는 `path`(`<workflow>@<ref>`)에서 읽고, PR 번호는 `pull_requests`에서 읽는다.
|
||||
|
||||
## 시험
|
||||
|
||||
```bash
|
||||
cd infrastructure/ci/metrics && python3 -m unittest -v test_ci_metrics
|
||||
```
|
||||
|
||||
서버 없이 계산만 본다. 만든 run·job 기록으로 대기·실행·빨강 시간·사용률을 확인한다.
|
||||
@@ -0,0 +1,384 @@
|
||||
#!/usr/bin/env python3
|
||||
"""CI metrics for the repositories one Gitea Actions runner serves.
|
||||
|
||||
Reads the Gitea Actions API (runs, their jobs, closed pull requests) and prints, per repository and
|
||||
stage, the CI manual's metrics 1-5: run execution time, queue wait, runner utilisation, pull
|
||||
request runs per merged pull request and time main stayed red. Read-only; it never writes to the
|
||||
server.
|
||||
|
||||
What our Gitea 1.27 returns differs from the gitea.com API reference in three places, and the
|
||||
arithmetic below follows the server:
|
||||
|
||||
* A run carries no ``created_at``. A run's jobs are all created with the run, so the earliest job
|
||||
``created_at`` is the run's creation time.
|
||||
* A run or job that has not started reports ``started_at`` and ``completed_at`` as
|
||||
``1970-01-01T00:00:00Z``. Those are treated as absent.
|
||||
* A job's ``created_at`` is the run's, not the moment it became runnable. One runner runs a run's
|
||||
jobs one after another, so ``started_at - created_at`` of a later job counts the earlier jobs'
|
||||
execution as waiting. Queue wait is therefore measured per run: first job start minus run
|
||||
creation.
|
||||
|
||||
Credentials: ``GITEA_TOKEN`` (a read-only token) if set, otherwise ``git credential fill`` for the
|
||||
server host. Neither is printed.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import base64
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
import urllib.parse
|
||||
import urllib.request
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import date, datetime, timedelta, timezone
|
||||
|
||||
EPOCH_PREFIXES = ("1970-01-01", "0001-01-01")
|
||||
FINISHED = {"success", "failure"}
|
||||
|
||||
|
||||
def parse_time(value: str | None) -> datetime | None:
|
||||
"""An API timestamp, or None for a missing or zero value."""
|
||||
if not value or value.startswith(EPOCH_PREFIXES):
|
||||
return None
|
||||
return datetime.fromisoformat(value.replace("Z", "+00:00"))
|
||||
|
||||
|
||||
def percentile(values: list[float], fraction: float) -> float | None:
|
||||
"""Nearest-rank percentile; None for no values."""
|
||||
if not values:
|
||||
return None
|
||||
ordered = sorted(values)
|
||||
rank = max(1, math.ceil(fraction * len(ordered)))
|
||||
return ordered[rank - 1]
|
||||
|
||||
|
||||
@dataclass
|
||||
class Job:
|
||||
name: str
|
||||
created: datetime | None
|
||||
started: datetime | None
|
||||
completed: datetime | None
|
||||
conclusion: str | None
|
||||
|
||||
@property
|
||||
def minutes(self) -> float:
|
||||
if self.started is None or self.completed is None:
|
||||
return 0.0
|
||||
return max(0.0, (self.completed - self.started).total_seconds() / 60)
|
||||
|
||||
|
||||
@dataclass
|
||||
class Run:
|
||||
repo: str
|
||||
id: int
|
||||
workflow: str
|
||||
ref: str
|
||||
event: str
|
||||
status: str
|
||||
conclusion: str | None
|
||||
pull_request: int | None
|
||||
jobs: list[Job] = field(default_factory=list)
|
||||
|
||||
@property
|
||||
def stage(self) -> str:
|
||||
return f"{self.workflow}:{self.event}"
|
||||
|
||||
@property
|
||||
def created(self) -> datetime | None:
|
||||
times = [job.created for job in self.jobs if job.created is not None]
|
||||
return min(times) if times else None
|
||||
|
||||
@property
|
||||
def first_start(self) -> datetime | None:
|
||||
times = [job.started for job in self.jobs if job.started is not None]
|
||||
return min(times) if times else None
|
||||
|
||||
@property
|
||||
def completed(self) -> datetime | None:
|
||||
times = [job.completed for job in self.jobs if job.completed is not None]
|
||||
return max(times) if times and self.status == "completed" else None
|
||||
|
||||
@property
|
||||
def execution_minutes(self) -> float:
|
||||
return sum(job.minutes for job in self.jobs)
|
||||
|
||||
@property
|
||||
def queue_minutes(self) -> float | None:
|
||||
if self.created is None or self.first_start is None:
|
||||
return None
|
||||
return max(0.0, (self.first_start - self.created).total_seconds() / 60)
|
||||
|
||||
@property
|
||||
def main_push(self) -> bool:
|
||||
return self.event == "push" and self.ref == "refs/heads/main"
|
||||
|
||||
|
||||
def run_from_api(repo: str, raw: dict, jobs: list[dict]) -> Run:
|
||||
workflow, _, ref = raw.get("path", "").partition("@")
|
||||
pulls = raw.get("pull_requests") or []
|
||||
return Run(
|
||||
repo=repo,
|
||||
id=raw["id"],
|
||||
workflow=workflow,
|
||||
ref=ref,
|
||||
event=raw.get("event", ""),
|
||||
status=raw.get("status", ""),
|
||||
conclusion=raw.get("conclusion"),
|
||||
pull_request=pulls[0]["number"] if pulls else None,
|
||||
jobs=[
|
||||
Job(
|
||||
name=job.get("name", ""),
|
||||
created=parse_time(job.get("created_at")),
|
||||
started=parse_time(job.get("started_at")),
|
||||
completed=parse_time(job.get("completed_at")),
|
||||
conclusion=job.get("conclusion"),
|
||||
)
|
||||
for job in jobs
|
||||
],
|
||||
)
|
||||
|
||||
|
||||
class Gitea:
|
||||
"""The few read-only API calls this script makes."""
|
||||
|
||||
def __init__(self, server: str):
|
||||
self.server = server.rstrip("/")
|
||||
self.header = self._authorization()
|
||||
|
||||
def _authorization(self) -> str:
|
||||
token = os.environ.get("GITEA_TOKEN")
|
||||
if token:
|
||||
return f"token {token}"
|
||||
host = urllib.parse.urlparse(self.server).netloc
|
||||
answer = subprocess.run(
|
||||
["git", "credential", "fill"],
|
||||
input=f"protocol=https\nhost={host}\n\n",
|
||||
capture_output=True,
|
||||
text=True,
|
||||
check=True,
|
||||
).stdout
|
||||
fields = dict(line.split("=", 1) for line in answer.splitlines() if "=" in line)
|
||||
pair = f"{fields['username']}:{fields['password']}".encode()
|
||||
return "Basic " + base64.b64encode(pair).decode()
|
||||
|
||||
def get(self, path: str) -> object:
|
||||
request = urllib.request.Request(
|
||||
f"{self.server}/api/v1{path}",
|
||||
headers={"Authorization": self.header, "Accept": "application/json"},
|
||||
)
|
||||
with urllib.request.urlopen(request, timeout=30) as response:
|
||||
return json.load(response)
|
||||
|
||||
|
||||
def fetch_runs(api: Gitea, owner: str, repo: str, since: datetime) -> list[Run]:
|
||||
"""Runs created at or after ``since``, newest first. Run ids grow with creation time."""
|
||||
runs: list[Run] = []
|
||||
page = 1
|
||||
while True:
|
||||
body = api.get(f"/repos/{owner}/{repo}/actions/runs?limit=50&page={page}")
|
||||
batch = body.get("workflow_runs") or []
|
||||
if not batch:
|
||||
return runs
|
||||
for raw in batch:
|
||||
jobs = api.get(f"/repos/{owner}/{repo}/actions/runs/{raw['id']}/jobs").get("jobs") or []
|
||||
run = run_from_api(repo, raw, jobs)
|
||||
if run.created is not None and run.created < since:
|
||||
return runs
|
||||
runs.append(run)
|
||||
page += 1
|
||||
|
||||
|
||||
def fetch_merged_pulls(api: Gitea, owner: str, repo: str, since: datetime, until: datetime) -> list[int]:
|
||||
numbers: list[int] = []
|
||||
page = 1
|
||||
while True:
|
||||
batch = api.get(f"/repos/{owner}/{repo}/pulls?state=closed&sort=recentupdate&limit=50&page={page}")
|
||||
if not batch:
|
||||
return numbers
|
||||
for pull in batch:
|
||||
merged = parse_time(pull.get("merged_at"))
|
||||
if pull.get("merged") and merged is not None and since <= merged < until:
|
||||
numbers.append(pull["number"])
|
||||
updated = [parse_time(pull.get("updated_at")) for pull in batch]
|
||||
if all(time is not None and time < since for time in updated):
|
||||
return numbers
|
||||
page += 1
|
||||
|
||||
|
||||
@dataclass
|
||||
class RedEpisode:
|
||||
workflow: str
|
||||
first_failed_run: int
|
||||
fixed_by_run: int | None
|
||||
hours: float
|
||||
|
||||
|
||||
def red_episodes(runs: list[Run], now: datetime) -> list[RedEpisode]:
|
||||
"""Main stays red from the first failing main push run's creation to the next green one's end."""
|
||||
episodes: list[RedEpisode] = []
|
||||
by_workflow: dict[str, list[Run]] = {}
|
||||
for run in runs:
|
||||
if run.main_push and run.status == "completed" and run.conclusion in FINISHED:
|
||||
by_workflow.setdefault(run.workflow, []).append(run)
|
||||
for workflow, history in by_workflow.items():
|
||||
history.sort(key=lambda run: run.created or now)
|
||||
start: Run | None = None
|
||||
for run in history:
|
||||
if run.conclusion == "failure" and start is None:
|
||||
start = run
|
||||
elif run.conclusion == "success" and start is not None:
|
||||
end = run.completed or now
|
||||
episodes.append(
|
||||
RedEpisode(workflow, start.id, run.id, (end - start.created).total_seconds() / 3600)
|
||||
)
|
||||
start = None
|
||||
if start is not None:
|
||||
episodes.append(RedEpisode(workflow, start.id, None, (now - start.created).total_seconds() / 3600))
|
||||
return episodes
|
||||
|
||||
|
||||
def stage_rows(runs: list[Run]) -> list[dict]:
|
||||
rows = []
|
||||
for stage in sorted({run.stage for run in runs}):
|
||||
finished = [run for run in runs if run.stage == stage and run.conclusion in FINISHED]
|
||||
waited = [run.queue_minutes for run in runs if run.stage == stage and run.queue_minutes is not None]
|
||||
executed = [run.execution_minutes for run in finished]
|
||||
rows.append(
|
||||
{
|
||||
"stage": stage,
|
||||
"runs": len([run for run in runs if run.stage == stage]),
|
||||
"finished": len(finished),
|
||||
"red": len([run for run in finished if run.conclusion == "failure"]),
|
||||
"exec_p50": percentile(executed, 0.5),
|
||||
"exec_p90": percentile(executed, 0.9),
|
||||
"wait_p50": percentile(waited, 0.5),
|
||||
"wait_p90": percentile(waited, 0.9),
|
||||
}
|
||||
)
|
||||
return rows
|
||||
|
||||
|
||||
def utilisation(runs: list[Run], offset: timedelta) -> dict[date, float]:
|
||||
"""Job execution minutes per local day (job start day) across every repository given."""
|
||||
minutes: dict[date, float] = {}
|
||||
for run in runs:
|
||||
for job in run.jobs:
|
||||
if job.started is not None and job.minutes > 0:
|
||||
day = (job.started + offset).date()
|
||||
minutes[day] = minutes.get(day, 0.0) + job.minutes
|
||||
return minutes
|
||||
|
||||
|
||||
def runs_per_merged_pull(runs: list[Run], merged: list[int]) -> list[int]:
|
||||
return [len([run for run in runs if run.pull_request == number]) for number in merged]
|
||||
|
||||
|
||||
def fmt(value: float | None, digits: int = 1) -> str:
|
||||
return "-" if value is None else f"{value:.{digits}f}"
|
||||
|
||||
|
||||
def report(window: str, per_repo: dict[str, dict], days: dict[date, float]) -> str:
|
||||
lines = [f"# CI 지표 ({window})", ""]
|
||||
for repo, data in per_repo.items():
|
||||
lines += [
|
||||
f"## {repo}",
|
||||
"",
|
||||
"| 단계 | run | 끝난 run | 빨강 | 실행 p50(분) | 실행 p90(분) | 대기 p50(분) | 대기 p90(분) |",
|
||||
"|---|---|---|---|---|---|---|---|",
|
||||
]
|
||||
for row in data["stages"]:
|
||||
lines.append(
|
||||
f"| {row['stage']} | {row['runs']} | {row['finished']} | {row['red']} | {fmt(row['exec_p50'])} | "
|
||||
f"{fmt(row['exec_p90'])} | {fmt(row['wait_p50'])} | {fmt(row['wait_p90'])} |"
|
||||
)
|
||||
counts = data["runs_per_pull"]
|
||||
lines += [
|
||||
"",
|
||||
f"- 병합된 PR {len(counts)}개, PR당 run 수 p50 {fmt(percentile(counts, 0.5), 0)}, "
|
||||
f"p90 {fmt(percentile(counts, 0.9), 0)}, 최대 {max(counts) if counts else '-'}",
|
||||
]
|
||||
episodes = data["red"]
|
||||
if episodes:
|
||||
longest = max(episodes, key=lambda episode: episode.hours)
|
||||
lines.append(
|
||||
f"- main 빨강 {len(episodes)}회, 합 {sum(e.hours for e in episodes):.1f}시간, p90 "
|
||||
f"{fmt(percentile([e.hours for e in episodes], 0.9))}시간, 가장 긴 것 {longest.hours:.1f}시간"
|
||||
f"({longest.workflow} run {longest.first_failed_run} → "
|
||||
f"{longest.fixed_by_run if longest.fixed_by_run else '아직 빨강'})"
|
||||
)
|
||||
else:
|
||||
lines.append("- main 빨강 0회")
|
||||
lines.append("")
|
||||
lines += ["## runner 사용률 (위 repo 합, job 시작 날짜 기준)", "", "| 날짜 | 실행 합(분) | 사용률 |", "|---|---|---|"]
|
||||
for day in sorted(days):
|
||||
lines.append(f"| {day} | {days[day]:.0f} | {days[day] / 1440:.0%} |")
|
||||
if days:
|
||||
busiest = max(days, key=days.get)
|
||||
lines += ["", f"- 바쁜 날 {busiest}: {days[busiest] / 1440:.0%}"]
|
||||
return "\n".join(lines) + "\n"
|
||||
|
||||
|
||||
def main(argv: list[str]) -> int:
|
||||
parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
|
||||
parser.add_argument("--server", default="https://git.learn.hyeonworks.com")
|
||||
parser.add_argument("--owner", default="donghyeon.kang")
|
||||
parser.add_argument("--repo", action="append", required=True, help="repeat for each repository")
|
||||
parser.add_argument("--since", required=True, type=date.fromisoformat, help="first local day, YYYY-MM-DD")
|
||||
parser.add_argument("--until", required=True, type=date.fromisoformat, help="last local day, inclusive")
|
||||
parser.add_argument("--utc-offset-hours", type=int, default=9, help="local day boundary (default KST)")
|
||||
parser.add_argument("--json", action="store_true", help="print the numbers as JSON instead of tables")
|
||||
args = parser.parse_args(argv)
|
||||
|
||||
offset = timedelta(hours=args.utc_offset_hours)
|
||||
since = datetime.combine(args.since, datetime.min.time(), timezone.utc) - offset
|
||||
until = datetime.combine(args.until + timedelta(days=1), datetime.min.time(), timezone.utc) - offset
|
||||
# A pull request merged in the window may have had runs in the week before it.
|
||||
lookback = since - timedelta(days=7)
|
||||
now = min(datetime.now(timezone.utc), until)
|
||||
|
||||
api = Gitea(args.server)
|
||||
per_repo: dict[str, dict] = {}
|
||||
window_runs: list[Run] = []
|
||||
for repo in args.repo:
|
||||
fetched = fetch_runs(api, args.owner, repo, lookback)
|
||||
in_window = [run for run in fetched if run.created is not None and since <= run.created < until]
|
||||
window_runs += in_window
|
||||
merged = fetch_merged_pulls(api, args.owner, repo, since, until)
|
||||
per_repo[repo] = {
|
||||
"stages": stage_rows(in_window),
|
||||
"runs_per_pull": runs_per_merged_pull(fetched, merged),
|
||||
"red": red_episodes(in_window, now),
|
||||
}
|
||||
days = utilisation(window_runs, offset)
|
||||
|
||||
window = f"{args.since} ~ {args.until}, UTC{args.utc_offset_hours:+d}"
|
||||
if args.json:
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"window": window,
|
||||
"repos": {
|
||||
repo: {
|
||||
"stages": data["stages"],
|
||||
"runs_per_pull": data["runs_per_pull"],
|
||||
"red": [episode.__dict__ for episode in data["red"]],
|
||||
}
|
||||
for repo, data in per_repo.items()
|
||||
},
|
||||
"utilisation_minutes": {str(day): round(value, 1) for day, value in sorted(days.items())},
|
||||
},
|
||||
ensure_ascii=False,
|
||||
indent=2,
|
||||
)
|
||||
)
|
||||
else:
|
||||
print(report(window, per_repo, days), end="")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main(sys.argv[1:]))
|
||||
@@ -0,0 +1,124 @@
|
||||
"""Offline tests for ci_metrics: the arithmetic on run and job records, no server."""
|
||||
|
||||
import unittest
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
import ci_metrics as m
|
||||
|
||||
T0 = datetime(2026, 11, 13, 0, 0, tzinfo=timezone.utc)
|
||||
|
||||
|
||||
def at(minutes: float) -> str:
|
||||
return (T0 + timedelta(minutes=minutes)).isoformat().replace("+00:00", "Z")
|
||||
|
||||
|
||||
def job(name, created, started=None, completed=None, conclusion="success"):
|
||||
return {
|
||||
"name": name,
|
||||
"created_at": at(created),
|
||||
"started_at": at(started) if started is not None else "1970-01-01T00:00:00Z",
|
||||
"completed_at": at(completed) if completed is not None else "1970-01-01T00:00:00Z",
|
||||
"conclusion": conclusion,
|
||||
}
|
||||
|
||||
|
||||
def run(run_id, path, event, conclusion, jobs, status="completed", pull=None):
|
||||
raw = {"id": run_id, "path": path, "event": event, "status": status, "conclusion": conclusion}
|
||||
if pull is not None:
|
||||
raw["pull_requests"] = [{"number": pull}]
|
||||
return m.run_from_api("backend", raw, jobs)
|
||||
|
||||
|
||||
class ParsingTest(unittest.TestCase):
|
||||
def test_a_zero_timestamp_from_a_job_that_has_not_started_is_absent(self):
|
||||
self.assertIsNone(m.parse_time("1970-01-01T00:00:00Z"))
|
||||
|
||||
def test_the_run_is_created_when_its_earliest_job_is(self):
|
||||
r = run(1, "pr.yml@refs/pull/7/head", "pull_request", "success", [job("done", 5, 20, 32)], pull=7)
|
||||
self.assertEqual((r.created, r.stage, r.pull_request), (T0 + timedelta(minutes=5), "pr.yml:pull_request", 7))
|
||||
|
||||
|
||||
class TimingTest(unittest.TestCase):
|
||||
def test_queue_wait_is_the_first_job_start_minus_run_creation_not_each_jobs_own_gap(self):
|
||||
# given: one runner runs the jobs one after another, so the second job's own gap is 40 minutes
|
||||
r = run(1, "main.yml@refs/heads/main", "push", "success", [job("a", 0, 10, 30), job("b", 0, 30, 40)])
|
||||
|
||||
# when
|
||||
waited = r.queue_minutes
|
||||
|
||||
# then
|
||||
self.assertEqual(waited, 10)
|
||||
|
||||
def test_execution_is_the_sum_of_job_run_times_and_a_skipped_job_adds_nothing(self):
|
||||
r = run(1, "main.yml@refs/heads/main", "push", "success",
|
||||
[job("a", 0, 10, 30), job("b", 0, 30, 40), job("c", 0, 40, 40, "skipped")])
|
||||
self.assertEqual(r.execution_minutes, 30)
|
||||
|
||||
def test_a_queued_run_has_no_wait_and_no_execution_yet(self):
|
||||
r = run(1, "pr.yml@refs/pull/7/head", "pull_request", None, [job("done", 0)], status="queued", pull=7)
|
||||
self.assertEqual((r.queue_minutes, r.execution_minutes), (None, 0.0))
|
||||
|
||||
def test_percentiles_use_the_nearest_rank(self):
|
||||
self.assertEqual((m.percentile([3, 1, 2, 10], 0.5), m.percentile([3, 1, 2, 10], 0.9)), (2, 10))
|
||||
|
||||
|
||||
class StageTest(unittest.TestCase):
|
||||
def test_a_cancelled_run_counts_as_a_run_but_not_in_execution_percentiles(self):
|
||||
# given
|
||||
runs = [
|
||||
run(1, "pr.yml@refs/pull/7/head", "pull_request", "success", [job("done", 0, 1, 13)], pull=7),
|
||||
run(2, "pr.yml@refs/pull/7/head", "pull_request", "cancelled", [job("done", 0, 1, 2)], pull=7),
|
||||
]
|
||||
|
||||
# when
|
||||
row = m.stage_rows(runs)[0]
|
||||
|
||||
# then
|
||||
self.assertEqual((row["runs"], row["finished"], row["exec_p50"], row["wait_p50"]), (2, 1, 12, 1))
|
||||
|
||||
def test_runs_per_merged_pull_request_count_every_attempt_of_that_pull_request(self):
|
||||
runs = [
|
||||
run(1, "pr.yml@refs/pull/7/head", "pull_request", "cancelled", [job("done", 0)], pull=7),
|
||||
run(2, "pr.yml@refs/pull/7/head", "pull_request", "success", [job("done", 1, 2, 14)], pull=7),
|
||||
run(3, "pr.yml@refs/pull/8/head", "pull_request", "success", [job("done", 1, 2, 14)], pull=8),
|
||||
]
|
||||
self.assertEqual(m.runs_per_merged_pull(runs, [7, 8]), [2, 1])
|
||||
|
||||
|
||||
class RedTest(unittest.TestCase):
|
||||
def test_main_is_red_from_the_first_failing_runs_creation_to_the_next_green_runs_end(self):
|
||||
# given: two failures in a row, then a green run that ends at minute 210
|
||||
runs = [
|
||||
run(1, "main.yml@refs/heads/main", "push", "failure", [job("main", 0, 5, 30, "failure")]),
|
||||
run(2, "main.yml@refs/heads/main", "push", "failure", [job("main", 60, 65, 95, "failure")]),
|
||||
run(3, "main.yml@refs/heads/main", "push", "success", [job("main", 120, 125, 210)]),
|
||||
run(4, "pr.yml@refs/pull/7/head", "pull_request", "failure", [job("done", 0, 1, 2, "failure")], pull=7),
|
||||
]
|
||||
|
||||
# when
|
||||
episodes = m.red_episodes(runs, now=T0 + timedelta(hours=10))
|
||||
|
||||
# then
|
||||
self.assertEqual([(e.first_failed_run, e.fixed_by_run, e.hours) for e in episodes], [(1, 3, 3.5)])
|
||||
|
||||
def test_main_still_red_at_the_end_of_the_window_counts_up_to_now(self):
|
||||
runs = [run(1, "main.yml@refs/heads/main", "push", "failure", [job("main", 0, 5, 30, "failure")])]
|
||||
episodes = m.red_episodes(runs, now=T0 + timedelta(hours=2))
|
||||
self.assertEqual([(e.fixed_by_run, e.hours) for e in episodes], [(None, 2.0)])
|
||||
|
||||
|
||||
class UtilisationTest(unittest.TestCase):
|
||||
def test_job_minutes_count_on_the_local_day_the_job_started(self):
|
||||
# given: 14:30 UTC is 23:30 KST on the 13th, 15:30 UTC is 00:30 KST on the 14th
|
||||
r = run(1, "main.yml@refs/heads/main", "push", "success",
|
||||
[job("a", 0, 14 * 60 + 30, 14 * 60 + 50), job("b", 0, 15 * 60 + 30, 15 * 60 + 40)])
|
||||
|
||||
# when
|
||||
days = m.utilisation([r], timedelta(hours=9))
|
||||
|
||||
# then
|
||||
self.assertEqual({str(day): value for day, value in days.items()}, {"2026-11-13": 20, "2026-11-14": 10})
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -2,9 +2,13 @@
|
||||
|
||||
set -o pipefail
|
||||
|
||||
readonly PLATFORM_SENSITIVE_SOURCE_EXACT_PLATFORM_ROOT=/home/donghyeon/workspace/platform
|
||||
readonly PLATFORM_SENSITIVE_SOURCE_EXACT_DOCS_ROOT=/home/donghyeon/workspace/docs/platform
|
||||
readonly PLATFORM_SENSITIVE_SOURCE_EXACT_SCANNER="$PLATFORM_SENSITIVE_SOURCE_EXACT_PLATFORM_ROOT/scripts/validate/scan-platform-sensitive-source.sh"
|
||||
# The platform root is the checkout this scanner lives in, so a temporary copy
|
||||
# or a CI runner scans itself. The central documentation root stays on the
|
||||
# desktop host unless --docs-root or --platform-only says otherwise.
|
||||
PLATFORM_SENSITIVE_SOURCE_SELF_ROOT="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/../.." && pwd -P)"
|
||||
readonly PLATFORM_SENSITIVE_SOURCE_SELF_ROOT
|
||||
readonly PLATFORM_SENSITIVE_SOURCE_SELF="$PLATFORM_SENSITIVE_SOURCE_SELF_ROOT/scripts/validate/scan-platform-sensitive-source.sh"
|
||||
readonly PLATFORM_SENSITIVE_SOURCE_DEFAULT_DOCS_ROOT=/home/donghyeon/workspace/docs/platform
|
||||
|
||||
_sensitive_source_fail() {
|
||||
printf 'ERROR: %s\n' "$*" >&2
|
||||
@@ -14,17 +18,22 @@ _sensitive_source_fail() {
|
||||
_sensitive_source_usage() {
|
||||
cat <<'USAGE'
|
||||
Usage:
|
||||
bash scripts/validate/scan-platform-sensitive-source.sh
|
||||
bash scripts/validate/scan-platform-sensitive-source.sh [--docs-root DIR | --platform-only]
|
||||
|
||||
Scans the complete platform source and central platform documentation trees.
|
||||
Only .git and .helm directory contents, plus the exact platform-root
|
||||
.superpowers/sdd scratch subtree, are excluded. Diagnostics contain file names
|
||||
and finding classes only; matched values and lines are never printed.
|
||||
Scans the platform checkout that contains this script and the central platform
|
||||
documentation tree (default /home/donghyeon/workspace/docs/platform, or
|
||||
--docs-root DIR). --platform-only skips the documentation tree, for a
|
||||
temporary copy or a CI runner that has none. Only .git and .helm directory
|
||||
contents, plus the platform-root .superpowers/sdd scratch subtree, are
|
||||
excluded. Diagnostics contain file names and finding classes only; matched
|
||||
values and lines are never printed.
|
||||
|
||||
The production run invokes all five renderer entrypoints in private temporary
|
||||
contexts. Four output-producing renderers publish private handoffs. The access
|
||||
renderer uses its output-free Grafana verifier because it does not require
|
||||
historical metric inventories; its exact two-line success contract is checked.
|
||||
contexts. In a git checkout they run from a private copy of the tracked and
|
||||
not-ignored files, so an ignored chart cache cannot stop them. Four
|
||||
output-producing renderers publish private handoffs. The access renderer uses
|
||||
its output-free Grafana verifier because it does not require historical metric
|
||||
inventories; its exact two-line success contract is checked.
|
||||
It still traverses the pinned core renderer, whose separately published handoff
|
||||
contains the only allowlisted credential-free Secret and is decoded here.
|
||||
USAGE
|
||||
@@ -43,7 +52,7 @@ _sensitive_source_report_path() {
|
||||
local path=$1 platform_root=$2 docs_root=$3 relative
|
||||
if [[ "$path" == "$platform_root"/* ]]; then
|
||||
relative="platform/${path#"$platform_root"/}"
|
||||
elif [[ "$path" == "$docs_root"/* ]]; then
|
||||
elif [[ -n "$docs_root" && "$path" == "$docs_root"/* ]]; then
|
||||
relative="docs/${path#"$docs_root"/}"
|
||||
else
|
||||
relative="$(basename -- "$path")"
|
||||
@@ -82,13 +91,19 @@ structured_source = suffix in (".yaml", ".yml", ".json")
|
||||
python_source = suffix == ".py"
|
||||
shell_source = suffix in (".sh", ".bash", ".zsh") or not suffix
|
||||
prose_source = suffix in (".md", ".markdown", ".txt", ".rst")
|
||||
embedded_python_source = path in (
|
||||
"/home/donghyeon/workspace/platform/scripts/validate/observability-smoke.sh",
|
||||
"/home/donghyeon/workspace/platform/scripts/validate/test-scan-platform-sensitive-source.sh",
|
||||
"/home/donghyeon/workspace/platform/scripts/validate/test-configure-keycloak-grafana-oidc.sh",
|
||||
"/home/donghyeon/workspace/platform/scripts/bootstrap/create-observability-secrets.sh",
|
||||
"/home/donghyeon/workspace/platform/scripts/bootstrap/configure-keycloak-grafana-oidc.sh",
|
||||
)
|
||||
# Shell files with embedded Python, relative to the checkout that holds this
|
||||
# scanner (scanner_source is <root>/scripts/validate/<scanner>).
|
||||
scanner_root = Path(scanner_source).parents[2]
|
||||
embedded_python_source = path in {
|
||||
str(scanner_root / relative)
|
||||
for relative in (
|
||||
"scripts/validate/observability-smoke.sh",
|
||||
"scripts/validate/test-scan-platform-sensitive-source.sh",
|
||||
"scripts/validate/test-configure-keycloak-grafana-oidc.sh",
|
||||
"scripts/bootstrap/create-observability-secrets.sh",
|
||||
"scripts/bootstrap/configure-keycloak-grafana-oidc.sh",
|
||||
)
|
||||
}
|
||||
|
||||
try:
|
||||
raw = open(path, "rb").read()
|
||||
@@ -717,6 +732,19 @@ def permitted_reference_container(key, value):
|
||||
)
|
||||
|
||||
|
||||
def permitted_s3_credentials_reference(path, key, value):
|
||||
# CloudNativePG barmanObjectStore.s3Credentials: each entry is a
|
||||
# SecretKeySelector (name and key only), never the value itself.
|
||||
return (
|
||||
len(path) >= 1
|
||||
and path[-1] == "s3Credentials"
|
||||
and key in ("accessKeyId", "secretAccessKey")
|
||||
and isinstance(value, dict)
|
||||
and set(value) == {"name", "key"}
|
||||
and all(isinstance(value[field], str) and value[field] for field in value)
|
||||
)
|
||||
|
||||
|
||||
def permitted_webhook_feature(path, key, value):
|
||||
return (
|
||||
tuple(str(part) for part in path) == ("operators", "object-store")
|
||||
@@ -752,6 +780,8 @@ def walk(value, path=()):
|
||||
pass
|
||||
elif permitted_reference_container(key, child):
|
||||
pass
|
||||
elif permitted_s3_credentials_reference(path, key, child):
|
||||
pass
|
||||
elif not isinstance(child, str) or unsafe_scalar(child):
|
||||
return True
|
||||
if walk(child, path + (key,)):
|
||||
@@ -1382,9 +1412,13 @@ scan_platform_sensitive_roots() (
|
||||
return 1
|
||||
}
|
||||
done
|
||||
[[ "$platform_root" == /* && "$docs_root" == /* && "$platform_root" != "$docs_root" ]] ||
|
||||
# An empty docs root means a platform-only scan.
|
||||
local -a scan_roots=("$platform_root")
|
||||
[[ -z "$docs_root" ]] || scan_roots+=("$docs_root")
|
||||
[[ "$platform_root" == /* ]] && { [[ -z "$docs_root" ]] ||
|
||||
[[ "$docs_root" == /* && "$platform_root" != "$docs_root" ]]; } ||
|
||||
_sensitive_source_fail 'scan roots must be distinct absolute paths' || return
|
||||
for argument in "$platform_root" "$docs_root"; do
|
||||
for argument in "${scan_roots[@]}"; do
|
||||
[[ -d "$argument" && ! -L "$argument" && "$(cd -- "$argument" && pwd -P)" == "$argument" ]] ||
|
||||
_sensitive_source_fail "scan root is not a physical directory: $argument" || return
|
||||
done
|
||||
@@ -1397,7 +1431,7 @@ scan_platform_sensitive_roots() (
|
||||
printf 'SCANNER_ERROR: file inventory hardening failed\n' >&2
|
||||
return 1
|
||||
}
|
||||
if ! find "$platform_root" "$docs_root" \
|
||||
if ! find "${scan_roots[@]}" \
|
||||
\( -type d -path "$platform_root/.superpowers/sdd" -prune \) -o \
|
||||
\( -type d \( -name .git -o -name .helm \) -prune \) -o \
|
||||
\( -type f -o -type l \) -print0 >"$list_file"; then
|
||||
@@ -1432,7 +1466,7 @@ scan_platform_sensitive_roots() (
|
||||
fi
|
||||
|
||||
finding="$(_sensitive_source_python_scan \
|
||||
"$file" "$test_file" "$PLATFORM_SENSITIVE_SOURCE_EXACT_SCANNER")" || {
|
||||
"$file" "$test_file" "$PLATFORM_SENSITIVE_SOURCE_SELF")" || {
|
||||
printf 'SCANNER_ERROR: content scan failed for FILE=%s\n' "$report" >&2
|
||||
return 1
|
||||
}
|
||||
@@ -1505,7 +1539,7 @@ scan_platform_sensitive_roots() (
|
||||
scan_platform_renderer_secrets() (
|
||||
set -o pipefail
|
||||
local platform_root=$1 renderer label output_prefix publishes log file results line rc entry artifact
|
||||
local helm_bin list_file=''
|
||||
local helm_bin list_file='' render_root source_copy=''
|
||||
local failed=false
|
||||
local -a renderers=(
|
||||
'phase1|render-phase1.sh|platform-phase1-apply|yes'
|
||||
@@ -1546,6 +1580,11 @@ scan_platform_renderer_secrets() (
|
||||
rm -f -- "$list_file"
|
||||
;;
|
||||
esac
|
||||
case "$source_copy" in
|
||||
/tmp/platform-sensitive-source-render-src.??????)
|
||||
rm -rf -- "$source_copy"
|
||||
;;
|
||||
esac
|
||||
}
|
||||
trap cleanup_renderer_scan EXIT HUP INT TERM
|
||||
|
||||
@@ -1570,9 +1609,56 @@ scan_platform_renderer_secrets() (
|
||||
return 1
|
||||
}
|
||||
|
||||
# In a git checkout, render from a private copy of the tracked and not-ignored
|
||||
# files (minus files deleted from the working tree). Ignored state such as a
|
||||
# leftover .helm chart cache would otherwise make the renderers refuse to run.
|
||||
render_root=$platform_root
|
||||
if [[ -e "$platform_root/.git" ]]; then
|
||||
for label in git tar; do
|
||||
command -v "$label" >/dev/null 2>&1 || {
|
||||
printf 'SCANNER_ERROR: renderer command unavailable: %s\n' "$label" >&2
|
||||
return 1
|
||||
}
|
||||
done
|
||||
[[ "$(git -C "$platform_root" rev-parse --show-toplevel 2>/dev/null)" == "$platform_root" ]] || {
|
||||
printf 'SCANNER_ERROR: renderer root is not the top of its git checkout\n' >&2
|
||||
return 1
|
||||
}
|
||||
source_copy="$(mktemp -d /tmp/platform-sensitive-source-render-src.XXXXXX)" || {
|
||||
printf 'SCANNER_ERROR: renderer source copy allocation failed\n' >&2
|
||||
return 1
|
||||
}
|
||||
chmod 0700 "$source_copy" || return 1
|
||||
if ! python3 -I - "$platform_root" <<'PY' |
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
root = sys.argv[1]
|
||||
|
||||
|
||||
def listed(*options):
|
||||
result = subprocess.run(
|
||||
["git", "-C", root, "ls-files", "-z", *options],
|
||||
check=True,
|
||||
stdout=subprocess.PIPE,
|
||||
)
|
||||
return [entry for entry in result.stdout.split(b"\0") if entry]
|
||||
|
||||
|
||||
keep = set(listed("--cached", "--others", "--exclude-standard")) - set(listed("--deleted"))
|
||||
for entry in sorted(keep):
|
||||
sys.stdout.buffer.write(entry + b"\0")
|
||||
PY
|
||||
tar -C "$platform_root" --null -T - -cf - | tar -x -C "$source_copy"; then
|
||||
printf 'SCANNER_ERROR: renderer source copy failed\n' >&2
|
||||
return 1
|
||||
fi
|
||||
render_root=$source_copy
|
||||
fi
|
||||
|
||||
for renderer in "${renderers[@]}"; do
|
||||
IFS='|' read -r label renderer output_prefix publishes <<<"$renderer"
|
||||
renderer="$platform_root/scripts/validate/$renderer"
|
||||
renderer="$render_root/scripts/validate/$renderer"
|
||||
[[ -f "$renderer" && ! -L "$renderer" ]] || {
|
||||
printf 'SCANNER_ERROR: required renderer is unsafe: %s\n' "$label" >&2
|
||||
return 1
|
||||
@@ -1706,14 +1792,23 @@ scan_platform_renderer_secrets() (
|
||||
)
|
||||
|
||||
scan_platform_sensitive_source_main() {
|
||||
(( $# == 0 )) || {
|
||||
local docs_root=$PLATFORM_SENSITIVE_SOURCE_DEFAULT_DOCS_ROOT
|
||||
if (( $# == 1 )) && [[ "$1" == --platform-only ]]; then
|
||||
docs_root=''
|
||||
elif (( $# == 2 )) && [[ "$1" == --docs-root && "$2" == /* ]]; then
|
||||
docs_root=$2
|
||||
elif (( $# != 0 )); then
|
||||
_sensitive_source_usage >&2
|
||||
return 2
|
||||
}
|
||||
fi
|
||||
if [[ -n "$docs_root" && ! -d "$docs_root" ]]; then
|
||||
printf 'ERROR: documentation root %s does not exist; pass --docs-root DIR or --platform-only\n' \
|
||||
"$docs_root" >&2
|
||||
return 2
|
||||
fi
|
||||
scan_platform_sensitive_roots \
|
||||
"$PLATFORM_SENSITIVE_SOURCE_EXACT_PLATFORM_ROOT" \
|
||||
"$PLATFORM_SENSITIVE_SOURCE_EXACT_DOCS_ROOT" >/dev/null || return
|
||||
scan_platform_renderer_secrets "$PLATFORM_SENSITIVE_SOURCE_EXACT_PLATFORM_ROOT" || return
|
||||
"$PLATFORM_SENSITIVE_SOURCE_SELF_ROOT" "$docs_root" >/dev/null || return
|
||||
scan_platform_renderer_secrets "$PLATFORM_SENSITIVE_SOURCE_SELF_ROOT" || return
|
||||
printf 'PLATFORM SENSITIVE SOURCE SCAN PASS\n'
|
||||
}
|
||||
|
||||
|
||||
@@ -378,6 +378,44 @@ envValueFrom:
|
||||
YAML
|
||||
expect_reject 'a secretKeyRef container with a literal child fails closed' \
|
||||
'platform/bootstrap/reference-container.yaml'
|
||||
rm -f -- "$WORK/platform/bootstrap/reference-container.yaml"
|
||||
|
||||
cat >"$WORK/platform/bootstrap/s3-credentials.yaml" <<'YAML'
|
||||
spec:
|
||||
backup:
|
||||
barmanObjectStore:
|
||||
s3Credentials:
|
||||
accessKeyId:
|
||||
name: platform-postgres-backup-s3
|
||||
key: ACCESS_KEY_ID
|
||||
secretAccessKey:
|
||||
name: platform-postgres-backup-s3
|
||||
key: ACCESS_SECRET_KEY
|
||||
YAML
|
||||
expect_clean 'a CloudNativePG s3Credentials Secret reference remains a reference'
|
||||
cat >"$WORK/platform/bootstrap/s3-credentials.yaml" <<'YAML'
|
||||
spec:
|
||||
s3Credentials:
|
||||
secretAccessKey:
|
||||
name: platform-postgres-backup-s3
|
||||
key: ACCESS_SECRET_KEY
|
||||
value: should-never-be-printed-s3-reference
|
||||
YAML
|
||||
expect_reject 'an s3Credentials reference with an extra literal child fails closed' \
|
||||
'platform/bootstrap/s3-credentials.yaml'
|
||||
cat >"$WORK/platform/bootstrap/s3-credentials.yaml" <<'YAML'
|
||||
spec:
|
||||
storage:
|
||||
secretAccessKey:
|
||||
name: platform-postgres-backup-s3
|
||||
key: ACCESS_SECRET_KEY
|
||||
YAML
|
||||
expect_reject 'a name and key container outside s3Credentials still fails closed' \
|
||||
'platform/bootstrap/s3-credentials.yaml'
|
||||
printf 'spec:\n s3Credentials:\n secretAccess%sKey: Actual%s\n' '' 'ProductionSecretValue9283' \
|
||||
>"$WORK/platform/bootstrap/s3-credentials.yaml"
|
||||
expect_reject 'a literal s3Credentials secret access key is rejected' \
|
||||
'platform/bootstrap/s3-credentials.yaml'
|
||||
|
||||
new_fixture
|
||||
cat >"$WORK/platform/bootstrap/webhook-feature.yaml" <<'YAML'
|
||||
@@ -967,6 +1005,16 @@ label="$(basename -- "$0")"
|
||||
handoff=no
|
||||
[[ -z "$out" ]] || handoff=yes
|
||||
printf '%s\n' "$label${component:+:$component}:handoff=$handoff" >>"$PLATFORM_SENSITIVE_SOURCE_RENDER_LOG"
|
||||
root="$(cd -- "$(dirname -- "$0")/../.." && pwd -P)"
|
||||
if [[ "${PLATFORM_SENSITIVE_SOURCE_FAKE_REJECT_CACHE:-0}" != 0 ]] && \
|
||||
[[ -n "$(find "$root" -name .helm -type d -print -quit)" ]]; then
|
||||
printf 'REJECT: exact chart cache already exists\n' >&2
|
||||
exit 23
|
||||
fi
|
||||
if [[ -n "${PLATFORM_SENSITIVE_SOURCE_FAKE_REQUIRE_INPUT:-}" && \
|
||||
! -f "$root/$PLATFORM_SENSITIVE_SOURCE_FAKE_REQUIRE_INPUT" ]]; then
|
||||
exit 42
|
||||
fi
|
||||
|
||||
emit_phase1_log() {
|
||||
local publish=$1 name
|
||||
@@ -1438,6 +1486,41 @@ for injected_renderer in \
|
||||
fail "$injected_renderer unexpected stdout leaked content"
|
||||
pass "$injected_renderer stdout is restricted to its bounded grammar"
|
||||
done
|
||||
|
||||
# A git checkout renders from a private copy of its tracked and not-ignored files,
|
||||
# so an ignored chart cache left in the checkout cannot stop the renderers (the
|
||||
# real renderers refuse a pre-existing exact cache with RC=23).
|
||||
command -v git >/dev/null 2>&1 || fail 'git is required for the clean render copy test'
|
||||
printf '.helm/\n' >"$WORK/platform/.gitignore"
|
||||
git -C "$WORK/platform" init -q
|
||||
git -C "$WORK/platform" add -A
|
||||
git -C "$WORK/platform" -c user.name=fixture -c user.email=fixture@example.invalid \
|
||||
commit -q -m fixture
|
||||
mkdir -p "$WORK/platform/services/observability/.helm/charts/stale-1.0.0"
|
||||
printf 'untracked\n' >"$WORK/platform/services/observability/untracked-input.yaml"
|
||||
: >"$WORK/renderer.log"
|
||||
renderer_rc=0
|
||||
renderer_output="$(PLATFORM_SENSITIVE_SOURCE_FAKE_REJECT_CACHE=1 \
|
||||
PLATFORM_SENSITIVE_SOURCE_FAKE_REQUIRE_INPUT=services/observability/untracked-input.yaml \
|
||||
scan_platform_renderer_secrets "$WORK/platform" 2>&1)" || renderer_rc=$?
|
||||
(( renderer_rc == 0 )) || {
|
||||
printf '%s\n' "$renderer_output" >&2
|
||||
fail 'an ignored chart cache in a git checkout blocked the renderers'
|
||||
}
|
||||
[[ -d "$WORK/platform/services/observability/.helm/charts/stale-1.0.0" ]] || \
|
||||
fail 'the renderer scan changed the ignored cache in the scanned checkout'
|
||||
pass 'a git checkout renders from a clean copy, so an ignored chart cache cannot block it'
|
||||
pass 'the clean render copy keeps untracked files that git does not ignore'
|
||||
rm -f -- "$WORK/platform/services/observability/untracked-input.yaml"
|
||||
rm -f -- "$WORK/platform/scripts/validate/render-phase2.sh"
|
||||
renderer_rc=0
|
||||
renderer_output="$(scan_platform_renderer_secrets "$WORK/platform" 2>&1)" || renderer_rc=$?
|
||||
(( renderer_rc != 0 )) || fail 'a renderer deleted from the working tree was still run'
|
||||
[[ "$renderer_output" == *'required renderer is unsafe: phase2'* ]] || {
|
||||
printf '%s\n' "$renderer_output" >&2
|
||||
fail 'a renderer deleted from the working tree did not fail at the renderer check'
|
||||
}
|
||||
pass 'the clean render copy leaves out files deleted from the working tree'
|
||||
unset PLATFORM_SENSITIVE_SOURCE_RENDER_LOG
|
||||
|
||||
new_fixture
|
||||
@@ -1468,6 +1551,39 @@ output="$(run_scanner 2>&1)" || scanner_error_rc=$?
|
||||
fail 'scanner error did not fail closed'
|
||||
}
|
||||
pass 'rg errors fail closed'
|
||||
PLATFORM_SENSITIVE_SOURCE_TEST_PATH=''
|
||||
|
||||
[[ "$PLATFORM_SENSITIVE_SOURCE_SELF_ROOT" == "$ROOT" ]] || \
|
||||
fail 'the scanner does not scan the checkout it lives in'
|
||||
pass 'the scanner scans the checkout it lives in, not a fixed path'
|
||||
main_rc=0
|
||||
main_output="$(scan_platform_sensitive_source_main --unknown 2>&1)" || main_rc=$?
|
||||
(( main_rc == 2 )) || fail 'an unknown entrypoint argument was accepted'
|
||||
main_rc=0
|
||||
main_output="$(scan_platform_sensitive_source_main --docs-root relative/docs 2>&1)" || main_rc=$?
|
||||
(( main_rc == 2 )) || fail 'a relative docs root was accepted'
|
||||
new_fixture
|
||||
main_rc=0
|
||||
main_output="$(scan_platform_sensitive_source_main --docs-root "$WORK/missing-docs" 2>&1)" || main_rc=$?
|
||||
(( main_rc == 2 )) || fail 'a missing docs root was accepted'
|
||||
[[ "$main_output" == *'--platform-only'* ]] || \
|
||||
fail 'a missing docs root did not name --platform-only'
|
||||
pass 'the entrypoint rejects unknown, relative and missing docs roots before scanning'
|
||||
printf 'ordinary input\n' >"$WORK/platform/bootstrap/ordinary.md"
|
||||
platform_only_output="$(scan_platform_sensitive_roots "$WORK/platform" '' 2>&1)" || {
|
||||
printf '%s\n' "$platform_only_output" >&2
|
||||
fail 'a platform-only root scan failed'
|
||||
}
|
||||
[[ "$platform_only_output" == *'PLATFORM SENSITIVE SOURCE SCAN PASS'* ]] || \
|
||||
fail 'a platform-only root scan did not emit the PASS marker'
|
||||
printf 'Slack: https://hooks.%s/services/T1/B2/C3\n' 'slack.com' \
|
||||
>"$WORK/platform/bootstrap/live.env"
|
||||
platform_only_rc=0
|
||||
platform_only_output="$(scan_platform_sensitive_roots "$WORK/platform" '' 2>&1)" || platform_only_rc=$?
|
||||
(( platform_only_rc != 0 )) || fail 'a platform-only root scan accepted a Slack webhook'
|
||||
[[ "$platform_only_output" == *'platform/bootstrap/live.env'* ]] || \
|
||||
fail 'a platform-only root scan did not report the filename'
|
||||
pass 'a platform-only root scan still scans the platform root'
|
||||
|
||||
cleanup_before="$(
|
||||
find /tmp -maxdepth 1 -type d -name 'platform-sensitive-source-test.??????' \
|
||||
|
||||
Reference in New Issue
Block a user