Author SHA1 Message Date
donghyeon.kang d0875bfe44 Merge pull request 'feat(ci): add a read-only CI metrics script for the desktop runner' (#12) from claude/w2-ci-metrics into main 2026-11-13 10:43:35 +00:00
DongHyeonka dd8e9140b7 feat(ci): add a read-only CI metrics script for the desktop runner
There is no number for how long CI runs wait, how busy the one runner is,
how many runs a pull request costs or how long main stays red. Run times
were added up by hand from single runs, so whether a change helped could
not be told apart from load.

infrastructure/ci/metrics/ci_metrics.py reads the Gitea Actions API (runs,
their jobs, closed pull requests) for the repositories the runner serves
and prints the CI manual's metrics 1-5 per repository and stage: run
execution p50/p90, queue wait p50/p90, runner utilisation per KST day,
pull request runs per merged pull request, and main red time. It only
reads; credentials come from GITEA_TOKEN or `git credential fill` and are
never printed.

The arithmetic follows what our Gitea 1.27 returns, which differs from the
API reference: a run has no created_at (the earliest job created_at is
used), unstarted runs and jobs report 1970-01-01 times (treated as
absent), and a job's created_at is its run's, so queue wait is measured
per run (first job start minus run creation) instead of per job.

Verification: python3 -m unittest test_ci_metrics (11 offline tests) and
one run against the server for 2026-11-07..13 (two repositories, 30 s,
rc=0). The sensitive-source scan needs rg and ran on the desktop host.

Refs: design-docs ci/03-time-reduction-plan.md M-0
2026-11-13 19:34:38 +09:00
donghyeon.kang 5d27096ed0 Merge pull request 'fix(validate): run the sensitive-source scan from any checkout' (#11) from donghyeon.kang/w6-scanner-fix into main 2026-11-11 23:46:20 +00:00
DongHyeonka ae4e256c95 fix(validate): accept CloudNativePG s3Credentials Secret references
PR #10 added the platform-postgres backup patch and its restore-drill
Cluster. Both hold barmanObjectStore.s3Credentials, where accessKeyId
and secretAccessKey are SecretKeySelectors (name and key). The
structured scan treats any non-string value under a key ending in
"access key" as a credential literal, so the source scan reported
CREDENTIAL_LITERAL for both files on main.

An entry directly under s3Credentials, named accessKeyId or
secretAccessKey, whose value is a mapping of exactly name and key
with non-empty strings, is now a reference. This mirrors the existing
exact rule for the Grafana client-secret secretKeyRef.

Rejected: a general "name and key is always a reference" rule. It
would also pass a literal hidden under some other credential key that
happens to have those two fields.

Risk: none beyond the two key names under that one parent. A literal
string, an extra child, or the same mapping under another parent is
still rejected (new test cases).

Verification: on desktop, the source scan of main 205927f reported the
two files before this change and passes after it. The unit test
passes in a non-canonical clone.
2026-11-12 08:36:38 +09:00
DongHyeonka 27eb8c1799 fix(validate): render the sensitive-source scan from a clean copy
The scanner ran the five renderers inside the scanned checkout. The
desktop checkout holds eight .helm chart caches left on 2026-08-11
(ignored by git), and render-observability-core.sh refuses an existing
exact chart cache. The scan stopped with "renderer failed:
observability-core RC=23" before it checked any rendered Secret, on
main and on every branch.

In a git checkout the renderers now run from a private 0700 copy in
/tmp of the tracked and not-ignored files, minus files deleted from
the working tree (git ls-files, then tar). The copy is removed with
the other renderer temp paths. A tree that is not a git checkout, such
as the test fixtures, still renders in place.

Rejected: deleting the stale caches only. That fixes one host until a
cache is left again, and the scanner is not the place to delete files
in the tree it scans. The renderer's refusal of an existing cache
stays; it guards the chart digest checks.

Risk: an ignored file is not in the copy. No renderer reads one today
(no renderer references .env, local-recovery.env, .superpowers or
secrets/). If one ever does, its render fails closed rather than
scanning less.

Verification: on desktop, real renderers against a clone of main
205927f with the eight stale caches copied in. Before: phase1, phase2
and admin pass, then "renderer failed: observability-core RC=23".
With this branch: all five renderers pass, "PLATFORM RENDERED SECRET
SCAN PASS", the eight caches in the scanned clone are untouched and no
render copy is left in /tmp. The new test cases fail before this
change and pass after it.
2026-11-12 08:36:38 +09:00
DongHyeonka 149b885e35 fix(validate): scan the checkout that holds the sensitive-source scanner
scan-platform-sensitive-source.sh always scanned the fixed path
/home/donghyeon/workspace/platform, and its list of shell files with
embedded Python held the same fixed paths. Run from a temporary copy
or a CI runner, it scanned the desktop checkout instead of itself, or
nothing at all. Its own test failed at the first assertion in any
other checkout: the test file lost its embedded-Python context there
and was reported as a credential literal.

The platform root is now the checkout that contains the scanner
(dirname of BASH_SOURCE, physical path), and the embedded-Python list
is resolved under that root. The central documentation root stays the
desktop default. --docs-root DIR replaces it and --platform-only skips
it, for a copy or runner that has no central docs. A missing docs root
stops with exit 2 and names both options instead of scanning less.

Rejected: a --platform-root option. Scanning a tree with a scanner
from another tree would let a scanned tree differ from the rules that
scan it.

Risk: on the desktop host the default run is unchanged. A caller that
passed arguments before got usage and exit 2. It now gets the new
options.

Verification: on desktop, a non-canonical clone of main 205927f with
this change: test-scan-platform-sensitive-source.sh passes. Before the
change the same test in a worktree copy failed with "canonical scanner
test content leaf found a credential literal".
2026-11-12 08:02:05 +09:00
donghyeon.kang 205927f4e7 Merge pull request 'feat(platform): guard the disk, clock, backups and monitoring after this week's incidents' (#10) from claude/w1-infra-hardening into main 2026-11-11 22:43:49 +00:00
5 changed files with 795 additions and 30 deletions
+46
View File
@@ -0,0 +1,46 @@
# CI 지표
desktop runner 하나가 받는 repo들의 CI 지표를 Gitea Actions API에서 뽑는다. 읽기만 하고 서버에 아무것도 쓰지 않는다. 지표의 정의와 목표는 design-docs `ci/manual.md` 10절이고, 주 1회 결과를 design-docs `ci/metrics.md`에 한 줄 남긴다.
## 돌리기
```bash
python3 infrastructure/ci/metrics/ci_metrics.py \
--repo clean-architecture-backend-template \
--repo clean-architecture-frontend-template \
--since 2026-11-07 --until 2026-11-13
```
- 날짜는 KST 하루 단위이고 `--until`도 그날을 포함한다(`--utc-offset-hours`로 바꾼다).
- 인증: `GITEA_TOKEN`(읽기 전용 토큰)이 있으면 그것을, 없으면 `git credential fill`의 그 호스트 자격 증명을 쓴다. 어느 쪽도 출력하지 않는다.
- `--json`은 표 대신 숫자를 JSON으로 낸다.
- 두 repo, 지난 7일이면 30초 안팎이다(run마다 jobs API를 한 번 부른다).
## 내는 것
| # | 지표 | 계산 |
| --- | --- | --- |
| 1 | 실행 시간 p50·p90 | 단계(`workflow:event`)별, 끝난 run(success·failure)의 job 실행 시간 합. cancelled run은 run 수에만 센다 |
| 2 | 대기 시간 p50·p90 | run의 첫 job 시작 − run 생성 |
| 3 | runner 사용률 | 날짜(job 시작 시각, KST)별 job 실행 시간 합 ÷ 1,440분. 준 repo를 모두 더한다(runner가 하나라서) |
| 4 | PR당 run 수 | 기간 안에 병합된 PR마다 그 PR의 `pull_request` run 수. 병합 전 7일의 run까지 센다 |
| 5 | main 빨강 시간 | workflow별 main push run이 처음 실패한 run의 생성부터 다음 성공 run의 끝까지. 기간 끝까지 빨강이면 그때까지 |
지표 6(흔들림)은 각 repo `docs/ledger/flaky-tests.jsonl`, 지표 7(캐시 적중)은 PR run 로그에서 나온다. 이 스크립트는 보지 않는다.
## 우리 서버(Gitea 1.27)가 API 명세와 다른 곳
2026-11-13 응답 한 건씩으로 확인했다. 계산은 서버를 따른다.
- run에 `created_at`이 없다. run의 job은 run과 함께 만들어지므로, job `created_at`의 가장 이른 값을 run 생성 시각으로 쓴다.
- 시작하지 않은 run·job의 `started_at`·`completed_at`은 `1970-01-01T00:00:00Z`다. 없는 값으로 본다.
- job의 `created_at`은 그 job이 실행할 수 있게 된 시각이 아니라 run의 생성 시각이다. runner 하나가 run의 job을 차례로 돌리므로 job마다 `started_at − created_at`을 대기로 세면 앞 job의 실행이 대기에 들어간다. 그래서 대기는 run 단위로 센다. `needs`가 있는 job이 늦게 풀려 대기열 뒤로 가는 시간은 이 지표에 안 보이고 실행 시간에도 안 들어간다.
- run 목록의 run에는 `head_branch`가 없다. 단계와 ref는 `path`(`<workflow>@<ref>`)에서 읽고, PR 번호는 `pull_requests`에서 읽는다.
## 시험
```bash
cd infrastructure/ci/metrics && python3 -m unittest -v test_ci_metrics
```
서버 없이 계산만 본다. 만든 run·job 기록으로 대기·실행·빨강 시간·사용률을 확인한다.
+384
View File
@@ -0,0 +1,384 @@
#!/usr/bin/env python3
"""CI metrics for the repositories one Gitea Actions runner serves.
Reads the Gitea Actions API (runs, their jobs, closed pull requests) and prints, per repository and
stage, the CI manual's metrics 1-5: run execution time, queue wait, runner utilisation, pull
request runs per merged pull request and time main stayed red. Read-only; it never writes to the
server.
What our Gitea 1.27 returns differs from the gitea.com API reference in three places, and the
arithmetic below follows the server:
* A run carries no ``created_at``. A run's jobs are all created with the run, so the earliest job
``created_at`` is the run's creation time.
* A run or job that has not started reports ``started_at`` and ``completed_at`` as
``1970-01-01T00:00:00Z``. Those are treated as absent.
* A job's ``created_at`` is the run's, not the moment it became runnable. One runner runs a run's
jobs one after another, so ``started_at - created_at`` of a later job counts the earlier jobs'
execution as waiting. Queue wait is therefore measured per run: first job start minus run
creation.
Credentials: ``GITEA_TOKEN`` (a read-only token) if set, otherwise ``git credential fill`` for the
server host. Neither is printed.
"""
from __future__ import annotations
import argparse
import base64
import json
import math
import os
import subprocess
import sys
import urllib.parse
import urllib.request
from dataclasses import dataclass, field
from datetime import date, datetime, timedelta, timezone
EPOCH_PREFIXES = ("1970-01-01", "0001-01-01")
FINISHED = {"success", "failure"}
def parse_time(value: str | None) -> datetime | None:
"""An API timestamp, or None for a missing or zero value."""
if not value or value.startswith(EPOCH_PREFIXES):
return None
return datetime.fromisoformat(value.replace("Z", "+00:00"))
def percentile(values: list[float], fraction: float) -> float | None:
"""Nearest-rank percentile; None for no values."""
if not values:
return None
ordered = sorted(values)
rank = max(1, math.ceil(fraction * len(ordered)))
return ordered[rank - 1]
@dataclass
class Job:
name: str
created: datetime | None
started: datetime | None
completed: datetime | None
conclusion: str | None
@property
def minutes(self) -> float:
if self.started is None or self.completed is None:
return 0.0
return max(0.0, (self.completed - self.started).total_seconds() / 60)
@dataclass
class Run:
repo: str
id: int
workflow: str
ref: str
event: str
status: str
conclusion: str | None
pull_request: int | None
jobs: list[Job] = field(default_factory=list)
@property
def stage(self) -> str:
return f"{self.workflow}:{self.event}"
@property
def created(self) -> datetime | None:
times = [job.created for job in self.jobs if job.created is not None]
return min(times) if times else None
@property
def first_start(self) -> datetime | None:
times = [job.started for job in self.jobs if job.started is not None]
return min(times) if times else None
@property
def completed(self) -> datetime | None:
times = [job.completed for job in self.jobs if job.completed is not None]
return max(times) if times and self.status == "completed" else None
@property
def execution_minutes(self) -> float:
return sum(job.minutes for job in self.jobs)
@property
def queue_minutes(self) -> float | None:
if self.created is None or self.first_start is None:
return None
return max(0.0, (self.first_start - self.created).total_seconds() / 60)
@property
def main_push(self) -> bool:
return self.event == "push" and self.ref == "refs/heads/main"
def run_from_api(repo: str, raw: dict, jobs: list[dict]) -> Run:
workflow, _, ref = raw.get("path", "").partition("@")
pulls = raw.get("pull_requests") or []
return Run(
repo=repo,
id=raw["id"],
workflow=workflow,
ref=ref,
event=raw.get("event", ""),
status=raw.get("status", ""),
conclusion=raw.get("conclusion"),
pull_request=pulls[0]["number"] if pulls else None,
jobs=[
Job(
name=job.get("name", ""),
created=parse_time(job.get("created_at")),
started=parse_time(job.get("started_at")),
completed=parse_time(job.get("completed_at")),
conclusion=job.get("conclusion"),
)
for job in jobs
],
)
class Gitea:
"""The few read-only API calls this script makes."""
def __init__(self, server: str):
self.server = server.rstrip("/")
self.header = self._authorization()
def _authorization(self) -> str:
token = os.environ.get("GITEA_TOKEN")
if token:
return f"token {token}"
host = urllib.parse.urlparse(self.server).netloc
answer = subprocess.run(
["git", "credential", "fill"],
input=f"protocol=https\nhost={host}\n\n",
capture_output=True,
text=True,
check=True,
).stdout
fields = dict(line.split("=", 1) for line in answer.splitlines() if "=" in line)
pair = f"{fields['username']}:{fields['password']}".encode()
return "Basic " + base64.b64encode(pair).decode()
def get(self, path: str) -> object:
request = urllib.request.Request(
f"{self.server}/api/v1{path}",
headers={"Authorization": self.header, "Accept": "application/json"},
)
with urllib.request.urlopen(request, timeout=30) as response:
return json.load(response)
def fetch_runs(api: Gitea, owner: str, repo: str, since: datetime) -> list[Run]:
"""Runs created at or after ``since``, newest first. Run ids grow with creation time."""
runs: list[Run] = []
page = 1
while True:
body = api.get(f"/repos/{owner}/{repo}/actions/runs?limit=50&page={page}")
batch = body.get("workflow_runs") or []
if not batch:
return runs
for raw in batch:
jobs = api.get(f"/repos/{owner}/{repo}/actions/runs/{raw['id']}/jobs").get("jobs") or []
run = run_from_api(repo, raw, jobs)
if run.created is not None and run.created < since:
return runs
runs.append(run)
page += 1
def fetch_merged_pulls(api: Gitea, owner: str, repo: str, since: datetime, until: datetime) -> list[int]:
numbers: list[int] = []
page = 1
while True:
batch = api.get(f"/repos/{owner}/{repo}/pulls?state=closed&sort=recentupdate&limit=50&page={page}")
if not batch:
return numbers
for pull in batch:
merged = parse_time(pull.get("merged_at"))
if pull.get("merged") and merged is not None and since <= merged < until:
numbers.append(pull["number"])
updated = [parse_time(pull.get("updated_at")) for pull in batch]
if all(time is not None and time < since for time in updated):
return numbers
page += 1
@dataclass
class RedEpisode:
workflow: str
first_failed_run: int
fixed_by_run: int | None
hours: float
def red_episodes(runs: list[Run], now: datetime) -> list[RedEpisode]:
"""Main stays red from the first failing main push run's creation to the next green one's end."""
episodes: list[RedEpisode] = []
by_workflow: dict[str, list[Run]] = {}
for run in runs:
if run.main_push and run.status == "completed" and run.conclusion in FINISHED:
by_workflow.setdefault(run.workflow, []).append(run)
for workflow, history in by_workflow.items():
history.sort(key=lambda run: run.created or now)
start: Run | None = None
for run in history:
if run.conclusion == "failure" and start is None:
start = run
elif run.conclusion == "success" and start is not None:
end = run.completed or now
episodes.append(
RedEpisode(workflow, start.id, run.id, (end - start.created).total_seconds() / 3600)
)
start = None
if start is not None:
episodes.append(RedEpisode(workflow, start.id, None, (now - start.created).total_seconds() / 3600))
return episodes
def stage_rows(runs: list[Run]) -> list[dict]:
rows = []
for stage in sorted({run.stage for run in runs}):
finished = [run for run in runs if run.stage == stage and run.conclusion in FINISHED]
waited = [run.queue_minutes for run in runs if run.stage == stage and run.queue_minutes is not None]
executed = [run.execution_minutes for run in finished]
rows.append(
{
"stage": stage,
"runs": len([run for run in runs if run.stage == stage]),
"finished": len(finished),
"red": len([run for run in finished if run.conclusion == "failure"]),
"exec_p50": percentile(executed, 0.5),
"exec_p90": percentile(executed, 0.9),
"wait_p50": percentile(waited, 0.5),
"wait_p90": percentile(waited, 0.9),
}
)
return rows
def utilisation(runs: list[Run], offset: timedelta) -> dict[date, float]:
"""Job execution minutes per local day (job start day) across every repository given."""
minutes: dict[date, float] = {}
for run in runs:
for job in run.jobs:
if job.started is not None and job.minutes > 0:
day = (job.started + offset).date()
minutes[day] = minutes.get(day, 0.0) + job.minutes
return minutes
def runs_per_merged_pull(runs: list[Run], merged: list[int]) -> list[int]:
return [len([run for run in runs if run.pull_request == number]) for number in merged]
def fmt(value: float | None, digits: int = 1) -> str:
return "-" if value is None else f"{value:.{digits}f}"
def report(window: str, per_repo: dict[str, dict], days: dict[date, float]) -> str:
lines = [f"# CI 지표 ({window})", ""]
for repo, data in per_repo.items():
lines += [
f"## {repo}",
"",
"| 단계 | run | 끝난 run | 빨강 | 실행 p50(분) | 실행 p90(분) | 대기 p50(분) | 대기 p90(분) |",
"|---|---|---|---|---|---|---|---|",
]
for row in data["stages"]:
lines.append(
f"| {row['stage']} | {row['runs']} | {row['finished']} | {row['red']} | {fmt(row['exec_p50'])} | "
f"{fmt(row['exec_p90'])} | {fmt(row['wait_p50'])} | {fmt(row['wait_p90'])} |"
)
counts = data["runs_per_pull"]
lines += [
"",
f"- 병합된 PR {len(counts)}개, PR당 run 수 p50 {fmt(percentile(counts, 0.5), 0)}, "
f"p90 {fmt(percentile(counts, 0.9), 0)}, 최대 {max(counts) if counts else '-'}",
]
episodes = data["red"]
if episodes:
longest = max(episodes, key=lambda episode: episode.hours)
lines.append(
f"- main 빨강 {len(episodes)}회, 합 {sum(e.hours for e in episodes):.1f}시간, p90 "
f"{fmt(percentile([e.hours for e in episodes], 0.9))}시간, 가장 긴 것 {longest.hours:.1f}시간"
f"({longest.workflow} run {longest.first_failed_run} → "
f"{longest.fixed_by_run if longest.fixed_by_run else '아직 빨강'})"
)
else:
lines.append("- main 빨강 0회")
lines.append("")
lines += ["## runner 사용률 (위 repo 합, job 시작 날짜 기준)", "", "| 날짜 | 실행 합(분) | 사용률 |", "|---|---|---|"]
for day in sorted(days):
lines.append(f"| {day} | {days[day]:.0f} | {days[day] / 1440:.0%} |")
if days:
busiest = max(days, key=days.get)
lines += ["", f"- 바쁜 날 {busiest}: {days[busiest] / 1440:.0%}"]
return "\n".join(lines) + "\n"
def main(argv: list[str]) -> int:
parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
parser.add_argument("--server", default="https://git.learn.hyeonworks.com")
parser.add_argument("--owner", default="donghyeon.kang")
parser.add_argument("--repo", action="append", required=True, help="repeat for each repository")
parser.add_argument("--since", required=True, type=date.fromisoformat, help="first local day, YYYY-MM-DD")
parser.add_argument("--until", required=True, type=date.fromisoformat, help="last local day, inclusive")
parser.add_argument("--utc-offset-hours", type=int, default=9, help="local day boundary (default KST)")
parser.add_argument("--json", action="store_true", help="print the numbers as JSON instead of tables")
args = parser.parse_args(argv)
offset = timedelta(hours=args.utc_offset_hours)
since = datetime.combine(args.since, datetime.min.time(), timezone.utc) - offset
until = datetime.combine(args.until + timedelta(days=1), datetime.min.time(), timezone.utc) - offset
# A pull request merged in the window may have had runs in the week before it.
lookback = since - timedelta(days=7)
now = min(datetime.now(timezone.utc), until)
api = Gitea(args.server)
per_repo: dict[str, dict] = {}
window_runs: list[Run] = []
for repo in args.repo:
fetched = fetch_runs(api, args.owner, repo, lookback)
in_window = [run for run in fetched if run.created is not None and since <= run.created < until]
window_runs += in_window
merged = fetch_merged_pulls(api, args.owner, repo, since, until)
per_repo[repo] = {
"stages": stage_rows(in_window),
"runs_per_pull": runs_per_merged_pull(fetched, merged),
"red": red_episodes(in_window, now),
}
days = utilisation(window_runs, offset)
window = f"{args.since} ~ {args.until}, UTC{args.utc_offset_hours:+d}"
if args.json:
print(
json.dumps(
{
"window": window,
"repos": {
repo: {
"stages": data["stages"],
"runs_per_pull": data["runs_per_pull"],
"red": [episode.__dict__ for episode in data["red"]],
}
for repo, data in per_repo.items()
},
"utilisation_minutes": {str(day): round(value, 1) for day, value in sorted(days.items())},
},
ensure_ascii=False,
indent=2,
)
)
else:
print(report(window, per_repo, days), end="")
return 0
if __name__ == "__main__":
sys.exit(main(sys.argv[1:]))
@@ -0,0 +1,124 @@
"""Offline tests for ci_metrics: the arithmetic on run and job records, no server."""
import unittest
from datetime import datetime, timedelta, timezone
import ci_metrics as m
T0 = datetime(2026, 11, 13, 0, 0, tzinfo=timezone.utc)
def at(minutes: float) -> str:
return (T0 + timedelta(minutes=minutes)).isoformat().replace("+00:00", "Z")
def job(name, created, started=None, completed=None, conclusion="success"):
return {
"name": name,
"created_at": at(created),
"started_at": at(started) if started is not None else "1970-01-01T00:00:00Z",
"completed_at": at(completed) if completed is not None else "1970-01-01T00:00:00Z",
"conclusion": conclusion,
}
def run(run_id, path, event, conclusion, jobs, status="completed", pull=None):
raw = {"id": run_id, "path": path, "event": event, "status": status, "conclusion": conclusion}
if pull is not None:
raw["pull_requests"] = [{"number": pull}]
return m.run_from_api("backend", raw, jobs)
class ParsingTest(unittest.TestCase):
def test_a_zero_timestamp_from_a_job_that_has_not_started_is_absent(self):
self.assertIsNone(m.parse_time("1970-01-01T00:00:00Z"))
def test_the_run_is_created_when_its_earliest_job_is(self):
r = run(1, "pr.yml@refs/pull/7/head", "pull_request", "success", [job("done", 5, 20, 32)], pull=7)
self.assertEqual((r.created, r.stage, r.pull_request), (T0 + timedelta(minutes=5), "pr.yml:pull_request", 7))
class TimingTest(unittest.TestCase):
def test_queue_wait_is_the_first_job_start_minus_run_creation_not_each_jobs_own_gap(self):
# given: one runner runs the jobs one after another, so the second job's own gap is 40 minutes
r = run(1, "main.yml@refs/heads/main", "push", "success", [job("a", 0, 10, 30), job("b", 0, 30, 40)])
# when
waited = r.queue_minutes
# then
self.assertEqual(waited, 10)
def test_execution_is_the_sum_of_job_run_times_and_a_skipped_job_adds_nothing(self):
r = run(1, "main.yml@refs/heads/main", "push", "success",
[job("a", 0, 10, 30), job("b", 0, 30, 40), job("c", 0, 40, 40, "skipped")])
self.assertEqual(r.execution_minutes, 30)
def test_a_queued_run_has_no_wait_and_no_execution_yet(self):
r = run(1, "pr.yml@refs/pull/7/head", "pull_request", None, [job("done", 0)], status="queued", pull=7)
self.assertEqual((r.queue_minutes, r.execution_minutes), (None, 0.0))
def test_percentiles_use_the_nearest_rank(self):
self.assertEqual((m.percentile([3, 1, 2, 10], 0.5), m.percentile([3, 1, 2, 10], 0.9)), (2, 10))
class StageTest(unittest.TestCase):
def test_a_cancelled_run_counts_as_a_run_but_not_in_execution_percentiles(self):
# given
runs = [
run(1, "pr.yml@refs/pull/7/head", "pull_request", "success", [job("done", 0, 1, 13)], pull=7),
run(2, "pr.yml@refs/pull/7/head", "pull_request", "cancelled", [job("done", 0, 1, 2)], pull=7),
]
# when
row = m.stage_rows(runs)[0]
# then
self.assertEqual((row["runs"], row["finished"], row["exec_p50"], row["wait_p50"]), (2, 1, 12, 1))
def test_runs_per_merged_pull_request_count_every_attempt_of_that_pull_request(self):
runs = [
run(1, "pr.yml@refs/pull/7/head", "pull_request", "cancelled", [job("done", 0)], pull=7),
run(2, "pr.yml@refs/pull/7/head", "pull_request", "success", [job("done", 1, 2, 14)], pull=7),
run(3, "pr.yml@refs/pull/8/head", "pull_request", "success", [job("done", 1, 2, 14)], pull=8),
]
self.assertEqual(m.runs_per_merged_pull(runs, [7, 8]), [2, 1])
class RedTest(unittest.TestCase):
def test_main_is_red_from_the_first_failing_runs_creation_to_the_next_green_runs_end(self):
# given: two failures in a row, then a green run that ends at minute 210
runs = [
run(1, "main.yml@refs/heads/main", "push", "failure", [job("main", 0, 5, 30, "failure")]),
run(2, "main.yml@refs/heads/main", "push", "failure", [job("main", 60, 65, 95, "failure")]),
run(3, "main.yml@refs/heads/main", "push", "success", [job("main", 120, 125, 210)]),
run(4, "pr.yml@refs/pull/7/head", "pull_request", "failure", [job("done", 0, 1, 2, "failure")], pull=7),
]
# when
episodes = m.red_episodes(runs, now=T0 + timedelta(hours=10))
# then
self.assertEqual([(e.first_failed_run, e.fixed_by_run, e.hours) for e in episodes], [(1, 3, 3.5)])
def test_main_still_red_at_the_end_of_the_window_counts_up_to_now(self):
runs = [run(1, "main.yml@refs/heads/main", "push", "failure", [job("main", 0, 5, 30, "failure")])]
episodes = m.red_episodes(runs, now=T0 + timedelta(hours=2))
self.assertEqual([(e.fixed_by_run, e.hours) for e in episodes], [(None, 2.0)])
class UtilisationTest(unittest.TestCase):
def test_job_minutes_count_on_the_local_day_the_job_started(self):
# given: 14:30 UTC is 23:30 KST on the 13th, 15:30 UTC is 00:30 KST on the 14th
r = run(1, "main.yml@refs/heads/main", "push", "success",
[job("a", 0, 14 * 60 + 30, 14 * 60 + 50), job("b", 0, 15 * 60 + 30, 15 * 60 + 40)])
# when
days = m.utilisation([r], timedelta(hours=9))
# then
self.assertEqual({str(day): value for day, value in days.items()}, {"2026-11-13": 20, "2026-11-14": 10})
if __name__ == "__main__":
unittest.main()
@@ -2,9 +2,13 @@
set -o pipefail
readonly PLATFORM_SENSITIVE_SOURCE_EXACT_PLATFORM_ROOT=/home/donghyeon/workspace/platform
readonly PLATFORM_SENSITIVE_SOURCE_EXACT_DOCS_ROOT=/home/donghyeon/workspace/docs/platform
readonly PLATFORM_SENSITIVE_SOURCE_EXACT_SCANNER="$PLATFORM_SENSITIVE_SOURCE_EXACT_PLATFORM_ROOT/scripts/validate/scan-platform-sensitive-source.sh"
# The platform root is the checkout this scanner lives in, so a temporary copy
# or a CI runner scans itself. The central documentation root stays on the
# desktop host unless --docs-root or --platform-only says otherwise.
PLATFORM_SENSITIVE_SOURCE_SELF_ROOT="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/../.." && pwd -P)"
readonly PLATFORM_SENSITIVE_SOURCE_SELF_ROOT
readonly PLATFORM_SENSITIVE_SOURCE_SELF="$PLATFORM_SENSITIVE_SOURCE_SELF_ROOT/scripts/validate/scan-platform-sensitive-source.sh"
readonly PLATFORM_SENSITIVE_SOURCE_DEFAULT_DOCS_ROOT=/home/donghyeon/workspace/docs/platform
_sensitive_source_fail() {
printf 'ERROR: %s\n' "$*" >&2
@@ -14,17 +18,22 @@ _sensitive_source_fail() {
_sensitive_source_usage() {
cat <<'USAGE'
Usage:
bash scripts/validate/scan-platform-sensitive-source.sh
bash scripts/validate/scan-platform-sensitive-source.sh [--docs-root DIR | --platform-only]
Scans the complete platform source and central platform documentation trees.
Only .git and .helm directory contents, plus the exact platform-root
.superpowers/sdd scratch subtree, are excluded. Diagnostics contain file names
and finding classes only; matched values and lines are never printed.
Scans the platform checkout that contains this script and the central platform
documentation tree (default /home/donghyeon/workspace/docs/platform, or
--docs-root DIR). --platform-only skips the documentation tree, for a
temporary copy or a CI runner that has none. Only .git and .helm directory
contents, plus the platform-root .superpowers/sdd scratch subtree, are
excluded. Diagnostics contain file names and finding classes only; matched
values and lines are never printed.
The production run invokes all five renderer entrypoints in private temporary
contexts. Four output-producing renderers publish private handoffs. The access
renderer uses its output-free Grafana verifier because it does not require
historical metric inventories; its exact two-line success contract is checked.
contexts. In a git checkout they run from a private copy of the tracked and
not-ignored files, so an ignored chart cache cannot stop them. Four
output-producing renderers publish private handoffs. The access renderer uses
its output-free Grafana verifier because it does not require historical metric
inventories; its exact two-line success contract is checked.
It still traverses the pinned core renderer, whose separately published handoff
contains the only allowlisted credential-free Secret and is decoded here.
USAGE
@@ -43,7 +52,7 @@ _sensitive_source_report_path() {
local path=$1 platform_root=$2 docs_root=$3 relative
if [[ "$path" == "$platform_root"/* ]]; then
relative="platform/${path#"$platform_root"/}"
elif [[ "$path" == "$docs_root"/* ]]; then
elif [[ -n "$docs_root" && "$path" == "$docs_root"/* ]]; then
relative="docs/${path#"$docs_root"/}"
else
relative="$(basename -- "$path")"
@@ -82,13 +91,19 @@ structured_source = suffix in (".yaml", ".yml", ".json")
python_source = suffix == ".py"
shell_source = suffix in (".sh", ".bash", ".zsh") or not suffix
prose_source = suffix in (".md", ".markdown", ".txt", ".rst")
embedded_python_source = path in (
"/home/donghyeon/workspace/platform/scripts/validate/observability-smoke.sh",
"/home/donghyeon/workspace/platform/scripts/validate/test-scan-platform-sensitive-source.sh",
"/home/donghyeon/workspace/platform/scripts/validate/test-configure-keycloak-grafana-oidc.sh",
"/home/donghyeon/workspace/platform/scripts/bootstrap/create-observability-secrets.sh",
"/home/donghyeon/workspace/platform/scripts/bootstrap/configure-keycloak-grafana-oidc.sh",
)
# Shell files with embedded Python, relative to the checkout that holds this
# scanner (scanner_source is <root>/scripts/validate/<scanner>).
scanner_root = Path(scanner_source).parents[2]
embedded_python_source = path in {
str(scanner_root / relative)
for relative in (
"scripts/validate/observability-smoke.sh",
"scripts/validate/test-scan-platform-sensitive-source.sh",
"scripts/validate/test-configure-keycloak-grafana-oidc.sh",
"scripts/bootstrap/create-observability-secrets.sh",
"scripts/bootstrap/configure-keycloak-grafana-oidc.sh",
)
}
try:
raw = open(path, "rb").read()
@@ -717,6 +732,19 @@ def permitted_reference_container(key, value):
)
def permitted_s3_credentials_reference(path, key, value):
# CloudNativePG barmanObjectStore.s3Credentials: each entry is a
# SecretKeySelector (name and key only), never the value itself.
return (
len(path) >= 1
and path[-1] == "s3Credentials"
and key in ("accessKeyId", "secretAccessKey")
and isinstance(value, dict)
and set(value) == {"name", "key"}
and all(isinstance(value[field], str) and value[field] for field in value)
)
def permitted_webhook_feature(path, key, value):
return (
tuple(str(part) for part in path) == ("operators", "object-store")
@@ -752,6 +780,8 @@ def walk(value, path=()):
pass
elif permitted_reference_container(key, child):
pass
elif permitted_s3_credentials_reference(path, key, child):
pass
elif not isinstance(child, str) or unsafe_scalar(child):
return True
if walk(child, path + (key,)):
@@ -1382,9 +1412,13 @@ scan_platform_sensitive_roots() (
return 1
}
done
[[ "$platform_root" == /* && "$docs_root" == /* && "$platform_root" != "$docs_root" ]] ||
# An empty docs root means a platform-only scan.
local -a scan_roots=("$platform_root")
[[ -z "$docs_root" ]] || scan_roots+=("$docs_root")
[[ "$platform_root" == /* ]] && { [[ -z "$docs_root" ]] ||
[[ "$docs_root" == /* && "$platform_root" != "$docs_root" ]]; } ||
_sensitive_source_fail 'scan roots must be distinct absolute paths' || return
for argument in "$platform_root" "$docs_root"; do
for argument in "${scan_roots[@]}"; do
[[ -d "$argument" && ! -L "$argument" && "$(cd -- "$argument" && pwd -P)" == "$argument" ]] ||
_sensitive_source_fail "scan root is not a physical directory: $argument" || return
done
@@ -1397,7 +1431,7 @@ scan_platform_sensitive_roots() (
printf 'SCANNER_ERROR: file inventory hardening failed\n' >&2
return 1
}
if ! find "$platform_root" "$docs_root" \
if ! find "${scan_roots[@]}" \
\( -type d -path "$platform_root/.superpowers/sdd" -prune \) -o \
\( -type d \( -name .git -o -name .helm \) -prune \) -o \
\( -type f -o -type l \) -print0 >"$list_file"; then
@@ -1432,7 +1466,7 @@ scan_platform_sensitive_roots() (
fi
finding="$(_sensitive_source_python_scan \
"$file" "$test_file" "$PLATFORM_SENSITIVE_SOURCE_EXACT_SCANNER")" || {
"$file" "$test_file" "$PLATFORM_SENSITIVE_SOURCE_SELF")" || {
printf 'SCANNER_ERROR: content scan failed for FILE=%s\n' "$report" >&2
return 1
}
@@ -1505,7 +1539,7 @@ scan_platform_sensitive_roots() (
scan_platform_renderer_secrets() (
set -o pipefail
local platform_root=$1 renderer label output_prefix publishes log file results line rc entry artifact
local helm_bin list_file=''
local helm_bin list_file='' render_root source_copy=''
local failed=false
local -a renderers=(
'phase1|render-phase1.sh|platform-phase1-apply|yes'
@@ -1546,6 +1580,11 @@ scan_platform_renderer_secrets() (
rm -f -- "$list_file"
;;
esac
case "$source_copy" in
/tmp/platform-sensitive-source-render-src.??????)
rm -rf -- "$source_copy"
;;
esac
}
trap cleanup_renderer_scan EXIT HUP INT TERM
@@ -1570,9 +1609,56 @@ scan_platform_renderer_secrets() (
return 1
}
# In a git checkout, render from a private copy of the tracked and not-ignored
# files (minus files deleted from the working tree). Ignored state such as a
# leftover .helm chart cache would otherwise make the renderers refuse to run.
render_root=$platform_root
if [[ -e "$platform_root/.git" ]]; then
for label in git tar; do
command -v "$label" >/dev/null 2>&1 || {
printf 'SCANNER_ERROR: renderer command unavailable: %s\n' "$label" >&2
return 1
}
done
[[ "$(git -C "$platform_root" rev-parse --show-toplevel 2>/dev/null)" == "$platform_root" ]] || {
printf 'SCANNER_ERROR: renderer root is not the top of its git checkout\n' >&2
return 1
}
source_copy="$(mktemp -d /tmp/platform-sensitive-source-render-src.XXXXXX)" || {
printf 'SCANNER_ERROR: renderer source copy allocation failed\n' >&2
return 1
}
chmod 0700 "$source_copy" || return 1
if ! python3 -I - "$platform_root" <<'PY' |
import subprocess
import sys
root = sys.argv[1]
def listed(*options):
result = subprocess.run(
["git", "-C", root, "ls-files", "-z", *options],
check=True,
stdout=subprocess.PIPE,
)
return [entry for entry in result.stdout.split(b"\0") if entry]
keep = set(listed("--cached", "--others", "--exclude-standard")) - set(listed("--deleted"))
for entry in sorted(keep):
sys.stdout.buffer.write(entry + b"\0")
PY
tar -C "$platform_root" --null -T - -cf - | tar -x -C "$source_copy"; then
printf 'SCANNER_ERROR: renderer source copy failed\n' >&2
return 1
fi
render_root=$source_copy
fi
for renderer in "${renderers[@]}"; do
IFS='|' read -r label renderer output_prefix publishes <<<"$renderer"
renderer="$platform_root/scripts/validate/$renderer"
renderer="$render_root/scripts/validate/$renderer"
[[ -f "$renderer" && ! -L "$renderer" ]] || {
printf 'SCANNER_ERROR: required renderer is unsafe: %s\n' "$label" >&2
return 1
@@ -1706,14 +1792,23 @@ scan_platform_renderer_secrets() (
)
scan_platform_sensitive_source_main() {
(( $# == 0 )) || {
local docs_root=$PLATFORM_SENSITIVE_SOURCE_DEFAULT_DOCS_ROOT
if (( $# == 1 )) && [[ "$1" == --platform-only ]]; then
docs_root=''
elif (( $# == 2 )) && [[ "$1" == --docs-root && "$2" == /* ]]; then
docs_root=$2
elif (( $# != 0 )); then
_sensitive_source_usage >&2
return 2
}
fi
if [[ -n "$docs_root" && ! -d "$docs_root" ]]; then
printf 'ERROR: documentation root %s does not exist; pass --docs-root DIR or --platform-only\n' \
"$docs_root" >&2
return 2
fi
scan_platform_sensitive_roots \
"$PLATFORM_SENSITIVE_SOURCE_EXACT_PLATFORM_ROOT" \
"$PLATFORM_SENSITIVE_SOURCE_EXACT_DOCS_ROOT" >/dev/null || return
scan_platform_renderer_secrets "$PLATFORM_SENSITIVE_SOURCE_EXACT_PLATFORM_ROOT" || return
"$PLATFORM_SENSITIVE_SOURCE_SELF_ROOT" "$docs_root" >/dev/null || return
scan_platform_renderer_secrets "$PLATFORM_SENSITIVE_SOURCE_SELF_ROOT" || return
printf 'PLATFORM SENSITIVE SOURCE SCAN PASS\n'
}
@@ -378,6 +378,44 @@ envValueFrom:
YAML
expect_reject 'a secretKeyRef container with a literal child fails closed' \
'platform/bootstrap/reference-container.yaml'
rm -f -- "$WORK/platform/bootstrap/reference-container.yaml"
cat >"$WORK/platform/bootstrap/s3-credentials.yaml" <<'YAML'
spec:
backup:
barmanObjectStore:
s3Credentials:
accessKeyId:
name: platform-postgres-backup-s3
key: ACCESS_KEY_ID
secretAccessKey:
name: platform-postgres-backup-s3
key: ACCESS_SECRET_KEY
YAML
expect_clean 'a CloudNativePG s3Credentials Secret reference remains a reference'
cat >"$WORK/platform/bootstrap/s3-credentials.yaml" <<'YAML'
spec:
s3Credentials:
secretAccessKey:
name: platform-postgres-backup-s3
key: ACCESS_SECRET_KEY
value: should-never-be-printed-s3-reference
YAML
expect_reject 'an s3Credentials reference with an extra literal child fails closed' \
'platform/bootstrap/s3-credentials.yaml'
cat >"$WORK/platform/bootstrap/s3-credentials.yaml" <<'YAML'
spec:
storage:
secretAccessKey:
name: platform-postgres-backup-s3
key: ACCESS_SECRET_KEY
YAML
expect_reject 'a name and key container outside s3Credentials still fails closed' \
'platform/bootstrap/s3-credentials.yaml'
printf 'spec:\n s3Credentials:\n secretAccess%sKey: Actual%s\n' '' 'ProductionSecretValue9283' \
>"$WORK/platform/bootstrap/s3-credentials.yaml"
expect_reject 'a literal s3Credentials secret access key is rejected' \
'platform/bootstrap/s3-credentials.yaml'
new_fixture
cat >"$WORK/platform/bootstrap/webhook-feature.yaml" <<'YAML'
@@ -967,6 +1005,16 @@ label="$(basename -- "$0")"
handoff=no
[[ -z "$out" ]] || handoff=yes
printf '%s\n' "$label${component:+:$component}:handoff=$handoff" >>"$PLATFORM_SENSITIVE_SOURCE_RENDER_LOG"
root="$(cd -- "$(dirname -- "$0")/../.." && pwd -P)"
if [[ "${PLATFORM_SENSITIVE_SOURCE_FAKE_REJECT_CACHE:-0}" != 0 ]] && \
[[ -n "$(find "$root" -name .helm -type d -print -quit)" ]]; then
printf 'REJECT: exact chart cache already exists\n' >&2
exit 23
fi
if [[ -n "${PLATFORM_SENSITIVE_SOURCE_FAKE_REQUIRE_INPUT:-}" && \
! -f "$root/$PLATFORM_SENSITIVE_SOURCE_FAKE_REQUIRE_INPUT" ]]; then
exit 42
fi
emit_phase1_log() {
local publish=$1 name
@@ -1438,6 +1486,41 @@ for injected_renderer in \
fail "$injected_renderer unexpected stdout leaked content"
pass "$injected_renderer stdout is restricted to its bounded grammar"
done
# A git checkout renders from a private copy of its tracked and not-ignored files,
# so an ignored chart cache left in the checkout cannot stop the renderers (the
# real renderers refuse a pre-existing exact cache with RC=23).
command -v git >/dev/null 2>&1 || fail 'git is required for the clean render copy test'
printf '.helm/\n' >"$WORK/platform/.gitignore"
git -C "$WORK/platform" init -q
git -C "$WORK/platform" add -A
git -C "$WORK/platform" -c user.name=fixture -c user.email=fixture@example.invalid \
commit -q -m fixture
mkdir -p "$WORK/platform/services/observability/.helm/charts/stale-1.0.0"
printf 'untracked\n' >"$WORK/platform/services/observability/untracked-input.yaml"
: >"$WORK/renderer.log"
renderer_rc=0
renderer_output="$(PLATFORM_SENSITIVE_SOURCE_FAKE_REJECT_CACHE=1 \
PLATFORM_SENSITIVE_SOURCE_FAKE_REQUIRE_INPUT=services/observability/untracked-input.yaml \
scan_platform_renderer_secrets "$WORK/platform" 2>&1)" || renderer_rc=$?
(( renderer_rc == 0 )) || {
printf '%s\n' "$renderer_output" >&2
fail 'an ignored chart cache in a git checkout blocked the renderers'
}
[[ -d "$WORK/platform/services/observability/.helm/charts/stale-1.0.0" ]] || \
fail 'the renderer scan changed the ignored cache in the scanned checkout'
pass 'a git checkout renders from a clean copy, so an ignored chart cache cannot block it'
pass 'the clean render copy keeps untracked files that git does not ignore'
rm -f -- "$WORK/platform/services/observability/untracked-input.yaml"
rm -f -- "$WORK/platform/scripts/validate/render-phase2.sh"
renderer_rc=0
renderer_output="$(scan_platform_renderer_secrets "$WORK/platform" 2>&1)" || renderer_rc=$?
(( renderer_rc != 0 )) || fail 'a renderer deleted from the working tree was still run'
[[ "$renderer_output" == *'required renderer is unsafe: phase2'* ]] || {
printf '%s\n' "$renderer_output" >&2
fail 'a renderer deleted from the working tree did not fail at the renderer check'
}
pass 'the clean render copy leaves out files deleted from the working tree'
unset PLATFORM_SENSITIVE_SOURCE_RENDER_LOG
new_fixture
@@ -1468,6 +1551,39 @@ output="$(run_scanner 2>&1)" || scanner_error_rc=$?
fail 'scanner error did not fail closed'
}
pass 'rg errors fail closed'
PLATFORM_SENSITIVE_SOURCE_TEST_PATH=''
[[ "$PLATFORM_SENSITIVE_SOURCE_SELF_ROOT" == "$ROOT" ]] || \
fail 'the scanner does not scan the checkout it lives in'
pass 'the scanner scans the checkout it lives in, not a fixed path'
main_rc=0
main_output="$(scan_platform_sensitive_source_main --unknown 2>&1)" || main_rc=$?
(( main_rc == 2 )) || fail 'an unknown entrypoint argument was accepted'
main_rc=0
main_output="$(scan_platform_sensitive_source_main --docs-root relative/docs 2>&1)" || main_rc=$?
(( main_rc == 2 )) || fail 'a relative docs root was accepted'
new_fixture
main_rc=0
main_output="$(scan_platform_sensitive_source_main --docs-root "$WORK/missing-docs" 2>&1)" || main_rc=$?
(( main_rc == 2 )) || fail 'a missing docs root was accepted'
[[ "$main_output" == *'--platform-only'* ]] || \
fail 'a missing docs root did not name --platform-only'
pass 'the entrypoint rejects unknown, relative and missing docs roots before scanning'
printf 'ordinary input\n' >"$WORK/platform/bootstrap/ordinary.md"
platform_only_output="$(scan_platform_sensitive_roots "$WORK/platform" '' 2>&1)" || {
printf '%s\n' "$platform_only_output" >&2
fail 'a platform-only root scan failed'
}
[[ "$platform_only_output" == *'PLATFORM SENSITIVE SOURCE SCAN PASS'* ]] || \
fail 'a platform-only root scan did not emit the PASS marker'
printf 'Slack: https://hooks.%s/services/T1/B2/C3\n' 'slack.com' \
>"$WORK/platform/bootstrap/live.env"
platform_only_rc=0
platform_only_output="$(scan_platform_sensitive_roots "$WORK/platform" '' 2>&1)" || platform_only_rc=$?
(( platform_only_rc != 0 )) || fail 'a platform-only root scan accepted a Slack webhook'
[[ "$platform_only_output" == *'platform/bootstrap/live.env'* ]] || \
fail 'a platform-only root scan did not report the filename'
pass 'a platform-only root scan still scans the platform root'
cleanup_before="$(
find /tmp -maxdepth 1 -type d -name 'platform-sensitive-source-test.??????' \