fix(ci): FE-GATE-013 provider 시험의 exit=125 묶음을 고친다 #58
@@ -50,6 +50,7 @@ Gitea runner에서 merge 단계를 초록으로 만들고, worker를 늘리지
|
||||
| HTTP 실행기의 deadline 분류 시험이 가짜 시계로 대기 지점에서 deadline을 낸다. 실제 30ms deadline이 부하에서 backoff 전에 지나 TIMEOUT 대신 전송 실패로 끝났다 | `tests/unit/http-execution-v3.test.ts` | `37a85004` |
|
||||
| 위험 커버리지가 `"use strict"` 밖의 지시문(`"use client"` 등)을 문장으로 센다. V8 coverage가 그렇게 센다(`check:v8-coverage-counter-semantics` fixture에 지시문 모듈 둘) | `scripts/lib/risk-coverage.ts`, `scripts/lib/v8-coverage-counter-semantics.ts` | `0c3c9d33` |
|
||||
| e2e fixture build를 브라우저 project마다가 아니라 run에 한 번 만든다. global setup이 공유 폴더를 만들고 처음 닿은 worker만 build한다. 자기 spec에서 `prepareServiceWorkerBuilds`를 부르면 `key`를 준다 | `playwright.config.ts`, `tests/e2e/support/shared-fixture-builds.ts`, `tests/e2e/support/fixture-builds.ts`, `tests/e2e/support/service-worker-builds.ts` | `c6d7f43a` |
|
||||
| FE-GATE-013의 시스템 컨테이너가 systemd 전에 root cgroup 제어기를 켠다. 부팅 중 `docker exec`가 root cgroup에 남으면 systemd가 user manager에 제어기를 넘기지 못해 provider 시험 18개가 exit 125로 떨어졌다(desktop 4회 중 1회). 사전 점검이 user manager의 cpu·memory·pids 제어기를 보고, provider 실패 메시지에 우리 단계가 쓴 고정 이유 줄을 남긴다 | `scripts/run-in-system-container.sh`, `scripts/check-system-test-prerequisites.ts`, `scripts/lib/provider-sandbox-diagnostics.ts`, `scripts/run-and-validate-provider.ts`, `scripts/lib/provider-scope-wrapper.ts` | `6d6fd61a`, `4e1a385f`, `13b40934` |
|
||||
|
||||
## 미출시 — `8327e221` → 하네스와 작업 매뉴얼
|
||||
|
||||
|
||||
@@ -6,3 +6,4 @@
|
||||
{"schema": 1, "kind": "actual", "task": "w2-fe-ci-perf", "recorded_at": "2026-11-12T22:25+09:00", "transcript": "00d79702-1dea-4e11-b79a-f021d0d9f4bb", "minutes": 1463, "active_minutes": 308, "calls": 447, "output_tokens": 413297, "cache_read_tokens": 246773496, "max_context": 943246, "within": "p30", "result": "partial", "commits": ["ddfcb6f6", "79a225b6", "0d8e4f5a", "b4210753", "66138440", "1d3eb901", "92abfd20", "1f09c1cc", "27876ffc", "35b60a04", "f90ffb6c", "f6e2560e", "ba72b865"], "why": "작업 중 시간은 p30 안이지만 일이 끝나지 않고 인계됐다(PR #19 Draft). 벽시계의 대부분은 runner 대기열(백엔드 정기 lane 뒤)과 desktop heavy 잠금 대기다. 남은 것: runner 초록 확인, concurrency, firefox 흔들림, coverage 경로, 느린 시험 둘. 세션 기록으로 다시 셌고 인계 절을 쓴 22:10 KST 뒤까지 들어 있다. 기록은 이어받은 세션이 남긴다"}
|
||||
{"schema": 1, "kind": "estimate", "task": "w2-fe2-takeover", "recorded_at": "2026-11-12T22:25+09:00", "transcript": "fa9a12aa-7abc-46ca-a5fd-ee00cab98618", "started_at": "2026-11-12T21:05+09:00", "base": "ba72b865", "task_kind": "improve+operate", "size": "large", "minutes": {"p30": 180, "p50": 300, "p80": 480}, "calls": {"p30": 250, "p50": 400, "p80": 650}, "output_tokens": {"p30": 150000, "p50": 250000, "p80": 400000}, "basis": "PR #19 이어받기(인계 절 2026-11-12 22:10). concurrency·coverage 경로·느린 시험 둘은 각 30~60분(작음, 선례 W2-BE concurrency). firefox 008 흔들림은 원인 미확정이라 1~4시간. runner 1대를 백엔드와 같이 써 PR run 하나가 대기 포함 1시간 이상(W2-BE 기록: PR run이 2~4시간 뒤로 밀림), 초록 3회와 실패-고침 1~2번을 p80에 넣었다. 첫 5분 뒤에 적었다(인계 대기 중 시작)"}
|
||||
{"schema": 1, "kind": "estimate", "task": "r9-fe-group-A", "recorded_at": "2026-11-13T18:50+09:00", "transcript": "66f9f89d-a13f-4cee-a2bf-c3c528c6bfb2", "started_at": "2026-11-13T17:57+09:00", "base": "ee95573f", "task_kind": "improve", "size": "large", "minutes": {"p30": 300, "p50": 420, "p80": 600}, "calls": {"p30": 350, "p50": 500, "p80": 750}, "output_tokens": {"p30": 200000, "p50": 300000, "p80": 450000}, "basis": "round9 그룹 A 발견 22건(화면·토큰·story·검사·접근성 기록)을 PR 셋으로. 바꿀 파일 약 60개, CSS 선언 118개 토큰화와 story 37개가 가장 크다. desktop heavy 잠금을 다른 세션 넷과 같이 써 실행마다 대기 10~30분, merge 단계 셋(각 약 21분)과 골든 갱신·변이 실험, runner run 셋(각 약 23분)을 p80에 넣었다. 선례 r9-fe F4a·F5(묶음당 2~4시간). 착수 약 50분 뒤 발견 원문을 다 읽고 PR1 고침을 쓰던 중에 적었다"}
|
||||
{"schema": 1, "kind": "estimate", "task": "fe-ci-fix3", "recorded_at": "2026-11-14T06:12+09:00", "transcript": "b28bbd6f-b8e2-4b03-998f-95cc55c1ef86", "started_at": "2026-11-14T04:50+09:00", "base": "e98d8ee9", "task_kind": "improve", "size": "medium", "minutes": {"p30": 150, "p50": 210, "p80": 300}, "calls": {"p30": 180, "p50": 260, "p80": 380}, "output_tokens": {"p30": 70000, "p50": 110000, "p80": 170000}, "basis": "FE-GATE-013 exit=125 묶음(18개) 진단·원인·고침 PR 하나. 원인 미확정, 실패율 약 25%라 desktop 파일 전체 반복(회당 약 2.5분) 16회를 고치기 전·뒤 두 번, merge 단계(약 25분) 한두 번, 새 눈 리뷰와 확인 리뷰, runner run 하나(약 25분, 1대 공유 대기)를 넣었다. 선례 fe-ci-fix2(작음, 예측 p50 150분). 착수 때 적지 못하고 원인을 확정한 뒤(착수 약 80분) 적어 이미 쓴 시간에 기댄다"}
|
||||
|
||||
@@ -8,3 +8,5 @@
|
||||
{"date": "2026-11-13", "test": "tests/browser-capabilities/indexeddb-runtime.spec.ts:256", "category": "resource-io", "first_run": "/tmp/claude-gates/ci-pr-gate-selection-merge/gate-artifacts/FE-GATE-008.txt", "verdict": "[firefox] desktop merge(트리 148e68f4, #47 없음)에서 30초 timeout 1회. 같은 게이트가 Firefox 프로필을 RAM에 둔 #47 트리에서는 69개 전부 통과. Firefox IndexedDB·OPFS가 프로필 디스크를 기다린 것으로 봄", "quarantined_until": "", "fixed_by": "프론트 #47"}
|
||||
{"date": "2026-11-13", "test": "tests/browser-capabilities/local-draft-composition.spec.ts:10", "category": "resource-io", "first_run": "/tmp/claude-gates/ci-pr-gate-selection-merge/gate-artifacts/FE-GATE-008.txt", "verdict": "[firefox] desktop merge(트리 148e68f4, #47 없음)에서 30초 timeout 1회. 같은 게이트가 Firefox 프로필을 RAM에 둔 #47 트리에서는 69개 전부 통과. Firefox IndexedDB·OPFS가 프로필 디스크를 기다린 것으로 봄", "quarantined_until": "", "fixed_by": "프론트 #47"}
|
||||
{"date": "2026-11-13", "test": "tests/browser-capabilities/opfs-runtime.spec.ts:338", "category": "resource-io", "first_run": "/tmp/claude-gates/ci-pr-gate-selection-merge/gate-artifacts/FE-GATE-008.txt", "verdict": "[firefox] desktop merge(트리 148e68f4, #47 없음)에서 30초 timeout 1회. 같은 게이트가 Firefox 프로필을 RAM에 둔 #47 트리에서는 69개 전부 통과. Firefox IndexedDB·OPFS가 프로필 디스크를 기다린 것으로 봄", "quarantined_until": "", "fixed_by": "프론트 #47"}
|
||||
{"date": "2026-11-13", "test": "tests/system/ci-runner/ci-artifact-contract.test.ts:671", "category": "concurrency-order", "first_run": "/tmp/claude-gates/w2fe2-t08m/sys-2.json", "verdict": "원인 확정(2026-11-14). desktop 시스템 컨테이너 파일 전체 4회 중 1회 79번째(그 파일에서 처음 provider를 띄우는 시험)부터 18개가 exit=125, 진단을 넣은 뒤 메시지는 `provider cgroup enforcement failed: cannot read memory.max`. scripts/run-in-system-container.sh가 부팅 중에 띄운 docker exec가 systemd가 root cgroup에 제어기를 켜기 전에 root cgroup에 들어가, cgroup v2가 그 cgroup에 제어기 켜기를 막아 cpu·memory·pids가 user manager까지 넘어가지 않았다. 부팅 직후 exec 15개 탐침이 6/6 결정적으로 재현. 고친 트리는 탐침 6/6 정상, 파일 전체 16회 0/16", "quarantined_until": "", "fixed_by": "프론트 #58"}
|
||||
{"date": "2026-11-14", "test": "tests/system/ci-runner/ci-artifact-contract.test.ts:983", "category": "concurrency-order", "first_run": "/tmp/claude-gates/fe125-loop1-runs/run-1.log", "verdict": "desktop 시스템 컨테이너 파일 전체 4회 중 1회 실패(시작 load 15.34). 기대한 `provider guardian failed|SIGKILL` 대신 supervisor가 guardian commit 쓰기(scripts/lib/provider-guardian-client.ts:286)의 처리되지 않은 `write EPIPE`로 죽었다. 원인 조사 전", "quarantined_until": "", "fixed_by": ""}
|
||||
|
||||
@@ -239,10 +239,22 @@ runner의 Docker proxy가 안쪽 dind daemon에 컨테이너를 띄운다.
|
||||
release 단계 provider job은 아직 job 컨테이너에서 돈다.
|
||||
- runner가 privileged 컨테이너를 막는 설정(rootless dind 등)으로 바뀌면 이
|
||||
길이 깨진다. 그때는 위 조건을 갖춘 전용 runner(VM 등)로 옮긴다.
|
||||
- 컨테이너의 첫 프로세스는 systemd를 띄우기 전에 root cgroup의 프로세스를
|
||||
`init.scope`로 옮기고 root에 cgroup 제어기를 켠다. 그러지 않으면 부팅 중에
|
||||
닿은 `docker exec`가 root cgroup에 들어가 systemd가 제어기를 넘기지 못하고,
|
||||
provider scope에 `memory.max`가 없어 아래 확인이 provider를 exit 125로
|
||||
멈춘다(2026-11-14 desktop 4회 중 1회). `test:system`의 사전 점검
|
||||
(`scripts/check-system-test-prerequisites.ts`)이 user manager에 cpu·memory·pids
|
||||
제어기가 넘어왔는지 보고, 없으면 빠진 이름을 대며 실패한다.
|
||||
|
||||
각 provider는 고유한 collected user scope에서 실행된다. Supervisor는 실행 전에
|
||||
실제 cgroup membership과 `memory.max=1073741824`, `memory.swap.max=0`,
|
||||
`pids.max=64`, `cpu.max="100000 100000"`을 확인한다. 내부 process에는 core 0,
|
||||
`pids.max=64`, `cpu.max="100000 100000"`을 확인한다. 확인이나 scope 준비가
|
||||
provider 전에 실패하면 supervisor 오류 끝에 이유가 붙는다
|
||||
(`sandbox reported: provider cgroup enforcement failed: ...`). 남기는 줄은 8줄까지,
|
||||
우리 단계가 쓰는 고정 문구와 sandbox 도구의 접두 줄(`bwrap:` 등)뿐이다. provider
|
||||
출력은 자격 증명을 실을 수 있어 그 밖의 줄은 남기지 않는다
|
||||
(`scripts/lib/provider-sandbox-diagnostics.ts`). 내부 process에는 core 0,
|
||||
file-size 8,388,607 bytes, open FD 64, CPU 1,200초 상한도 적용된다.
|
||||
같은 UID 전체에 합산되는 `RLIMIT_NPROC`로 provider별 32개를 보장한다고 주장하지
|
||||
않으며 aggregate PID authority는 cgroup `TasksMax=64`다.
|
||||
|
||||
@@ -71,10 +71,13 @@ export async function findSystemTestPrerequisiteFailures(
|
||||
}
|
||||
}
|
||||
|
||||
let cgroupV2 = false;
|
||||
try {
|
||||
const controllers = await host.readFile("/sys/fs/cgroup/cgroup.controllers");
|
||||
if (controllers.trim().length === 0) {
|
||||
failures.push("cgroup v2 controllers are unavailable");
|
||||
} else {
|
||||
cgroupV2 = true;
|
||||
}
|
||||
} catch {
|
||||
failures.push("cgroup v2 is required at /sys/fs/cgroup/cgroup.controllers");
|
||||
@@ -87,6 +90,10 @@ export async function findSystemTestPrerequisiteFailures(
|
||||
}
|
||||
}
|
||||
|
||||
// The user manager's cgroup path below is a cgroup v2 path; without v2 it would name the wrong cause.
|
||||
const controllerFailure = cgroupV2 ? await findProviderScopeControllerFailure(host) : undefined;
|
||||
if (controllerFailure !== undefined) failures.push(controllerFailure);
|
||||
|
||||
if (executable.get(BWRAP)) {
|
||||
const failure = await findProviderSandboxNamespaceFailure(host);
|
||||
if (failure !== undefined) failures.push(failure);
|
||||
@@ -141,6 +148,35 @@ async function namespaceFailure(
|
||||
].join("\n");
|
||||
}
|
||||
|
||||
/** The controllers every provider scope is limited by (scripts/lib/provider-cgroup.ts). */
|
||||
const PROVIDER_SCOPE_CONTROLLERS = Object.freeze(["cpu", "memory", "pids"] as const);
|
||||
|
||||
/**
|
||||
* Whether the systemd user manager that provider scopes run under was handed
|
||||
* the cgroup controllers their limits need. A manager without them passed
|
||||
* every check above, and then each provider stopped with `provider cgroup
|
||||
* enforcement failed: cannot read memory.max` (2026-11-14: a `docker exec`
|
||||
* sat in the system container's root cgroup while systemd started, so systemd
|
||||
* delegated nothing).
|
||||
*/
|
||||
async function findProviderScopeControllerFailure(host: SystemTestHost): Promise<string | undefined> {
|
||||
const status = await host.readFile("/proc/self/status").catch(() => "");
|
||||
const uid = /^Uid:\s+(\d+)\s/mu.exec(status)?.[1];
|
||||
if (uid === undefined) return "this process's uid is unreadable at /proc/self/status";
|
||||
const manager = `/sys/fs/cgroup/user.slice/user-${uid}.slice/user@${uid}.service`;
|
||||
let delegated: string[];
|
||||
try {
|
||||
delegated = (await host.readFile(`${manager}/cgroup.controllers`)).trim().split(/\s+/u);
|
||||
} catch {
|
||||
return `a systemd user manager is required at ${manager}`;
|
||||
}
|
||||
const missing = PROVIDER_SCOPE_CONTROLLERS.filter((controller) => !delegated.includes(controller));
|
||||
return missing.length === 0
|
||||
? undefined
|
||||
: `the systemd user manager at ${manager} has no ${missing.join(", ")} cgroup controller ` +
|
||||
`for provider scopes (it has: ${delegated.join(" ") || "none"})`;
|
||||
}
|
||||
|
||||
export function formatSystemTestPrerequisiteReport(failures: readonly string[]): string {
|
||||
return [
|
||||
"System/CI-runner test prerequisites are unavailable:",
|
||||
|
||||
@@ -0,0 +1,93 @@
|
||||
/**
|
||||
* Lines the sandbox tooling itself emits, kept so a launch failure can say
|
||||
* why. Everything else the child writes is provider output and may carry
|
||||
* credentials, so it is counted and discarded as before.
|
||||
*
|
||||
* Without this a sandbox that never started reported only `exit=1`, and the
|
||||
* actual cause — `bwrap: loopback: Failed RTM_NEWADDR: Operation not
|
||||
* permitted` on a host with `kernel.apparmor_restrict_unprivileged_userns=1`
|
||||
* — was invisible. That turned a host restriction into an unexplained
|
||||
* product failure.
|
||||
*/
|
||||
const SANDBOX_TOOL = /^(?:bwrap|prlimit|systemd-run|systemctl):\s.*$/u;
|
||||
/**
|
||||
* Our own launch stages exit 125 before the provider runs: the cgroup gate
|
||||
* (provider-cgroup.ts ENFORCEMENT_GATE) and the scope wrapper
|
||||
* (provider-scope-wrapper.ts). Their lines are matched as whole templates,
|
||||
* not by prefix, so provider output that merely starts the same way is still
|
||||
* discarded. Only the liveness line carries free text, Node's own error for
|
||||
* descriptor 0.
|
||||
*/
|
||||
/** An errno or Node error name, the only free part of the scope wrapper's stop line. */
|
||||
const ERROR_CODE = String.raw`(?:E[A-Z0-9]{1,15}|ERR_[A-Z0-9_]{1,40})`;
|
||||
const CGROUP_FILE = String.raw`(?:memory\.max|memory\.swap\.max|pids\.max|cpu\.max)`;
|
||||
const CGROUP_VALUE = String.raw`(?:max|[0-9]{1,20})(?: [0-9]{1,20})?`;
|
||||
const LAUNCH_STAGE = [
|
||||
new RegExp(
|
||||
String.raw`^provider cgroup enforcement failed: (?:unified cgroup v2 membership is required|` +
|
||||
String.raw`unit membership is invalid|cannot read ${CGROUP_FILE}|` +
|
||||
String.raw`${CGROUP_FILE} is (?:${CGROUP_VALUE})?, expected ${CGROUP_VALUE})$`,
|
||||
"u",
|
||||
),
|
||||
new RegExp(
|
||||
String.raw`^provider scope stopped before launch: (?:supervisor channel ended|` +
|
||||
String.raw`supervisor channel error (?:${ERROR_CODE}|UNKNOWN)|protocol failure)$`,
|
||||
"u",
|
||||
),
|
||||
/^provider scope (?:received trailing protocol bytes|frame length is invalid|frame has trailing bytes)$/u,
|
||||
/^provider scope liveness channel is unavailable: [\x20-\x7e]{1,200}$/u,
|
||||
] as const;
|
||||
const MAX_LINES = 8;
|
||||
/** A diagnostic line is short; anything longer is provider output. */
|
||||
const MAX_LINE_CHARACTERS = 512;
|
||||
|
||||
export type ProviderSandboxDiagnostics = Readonly<{
|
||||
consume(stream: "stdout" | "stderr", chunk: Buffer | string): void;
|
||||
lines(): readonly string[];
|
||||
}>;
|
||||
|
||||
/** Whether the scope wrapper may name this error code in its stop line. */
|
||||
export function isProviderScopeErrorCode(code: string): boolean {
|
||||
return new RegExp(`^${ERROR_CODE}$`, "u").test(code);
|
||||
}
|
||||
|
||||
export function isProviderSandboxDiagnostic(line: string): boolean {
|
||||
return SANDBOX_TOOL.test(line) || LAUNCH_STAGE.some((pattern) => pattern.test(line));
|
||||
}
|
||||
|
||||
/**
|
||||
* Reassembles each stream into whole lines before matching, so a line split
|
||||
* across two reads is neither missed nor kept in a truncated form. A line is
|
||||
* only judged once its newline arrives.
|
||||
*/
|
||||
export function createProviderSandboxDiagnostics(): ProviderSandboxDiagnostics {
|
||||
const partial = { stdout: "", stderr: "" };
|
||||
const overlong = { stdout: false, stderr: false };
|
||||
const kept: string[] = [];
|
||||
const keep = (line: string): void => {
|
||||
if (
|
||||
kept.length < MAX_LINES && line.length <= MAX_LINE_CHARACTERS &&
|
||||
!kept.includes(line) && isProviderSandboxDiagnostic(line)
|
||||
) {
|
||||
kept.push(line);
|
||||
}
|
||||
};
|
||||
return Object.freeze({
|
||||
consume: (stream: "stdout" | "stderr", chunk: Buffer | string) => {
|
||||
const pieces = String(chunk).split("\n");
|
||||
const tail = pieces.pop() ?? "";
|
||||
for (const piece of pieces) {
|
||||
if (!overlong[stream]) keep(partial[stream] + piece);
|
||||
partial[stream] = "";
|
||||
overlong[stream] = false;
|
||||
}
|
||||
if (overlong[stream]) return;
|
||||
partial[stream] += tail;
|
||||
if (partial[stream].length > MAX_LINE_CHARACTERS) {
|
||||
partial[stream] = "";
|
||||
overlong[stream] = true;
|
||||
}
|
||||
},
|
||||
lines: () => Object.freeze([...kept]),
|
||||
});
|
||||
}
|
||||
@@ -4,6 +4,7 @@ import { Socket } from "node:net";
|
||||
|
||||
import { traceOwnProviderProcess, traceSpawnedProviderProcess } from "./provider-process-trace.ts";
|
||||
import { cleanupOwnedProviderReport } from "./provider-raw-cleanup.ts";
|
||||
import { isProviderScopeErrorCode } from "./provider-sandbox-diagnostics.ts";
|
||||
|
||||
const MAX_FRAME_BYTES = 16_777_216;
|
||||
const reportIdentity = parseReportIdentity(process.argv.slice(2));
|
||||
@@ -13,6 +14,8 @@ let expectedBytes: number | undefined;
|
||||
let provider: ReturnType<typeof spawn> | undefined;
|
||||
let providerClosed = false;
|
||||
let livenessLost = false;
|
||||
/** Why the scope stopped before it launched the provider, for its exit-125 line. */
|
||||
let stopReason = "supervisor channel ended";
|
||||
/**
|
||||
* The supervisor keeps this pipe open for the scope's whole life — that is how
|
||||
* parent loss is observed — and only ever writes one frame into it.
|
||||
@@ -47,8 +50,8 @@ liveness.on("data", (chunk: Buffer | string) => {
|
||||
}
|
||||
});
|
||||
|
||||
liveness.once("end", () => terminateForParentLoss());
|
||||
liveness.once("error", () => terminateForParentLoss());
|
||||
liveness.once("end", () => terminateForParentLoss("supervisor channel ended"));
|
||||
liveness.once("error", (error) => terminateForParentLoss(`supervisor channel error ${errorCode(error)}`));
|
||||
|
||||
function launchProvider(payload: Buffer): void {
|
||||
const frame = parseFrame(payload);
|
||||
@@ -96,9 +99,10 @@ async function finishProvider(
|
||||
process.exit(code ?? 1);
|
||||
}
|
||||
|
||||
function terminateForParentLoss(): void {
|
||||
function terminateForParentLoss(reason: string): void {
|
||||
if (livenessLost) return;
|
||||
livenessLost = true;
|
||||
stopReason = reason;
|
||||
if (!provider || providerClosed) {
|
||||
void cleanupAfterParentLossAndExit();
|
||||
return;
|
||||
@@ -117,12 +121,15 @@ async function cleanupAfterParentLossAndExit(): Promise<void> {
|
||||
writeSync(2, `${error instanceof Error ? error.message : String(error)}\n`);
|
||||
}
|
||||
closeLivenessInput();
|
||||
// The supervisor keeps only fixed launch-stage lines (provider-sandbox-diagnostics.ts);
|
||||
// without this one an exit 125 here was indistinguishable from the cgroup gate's.
|
||||
if (!provider) writeSync(2, `provider scope stopped before launch: ${stopReason}\n`);
|
||||
process.exit(125);
|
||||
}
|
||||
|
||||
function terminateForProtocolFailure(message: string): void {
|
||||
writeSync(2, `${message}\n`);
|
||||
terminateForParentLoss();
|
||||
terminateForParentLoss("protocol failure");
|
||||
}
|
||||
|
||||
function openLivenessChannel(): Socket {
|
||||
@@ -212,6 +219,11 @@ function signalNumber(signal: NodeJS.Signals): number {
|
||||
return signal === "SIGKILL" ? 9 : signal === "SIGXCPU" ? 24 : 1;
|
||||
}
|
||||
|
||||
function errorCode(error: unknown): string {
|
||||
const code = error && typeof error === "object" && "code" in error ? String(error.code) : "";
|
||||
return isProviderScopeErrorCode(code) ? code : "UNKNOWN";
|
||||
}
|
||||
|
||||
function hasErrorCode(error: unknown, code: string): boolean {
|
||||
return Boolean(error && typeof error === "object" && "code" in error && error.code === code);
|
||||
}
|
||||
|
||||
@@ -30,6 +30,7 @@ import {
|
||||
type ProviderGuardianLease,
|
||||
} from "./lib/provider-guardian-client.ts";
|
||||
import { createProviderOutputLimiter } from "./lib/provider-output-limiter.ts";
|
||||
import { createProviderSandboxDiagnostics } from "./lib/provider-sandbox-diagnostics.ts";
|
||||
import { findProviderSandboxNamespaceFailure } from "./check-system-test-prerequisites.ts";
|
||||
import { PROVIDER_SANDBOX_NAMESPACE_ARGUMENTS } from "./lib/provider-sandbox-namespaces.ts";
|
||||
import {
|
||||
@@ -400,30 +401,15 @@ async function waitForProvider(
|
||||
PROVIDER_MAX_OUTPUT_BYTES,
|
||||
() => terminate("output"),
|
||||
);
|
||||
/**
|
||||
* Lines the sandbox tooling itself emits, kept so a launch failure can say
|
||||
* why. Everything else the child writes is provider output and may carry
|
||||
* credentials, so it is counted and discarded as before.
|
||||
*
|
||||
* Without this a sandbox that never started reported only `exit=1`, and the
|
||||
* actual cause — `bwrap: loopback: Failed RTM_NEWADDR: Operation not
|
||||
* permitted` on a host with `kernel.apparmor_restrict_unprivileged_userns=1`
|
||||
* — was invisible. That turned a host restriction into an unexplained
|
||||
* product failure.
|
||||
*/
|
||||
const SANDBOX_DIAGNOSTIC = /^(?:bwrap|prlimit|systemd-run|systemctl):\s.*$/gmu;
|
||||
const sandboxDiagnostics: string[] = [];
|
||||
const capture = (chunk: Buffer | string): void => {
|
||||
for (const line of String(chunk).matchAll(SANDBOX_DIAGNOSTIC)) {
|
||||
if (sandboxDiagnostics.length < 8 && !sandboxDiagnostics.includes(line[0])) {
|
||||
sandboxDiagnostics.push(line[0]);
|
||||
}
|
||||
}
|
||||
// Only lines our own launch stages write survive (lib/provider-sandbox-diagnostics.ts).
|
||||
const sandboxDiagnostics = createProviderSandboxDiagnostics();
|
||||
const capture = (stream: "stdout" | "stderr") => (chunk: Buffer | string): void => {
|
||||
sandboxDiagnostics.consume(stream, chunk);
|
||||
if (termination) return;
|
||||
outputLimiter.consume(chunk);
|
||||
};
|
||||
child.stdout?.on("data", capture);
|
||||
child.stderr?.on("data", capture);
|
||||
child.stdout?.on("data", capture("stdout"));
|
||||
child.stderr?.on("data", capture("stderr"));
|
||||
const timeout = setTimeout(() => terminate("timeout"), timeoutMs);
|
||||
try {
|
||||
const first = await Promise.race([
|
||||
@@ -475,8 +461,8 @@ async function waitForProvider(
|
||||
throw new Error(
|
||||
`sandboxed external provider failed: exit=${result.code ?? "none"}, ` +
|
||||
`signal=${result.signal ?? "none"}` +
|
||||
(sandboxDiagnostics.length > 0
|
||||
? `; sandbox reported: ${sandboxDiagnostics.join("; ")}`
|
||||
(sandboxDiagnostics.lines().length > 0
|
||||
? `; sandbox reported: ${sandboxDiagnostics.lines().join("; ")}`
|
||||
: ""),
|
||||
);
|
||||
}
|
||||
|
||||
@@ -42,9 +42,32 @@ if ! $docker image inspect "$tag" > /dev/null 2>&1; then
|
||||
say "building $tag"
|
||||
$docker build -q -t "$tag" -f "$dockerfile" config/ci > /dev/null || fail "image build failed"
|
||||
fi
|
||||
cid=$($docker run -d --privileged --cgroupns=private --tmpfs /run --tmpfs /run/lock "$tag") \
|
||||
# systemd hands the cgroup controllers the provider scope needs (cpu, memory, pids) down from the container's
|
||||
# root cgroup. A `docker exec` that arrives while systemd starts, before it has enabled controllers there,
|
||||
# joins that root cgroup, and while any process sits in it the kernel refuses to enable controllers on it
|
||||
# (cgroup v2: a cgroup that hands controllers to children holds no processes). systemd then delegates
|
||||
# nothing, every later exec joins the root cgroup too, and each provider scope lacks memory.max, so the
|
||||
# provider cgroup gate exits 125 (1 run in 4 on desktop, 2026-11-14). So the container's first process
|
||||
# moves every process out of the root cgroup and enables its controllers before it execs systemd, as
|
||||
# moby's hack/dind does. From then on an exec cannot join the root cgroup and lands in PID 1's.
|
||||
boot='cg=/sys/fs/cgroup
|
||||
controllers=$(cat $cg/cgroup.controllers 2> /dev/null) || controllers=
|
||||
if [ -n "$controllers" ]; then
|
||||
mkdir -p $cg/init.scope
|
||||
enable=$(printf "+%s " $controllers)
|
||||
tries=0
|
||||
until { xargs -rn1 < $cg/cgroup.procs > $cg/init.scope/cgroup.procs; echo "$enable" > $cg/cgroup.subtree_control; } 2> /dev/null; do
|
||||
tries=$((tries + 1))
|
||||
[ $tries -lt 50 ] || { echo "system-container: cannot enable the root cgroup controllers" >&2; break; }
|
||||
sleep 0.1
|
||||
done
|
||||
fi
|
||||
exec /sbin/init'
|
||||
cid=$($docker run -d --privileged --cgroupns=private --tmpfs /run --tmpfs /run/lock "$tag" /bin/sh -c "$boot") \
|
||||
|| fail "container did not start"
|
||||
cleanup() {
|
||||
# The boot step writes to the container log only when it gave up; say so whatever ends the run.
|
||||
$docker logs "$cid" 2>&1 | grep '^system-container:' >&2 || :
|
||||
if [ "${SYSTEM_CONTAINER_KEEP:-0}" = 1 ]; then say "kept container $cid"; else $docker rm -f "$cid" > /dev/null 2>&1 || :; fi
|
||||
}
|
||||
trap cleanup EXIT
|
||||
|
||||
@@ -0,0 +1,147 @@
|
||||
import { spawnSync } from "node:child_process";
|
||||
import { mkdtemp, rm, writeFile } from "node:fs/promises";
|
||||
import { tmpdir } from "node:os";
|
||||
import path from "node:path";
|
||||
|
||||
import { describe, expect, it } from "vitest";
|
||||
|
||||
import {
|
||||
formatProviderCgroupUnitName,
|
||||
systemdRunProviderArguments,
|
||||
} from "../../scripts/lib/provider-cgroup.ts";
|
||||
import { createProviderSandboxDiagnostics } from "../../scripts/lib/provider-sandbox-diagnostics.ts";
|
||||
import { childProcessTimeout } from "../helpers/child-process-timeout.ts";
|
||||
|
||||
const CREDENTIAL = "provider-credential-must-not-reach-ci";
|
||||
const GATE_LINE = "provider cgroup enforcement failed: ";
|
||||
|
||||
/** The command systemd-run starts in the scope: `/bin/sh -eu -c <gate> <unit> …`. */
|
||||
function gateShell(): string[] {
|
||||
const unit = formatProviderCgroupUnitName("vulnerability", 4242, "0".repeat(24));
|
||||
const systemdRun = systemdRunProviderArguments(unit, 1_000, 1, "/bin/node", "/wrapper.ts", "/r", 1, 1);
|
||||
return systemdRun.slice(systemdRun.indexOf("--") + 1);
|
||||
}
|
||||
|
||||
describe("provider sandbox diagnostics", () => {
|
||||
it("keeps the line the real cgroup gate writes when it refuses a scope", () => {
|
||||
// Outside a provider scope the gate's own membership check fails, which is
|
||||
// the same shell, text, and exit code a broken scope produces.
|
||||
const shell = gateShell();
|
||||
const gate = spawnSync(shell[0]!, shell.slice(1), { encoding: "utf8" });
|
||||
|
||||
expect(gate.status).toBe(125);
|
||||
expect(gate.stderr).toMatch(/^provider cgroup enforcement failed: \S/u);
|
||||
const diagnostics = createProviderSandboxDiagnostics();
|
||||
diagnostics.consume("stderr", gate.stderr);
|
||||
expect(diagnostics.lines()).toEqual([gate.stderr.trimEnd()]);
|
||||
}, childProcessTimeout(4));
|
||||
|
||||
it("keeps the line the real gate check writes for every unreadable or wrong limit", async () => {
|
||||
// The gate's own limit check and its own calls, run against a stand-in
|
||||
// cgroup directory: 2026-11-14's exit 125 was `cannot read memory.max`.
|
||||
const gate = gateShell()[3]!;
|
||||
const start = gate.indexOf("require_cgroup_value() {");
|
||||
const definition = gate.slice(start, gate.indexOf("\n}\n", start) + 2);
|
||||
const calls = gate.split("\n").filter((line) => line.startsWith("require_cgroup_value "));
|
||||
expect(calls).toHaveLength(4);
|
||||
const root = await mkdtemp(path.join(tmpdir(), "provider-cgroup-limits-"));
|
||||
try {
|
||||
for (const content of [undefined, "max\n"]) {
|
||||
for (const call of calls) {
|
||||
const file = path.join(root, call.split(" ")[1]!);
|
||||
if (content === undefined) await rm(file, { force: true });
|
||||
else await writeFile(file, content);
|
||||
const check = spawnSync("/bin/sh", ["-eu", "-c", `cgroup_root=$1\n${definition}\n${call}`, "check", root], {
|
||||
encoding: "utf8",
|
||||
});
|
||||
expect(check.status, call).toBe(125);
|
||||
const written = check.stderr.split("\n").filter((line) => line.startsWith(GATE_LINE));
|
||||
expect(written, call).toHaveLength(1);
|
||||
const diagnostics = createProviderSandboxDiagnostics();
|
||||
diagnostics.consume("stderr", check.stderr);
|
||||
expect(diagnostics.lines(), call).toEqual(written);
|
||||
}
|
||||
}
|
||||
} finally {
|
||||
await rm(root, { recursive: true, force: true });
|
||||
}
|
||||
}, childProcessTimeout(40));
|
||||
|
||||
it.each([
|
||||
"provider cgroup enforcement failed: unified cgroup v2 membership is required",
|
||||
"provider cgroup enforcement failed: unit membership is invalid",
|
||||
"provider cgroup enforcement failed: cannot read memory.max",
|
||||
"provider cgroup enforcement failed: memory.max is max, expected 1073741824",
|
||||
"provider cgroup enforcement failed: memory.swap.max is , expected 0",
|
||||
"provider cgroup enforcement failed: cpu.max is max 100000, expected 100000 100000",
|
||||
"provider scope stopped before launch: supervisor channel ended",
|
||||
"provider scope stopped before launch: supervisor channel error EPIPE",
|
||||
"provider scope stopped before launch: supervisor channel error ERR_STREAM_DESTROYED",
|
||||
"provider scope stopped before launch: supervisor channel error UNKNOWN",
|
||||
"provider scope stopped before launch: protocol failure",
|
||||
"provider scope frame length is invalid",
|
||||
"provider scope liveness channel is unavailable: Unsupported fd type: FILE",
|
||||
"bwrap: loopback: Failed RTM_NEWADDR: Operation not permitted",
|
||||
])("keeps the launch-stage line %j", (line) => {
|
||||
const diagnostics = createProviderSandboxDiagnostics();
|
||||
diagnostics.consume("stderr", `${line}\n`);
|
||||
expect(diagnostics.lines()).toEqual([line]);
|
||||
});
|
||||
|
||||
it.each([
|
||||
CREDENTIAL,
|
||||
`provider cgroup enforcement failed: memory.max is ${CREDENTIAL}, expected 1073741824`,
|
||||
`provider cgroup enforcement failed: cannot read ${CREDENTIAL}`,
|
||||
`provider scope stopped before launch: ${CREDENTIAL}`,
|
||||
`provider scope stopped before launch: supervisor channel error ${CREDENTIAL}`,
|
||||
// Upper case alone is not enough: a key-like value is not an errno or Node error name.
|
||||
"provider scope stopped before launch: supervisor channel error ABCDEFGHIJKLMNOPQRST",
|
||||
` provider cgroup enforcement failed: unit membership is invalid`,
|
||||
// A whole template followed by more text is provider output, not ours.
|
||||
`provider cgroup enforcement failed: cannot read memory.max ${CREDENTIAL}`,
|
||||
`provider cgroup enforcement failed: memory.max is max, expected 1073741824 ${CREDENTIAL}`,
|
||||
`provider scope stopped before launch: supervisor channel ended ${CREDENTIAL}`,
|
||||
`provider scope frame length is invalid ${CREDENTIAL}`,
|
||||
`provider scope liveness channel is unavailable: ${"x".repeat(201)}`,
|
||||
`bwrap:${CREDENTIAL}`,
|
||||
])("drops provider output that only resembles a launch-stage line: %j", (line) => {
|
||||
const diagnostics = createProviderSandboxDiagnostics();
|
||||
diagnostics.consume("stdout", `${line}\n`);
|
||||
diagnostics.consume("stderr", `${line}\n`);
|
||||
expect(diagnostics.lines()).toEqual([]);
|
||||
expect(JSON.stringify(diagnostics.lines())).not.toContain(CREDENTIAL);
|
||||
});
|
||||
|
||||
it("judges a line only once it is whole, per stream", () => {
|
||||
const diagnostics = createProviderSandboxDiagnostics();
|
||||
diagnostics.consume("stderr", "provider cgroup enforcement failed: cannot ");
|
||||
diagnostics.consume("stdout", "provider output\n");
|
||||
expect(diagnostics.lines()).toEqual([]);
|
||||
diagnostics.consume("stderr", "read pids.max\nprovider cgroup enforcement failed: unit");
|
||||
expect(diagnostics.lines()).toEqual(["provider cgroup enforcement failed: cannot read pids.max"]);
|
||||
});
|
||||
|
||||
it("drops the rest of an overlong line even when that rest arrives as its own write", () => {
|
||||
const diagnostics = createProviderSandboxDiagnostics();
|
||||
diagnostics.consume("stderr", "x".repeat(600));
|
||||
diagnostics.consume("stderr", `bwrap: ${CREDENTIAL}\n`);
|
||||
expect(diagnostics.lines()).toEqual([]);
|
||||
});
|
||||
|
||||
it("drops an overlong line even when it starts like a diagnostic", () => {
|
||||
const diagnostics = createProviderSandboxDiagnostics();
|
||||
diagnostics.consume("stderr", `bwrap: ${"x".repeat(600)}`);
|
||||
diagnostics.consume("stderr", `${CREDENTIAL}\nbwrap: ${"y".repeat(600)}\n`);
|
||||
expect(diagnostics.lines()).toEqual([]);
|
||||
});
|
||||
|
||||
it("keeps at most eight distinct lines", () => {
|
||||
const diagnostics = createProviderSandboxDiagnostics();
|
||||
for (let index = 0; index < 12; index += 1) {
|
||||
diagnostics.consume("stderr", `bwrap: failure ${index % 10}\nbwrap: failure ${index % 10}\n`);
|
||||
}
|
||||
expect(diagnostics.lines()).toEqual(
|
||||
Array.from({ length: 8 }, (_unused, index) => `bwrap: failure ${index}`),
|
||||
);
|
||||
});
|
||||
});
|
||||
@@ -15,6 +15,7 @@ type SpawnCall = Readonly<{ command: string; args: readonly string[] }>;
|
||||
type SpawnResult = ReturnType<SystemTestHost["spawn"]>;
|
||||
|
||||
const RTM_NEWADDR = "bwrap: loopback: Failed RTM_NEWADDR: Operation not permitted";
|
||||
const USER_MANAGER_CONTROLLERS = "/sys/fs/cgroup/user.slice/user-1000.slice/user@1000.service/cgroup.controllers";
|
||||
|
||||
/** A host that has every prerequisite, except what `bwrap` and `files` say. */
|
||||
function fakeHost(
|
||||
@@ -25,8 +26,12 @@ function fakeHost(
|
||||
const host: SystemTestHost = {
|
||||
access: async () => undefined,
|
||||
readFile: async (target) => {
|
||||
if (target === "/sys/fs/cgroup/cgroup.controllers") return "cpuset cpu io memory pids\n";
|
||||
const content = files[target];
|
||||
const content = {
|
||||
"/sys/fs/cgroup/cgroup.controllers": "cpuset cpu io memory pids\n",
|
||||
"/proc/self/status": "Name:\tnode\nUid:\t1000\t1000\t1000\t1000\n",
|
||||
[USER_MANAGER_CONTROLLERS]: "cpu memory pids\n",
|
||||
...files,
|
||||
}[target];
|
||||
if (content === undefined) throw new Error(`ENOENT: ${target}`);
|
||||
return content;
|
||||
},
|
||||
@@ -125,6 +130,46 @@ describe("system test prerequisites (1105/12-01)", () => {
|
||||
});
|
||||
});
|
||||
|
||||
// 2026-11-14. In one system container in four, systemd delegated no cgroup
|
||||
// controller to the root user manager; this check printed PASS and then every
|
||||
// provider stopped with `provider cgroup enforcement failed: cannot read
|
||||
// memory.max`.
|
||||
describe("system test prerequisites: provider scope controllers", () => {
|
||||
it.each([
|
||||
["none", "\n", "cpu, memory, pids", "none"],
|
||||
["memory", "cpu pids\n", "memory", "cpu pids"],
|
||||
])("names the controllers the user manager lacks (%s missing)", async (_label, delegated, missing, has) => {
|
||||
const { host } = fakeHost({ status: 0, stderr: "" }, { [USER_MANAGER_CONTROLLERS]: delegated });
|
||||
|
||||
expect(await findSystemTestPrerequisiteFailures(host)).toEqual([
|
||||
"the systemd user manager at /sys/fs/cgroup/user.slice/user-1000.slice/user@1000.service " +
|
||||
`has no ${missing} cgroup controller for provider scopes (it has: ${has})`,
|
||||
]);
|
||||
});
|
||||
|
||||
it("reads the user manager only where cgroup v2 is there", async () => {
|
||||
const { host } = fakeHost({ status: 0, stderr: "" }, { "/sys/fs/cgroup/cgroup.controllers": "\n", [USER_MANAGER_CONTROLLERS]: "\n" });
|
||||
|
||||
expect(await findSystemTestPrerequisiteFailures(host)).toEqual(["cgroup v2 controllers are unavailable"]);
|
||||
});
|
||||
|
||||
it("names an unreadable uid", async () => {
|
||||
const { host } = fakeHost({ status: 0, stderr: "" }, { "/proc/self/status": "Name:\tnode\n" });
|
||||
|
||||
expect(await findSystemTestPrerequisiteFailures(host)).toEqual([
|
||||
"this process's uid is unreadable at /proc/self/status",
|
||||
]);
|
||||
});
|
||||
|
||||
it("requires a user manager for this process's own uid", async () => {
|
||||
const { host } = fakeHost({ status: 0, stderr: "" }, { "/proc/self/status": "Uid:\t0\t0\t0\t0\n" });
|
||||
|
||||
expect(await findSystemTestPrerequisiteFailures(host)).toEqual([
|
||||
"a systemd user manager is required at /sys/fs/cgroup/user.slice/user-0.slice/user@0.service",
|
||||
]);
|
||||
});
|
||||
});
|
||||
|
||||
// 1101/13-03. The release provider jobs run the same sandbox on whatever
|
||||
// runner picks them up. The supervisor checked only that the executables
|
||||
// exist, so a host that refuses the namespaces failed later with bwrap's raw
|
||||
|
||||
@@ -10,6 +10,7 @@ import { afterEach, describe, expect, it } from "vitest";
|
||||
import { loadCiGateContract, parseCiGateContract } from "../../scripts/contracts/ci-gates.ts";
|
||||
import { generateCiWorkflow } from "../../scripts/generate-ci-workflow.ts";
|
||||
import { validatePackageScriptGraph } from "../../scripts/lib/package-script-graph.ts";
|
||||
import { createProviderSandboxDiagnostics } from "../../scripts/lib/provider-sandbox-diagnostics.ts";
|
||||
import { childProcessTimeout } from "../helpers/child-process-timeout.ts";
|
||||
|
||||
const roots: string[] = [];
|
||||
@@ -167,6 +168,9 @@ describe("selective Task 3 contract closure", () => {
|
||||
String(identity.ino),
|
||||
]);
|
||||
const completion = waitForChildResult(child);
|
||||
let stderr = "";
|
||||
child.stderr!.setEncoding("utf8");
|
||||
child.stderr!.on("data", (chunk: string) => { stderr += chunk; });
|
||||
await waitForScopeWrapperLoaded(child);
|
||||
// Time to exit on its own, which it must not do before EOF.
|
||||
await new Promise<void>((resolve) => setTimeout(resolve, 75));
|
||||
@@ -175,6 +179,10 @@ describe("selective Task 3 contract closure", () => {
|
||||
child.stdin!.end();
|
||||
const result = await within(completion, PROCESS_WATCHDOG_MS, "provider scope EOF close");
|
||||
expect(result).toEqual({ code: 125, signal: null });
|
||||
// The supervisor keeps this fixed line, so its exit-125 report names the stage.
|
||||
const diagnostics = createProviderSandboxDiagnostics();
|
||||
diagnostics.consume("stderr", stderr);
|
||||
expect(diagnostics.lines()).toEqual(["provider scope stopped before launch: supervisor channel ended"]);
|
||||
await expect(lstat(reportPath)).rejects.toMatchObject({ code: "ENOENT" });
|
||||
await expect(readdir(root)).resolves.toEqual([]);
|
||||
}, childProcessTimeout(433));
|
||||
|
||||
Reference in New Issue
Block a user