fix(ci): provider 실패 메시지에 우리 단계가 쓴 125 이유를 남긴다

문제: provider를 띄우는 우리 단계(cgroup 강제 게이트, scope wrapper)는
provider 전에 exit 125로 멈추며 stderr에 이유를 쓴다. waitForProvider는
bwrap·prlimit·systemd-run·systemctl 접두 줄만 남기고 나머지를 provider
출력으로 보고 버린다. 그래서 FE-GATE-013의 exit=125 묶음
(ci-artifact-contract 671줄부터 18개)이 `exit=125, signal=none`만 남겨
원인을 볼 수 없었다. scope wrapper가 감독자 채널을 잃고 provider 전에
125로 끝나는 길은 아무 줄도 쓰지 않았다.

고른 방법: 줄 고르기를 scripts/lib/provider-sandbox-diagnostics.ts로
옮기고, 우리 단계가 쓰는 고정 문구를 접두가 아니라 끝까지 맞는 template
으로 받는다. cgroup 값은 max와 숫자만, 오류 코드는 errno·Node 오류
이름(E…, ERR_…)만 받고, 그 모양은 scope wrapper와 한 곳에서 정한다. 줄은 개행이 온 뒤에만 판정하고(조각난 줄이
잘린 채 남지 않게), 512자 넘는 줄은 나머지 조각까지 버리며, 8줄까지
둔다. scope wrapper는 provider 전에 멈출 때
`provider scope stopped before launch: <이유>`를 쓴다. 시험은 실제 gate
shell과 그 limit 확인 함수를 돌려 나온 줄이 남는지 본다. gate 문구를
바꾸면 시험이 실패한다.
버린 대안: 허용 접두를 `provider `로 넓히기는 provider가 같은 접두로
비밀을 쓰면 실패 메시지에 싣는다. 실패 때 stderr 전부를 남기기는
provider 출력의 자격 증명을 CI 로그에 싣는다.

위험·한계: 기존 도구 접두 네 개(bwrap: 등)는 전처럼 접두만 보고 그
뒤는 512자까지 자유 문구다. liveness 채널 줄도 우리 고정 접두 뒤에 Node
오류 문구를 싣는다(출력 가능한 ASCII 200자까지). provider가 그 접두나
우리 template과 똑같은 줄을 쓰면 남는다.

검증: 노트북 감싸개에서 새 시험이 옛 허용 목록으로 13개 실패(RED),
고친 뒤 provider-sandbox-diagnostics·task3-selective-integration·
test-timeout-declarations 통과. 변이 셋(끝 앵커 제거, 긴 줄 표시 제거,
gate printf 문구 변경)에 시험이 각각 5·1·1개 실패. check:types·lint
rc=0. desktop 시스템 컨테이너에서 이 파일 전체 4회 중 1회 125 묶음이
났고 메시지에 `sandbox reported: provider cgroup enforcement failed:
cannot read memory.max`가 남았다(11건).
This commit is contained in:
DongHyeonka
2026-11-14 06:04:30 +09:00
parent e98d8ee9a9
commit 6d6fd61ab9
6 changed files with 279 additions and 28 deletions
+6 -1
View File
@@ -242,7 +242,12 @@ runner의 Docker proxy가 안쪽 dind daemon에 컨테이너를 띄운다.
각 provider는 고유한 collected user scope에서 실행된다. Supervisor는 실행 전에
실제 cgroup membership과 `memory.max=1073741824`, `memory.swap.max=0`,
`pids.max=64`, `cpu.max="100000 100000"`을 확인한다. 내부 process에는 core 0,
`pids.max=64`, `cpu.max="100000 100000"`을 확인한다. 확인이나 scope 준비가
provider 전에 실패하면 supervisor 오류 끝에 이유가 붙는다
(`sandbox reported: provider cgroup enforcement failed: ...`). 남기는 줄은 8줄까지,
우리 단계가 쓰는 고정 문구와 sandbox 도구의 접두 줄(`bwrap:` 등)뿐이다. provider
출력은 자격 증명을 실을 수 있어 그 밖의 줄은 남기지 않는다
(`scripts/lib/provider-sandbox-diagnostics.ts`). 내부 process에는 core 0,
file-size 8,388,607 bytes, open FD 64, CPU 1,200초 상한도 적용된다.
같은 UID 전체에 합산되는 `RLIMIT_NPROC`로 provider별 32개를 보장한다고 주장하지
않으며 aggregate PID authority는 cgroup `TasksMax=64`다.
@@ -0,0 +1,93 @@
/**
* Lines the sandbox tooling itself emits, kept so a launch failure can say
* why. Everything else the child writes is provider output and may carry
* credentials, so it is counted and discarded as before.
*
* Without this a sandbox that never started reported only `exit=1`, and the
* actual cause — `bwrap: loopback: Failed RTM_NEWADDR: Operation not
* permitted` on a host with `kernel.apparmor_restrict_unprivileged_userns=1`
* — was invisible. That turned a host restriction into an unexplained
* product failure.
*/
const SANDBOX_TOOL = /^(?:bwrap|prlimit|systemd-run|systemctl):\s.*$/u;
/**
* Our own launch stages exit 125 before the provider runs: the cgroup gate
* (provider-cgroup.ts ENFORCEMENT_GATE) and the scope wrapper
* (provider-scope-wrapper.ts). Their lines are matched as whole templates,
* not by prefix, so provider output that merely starts the same way is still
* discarded. Only the liveness line carries free text, Node's own error for
* descriptor 0.
*/
/** An errno or Node error name, the only free part of the scope wrapper's stop line. */
const ERROR_CODE = String.raw`(?:E[A-Z0-9]{1,15}|ERR_[A-Z0-9_]{1,40})`;
const CGROUP_FILE = String.raw`(?:memory\.max|memory\.swap\.max|pids\.max|cpu\.max)`;
const CGROUP_VALUE = String.raw`(?:max|[0-9]{1,20})(?: [0-9]{1,20})?`;
const LAUNCH_STAGE = [
new RegExp(
String.raw`^provider cgroup enforcement failed: (?:unified cgroup v2 membership is required|` +
String.raw`unit membership is invalid|cannot read ${CGROUP_FILE}|` +
String.raw`${CGROUP_FILE} is (?:${CGROUP_VALUE})?, expected ${CGROUP_VALUE})$`,
"u",
),
new RegExp(
String.raw`^provider scope stopped before launch: (?:supervisor channel ended|` +
String.raw`supervisor channel error (?:${ERROR_CODE}|UNKNOWN)|protocol failure)$`,
"u",
),
/^provider scope (?:received trailing protocol bytes|frame length is invalid|frame has trailing bytes)$/u,
/^provider scope liveness channel is unavailable: [\x20-\x7e]{1,200}$/u,
] as const;
const MAX_LINES = 8;
/** A diagnostic line is short; anything longer is provider output. */
const MAX_LINE_CHARACTERS = 512;
export type ProviderSandboxDiagnostics = Readonly<{
consume(stream: "stdout" | "stderr", chunk: Buffer | string): void;
lines(): readonly string[];
}>;
/** Whether the scope wrapper may name this error code in its stop line. */
export function isProviderScopeErrorCode(code: string): boolean {
return new RegExp(`^${ERROR_CODE}$`, "u").test(code);
}
export function isProviderSandboxDiagnostic(line: string): boolean {
return SANDBOX_TOOL.test(line) || LAUNCH_STAGE.some((pattern) => pattern.test(line));
}
/**
* Reassembles each stream into whole lines before matching, so a line split
* across two reads is neither missed nor kept in a truncated form. A line is
* only judged once its newline arrives.
*/
export function createProviderSandboxDiagnostics(): ProviderSandboxDiagnostics {
const partial = { stdout: "", stderr: "" };
const overlong = { stdout: false, stderr: false };
const kept: string[] = [];
const keep = (line: string): void => {
if (
kept.length < MAX_LINES && line.length <= MAX_LINE_CHARACTERS &&
!kept.includes(line) && isProviderSandboxDiagnostic(line)
) {
kept.push(line);
}
};
return Object.freeze({
consume: (stream: "stdout" | "stderr", chunk: Buffer | string) => {
const pieces = String(chunk).split("\n");
const tail = pieces.pop() ?? "";
for (const piece of pieces) {
if (!overlong[stream]) keep(partial[stream] + piece);
partial[stream] = "";
overlong[stream] = false;
}
if (overlong[stream]) return;
partial[stream] += tail;
if (partial[stream].length > MAX_LINE_CHARACTERS) {
partial[stream] = "";
overlong[stream] = true;
}
},
lines: () => Object.freeze([...kept]),
});
}
+16 -4
View File
@@ -4,6 +4,7 @@ import { Socket } from "node:net";
import { traceOwnProviderProcess, traceSpawnedProviderProcess } from "./provider-process-trace.ts";
import { cleanupOwnedProviderReport } from "./provider-raw-cleanup.ts";
import { isProviderScopeErrorCode } from "./provider-sandbox-diagnostics.ts";
const MAX_FRAME_BYTES = 16_777_216;
const reportIdentity = parseReportIdentity(process.argv.slice(2));
@@ -13,6 +14,8 @@ let expectedBytes: number | undefined;
let provider: ReturnType<typeof spawn> | undefined;
let providerClosed = false;
let livenessLost = false;
/** Why the scope stopped before it launched the provider, for its exit-125 line. */
let stopReason = "supervisor channel ended";
/**
* The supervisor keeps this pipe open for the scope's whole life — that is how
* parent loss is observed — and only ever writes one frame into it.
@@ -47,8 +50,8 @@ liveness.on("data", (chunk: Buffer | string) => {
}
});
liveness.once("end", () => terminateForParentLoss());
liveness.once("error", () => terminateForParentLoss());
liveness.once("end", () => terminateForParentLoss("supervisor channel ended"));
liveness.once("error", (error) => terminateForParentLoss(`supervisor channel error ${errorCode(error)}`));
function launchProvider(payload: Buffer): void {
const frame = parseFrame(payload);
@@ -96,9 +99,10 @@ async function finishProvider(
process.exit(code ?? 1);
}
function terminateForParentLoss(): void {
function terminateForParentLoss(reason: string): void {
if (livenessLost) return;
livenessLost = true;
stopReason = reason;
if (!provider || providerClosed) {
void cleanupAfterParentLossAndExit();
return;
@@ -117,12 +121,15 @@ async function cleanupAfterParentLossAndExit(): Promise<void> {
writeSync(2, `${error instanceof Error ? error.message : String(error)}\n`);
}
closeLivenessInput();
// The supervisor keeps only fixed launch-stage lines (provider-sandbox-diagnostics.ts);
// without this one an exit 125 here was indistinguishable from the cgroup gate's.
if (!provider) writeSync(2, `provider scope stopped before launch: ${stopReason}\n`);
process.exit(125);
}
function terminateForProtocolFailure(message: string): void {
writeSync(2, `${message}\n`);
terminateForParentLoss();
terminateForParentLoss("protocol failure");
}
function openLivenessChannel(): Socket {
@@ -212,6 +219,11 @@ function signalNumber(signal: NodeJS.Signals): number {
return signal === "SIGKILL" ? 9 : signal === "SIGXCPU" ? 24 : 1;
}
function errorCode(error: unknown): string {
const code = error && typeof error === "object" && "code" in error ? String(error.code) : "";
return isProviderScopeErrorCode(code) ? code : "UNKNOWN";
}
function hasErrorCode(error: unknown, code: string): boolean {
return Boolean(error && typeof error === "object" && "code" in error && error.code === code);
}
+9 -23
View File
@@ -30,6 +30,7 @@ import {
type ProviderGuardianLease,
} from "./lib/provider-guardian-client.ts";
import { createProviderOutputLimiter } from "./lib/provider-output-limiter.ts";
import { createProviderSandboxDiagnostics } from "./lib/provider-sandbox-diagnostics.ts";
import { findProviderSandboxNamespaceFailure } from "./check-system-test-prerequisites.ts";
import { PROVIDER_SANDBOX_NAMESPACE_ARGUMENTS } from "./lib/provider-sandbox-namespaces.ts";
import {
@@ -400,30 +401,15 @@ async function waitForProvider(
PROVIDER_MAX_OUTPUT_BYTES,
() => terminate("output"),
);
/**
* Lines the sandbox tooling itself emits, kept so a launch failure can say
* why. Everything else the child writes is provider output and may carry
* credentials, so it is counted and discarded as before.
*
* Without this a sandbox that never started reported only `exit=1`, and the
* actual cause — `bwrap: loopback: Failed RTM_NEWADDR: Operation not
* permitted` on a host with `kernel.apparmor_restrict_unprivileged_userns=1`
* — was invisible. That turned a host restriction into an unexplained
* product failure.
*/
const SANDBOX_DIAGNOSTIC = /^(?:bwrap|prlimit|systemd-run|systemctl):\s.*$/gmu;
const sandboxDiagnostics: string[] = [];
const capture = (chunk: Buffer | string): void => {
for (const line of String(chunk).matchAll(SANDBOX_DIAGNOSTIC)) {
if (sandboxDiagnostics.length < 8 && !sandboxDiagnostics.includes(line[0])) {
sandboxDiagnostics.push(line[0]);
}
}
// Only lines our own launch stages write survive (lib/provider-sandbox-diagnostics.ts).
const sandboxDiagnostics = createProviderSandboxDiagnostics();
const capture = (stream: "stdout" | "stderr") => (chunk: Buffer | string): void => {
sandboxDiagnostics.consume(stream, chunk);
if (termination) return;
outputLimiter.consume(chunk);
};
child.stdout?.on("data", capture);
child.stderr?.on("data", capture);
child.stdout?.on("data", capture("stdout"));
child.stderr?.on("data", capture("stderr"));
const timeout = setTimeout(() => terminate("timeout"), timeoutMs);
try {
const first = await Promise.race([
@@ -475,8 +461,8 @@ async function waitForProvider(
throw new Error(
`sandboxed external provider failed: exit=${result.code ?? "none"}, ` +
`signal=${result.signal ?? "none"}` +
(sandboxDiagnostics.length > 0
? `; sandbox reported: ${sandboxDiagnostics.join("; ")}`
(sandboxDiagnostics.lines().length > 0
? `; sandbox reported: ${sandboxDiagnostics.lines().join("; ")}`
: ""),
);
}
@@ -0,0 +1,147 @@
import { spawnSync } from "node:child_process";
import { mkdtemp, rm, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import path from "node:path";
import { describe, expect, it } from "vitest";
import {
formatProviderCgroupUnitName,
systemdRunProviderArguments,
} from "../../scripts/lib/provider-cgroup.ts";
import { createProviderSandboxDiagnostics } from "../../scripts/lib/provider-sandbox-diagnostics.ts";
import { childProcessTimeout } from "../helpers/child-process-timeout.ts";
const CREDENTIAL = "provider-credential-must-not-reach-ci";
const GATE_LINE = "provider cgroup enforcement failed: ";
/** The command systemd-run starts in the scope: `/bin/sh -eu -c <gate> <unit> …`. */
function gateShell(): string[] {
const unit = formatProviderCgroupUnitName("vulnerability", 4242, "0".repeat(24));
const systemdRun = systemdRunProviderArguments(unit, 1_000, 1, "/bin/node", "/wrapper.ts", "/r", 1, 1);
return systemdRun.slice(systemdRun.indexOf("--") + 1);
}
describe("provider sandbox diagnostics", () => {
it("keeps the line the real cgroup gate writes when it refuses a scope", () => {
// Outside a provider scope the gate's own membership check fails, which is
// the same shell, text, and exit code a broken scope produces.
const shell = gateShell();
const gate = spawnSync(shell[0]!, shell.slice(1), { encoding: "utf8" });
expect(gate.status).toBe(125);
expect(gate.stderr).toMatch(/^provider cgroup enforcement failed: \S/u);
const diagnostics = createProviderSandboxDiagnostics();
diagnostics.consume("stderr", gate.stderr);
expect(diagnostics.lines()).toEqual([gate.stderr.trimEnd()]);
}, childProcessTimeout(4));
it("keeps the line the real gate check writes for every unreadable or wrong limit", async () => {
// The gate's own limit check and its own calls, run against a stand-in
// cgroup directory: 2026-11-14's exit 125 was `cannot read memory.max`.
const gate = gateShell()[3]!;
const start = gate.indexOf("require_cgroup_value() {");
const definition = gate.slice(start, gate.indexOf("\n}\n", start) + 2);
const calls = gate.split("\n").filter((line) => line.startsWith("require_cgroup_value "));
expect(calls).toHaveLength(4);
const root = await mkdtemp(path.join(tmpdir(), "provider-cgroup-limits-"));
try {
for (const content of [undefined, "max\n"]) {
for (const call of calls) {
const file = path.join(root, call.split(" ")[1]!);
if (content === undefined) await rm(file, { force: true });
else await writeFile(file, content);
const check = spawnSync("/bin/sh", ["-eu", "-c", `cgroup_root=$1\n${definition}\n${call}`, "check", root], {
encoding: "utf8",
});
expect(check.status, call).toBe(125);
const written = check.stderr.split("\n").filter((line) => line.startsWith(GATE_LINE));
expect(written, call).toHaveLength(1);
const diagnostics = createProviderSandboxDiagnostics();
diagnostics.consume("stderr", check.stderr);
expect(diagnostics.lines(), call).toEqual(written);
}
}
} finally {
await rm(root, { recursive: true, force: true });
}
}, childProcessTimeout(40));
it.each([
"provider cgroup enforcement failed: unified cgroup v2 membership is required",
"provider cgroup enforcement failed: unit membership is invalid",
"provider cgroup enforcement failed: cannot read memory.max",
"provider cgroup enforcement failed: memory.max is max, expected 1073741824",
"provider cgroup enforcement failed: memory.swap.max is , expected 0",
"provider cgroup enforcement failed: cpu.max is max 100000, expected 100000 100000",
"provider scope stopped before launch: supervisor channel ended",
"provider scope stopped before launch: supervisor channel error EPIPE",
"provider scope stopped before launch: supervisor channel error ERR_STREAM_DESTROYED",
"provider scope stopped before launch: supervisor channel error UNKNOWN",
"provider scope stopped before launch: protocol failure",
"provider scope frame length is invalid",
"provider scope liveness channel is unavailable: Unsupported fd type: FILE",
"bwrap: loopback: Failed RTM_NEWADDR: Operation not permitted",
])("keeps the launch-stage line %j", (line) => {
const diagnostics = createProviderSandboxDiagnostics();
diagnostics.consume("stderr", `${line}\n`);
expect(diagnostics.lines()).toEqual([line]);
});
it.each([
CREDENTIAL,
`provider cgroup enforcement failed: memory.max is ${CREDENTIAL}, expected 1073741824`,
`provider cgroup enforcement failed: cannot read ${CREDENTIAL}`,
`provider scope stopped before launch: ${CREDENTIAL}`,
`provider scope stopped before launch: supervisor channel error ${CREDENTIAL}`,
// Upper case alone is not enough: a key-like value is not an errno or Node error name.
"provider scope stopped before launch: supervisor channel error ABCDEFGHIJKLMNOPQRST",
` provider cgroup enforcement failed: unit membership is invalid`,
// A whole template followed by more text is provider output, not ours.
`provider cgroup enforcement failed: cannot read memory.max ${CREDENTIAL}`,
`provider cgroup enforcement failed: memory.max is max, expected 1073741824 ${CREDENTIAL}`,
`provider scope stopped before launch: supervisor channel ended ${CREDENTIAL}`,
`provider scope frame length is invalid ${CREDENTIAL}`,
`provider scope liveness channel is unavailable: ${"x".repeat(201)}`,
`bwrap:${CREDENTIAL}`,
])("drops provider output that only resembles a launch-stage line: %j", (line) => {
const diagnostics = createProviderSandboxDiagnostics();
diagnostics.consume("stdout", `${line}\n`);
diagnostics.consume("stderr", `${line}\n`);
expect(diagnostics.lines()).toEqual([]);
expect(JSON.stringify(diagnostics.lines())).not.toContain(CREDENTIAL);
});
it("judges a line only once it is whole, per stream", () => {
const diagnostics = createProviderSandboxDiagnostics();
diagnostics.consume("stderr", "provider cgroup enforcement failed: cannot ");
diagnostics.consume("stdout", "provider output\n");
expect(diagnostics.lines()).toEqual([]);
diagnostics.consume("stderr", "read pids.max\nprovider cgroup enforcement failed: unit");
expect(diagnostics.lines()).toEqual(["provider cgroup enforcement failed: cannot read pids.max"]);
});
it("drops the rest of an overlong line even when that rest arrives as its own write", () => {
const diagnostics = createProviderSandboxDiagnostics();
diagnostics.consume("stderr", "x".repeat(600));
diagnostics.consume("stderr", `bwrap: ${CREDENTIAL}\n`);
expect(diagnostics.lines()).toEqual([]);
});
it("drops an overlong line even when it starts like a diagnostic", () => {
const diagnostics = createProviderSandboxDiagnostics();
diagnostics.consume("stderr", `bwrap: ${"x".repeat(600)}`);
diagnostics.consume("stderr", `${CREDENTIAL}\nbwrap: ${"y".repeat(600)}\n`);
expect(diagnostics.lines()).toEqual([]);
});
it("keeps at most eight distinct lines", () => {
const diagnostics = createProviderSandboxDiagnostics();
for (let index = 0; index < 12; index += 1) {
diagnostics.consume("stderr", `bwrap: failure ${index % 10}\nbwrap: failure ${index % 10}\n`);
}
expect(diagnostics.lines()).toEqual(
Array.from({ length: 8 }, (_unused, index) => `bwrap: failure ${index}`),
);
});
});
@@ -10,6 +10,7 @@ import { afterEach, describe, expect, it } from "vitest";
import { loadCiGateContract, parseCiGateContract } from "../../scripts/contracts/ci-gates.ts";
import { generateCiWorkflow } from "../../scripts/generate-ci-workflow.ts";
import { validatePackageScriptGraph } from "../../scripts/lib/package-script-graph.ts";
import { createProviderSandboxDiagnostics } from "../../scripts/lib/provider-sandbox-diagnostics.ts";
import { childProcessTimeout } from "../helpers/child-process-timeout.ts";
const roots: string[] = [];
@@ -167,6 +168,9 @@ describe("selective Task 3 contract closure", () => {
String(identity.ino),
]);
const completion = waitForChildResult(child);
let stderr = "";
child.stderr!.setEncoding("utf8");
child.stderr!.on("data", (chunk: string) => { stderr += chunk; });
await waitForScopeWrapperLoaded(child);
// Time to exit on its own, which it must not do before EOF.
await new Promise<void>((resolve) => setTimeout(resolve, 75));
@@ -175,6 +179,10 @@ describe("selective Task 3 contract closure", () => {
child.stdin!.end();
const result = await within(completion, PROCESS_WATCHDOG_MS, "provider scope EOF close");
expect(result).toEqual({ code: 125, signal: null });
// The supervisor keeps this fixed line, so its exit-125 report names the stage.
const diagnostics = createProviderSandboxDiagnostics();
diagnostics.consume("stderr", stderr);
expect(diagnostics.lines()).toEqual(["provider scope stopped before launch: supervisor channel ended"]);
await expect(lstat(reportPath)).rejects.toMatchObject({ code: "ENOENT" });
await expect(readdir(root)).resolves.toEqual([]);
}, childProcessTimeout(433));