fix(ci): provider 실패 메시지에 우리 단계가 쓴 125 이유를 남긴다
문제: provider를 띄우는 우리 단계(cgroup 강제 게이트, scope wrapper)는 provider 전에 exit 125로 멈추며 stderr에 이유를 쓴다. waitForProvider는 bwrap·prlimit·systemd-run·systemctl 접두 줄만 남기고 나머지를 provider 출력으로 보고 버린다. 그래서 FE-GATE-013의 exit=125 묶음 (ci-artifact-contract 671줄부터 18개)이 `exit=125, signal=none`만 남겨 원인을 볼 수 없었다. scope wrapper가 감독자 채널을 잃고 provider 전에 125로 끝나는 길은 아무 줄도 쓰지 않았다. 고른 방법: 줄 고르기를 scripts/lib/provider-sandbox-diagnostics.ts로 옮기고, 우리 단계가 쓰는 고정 문구를 접두가 아니라 끝까지 맞는 template 으로 받는다. cgroup 값은 max와 숫자만, 오류 코드는 errno·Node 오류 이름(E…, ERR_…)만 받고, 그 모양은 scope wrapper와 한 곳에서 정한다. 줄은 개행이 온 뒤에만 판정하고(조각난 줄이 잘린 채 남지 않게), 512자 넘는 줄은 나머지 조각까지 버리며, 8줄까지 둔다. scope wrapper는 provider 전에 멈출 때 `provider scope stopped before launch: <이유>`를 쓴다. 시험은 실제 gate shell과 그 limit 확인 함수를 돌려 나온 줄이 남는지 본다. gate 문구를 바꾸면 시험이 실패한다. 버린 대안: 허용 접두를 `provider `로 넓히기는 provider가 같은 접두로 비밀을 쓰면 실패 메시지에 싣는다. 실패 때 stderr 전부를 남기기는 provider 출력의 자격 증명을 CI 로그에 싣는다. 위험·한계: 기존 도구 접두 네 개(bwrap: 등)는 전처럼 접두만 보고 그 뒤는 512자까지 자유 문구다. liveness 채널 줄도 우리 고정 접두 뒤에 Node 오류 문구를 싣는다(출력 가능한 ASCII 200자까지). provider가 그 접두나 우리 template과 똑같은 줄을 쓰면 남는다. 검증: 노트북 감싸개에서 새 시험이 옛 허용 목록으로 13개 실패(RED), 고친 뒤 provider-sandbox-diagnostics·task3-selective-integration· test-timeout-declarations 통과. 변이 셋(끝 앵커 제거, 긴 줄 표시 제거, gate printf 문구 변경)에 시험이 각각 5·1·1개 실패. check:types·lint rc=0. desktop 시스템 컨테이너에서 이 파일 전체 4회 중 1회 125 묶음이 났고 메시지에 `sandbox reported: provider cgroup enforcement failed: cannot read memory.max`가 남았다(11건).
This commit is contained in:
@@ -0,0 +1,93 @@
|
||||
/**
|
||||
* Lines the sandbox tooling itself emits, kept so a launch failure can say
|
||||
* why. Everything else the child writes is provider output and may carry
|
||||
* credentials, so it is counted and discarded as before.
|
||||
*
|
||||
* Without this a sandbox that never started reported only `exit=1`, and the
|
||||
* actual cause — `bwrap: loopback: Failed RTM_NEWADDR: Operation not
|
||||
* permitted` on a host with `kernel.apparmor_restrict_unprivileged_userns=1`
|
||||
* — was invisible. That turned a host restriction into an unexplained
|
||||
* product failure.
|
||||
*/
|
||||
const SANDBOX_TOOL = /^(?:bwrap|prlimit|systemd-run|systemctl):\s.*$/u;
|
||||
/**
|
||||
* Our own launch stages exit 125 before the provider runs: the cgroup gate
|
||||
* (provider-cgroup.ts ENFORCEMENT_GATE) and the scope wrapper
|
||||
* (provider-scope-wrapper.ts). Their lines are matched as whole templates,
|
||||
* not by prefix, so provider output that merely starts the same way is still
|
||||
* discarded. Only the liveness line carries free text, Node's own error for
|
||||
* descriptor 0.
|
||||
*/
|
||||
/** An errno or Node error name, the only free part of the scope wrapper's stop line. */
|
||||
const ERROR_CODE = String.raw`(?:E[A-Z0-9]{1,15}|ERR_[A-Z0-9_]{1,40})`;
|
||||
const CGROUP_FILE = String.raw`(?:memory\.max|memory\.swap\.max|pids\.max|cpu\.max)`;
|
||||
const CGROUP_VALUE = String.raw`(?:max|[0-9]{1,20})(?: [0-9]{1,20})?`;
|
||||
const LAUNCH_STAGE = [
|
||||
new RegExp(
|
||||
String.raw`^provider cgroup enforcement failed: (?:unified cgroup v2 membership is required|` +
|
||||
String.raw`unit membership is invalid|cannot read ${CGROUP_FILE}|` +
|
||||
String.raw`${CGROUP_FILE} is (?:${CGROUP_VALUE})?, expected ${CGROUP_VALUE})$`,
|
||||
"u",
|
||||
),
|
||||
new RegExp(
|
||||
String.raw`^provider scope stopped before launch: (?:supervisor channel ended|` +
|
||||
String.raw`supervisor channel error (?:${ERROR_CODE}|UNKNOWN)|protocol failure)$`,
|
||||
"u",
|
||||
),
|
||||
/^provider scope (?:received trailing protocol bytes|frame length is invalid|frame has trailing bytes)$/u,
|
||||
/^provider scope liveness channel is unavailable: [\x20-\x7e]{1,200}$/u,
|
||||
] as const;
|
||||
const MAX_LINES = 8;
|
||||
/** A diagnostic line is short; anything longer is provider output. */
|
||||
const MAX_LINE_CHARACTERS = 512;
|
||||
|
||||
export type ProviderSandboxDiagnostics = Readonly<{
|
||||
consume(stream: "stdout" | "stderr", chunk: Buffer | string): void;
|
||||
lines(): readonly string[];
|
||||
}>;
|
||||
|
||||
/** Whether the scope wrapper may name this error code in its stop line. */
|
||||
export function isProviderScopeErrorCode(code: string): boolean {
|
||||
return new RegExp(`^${ERROR_CODE}$`, "u").test(code);
|
||||
}
|
||||
|
||||
export function isProviderSandboxDiagnostic(line: string): boolean {
|
||||
return SANDBOX_TOOL.test(line) || LAUNCH_STAGE.some((pattern) => pattern.test(line));
|
||||
}
|
||||
|
||||
/**
|
||||
* Reassembles each stream into whole lines before matching, so a line split
|
||||
* across two reads is neither missed nor kept in a truncated form. A line is
|
||||
* only judged once its newline arrives.
|
||||
*/
|
||||
export function createProviderSandboxDiagnostics(): ProviderSandboxDiagnostics {
|
||||
const partial = { stdout: "", stderr: "" };
|
||||
const overlong = { stdout: false, stderr: false };
|
||||
const kept: string[] = [];
|
||||
const keep = (line: string): void => {
|
||||
if (
|
||||
kept.length < MAX_LINES && line.length <= MAX_LINE_CHARACTERS &&
|
||||
!kept.includes(line) && isProviderSandboxDiagnostic(line)
|
||||
) {
|
||||
kept.push(line);
|
||||
}
|
||||
};
|
||||
return Object.freeze({
|
||||
consume: (stream: "stdout" | "stderr", chunk: Buffer | string) => {
|
||||
const pieces = String(chunk).split("\n");
|
||||
const tail = pieces.pop() ?? "";
|
||||
for (const piece of pieces) {
|
||||
if (!overlong[stream]) keep(partial[stream] + piece);
|
||||
partial[stream] = "";
|
||||
overlong[stream] = false;
|
||||
}
|
||||
if (overlong[stream]) return;
|
||||
partial[stream] += tail;
|
||||
if (partial[stream].length > MAX_LINE_CHARACTERS) {
|
||||
partial[stream] = "";
|
||||
overlong[stream] = true;
|
||||
}
|
||||
},
|
||||
lines: () => Object.freeze([...kept]),
|
||||
});
|
||||
}
|
||||
@@ -4,6 +4,7 @@ import { Socket } from "node:net";
|
||||
|
||||
import { traceOwnProviderProcess, traceSpawnedProviderProcess } from "./provider-process-trace.ts";
|
||||
import { cleanupOwnedProviderReport } from "./provider-raw-cleanup.ts";
|
||||
import { isProviderScopeErrorCode } from "./provider-sandbox-diagnostics.ts";
|
||||
|
||||
const MAX_FRAME_BYTES = 16_777_216;
|
||||
const reportIdentity = parseReportIdentity(process.argv.slice(2));
|
||||
@@ -13,6 +14,8 @@ let expectedBytes: number | undefined;
|
||||
let provider: ReturnType<typeof spawn> | undefined;
|
||||
let providerClosed = false;
|
||||
let livenessLost = false;
|
||||
/** Why the scope stopped before it launched the provider, for its exit-125 line. */
|
||||
let stopReason = "supervisor channel ended";
|
||||
/**
|
||||
* The supervisor keeps this pipe open for the scope's whole life — that is how
|
||||
* parent loss is observed — and only ever writes one frame into it.
|
||||
@@ -47,8 +50,8 @@ liveness.on("data", (chunk: Buffer | string) => {
|
||||
}
|
||||
});
|
||||
|
||||
liveness.once("end", () => terminateForParentLoss());
|
||||
liveness.once("error", () => terminateForParentLoss());
|
||||
liveness.once("end", () => terminateForParentLoss("supervisor channel ended"));
|
||||
liveness.once("error", (error) => terminateForParentLoss(`supervisor channel error ${errorCode(error)}`));
|
||||
|
||||
function launchProvider(payload: Buffer): void {
|
||||
const frame = parseFrame(payload);
|
||||
@@ -96,9 +99,10 @@ async function finishProvider(
|
||||
process.exit(code ?? 1);
|
||||
}
|
||||
|
||||
function terminateForParentLoss(): void {
|
||||
function terminateForParentLoss(reason: string): void {
|
||||
if (livenessLost) return;
|
||||
livenessLost = true;
|
||||
stopReason = reason;
|
||||
if (!provider || providerClosed) {
|
||||
void cleanupAfterParentLossAndExit();
|
||||
return;
|
||||
@@ -117,12 +121,15 @@ async function cleanupAfterParentLossAndExit(): Promise<void> {
|
||||
writeSync(2, `${error instanceof Error ? error.message : String(error)}\n`);
|
||||
}
|
||||
closeLivenessInput();
|
||||
// The supervisor keeps only fixed launch-stage lines (provider-sandbox-diagnostics.ts);
|
||||
// without this one an exit 125 here was indistinguishable from the cgroup gate's.
|
||||
if (!provider) writeSync(2, `provider scope stopped before launch: ${stopReason}\n`);
|
||||
process.exit(125);
|
||||
}
|
||||
|
||||
function terminateForProtocolFailure(message: string): void {
|
||||
writeSync(2, `${message}\n`);
|
||||
terminateForParentLoss();
|
||||
terminateForParentLoss("protocol failure");
|
||||
}
|
||||
|
||||
function openLivenessChannel(): Socket {
|
||||
@@ -212,6 +219,11 @@ function signalNumber(signal: NodeJS.Signals): number {
|
||||
return signal === "SIGKILL" ? 9 : signal === "SIGXCPU" ? 24 : 1;
|
||||
}
|
||||
|
||||
function errorCode(error: unknown): string {
|
||||
const code = error && typeof error === "object" && "code" in error ? String(error.code) : "";
|
||||
return isProviderScopeErrorCode(code) ? code : "UNKNOWN";
|
||||
}
|
||||
|
||||
function hasErrorCode(error: unknown, code: string): boolean {
|
||||
return Boolean(error && typeof error === "object" && "code" in error && error.code === code);
|
||||
}
|
||||
|
||||
@@ -30,6 +30,7 @@ import {
|
||||
type ProviderGuardianLease,
|
||||
} from "./lib/provider-guardian-client.ts";
|
||||
import { createProviderOutputLimiter } from "./lib/provider-output-limiter.ts";
|
||||
import { createProviderSandboxDiagnostics } from "./lib/provider-sandbox-diagnostics.ts";
|
||||
import { findProviderSandboxNamespaceFailure } from "./check-system-test-prerequisites.ts";
|
||||
import { PROVIDER_SANDBOX_NAMESPACE_ARGUMENTS } from "./lib/provider-sandbox-namespaces.ts";
|
||||
import {
|
||||
@@ -400,30 +401,15 @@ async function waitForProvider(
|
||||
PROVIDER_MAX_OUTPUT_BYTES,
|
||||
() => terminate("output"),
|
||||
);
|
||||
/**
|
||||
* Lines the sandbox tooling itself emits, kept so a launch failure can say
|
||||
* why. Everything else the child writes is provider output and may carry
|
||||
* credentials, so it is counted and discarded as before.
|
||||
*
|
||||
* Without this a sandbox that never started reported only `exit=1`, and the
|
||||
* actual cause — `bwrap: loopback: Failed RTM_NEWADDR: Operation not
|
||||
* permitted` on a host with `kernel.apparmor_restrict_unprivileged_userns=1`
|
||||
* — was invisible. That turned a host restriction into an unexplained
|
||||
* product failure.
|
||||
*/
|
||||
const SANDBOX_DIAGNOSTIC = /^(?:bwrap|prlimit|systemd-run|systemctl):\s.*$/gmu;
|
||||
const sandboxDiagnostics: string[] = [];
|
||||
const capture = (chunk: Buffer | string): void => {
|
||||
for (const line of String(chunk).matchAll(SANDBOX_DIAGNOSTIC)) {
|
||||
if (sandboxDiagnostics.length < 8 && !sandboxDiagnostics.includes(line[0])) {
|
||||
sandboxDiagnostics.push(line[0]);
|
||||
}
|
||||
}
|
||||
// Only lines our own launch stages write survive (lib/provider-sandbox-diagnostics.ts).
|
||||
const sandboxDiagnostics = createProviderSandboxDiagnostics();
|
||||
const capture = (stream: "stdout" | "stderr") => (chunk: Buffer | string): void => {
|
||||
sandboxDiagnostics.consume(stream, chunk);
|
||||
if (termination) return;
|
||||
outputLimiter.consume(chunk);
|
||||
};
|
||||
child.stdout?.on("data", capture);
|
||||
child.stderr?.on("data", capture);
|
||||
child.stdout?.on("data", capture("stdout"));
|
||||
child.stderr?.on("data", capture("stderr"));
|
||||
const timeout = setTimeout(() => terminate("timeout"), timeoutMs);
|
||||
try {
|
||||
const first = await Promise.race([
|
||||
@@ -475,8 +461,8 @@ async function waitForProvider(
|
||||
throw new Error(
|
||||
`sandboxed external provider failed: exit=${result.code ?? "none"}, ` +
|
||||
`signal=${result.signal ?? "none"}` +
|
||||
(sandboxDiagnostics.length > 0
|
||||
? `; sandbox reported: ${sandboxDiagnostics.join("; ")}`
|
||||
(sandboxDiagnostics.lines().length > 0
|
||||
? `; sandbox reported: ${sandboxDiagnostics.lines().join("; ")}`
|
||||
: ""),
|
||||
);
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user