test(scale): 극한 규모 테스트가 CPU와 무관한 비교만 단언한다 (0929/10-01)

문제
extreme-scale-registry.test.ts는 머지 게이트(FE-GATE-005 coverage, pr
scale tier)에서 돈다. 그런데 절대 시간 예산(100만 조회 1,500ms, 100k
조합·멤버십 3,000ms)을 단언한다. 1코어에서는 코드와 무관하게 떨어진다
(수정 전 9회 중 5회, 1,509~1,826ms). run-scale-profile.ts와
ungated-tests.json은 pr tier를 "CPU와 무관하다"고 설명하는데 이 파일만
어긋났다. 같은 이유로 진짜 회귀를 잡는지도 기계 속도에 달려 있었다.

고른 방법
- 조합: 이미 있던 10k -> 100k 증가 비율(30 미만)만 남긴다.
- 조회: 같은 100k 키로 만든 일반 객체 조회와 비교해 3배 미만인지 본다.
  조회 횟수는 1,000,000 -> 200,000(양쪽 합 400,000).
- 멤버십: 10k와 100k 선택·해석 비용을 repo의 assertLinearOrBetter(지수
  1.5 이하)로 본다.
- 측정: 비교하는 두 쪽을 번갈아 재고, 10k 쪽은 10번 반복해 일의 양을
  맞추고, 각자 가장 빠른 회차를 쓴다. 공유 코어에서 10ms 측정은 한
  scheduler slice 안에 끝나고 100ms 측정은 다른 프로세스 뒤에서 기다려
  성장이 실제보다 가팔라 보인다(일의 양을 맞추기 전 조합 비율 61 관측).
  가장 빠른 회차는 check:scale:baseline이 성장을 minMs로 읽는 것과 같다.
절대 시간은 기준 CPU에서만 비교하는 check:scale:baseline 몫으로 둔다.

버린 대안
- 예산을 올리기: 더 느린 러너에서 같은 문제가 다시 생긴다.
- 절대 예산을 nightly 전용 파일로 떼기: tier 설정·ungated 목록·문서를
  함께 고쳐야 하고, 머지 게이트 쪽에는 결국 이번 비교가 필요하다.
- 조회를 10k/100k 성장으로 보기: 감싸기(proxy)처럼 크기와 무관한 느려짐을
  못 잡는다. 선형 탐색은 어느 쪽이든 시간 초과로 드러난다.

위험·한계
- 비교도 실제 시계로 잰다. 바쁜 이웃 3개를 둔 1코어에서 관측한 최댓값은
  조합 비율 17(한도 30), 조회 비율 1.04(한도 3), 멤버십 지수 1.21(한도
  1.5)이다. 여유는 한도의 약 1.2~3배다.
- 크기와 무관한 작은 상수 배 느려짐(get trap이 있는 proxy 정도)은 조회
  비교가 잡지 못한다. 수정 전 절대 예산도 이 기계에서 잡지 못했다.

검증
- RED(수정 전, taskset -c 0): 대상 파일만 3회 중 1회, component·features·
  integration·platform/cardinality 3회 중 1회, 바쁜 이웃 1개를 더하면
  3회 중 3회 "expected 1509.6… to be less than 1500"
  (extreme-scale-registry.test.ts:78).
- 변이(기본 코어): 조합에 100번째마다 Object.keys -> "expected 145.5 to be
  less than 30". 조회에 20회 hasOwn get trap -> "expected 14.46 to be less
  than 3". 해석을 includes로 -> "super-linear growth: exponent 1.98".
- GREEN(taskset -c 0 + 같은 코어의 바쁜 이웃, --maxWorkers=4): 대상 6개
  파일 6회, component·features·integration·platform/cardinality 5회 모두
  통과. 이 파일만 바쁜 이웃 3개 옆에서 6회, coverage를 켜고 3회 통과.
- 실행 시간: 기본 코어에서 파일 1.76s -> 1.37s.
- 기본 코어: tests/features·platform·architecture·contract·runtime-schema
  60파일 403개 통과. tsc -p tsconfig.test.json, eslint 통과.

Refs: docs/reviews/2026-09-29-maturity-review/10-tests-fast-layers.md
This commit is contained in:
DongHyeonka
2026-11-01 21:23:10 +09:00
parent c8620042ea
commit 582630062b
@@ -7,106 +7,143 @@ import {
selectCompiledProductFeatures,
} from "../../../src/contracts/product-features.ts";
import { composeUniqueRecord } from "../../../src/platform/cardinality/compose-unique-record.ts";
import { assertLinearOrBetter } from "../../../src/platform/cardinality/growth-regression.ts";
/*
* This file runs in merge gates on CPUs nobody chose (FE-GATE-005 coverage and
* the pr scale tier), so it asserts only what holds on any CPU: how a cost
* grows from 10k to 100k entries, and how a registry lookup compares with a
* plain-object lookup timed in the same process. Absolute timings belong to
* check:scale:baseline, which compares them only on the CPU that recorded
* config/scale/benchmark-baseline.json.
*/
const SMALL_REGISTRY_SIZE = 10_000;
const LARGE_REGISTRY_SIZE = 100_000;
const FEATURE_SCALE_SIZE = 100_000;
const LOOKUP_OPERATIONS = 1_000_000;
const SIZE_RATIO = LARGE_REGISTRY_SIZE / SMALL_REGISTRY_SIZE;
const LOOKUP_OPERATIONS = 200_000;
const MEASUREMENT_RUNS = 3;
const MAX_COMPOSE_GROWTH_RATIO = 30;
const MAX_LARGE_COMPOSE_MS = 3_000;
const MAX_MEMBERSHIP_MS = 3_000;
const MAX_LOOKUP_MS = 1_500;
/** Registry lookups against plain-object lookups on the same 100k keys. */
const MAX_LOOKUP_OVERHEAD_RATIO = 3;
function syntheticRecords(count: number): readonly Readonly<Record<string, number>>[] {
return Array.from({ length: count }, (_, index) => ({ [`key-${index}`]: index }));
}
function median(values: readonly number[]): number {
const sorted = [...values].sort((left, right) => left - right);
return sorted[Math.floor(sorted.length / 2)] ?? Number.POSITIVE_INFINITY;
function elapsedMs(action: () => void, repetitions: number): number {
const startedAt = performance.now();
for (let repetition = 0; repetition < repetitions; repetition += 1) action();
return (performance.now() - startedAt) / repetitions;
}
function measureMedianElapsedMs(action: () => void): number {
action();
const samples = Array.from({ length: MEASUREMENT_RUNS }, () => {
const startedAt = performance.now();
action();
return performance.now() - startedAt;
});
return median(samples);
/**
* Per-call cost of `left` and `right`, timed in alternating runs so both see
* the same background load. `leftRepetitions` makes the 10k side do as much
* work per run as the 100k side: on a shared core a 10ms run can finish inside
* one scheduler slice while a 100ms run waits behind other processes, which
* reads as steeper growth than the code has. Other processes and GC pauses only
* ever add time, so each side keeps its fastest run (check:scale:baseline reads
* growth from minMs for the same reason).
*/
function timeAlternately(
left: () => void,
right: () => void,
leftRepetitions = 1,
): Readonly<{ leftMs: number; rightMs: number }> {
left();
right();
let leftMs = Number.POSITIVE_INFINITY;
let rightMs = Number.POSITIVE_INFINITY;
for (let run = 0; run < MEASUREMENT_RUNS; run += 1) {
leftMs = Math.min(leftMs, elapsedMs(left, leftRepetitions));
rightMs = Math.min(rightMs, elapsedMs(right, 1));
}
return Object.freeze({ leftMs, rightMs });
}
function measureCompose(records: readonly Readonly<Record<string, number>>[]): Readonly<{
elapsedMs: number;
registry: Readonly<Record<string, number>>;
}> {
let registry: Readonly<Record<string, number>> = {};
const elapsedMs = measureMedianElapsedMs(() => {
registry = composeUniqueRecord(records, () => "synthetic registry");
});
return Object.freeze({ elapsedMs, registry });
function lookupChecksum(registry: Readonly<Record<string, number>>): number {
let checksum = 0;
for (let index = 0; index < LOOKUP_OPERATIONS; index += 1) {
checksum += registry[`key-${index % LARGE_REGISTRY_SIZE}`] ?? -1;
}
return checksum;
}
describe("extreme-scale registry composition", () => {
it("keeps 10k -> 100k composition growth below a quadratic regression budget", () => {
const smallRecords = syntheticRecords(SMALL_REGISTRY_SIZE);
const largeRecords = syntheticRecords(LARGE_REGISTRY_SIZE);
const small = measureCompose(smallRecords);
const large = measureCompose(largeRecords);
const growthRatio = large.elapsedMs / Math.max(small.elapsedMs, 1);
const compose = (records: typeof smallRecords) =>
composeUniqueRecord(records, () => "synthetic registry");
const { leftMs: smallMs, rightMs: largeMs } = timeAlternately(
() => compose(smallRecords),
() => compose(largeRecords),
SIZE_RATIO,
);
const growthRatio = largeMs / Math.max(smallMs, 1);
expect(Object.keys(small.registry)).toHaveLength(SMALL_REGISTRY_SIZE);
expect(Object.keys(large.registry)).toHaveLength(LARGE_REGISTRY_SIZE);
expect(large.elapsedMs).toBeLessThan(MAX_LARGE_COMPOSE_MS);
expect(Object.keys(compose(smallRecords))).toHaveLength(SMALL_REGISTRY_SIZE);
expect(Object.keys(compose(largeRecords))).toHaveLength(LARGE_REGISTRY_SIZE);
expect(growthRatio).toBeLessThan(MAX_COMPOSE_GROWTH_RATIO);
});
it("keeps one million lookups bounded on a 100k-entry runtime registry", () => {
const { registry } = measureCompose(syntheticRecords(LARGE_REGISTRY_SIZE));
let checksum = 0;
const elapsedMs = measureMedianElapsedMs(() => {
let currentChecksum = 0;
for (let index = 0; index < LOOKUP_OPERATIONS; index += 1) {
currentChecksum += registry[`key-${index % LARGE_REGISTRY_SIZE}`] ?? -1;
}
checksum = currentChecksum;
});
it("keeps lookups on a 100k-entry registry as cheap as on a plain object", () => {
const records = syntheticRecords(LARGE_REGISTRY_SIZE);
const registry = composeUniqueRecord(records, () => "synthetic registry");
const plain: Record<string, number> = {};
for (const record of records) Object.assign(plain, record);
const { leftMs: registryMs, rightMs: plainMs } = timeAlternately(
() => lookupChecksum(registry),
() => lookupChecksum(plain),
);
const checksum = lookupChecksum(registry);
expect(checksum).toBeGreaterThan(0);
expect(elapsedMs).toBeLessThan(MAX_LOOKUP_MS);
expect(checksum).toBe(lookupChecksum(plain));
expect(registryMs / plainMs).toBeLessThan(MAX_LOOKUP_OVERHEAD_RATIO);
});
});
function membershipInput(size: number) {
const compiled = Array.from({ length: size }, (_, index) =>
Object.freeze({ featureId: `feature-${index.toString().padStart(6, "0")}` }),
);
const declared = compiled
.filter((_, index) => index % 10 === 0)
.map((feature) => feature.featureId)
.join(",");
const compiledIds = compiled.map((feature) => feature.featureId);
const installedIds = selectCompiledProductFeatures(compiled, declared).map(
(feature) => feature.featureId,
);
return {
select: () => selectCompiledProductFeatures(compiled, declared),
resolve: () => resolveProductFeatures(compiledIds, installedIds),
};
}
describe("extreme-scale product feature membership", () => {
it("selects and resolves 100k feature ids within the membership budget", () => {
const compiled = Array.from({ length: FEATURE_SCALE_SIZE }, (_, index) =>
Object.freeze({ featureId: `feature-${index.toString().padStart(6, "0")}` }),
);
const requestedIds = compiled
.filter((_, index) => index % 10 === 0)
.map((feature) => feature.featureId);
const declared = requestedIds.join(",");
it("selects and resolves 100k feature ids with near-linear growth from 10k", () => {
const small = membershipInput(SMALL_REGISTRY_SIZE);
const large = membershipInput(LARGE_REGISTRY_SIZE);
const selection = timeAlternately(small.select, large.select, SIZE_RATIO);
const resolution = timeAlternately(small.resolve, large.resolve, SIZE_RATIO);
const statuses = large.resolve();
let selected = compiled.slice(0, 0);
const selectionElapsedMs = measureMedianElapsedMs(() => {
selected = [...selectCompiledProductFeatures(compiled, declared)];
});
let statuses = resolveProductFeatures([], []);
const installedIds = selected.map((feature) => feature.featureId);
const compiledIds = compiled.map((feature) => feature.featureId);
const resolveElapsedMs = measureMedianElapsedMs(() => {
statuses = resolveProductFeatures(compiledIds, installedIds);
});
expect(selected).toHaveLength(FEATURE_SCALE_SIZE / 10);
expect(statuses).toHaveLength(FEATURE_SCALE_SIZE);
expect(large.select()).toHaveLength(LARGE_REGISTRY_SIZE / 10);
expect(statuses).toHaveLength(LARGE_REGISTRY_SIZE);
expect(statuses.filter((status) => status.state === "ACTIVE")).toHaveLength(
FEATURE_SCALE_SIZE / 10,
LARGE_REGISTRY_SIZE / 10,
);
expect(selectionElapsedMs).toBeLessThan(MAX_MEMBERSHIP_MS);
expect(resolveElapsedMs).toBeLessThan(MAX_MEMBERSHIP_MS);
assertLinearOrBetter([
{ n: SMALL_REGISTRY_SIZE, cost: selection.leftMs },
{ n: LARGE_REGISTRY_SIZE, cost: selection.rightMs },
]);
assertLinearOrBetter([
{ n: SMALL_REGISTRY_SIZE, cost: resolution.leftMs },
{ n: LARGE_REGISTRY_SIZE, cost: resolution.rightMs },
]);
});
});