fix(compose): heap dump 경로를 컨테이너 메모리 한계 안의 tmpfs에 두지 않는다
문제: 기본 compose.yaml은 -XX:HeapDumpPath=/var/tmp/heap을 512m tmpfs로
붙였다. 컨테이너는 mem_limit 512m·swap 없음이고 힙은 그 75%(384m)다.
tmpfs에 쓴 페이지는 같은 컨테이너 메모리 한계에 잡힌다. 그래서
OutOfMemoryError 때 덤프를 쓰는 동안 컨테이너가 OOM kill(137)될 수 있었고,
runbook이 기대하는 깨끗한 끝(코드 3, ExitOnOutOfMemoryError)이 사라졌다.
덤프가 끝까지 써져도 tmpfs는 컨테이너가 끝나면 함께 사라져 읽을 수 없었다.
Dockerfile 주석은 오케스트레이터에 그 경로를 tmpfs로 붙이라고 했고,
internal-error-spike runbook은 없는 파일 이름(heapdump-<pid>.hprof)을 적었다.
고른 방법:
- 기본 compose.yaml은 그 경로에 아무것도 붙이지 않는다. 루트가 읽기
전용이라 JVM은 파일을 만들지 못하고 바로 코드 3으로 끝난다. 덤프를 남기는
dev lane(named volume heap-dumps)은 그대로다.
- scripts/verify-compose-profile-contracts.sh(prCheck의
verifyComposeProfiles)가 모든 lane에서 그 경로의 tmpfs를 거절한다. 경로는
Dockerfile의 -XX:HeapDumpPath에서 읽는다.
- Dockerfile 주석과 runbook 둘(runtime-jvm-oom 「진단」, internal-error-spike)에
덤프를 남기는 법(Kubernetes emptyDir without medium: Memory·PV, dev lane
volume)과 실제 파일 이름(java_pid<pid>.hprof)을 적었다.
- 버린 대안: 기본 compose에 named volume을 붙이기 — 기본 조립은 디스크에
아무것도 남기지 않는다는 결정(compose.yaml 「No volume」 주석)과 다르다.
- 고치며 찾은 것(RR9-03-30): ComposeMergeCharacterizationTest는 어느 lane에서도
돌지 않았다. integrationTest lane은 requires로 적은 클래스 넷만 고른다
(StrictTestLanePlugin). 그 시험이 보던 기본·dev 조합 렌더링과 계약 스크립트는
verifyComposeProfiles가 18개 lane으로 이미 본다. 시험과, 그 lane에 compose
파일을 입력으로 선언하던 블록을 지웠다.
검증(desktop):
- RED: 기준 ca1fbfde + 검사 스크립트만으로 :verifyComposeProfiles 실패
(기본 compose를 쓰는 lane마다 「heap dump path ... as a tmpfs」).
- 탐침(eclipse-temurin:21-jdk, 읽기 전용 루트, 메모리 192m·swap 없음,
-Xmx96m으로 OutOfMemoryError):
경로에 아무것도 없음 → "Unable to create /var/tmp/heap: Read-only file
system", 코드 3. tmpfs → 덤프를 쓰다 코드 137(OOM kill). 디스크 마운트 →
덤프 101MB가 남고 코드 3.
- GREEN: :verifyComposeProfiles(18 lane), :app-bootstrap:integrationTest,
repositoryContractTest, spotlessCheck — 71건 실패 0, skip 1, rc=0.
This commit is contained in:
@@ -399,9 +399,9 @@ tasks.named('check') {
|
||||
}
|
||||
|
||||
// The other suites find the repository themselves and read what Gradle cannot otherwise see: the
|
||||
// architecture suite scans every leaf's production sources and its own rule sources as text, the
|
||||
// opt-in classpath contract reads this build script, and the Compose characterization reads the
|
||||
// Compose files.
|
||||
// architecture suite scans every leaf's production sources and its own rule sources as text, and the
|
||||
// opt-in classpath contract reads this build script. The Compose files are rendered by
|
||||
// :verifyComposeProfiles (pull-request stage), not by a test here.
|
||||
tasks.named('architectureTest', Test) {
|
||||
inputs.files(fileTree(rootProject.projectDir) {
|
||||
include 'src/**/src/main/java/**/*.java', 'build-logic/src/main/java/**/*.java',
|
||||
@@ -412,12 +412,6 @@ tasks.named('architectureTest', Test) {
|
||||
tasks.named('conditionalTransportBuildContractTest', Test) {
|
||||
inputs.file('build.gradle').withPathSensitivity(PathSensitivity.RELATIVE).withPropertyName('buildScript')
|
||||
}
|
||||
tasks.named('integrationTest', Test) {
|
||||
inputs.files(fileTree(repositoryRootForContractTests) {
|
||||
include 'compose.yaml', 'compose/**', 'scripts/*compose*.sh', 'src/bootstrap/app/compose-lanes/**'
|
||||
exclude repositoryContractExcludes
|
||||
}).withPathSensitivity(PathSensitivity.RELATIVE).withPropertyName('composeFiles')
|
||||
}
|
||||
|
||||
// Fast developer test lane. Whole-repository ArchUnit graphs and full composition-root boot
|
||||
// characterizations have their own source sets, so the ordinary lane does not compile or execute
|
||||
|
||||
@@ -22,10 +22,9 @@ services:
|
||||
# Relax read-only constraint for local development.
|
||||
read_only: false
|
||||
# `!override`, not a plain empty list. An empty sequence merges with the base sequence rather
|
||||
# than replacing it, so the base's /var/tmp/heap tmpfs survived and collided with the bind mount
|
||||
# below — Compose refuses to have the same target twice and will not silently pick one. That is
|
||||
# the right refusal: a heap dump written into a tmpfs dies with the container that produced it,
|
||||
# which is the one moment somebody wants the file.
|
||||
# than replacing it; the writable root needs no tmpfs at all. The base once declared a tmpfs at
|
||||
# /var/tmp/heap too, and the merge collided with the volume below — Compose refuses to have the
|
||||
# same target twice and will not silently pick one.
|
||||
#
|
||||
# `!override` needs Compose >= 2.24.4. Whether the collision is actually gone is checked in the
|
||||
# merged model rather than assumed from this line.
|
||||
@@ -74,11 +73,9 @@ services:
|
||||
disable: true
|
||||
# Do not restart automatically so crash loops stay visible.
|
||||
restart: "no"
|
||||
# Heap dumps survive the container that wrote them.
|
||||
# The base declares /var/tmp/heap as a tmpfs, which is right for an ephemeral runtime and wrong
|
||||
# for dev: a heap dump written into a tmpfs dies with the container that produced it, which is
|
||||
# the one moment somebody wants the file. Compose refuses to have both, and correctly — it will
|
||||
# not silently pick one — so the tmpfs list is replaced rather than appended to.
|
||||
# Heap dumps survive the container that wrote them. The base mounts nothing at /var/tmp/heap, so
|
||||
# there the JVM cannot write a dump at all; a tmpfs would not do instead, since it dies with the
|
||||
# container that produced it, which is the one moment somebody wants the file.
|
||||
#
|
||||
# A named volume, not a bind mount of ./tmp/heap-dumps: Docker created that folder as root inside
|
||||
# the checkout, and a git worktree holding it could not be removed without root. Copy a dump out
|
||||
|
||||
-119
@@ -1,119 +0,0 @@
|
||||
package dev.caskeleton.bootstrap.compose;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.nio.file.Files;
|
||||
import java.nio.file.Path;
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
import java.util.concurrent.TimeUnit;
|
||||
import org.junit.jupiter.api.Assumptions;
|
||||
import org.junit.jupiter.api.DisplayName;
|
||||
import org.junit.jupiter.api.Test;
|
||||
|
||||
/**
|
||||
* Every shipped Compose file stack must at least render.
|
||||
*
|
||||
* <p>A stack that cannot merge has no failure mode short of total: nothing starts, and the error
|
||||
* arrives at the moment somebody most wants the environment. The dev stack was in exactly that
|
||||
* state — the base declares {@code /var/tmp/heap} as a tmpfs, the dev overlay mounts a volume at
|
||||
* the same target so heap dumps survive the container, and the overlay\'s {@code tmpfs: []} merged
|
||||
* with the base list instead of replacing it. {@code !override} replaces it, and the merged model
|
||||
* is checked for duplicate mount targets rather than the fix being assumed from the syntax.
|
||||
*/
|
||||
class ComposeMergeCharacterizationTest {
|
||||
|
||||
private static final Path REPOSITORY_ROOT = repositoryRoot();
|
||||
|
||||
@Test
|
||||
@DisplayName("the development stack renders")
|
||||
void developmentStackRenders() {
|
||||
assertStackRenders("compose.yaml");
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("the dev lane stack renders")
|
||||
void devStackRenders() {
|
||||
assertStackRenders("compose.yaml", "src/bootstrap/app/compose-lanes/dev.yaml");
|
||||
}
|
||||
|
||||
/**
|
||||
* The lane contract, checked by the script that owns it.
|
||||
*
|
||||
* <p>Delegating rather than reimplementing: the script is the entry point CI calls, and a test
|
||||
* that re-derived the same checks in Java would be a second opinion that can agree with the
|
||||
* contract while the script disagrees. What this adds is that the check runs in the explicit
|
||||
* integration lane, where Docker Compose and companion CLI tools are environment prerequisites
|
||||
* rather than hidden unit-test dependencies.
|
||||
*/
|
||||
@Test
|
||||
@DisplayName("every lane renders exactly the services its contract names")
|
||||
void everyLaneMatchesItsContract() {
|
||||
Assumptions.assumeTrue(dockerComposeIsAvailable(), "docker compose is not on this machine");
|
||||
ProcessResult result = run(List.of("./scripts/verify-compose-profile-contracts.sh"));
|
||||
assertThat(result.exitCode())
|
||||
.as("verify-compose-profile-contracts.sh said:%n%s", result.output())
|
||||
.isZero();
|
||||
}
|
||||
|
||||
private static void assertStackRenders(String... files) {
|
||||
Assumptions.assumeTrue(dockerComposeIsAvailable(), "docker compose is not on this machine");
|
||||
List<String> command = new ArrayList<>(List.of("docker", "compose"));
|
||||
for (String file : files) {
|
||||
command.add("-f");
|
||||
command.add(file);
|
||||
}
|
||||
command.add("config");
|
||||
ProcessResult result = run(command);
|
||||
assertThat(result.exitCode())
|
||||
.as(
|
||||
"the %s stack must render; docker compose said:%n%s",
|
||||
String.join(" + ", files), result.output())
|
||||
.isZero();
|
||||
}
|
||||
|
||||
private static boolean dockerComposeIsAvailable() {
|
||||
try {
|
||||
return run(List.of("docker", "compose", "version", "--short")).exitCode() == 0;
|
||||
} catch (RuntimeException failure) {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
private static ProcessResult run(List<String> command) {
|
||||
try {
|
||||
Process process =
|
||||
new ProcessBuilder(command)
|
||||
.directory(REPOSITORY_ROOT.toFile())
|
||||
.redirectErrorStream(true)
|
||||
.start();
|
||||
String output =
|
||||
new String(
|
||||
process.getInputStream().readAllBytes(), java.nio.charset.StandardCharsets.UTF_8);
|
||||
if (!process.waitFor(120, TimeUnit.SECONDS)) {
|
||||
process.destroyForcibly();
|
||||
throw new IllegalStateException("docker compose did not finish within 120s");
|
||||
}
|
||||
return new ProcessResult(process.exitValue(), output);
|
||||
} catch (IOException failure) {
|
||||
throw new IllegalStateException("could not run " + command, failure);
|
||||
} catch (InterruptedException interrupted) {
|
||||
Thread.currentThread().interrupt();
|
||||
throw new IllegalStateException("interrupted while running " + command, interrupted);
|
||||
}
|
||||
}
|
||||
|
||||
private static Path repositoryRoot() {
|
||||
Path candidate = Path.of("").toAbsolutePath();
|
||||
while (candidate != null && !Files.isRegularFile(candidate.resolve("compose.yaml"))) {
|
||||
candidate = candidate.getParent();
|
||||
}
|
||||
if (candidate == null) {
|
||||
throw new IllegalStateException("could not locate the repository root from the test cwd");
|
||||
}
|
||||
return candidate;
|
||||
}
|
||||
|
||||
private record ProcessResult(int exitCode, String output) {}
|
||||
}
|
||||
Reference in New Issue
Block a user