fix(strict-lane)!: 성능 lane이 상한을 재지 않고 통과하지 못하게 한다
문제: 성능 lane의 기계 의존 상한(httpclient 스트리밍 힙 증가, Mongo 시간 gate)이 performance.assertions.enabled 뒤에 있었고 기본값이 자리마다 달랐다(httpclient build false, Mongo build true, Mongo 테스트 코드 false). workflow만 GRADLE_OPTS로 켰다. 그래서 desktop에서 손으로 돌린 ./gradlew :performanceCheck가 httpclient 힙 상한을 재지 않고 통과했다 (2026-11-05 r10 게이트). JPA pool lane도 전에 같은 속성 뒤에서 아무 상한도 비교하지 않았다(PoolLaneClaimTest). 방법: 규칙을 build-logic strict-lane convention 한 곳에 둔다. - PERFORMANCE lane 테스트 JVM에 performance.assertions.enabled=true를 넣는다. 빌드(-P, GRADLE_OPTS)나 lane(systemProperty, jvmArgs, task 설정)이 true 아닌 값을 주면 테스트 전에 실패하고, 지우거나 true로 바꾸라고 알려 준다. true는 받아 준다(파생 서비스 workflow가 넘긴다). - PERFORMANCE lane은 skip된 테스트가 있으면 실패한다. - httpclient·Mongo 테스트는 조건 없이 단언하고 잰 값을 한 줄 남긴다. 두 leaf의 속성 선언과 integration-main·release workflow의 GRADLE_OPTS 줄을 지웠다. 사람과 workflow가 같은 명령을 쓴다. verifyStrictLaneStages가 읽는 run: 줄은 그대로다. 버린 대안: 두 leaf 기본값만 true로(정의가 leaf마다 남아 다음 lane이 다시 복사한다), 속성 삭제만(옛 속성을 기본값 false로 읽는 파생 서비스 테스트가 조용히 건너뛴다), 측정 전용 태스크(쓸 곳이 없다: 숫자만 볼 때는 JMH, 실패한 단언과 매 실행 출력이 값을 남긴다). 같은 원인으로 함께 고친 문서: httpclient support-matrix·release-checklist의 없는 lane 이름 spring62CompatibilityTest와 빠진 main push 주기. 위험·한계: 손으로 돌린 performanceCheck가 이제 상한을 실제로 재므로 느린 기계에서는 실패할 수 있다. JVM 인자 공급자(jvmArgumentProviders)가 넣는 -D는 보지 않는다. Docker가 없을 때 skip되는 통합·시스템 lane은 지금 모두 실패한다(전부 skip이면 executed no test, 아니면 skip 안 되는 컨테이너 테스트가 실패) — 거짓 통과가 아니어서 고치지 않았다. 검증(desktop ~/b4-logs, flock 아래, 노트북과 같은 트리 — git diff sha256 일치): - RED(고치기 전 트리 + 새 테스트): build-logic 새 테스트 4건이 UnexpectedBuildSuccess로 실패 — aPerformanceLaneFailsOnAnExceededBound WhenTheBuildSetsNoAssertionsProperty, …FailsBeforeTestingWhenTheBuild TurnsItsAssertionsOff, …FailsWhenItsOwnDeclarationTurnsTheAssertionsOff, …FailsWhenOneOfItsTestsIsSkipped. jvmArgs 경로(…FailsWhenItsJvmArguments TurnTheAssertionsOff)도 같은 실패. 실제 lane: 속성 없이 httpClientPerformanceTest rc=0인데 "skipped machine-dependent bound" 출력. - GREEN: -p build-logic check javadoc rc=0. :performanceCheck rc=0(lane 5개 테스트 26, skip 0; httpclient 힙 증가 5,243,880 B / 상한 16,777,216 B, Mongo p50·p95·p99 1ms, examined/returned 1.0, pool 대기 0). -p optional-platforms performanceCheck rc=0(grpc 1). 두 명령에 -Pperformance.assertions.enabled=false → rc=1, 6개 lane 모두 테스트 전 실패. =true → rc=0. verifyStrictLaneStages --dry-run 두 빌드 rc=0, architectureCheck rc=0, ci·mainCheck·integrationCheck·releaseCheck· performanceCheck --dry-run --configuration-cache rc=0, :mainCheck --continue --warning-mode=fail rc=0, -p optional-platforms ci rc=0, persistence-mongo:check rc=0, verify-httpclient-docs.py rc=0. - 이 변경 밖 실패: :ci :verifyDependencyLocks --continue와 httpclient:check가 CircuitBreakerStatusAccountingTest JDK 2건으로 rc=1. 기준 커밋 3d73b6f1에서도 10/10 실패(이 트리도 10/10). ci의 다른 task는 모두 통과했다. Refs: docs/upgrade/2026-11-maturity-review.md B-43, docs/testing/TESTING_STRATEGY.md 6절 BREAKING CHANGE: PERFORMANCE lane은 performance.assertions.enabled가 true가 아니거나 skip된 테스트가 있으면 실패한다. 자기 성능 lane의 그 속성 선언(false)과 skip을 없앤다.
This commit is contained in:
+1
-1
@@ -27,7 +27,7 @@ Checkstyle은 **포매터가 표현할 수 없는 것만** — 네이밍과 몇
|
||||
| `configContractCheck` | `:app-bootstrap:verifyEnvKeys`로 환경 설정 계약을 검사 |
|
||||
| `qualificationCheck` | 전체 기동(system) 레인과 보조 source set Checkstyle을 실행 |
|
||||
| `integrationCheck` | 선언된 strict integration lane(sample 포함)을 실행. 대부분 container runtime이 필요. 입력이 필요한 lane(Redis topology: `-Predis.topology.mode`, AWS: 보호 자격 증명 env)은 입력이 없으면 빠지고 집계가 그 이유를 출력 |
|
||||
| `performanceCheck` | 선언된 strict performance lane(부하·종료·pool·자원 상한)을 실행. 실서버·container가 필요. `integration-main.yml` `performance-check`가 main·nightly에 돌림 |
|
||||
| `performanceCheck` | 선언된 strict performance lane(부하·종료·pool·자원 상한)을 실행. 실서버·container가 필요. 상한은 매번 단언하고 끄는 속성이 없다(`-Pperformance.assertions.enabled`가 `true`가 아니면 실패, skip된 테스트가 있어도 실패). `integration-main.yml` `performance-check`가 main·nightly에 같은 명령으로 돌림 |
|
||||
| `verifyStrictLaneStages` | `--dry-run` 전용. `explicitOnly`가 아닌 strict lane이 workflow가 실제 실행하는 단계(`ci`·`:sample-portfolio:check`·`mainCheck`·`integrationCheck`·`performanceCheck`·`releaseCheck`) 중 하나의 계획에 있는지, 각 단계를 workflow가 `--dry-run` 없이 실행하는지 검사. Gradle이 작업을 돌리지 않는 명령(`--dry-run`·`-v`·`--help` 등)과, 실패해도 단계가 실패하지 않을 수 있거나(`\|\| true`, pipefail 없는 파이프, `if` 조건 등) 돌지 않을 수 있거나(조건부 `exit 0` 뒤, 요청으로만 시작하는 workflow 등) 검사가 읽지 않는(`\|\| exit 1`, `{ }` 안 등) 명령은 실행으로 세지 않는다. 자세한 목록은 `build-logic/README.md`. PR 게이트가 돌림 |
|
||||
| `ci` | PR 게이트: 모든 leaf `check`(API surface 검사 포함, 소유자 결정 20) + `architectureCheck` + `qualityCheck` |
|
||||
| `mainCheck` | main 단계: `configContractCheck` + `qualificationCheck` + sample web 계약 + sample 전체 기동(`:sample-portfolio:systemTest`) + `verifyPublicPathSnapshot` + 전송·메시징 qualification 레인 + 각 leaf의 `mainCheck`(문서 대조 레인 `*DocumentContractTest`, objectstorage MinIO 레인) |
|
||||
|
||||
@@ -81,8 +81,9 @@ Default lane: `./gradlew :adapter:outbound:httpclient:test`. Additional lanes, a
|
||||
| `jmh` | per-call overhead benchmarks |
|
||||
|
||||
A selected lane never skips silently: the fault lane throws without Docker, the contract lane throws
|
||||
on an empty or unknown transport selection, and the performance lane prints which machine-dependent
|
||||
bounds were not asserted.
|
||||
on an empty or unknown transport selection, and the performance lane asserts every bound in every
|
||||
run — the strict-lane convention fails it when the build sets `performance.assertions.enabled` to
|
||||
anything but `true` or when one of its tests is skipped.
|
||||
|
||||
The transport port (`BlockingTransportProvider`) has one abstract contract in the test fixtures,
|
||||
`testkit/BlockingTransportProviderContract`. The engines pass it over a real socket
|
||||
|
||||
@@ -111,8 +111,9 @@ cd src
|
||||
```
|
||||
|
||||
No lane skips silently. The fault lane fails without Docker, the contract lane fails on an empty or
|
||||
unknown transport selection, and the performance lane reports which machine-dependent bounds it did
|
||||
not assert.
|
||||
unknown transport selection, and the performance lane asserts every bound in every run (it fails
|
||||
when the build sets `performance.assertions.enabled` to anything but `true`, or when a test is
|
||||
skipped).
|
||||
|
||||
## Further reading
|
||||
|
||||
|
||||
@@ -138,8 +138,9 @@ dependencies {
|
||||
}
|
||||
|
||||
// Which transports the contract suites cover and whether the HTTP/3 opt-in runs: the same two
|
||||
// properties for every Test task of this leaf, declared once (review 2026-11-01 O-10a). Values a
|
||||
// single lane needs (the performance assertions, the httpbin image) stay in that lane.
|
||||
// properties for every Test task of this leaf, declared once (review 2026-11-01 O-10a). A value a
|
||||
// single lane needs (the httpbin image) stays in that lane. The performance lane's bounds have no
|
||||
// switch here: the strict-lane convention asserts them in every run.
|
||||
tasks.withType(Test).configureEach {
|
||||
systemProperty 'httpclient.contract.transports',
|
||||
providers.gradleProperty('httpclient.contract.transports').orElse('apache,jdk').get()
|
||||
@@ -193,8 +194,6 @@ strictTestLanes {
|
||||
performance()
|
||||
sourceSet = 'httpClientPerformanceTest'
|
||||
description = 'Certifies pool, streaming, retry, and rotation resource bounds (design §28.8).'
|
||||
systemProperty 'performance.assertions.enabled',
|
||||
providers.gradleProperty('performance.assertions.enabled').orElse('false').get()
|
||||
}
|
||||
lane('httpClientFailureInjectionTest') {
|
||||
integration()
|
||||
|
||||
+10
-3
@@ -52,11 +52,18 @@ class LargeBodyResourceTest {
|
||||
}
|
||||
}
|
||||
long peakIncrease = usedHeap() - before;
|
||||
// The measured value, so a passing run is a recorded number and not only a green line.
|
||||
System.out.println(
|
||||
"[httpclient-performance] streaming download heap increase "
|
||||
+ peakIncrease
|
||||
+ " bytes, bound "
|
||||
+ payloadBytes / 2
|
||||
+ " bytes");
|
||||
|
||||
assertThat(consumed).isEqualTo(payloadBytes);
|
||||
PerformanceAssertions.machineDependent(
|
||||
"streaming download heap increase stays well below the payload size",
|
||||
() -> assertThat(peakIncrease).isLessThan((long) payloadBytes / 2));
|
||||
assertThat(peakIncrease)
|
||||
.as("streaming download heap increase stays well below the payload size")
|
||||
.isLessThan((long) payloadBytes / 2);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
+7
-20
@@ -3,32 +3,19 @@ package dev.caskeleton.adapter.outbound.httpclient.performance;
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
|
||||
/**
|
||||
* Separates deterministic bounds from machine-dependent ones (design §28.8, §37).
|
||||
* Structural bounds — retry budget arithmetic, a refresh collapsing to one request, drain
|
||||
* completing — that hold on any machine (design §28.8, §37).
|
||||
*
|
||||
* <p>Structural bounds — retry budget arithmetic, streaming not buffering the whole payload, drain
|
||||
* completing — always run. Latency and absolute memory numbers depend on the host, so they are
|
||||
* asserted only when the certification flag is set; when it is off the lane still executes and
|
||||
* still reports, rather than becoming a silent pass.
|
||||
* <p>The machine-dependent bound (the streaming heap increase) is asserted in place, in every run.
|
||||
* It used to sit behind {@code performance.assertions.enabled}, which defaulted to {@code false} in
|
||||
* this leaf, so a lane run by hand reported success with the bound compared to nothing. A
|
||||
* performance lane now asserts all of its bounds; the strict-lane convention rejects a build that
|
||||
* tries to turn them off.
|
||||
*/
|
||||
final class PerformanceAssertions {
|
||||
|
||||
private PerformanceAssertions() {}
|
||||
|
||||
static boolean certificationEnabled() {
|
||||
return Boolean.parseBoolean(System.getProperty("performance.assertions.enabled", "false"));
|
||||
}
|
||||
|
||||
static void machineDependent(String description, Runnable assertion) {
|
||||
if (certificationEnabled()) {
|
||||
assertion.run();
|
||||
return;
|
||||
}
|
||||
System.out.println(
|
||||
"[httpclient-performance] skipped machine-dependent bound (enable with "
|
||||
+ "-Pperformance.assertions.enabled=true): "
|
||||
+ description);
|
||||
}
|
||||
|
||||
static void structural(String description, boolean condition) {
|
||||
assertThat(condition).describedAs(description).isTrue();
|
||||
}
|
||||
|
||||
@@ -239,9 +239,8 @@ strictTestLanes {
|
||||
providers.gradleProperty('mongoCompatibilityImage').orElse(testContainerImages['mongo-7']).get()
|
||||
systemProperty 'mongodb.toxiproxy.image',
|
||||
providers.gradleProperty('mongoToxiproxyImage').orElse(testContainerImages['toxiproxy']).get()
|
||||
// Assertions on by default: a performance gate without bounds is only a report.
|
||||
systemProperty 'performance.assertions.enabled',
|
||||
providers.gradleProperty('performance.assertions.enabled').orElse('true').get()
|
||||
// No assertions switch: the strict-lane convention asserts a performance lane's bounds in
|
||||
// every run, because a performance gate without bounds is only a report.
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
+8
-10
@@ -27,16 +27,16 @@ import org.junit.jupiter.api.Test;
|
||||
* it is the same on an idle machine and a busy one, and it is what actually predicts the query
|
||||
* falling over as the collection grows.
|
||||
*
|
||||
* <p>Timing bounds are asserted only when {@code -Pperformance.assertions.enabled=true}, for the
|
||||
* same reason. The ratio bound is always asserted.
|
||||
* <p>The timing bounds are asserted too, in every run, but the gate's bounds are generous for the
|
||||
* same reason: they catch a query that falls over, not one that was slower than last time. They
|
||||
* used to sit behind {@code performance.assertions.enabled}, which the build defaulted to on and
|
||||
* this class to off, so a run outside Gradle skipped them; the strict-lane convention now asserts a
|
||||
* performance lane's bounds in every run.
|
||||
*/
|
||||
class MongoResourceBudgetLaneTest {
|
||||
|
||||
private static final int DOCUMENT_COUNT = 5_000;
|
||||
|
||||
private static final boolean TIMING_ASSERTIONS =
|
||||
Boolean.parseBoolean(System.getProperty("performance.assertions.enabled", "false"));
|
||||
|
||||
private static MongoSingleReplicaSetContainer container;
|
||||
|
||||
private static MongoClient client;
|
||||
@@ -75,18 +75,16 @@ class MongoResourceBudgetLaneTest {
|
||||
@Test
|
||||
void anIndexedQueryStaysInsideTheResourceBudget() {
|
||||
MongoResourceBudgetReport report = measure(new Document("status", "PENDING"), "ix_status");
|
||||
// The measured values, so a passing run is a recorded number and not only a green line.
|
||||
System.out.println("[mongo-performance] indexed query " + report.asArtifact());
|
||||
|
||||
assertThat(report.documentsReturned()).isEqualTo(DOCUMENT_COUNT / 50);
|
||||
assertThat(report.examinedToReturnedRatio())
|
||||
.as("an index seek examines about what it returns")
|
||||
.isLessThanOrEqualTo(MongoPerformanceGate.standard().maximumExaminedToReturnedRatio());
|
||||
assertThat(MongoPerformanceGate.standard().violations(report))
|
||||
.filteredOn(violation -> violation.startsWith("examined/returned"))
|
||||
.as("an indexed query stays inside every bound of the standard gate")
|
||||
.isEmpty();
|
||||
|
||||
if (TIMING_ASSERTIONS) {
|
||||
assertThat(MongoPerformanceGate.standard().passes(report)).isTrue();
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
|
||||
File diff suppressed because one or more lines are too long
+81
-3
@@ -10,12 +10,26 @@ import org.gradle.api.NamedDomainObjectContainer;
|
||||
import org.gradle.api.Plugin;
|
||||
import org.gradle.api.Project;
|
||||
import org.gradle.api.plugins.JavaPluginExtension;
|
||||
import org.gradle.api.provider.Provider;
|
||||
import org.gradle.api.tasks.SourceSet;
|
||||
import org.gradle.api.tasks.TaskProvider;
|
||||
import org.gradle.api.tasks.testing.Test;
|
||||
import org.gradle.testing.base.TestingExtension;
|
||||
|
||||
public final class StrictTestLanePlugin implements Plugin<Project> {
|
||||
/**
|
||||
* The switch a performance lane's machine-dependent bounds used to sit behind, now always on.
|
||||
*
|
||||
* <p>Each leaf declared it with its own default — {@code false} for the httpclient lane, {@code
|
||||
* true} for the Mongo lane — and only the workflows set it. {@code ./gradlew :performanceCheck}
|
||||
* typed by hand therefore passed with the httpclient heap bound compared to nothing (the
|
||||
* 2026-11-05 desktop gate). The convention owns it now: every performance lane's test JVM gets
|
||||
* {@code true}, so a test that still reads the switch asserts, and a build that sets it to
|
||||
* anything else fails before the lane runs. Nothing in this repository reads it any more; the
|
||||
* value is for tests written against the old switch, such as a derived service's.
|
||||
*/
|
||||
static final String PERFORMANCE_ASSERTIONS = "performance.assertions.enabled";
|
||||
|
||||
@Override
|
||||
public void apply(Project project) {
|
||||
project.getPluginManager().withPlugin("java", ignored -> configure(project));
|
||||
@@ -229,7 +243,8 @@ public final class StrictTestLanePlugin implements Plugin<Project> {
|
||||
// judge the lane again rather than reuse a result produced under the looser policy. Without
|
||||
// this a lane that skipped a test stayed green, up to date and from the build cache, after
|
||||
// rejectSkipped was switched on (review 2026-11-03 RR-06-02).
|
||||
test.getInputs().property("strictLane.rejectSkipped", project.provider(lane::isRejectSkipped));
|
||||
test.getInputs()
|
||||
.property("strictLane.rejectSkipped", project.provider(() -> rejectsSkipped(lane)));
|
||||
test.getInputs()
|
||||
.property("strictLane.requiredTests", project.provider(lane::getRequiredTests));
|
||||
test.getInputs()
|
||||
@@ -243,6 +258,17 @@ public final class StrictTestLanePlugin implements Plugin<Project> {
|
||||
test.getOutputs().upToDateWhen(ignored -> false);
|
||||
test.getOutputs().doNotCacheIf(rerunReason, ignored -> true);
|
||||
}
|
||||
boolean performance = lane.getCategory() == TestLaneCategory.PERFORMANCE;
|
||||
if (performance) {
|
||||
// Set before the lane's own JVM arguments and system properties — Gradle turns a -D in
|
||||
// jvmArgs into a system property — so a lane that still declares the switch replaces this
|
||||
// value and the check below sees the override, instead of the override disappearing silently.
|
||||
test.systemProperty(PERFORMANCE_ASSERTIONS, "true");
|
||||
Provider<String> requested = project.getProviders().gradleProperty(PERFORMANCE_ASSERTIONS);
|
||||
String laneName = lane.getName();
|
||||
test.doFirst(
|
||||
running -> requireAssertionsOn(laneName, ownerPath, requested.getOrNull(), (Test) running));
|
||||
}
|
||||
if (lane.getMaxHeapSize() != null && !lane.getMaxHeapSize().isBlank()) {
|
||||
test.setMaxHeapSize(lane.getMaxHeapSize());
|
||||
}
|
||||
@@ -288,7 +314,7 @@ public final class StrictTestLanePlugin implements Plugin<Project> {
|
||||
+ detail
|
||||
+ " — a lane that runs nothing reports success for whatever it was meant to prove.");
|
||||
}
|
||||
if (lane.isRejectSkipped() && tracker.skippedCount() > 0L) {
|
||||
if (rejectsSkipped(lane) && tracker.skippedCount() > 0L) {
|
||||
throw new GradleException(
|
||||
"strict test lane '"
|
||||
+ lane.getName()
|
||||
@@ -296,7 +322,10 @@ public final class StrictTestLanePlugin implements Plugin<Project> {
|
||||
+ ownerPath
|
||||
+ " forbids skipped tests but observed "
|
||||
+ tracker.skippedCount()
|
||||
+ " skipped test(s).");
|
||||
+ " skipped test(s)."
|
||||
+ (performance
|
||||
? " A performance lane always does: a skipped test is a bound nobody measured."
|
||||
: ""));
|
||||
}
|
||||
if (!required.isEmpty()) {
|
||||
List<String> absent =
|
||||
@@ -318,6 +347,55 @@ public final class StrictTestLanePlugin implements Plugin<Project> {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* A performance lane rejects skipped tests whatever it declares: its tests are its bounds, and a
|
||||
* skip is a bound the run reports as passed without comparing it to anything.
|
||||
*/
|
||||
private static boolean rejectsSkipped(StrictTestLaneSpec lane) {
|
||||
return lane.isRejectSkipped() || lane.getCategory() == TestLaneCategory.PERFORMANCE;
|
||||
}
|
||||
|
||||
/**
|
||||
* Fails a performance lane, before any test runs, when the build or the lane turns its bounds off.
|
||||
* {@code true} from the build is accepted, so a command that still passes it keeps working.
|
||||
*/
|
||||
private static void requireAssertionsOn(
|
||||
String laneName, String ownerPath, String requestedByBuild, Test test) {
|
||||
if (requestedByBuild != null && !Boolean.parseBoolean(requestedByBuild.trim())) {
|
||||
throw new GradleException(
|
||||
"strict test lane '"
|
||||
+ laneName
|
||||
+ "' in "
|
||||
+ ownerPath
|
||||
+ " is a performance lane, and this build sets "
|
||||
+ PERFORMANCE_ASSERTIONS
|
||||
+ "="
|
||||
+ requestedByBuild
|
||||
+ " (-P"
|
||||
+ PERFORMANCE_ASSERTIONS
|
||||
+ ", or -Dorg.gradle.project."
|
||||
+ PERFORMANCE_ASSERTIONS
|
||||
+ " in GRADLE_OPTS). A performance lane asserts its bounds in every run: one without"
|
||||
+ " them measures nothing it can fail on and still reports success. Remove the"
|
||||
+ " property or set it to true.");
|
||||
}
|
||||
Object effective = test.getSystemProperties().get(PERFORMANCE_ASSERTIONS);
|
||||
if (effective == null || !Boolean.parseBoolean(String.valueOf(effective).trim())) {
|
||||
throw new GradleException(
|
||||
"strict test lane '"
|
||||
+ laneName
|
||||
+ "' in "
|
||||
+ ownerPath
|
||||
+ " is a performance lane, but its test JVM would get "
|
||||
+ PERFORMANCE_ASSERTIONS
|
||||
+ "="
|
||||
+ effective
|
||||
+ " from this project's build (a lane systemProperty or jvmArgs, or a block that"
|
||||
+ " configures the task). The strict-lane convention sets it to true for every"
|
||||
+ " performance lane; delete the override.");
|
||||
}
|
||||
}
|
||||
|
||||
private static boolean startsARealServer(StrictTestLaneSpec lane) {
|
||||
return lane.getCategory() != TestLaneCategory.TEST
|
||||
&& lane.getCategory() != TestLaneCategory.ARCHITECTURE;
|
||||
|
||||
+6
@@ -137,6 +137,12 @@ public class StrictTestLaneSpec implements Named {
|
||||
category = TestLaneCategory.ARCHITECTURE;
|
||||
}
|
||||
|
||||
/**
|
||||
* A lane whose tests are resource or latency bounds. Besides rerunning every time and taking the
|
||||
* build-wide real-server slot, it rejects skipped tests and always runs with its bounds asserted:
|
||||
* the convention sets {@code performance.assertions.enabled=true}, and a build or lane that sets
|
||||
* it to anything else fails before a test runs.
|
||||
*/
|
||||
public void performance() {
|
||||
category = TestLaneCategory.PERFORMANCE;
|
||||
}
|
||||
|
||||
+167
@@ -611,6 +611,147 @@ class StrictTestLaneConventionTest {
|
||||
assertThat(result.getOutput()).contains("rulesLane=false").contains("databaseLane=true");
|
||||
}
|
||||
|
||||
@Test
|
||||
void aPerformanceLaneFailsOnAnExceededBoundWhenTheBuildSetsNoAssertionsProperty()
|
||||
throws Exception {
|
||||
// given
|
||||
buildFile(performanceLane(""));
|
||||
writePerformanceTest(
|
||||
"BoundTest.java",
|
||||
"""
|
||||
import org.junit.jupiter.api.Test;
|
||||
class BoundTest {
|
||||
@Test void staysInsideTheBound() {
|
||||
long p99Millis = 900;
|
||||
// The shape the httpclient lane had: the machine-dependent bound behind the switch.
|
||||
if (Boolean.parseBoolean(System.getProperty("performance.assertions.enabled", "false"))
|
||||
&& p99Millis > 500) {
|
||||
throw new AssertionError("p99 900ms above the 500ms bound");
|
||||
}
|
||||
}
|
||||
}
|
||||
""");
|
||||
|
||||
// when
|
||||
BuildResult result = runner("performanceTestLaneCheck").buildAndFail();
|
||||
|
||||
// then
|
||||
assertThat(result.getOutput()).contains("BoundTest > staysInsideTheBound() FAILED");
|
||||
}
|
||||
|
||||
@Test
|
||||
void aPerformanceLaneFailsBeforeTestingWhenTheBuildTurnsItsAssertionsOff() throws Exception {
|
||||
// given
|
||||
buildFile(performanceLane(""));
|
||||
writePerformanceTest(
|
||||
"MeasuredTest.java",
|
||||
"""
|
||||
import org.junit.jupiter.api.Test;
|
||||
class MeasuredTest {
|
||||
@Test void measured() { throw new AssertionError("the lane ran a test"); }
|
||||
}
|
||||
""");
|
||||
|
||||
// when
|
||||
BuildResult result =
|
||||
runner("performanceTestLaneCheck", "-Pperformance.assertions.enabled=false")
|
||||
.buildAndFail();
|
||||
|
||||
// then
|
||||
assertThat(result.getOutput())
|
||||
.contains("this build sets performance.assertions.enabled=false")
|
||||
.contains("set it to true")
|
||||
.doesNotContain("MeasuredTest > measured() FAILED");
|
||||
}
|
||||
|
||||
@Test
|
||||
void aPerformanceLaneFailsWhenItsOwnDeclarationTurnsTheAssertionsOff() throws Exception {
|
||||
// given
|
||||
buildFile(
|
||||
performanceLane(
|
||||
"""
|
||||
systemProperty 'performance.assertions.enabled',
|
||||
providers.gradleProperty('performance.assertions.enabled').orElse('false').get()
|
||||
"""));
|
||||
writePerformanceTest(
|
||||
"MeasuredTest.java",
|
||||
"""
|
||||
import org.junit.jupiter.api.Test;
|
||||
class MeasuredTest { @Test void measured() {} }
|
||||
""");
|
||||
|
||||
// when
|
||||
BuildResult result = runner("performanceTestLaneCheck").buildAndFail();
|
||||
|
||||
// then
|
||||
assertThat(result.getOutput())
|
||||
.contains("its test JVM would get performance.assertions.enabled=false")
|
||||
.contains("delete the override");
|
||||
}
|
||||
|
||||
@Test
|
||||
void aPerformanceLaneFailsWhenItsJvmArgumentsTurnTheAssertionsOff() throws Exception {
|
||||
// given
|
||||
buildFile(performanceLane("jvmArgs '-Dperformance.assertions.enabled=false'"));
|
||||
writePerformanceTest(
|
||||
"MeasuredTest.java",
|
||||
"""
|
||||
import org.junit.jupiter.api.Test;
|
||||
class MeasuredTest { @Test void measured() {} }
|
||||
""");
|
||||
|
||||
// when
|
||||
BuildResult result = runner("performanceTestLaneCheck").buildAndFail();
|
||||
|
||||
// then
|
||||
assertThat(result.getOutput())
|
||||
.contains("its test JVM would get performance.assertions.enabled=false")
|
||||
.contains("delete the override");
|
||||
}
|
||||
|
||||
@Test
|
||||
void aPerformanceLaneFailsWhenOneOfItsTestsIsSkipped() throws Exception {
|
||||
// given
|
||||
buildFile(performanceLane(""));
|
||||
writePerformanceTest(
|
||||
"PartlyMeasuredTest.java",
|
||||
"""
|
||||
import org.junit.jupiter.api.Disabled;
|
||||
import org.junit.jupiter.api.Test;
|
||||
class PartlyMeasuredTest {
|
||||
@Test void measured() {}
|
||||
@Test @Disabled("too slow on this machine") void quarantined() {}
|
||||
}
|
||||
""");
|
||||
|
||||
// when
|
||||
BuildResult result = runner("performanceTestLaneCheck").buildAndFail();
|
||||
|
||||
// then
|
||||
assertThat(result.getOutput())
|
||||
.contains("forbids skipped tests")
|
||||
.contains("a skipped test is a bound nobody measured");
|
||||
}
|
||||
|
||||
@Test
|
||||
void aPerformanceLaneAcceptsABuildThatTurnsItsAssertionsOnExplicitly() throws Exception {
|
||||
// given
|
||||
buildFile(performanceLane(""));
|
||||
writePerformanceTest(
|
||||
"MeasuredTest.java",
|
||||
"""
|
||||
import org.junit.jupiter.api.Test;
|
||||
class MeasuredTest { @Test void measured() {} }
|
||||
""");
|
||||
|
||||
// when
|
||||
BuildResult result =
|
||||
runner("performanceTestLaneCheck", "-Pperformance.assertions.enabled=true").build();
|
||||
|
||||
// then
|
||||
assertThat(result.task(":loadLane").getOutcome()).isEqualTo(TaskOutcome.SUCCESS);
|
||||
}
|
||||
|
||||
private void buildFile(String laneBlock) throws Exception {
|
||||
Files.writeString(
|
||||
projectDir.resolve("build.gradle"),
|
||||
@@ -631,6 +772,32 @@ class StrictTestLaneConventionTest {
|
||||
Files.writeString(testSource.resolve(fileName), source);
|
||||
}
|
||||
|
||||
private void writePerformanceTest(String fileName, String source) throws Exception {
|
||||
Path performanceSource = projectDir.resolve("src/performance/java");
|
||||
Files.createDirectories(performanceSource);
|
||||
Files.writeString(performanceSource.resolve(fileName), source);
|
||||
}
|
||||
|
||||
private static String performanceLane(String extraDeclarations) {
|
||||
return """
|
||||
sourceSets { performance }
|
||||
dependencies {
|
||||
performanceImplementation platform('org.junit:junit-bom:5.11.3')
|
||||
performanceImplementation 'org.junit.jupiter:junit-jupiter'
|
||||
performanceRuntimeOnly 'org.junit.platform:junit-platform-launcher'
|
||||
}
|
||||
strictTestLanes {
|
||||
lane('loadLane') {
|
||||
performance()
|
||||
sourceSet = 'performance'
|
||||
description = 'Measures a resource bound.'
|
||||
%s
|
||||
}
|
||||
}
|
||||
"""
|
||||
.formatted(extraDeclarations);
|
||||
}
|
||||
|
||||
private static String junitDependencies() {
|
||||
return """
|
||||
dependencies {
|
||||
|
||||
Reference in New Issue
Block a user