fix(errors)!: 서버 포화는 모든 전송이 공용 SERVER_BUSY 하나로 답한다

문제: 서버 쪽 상한이 찬 거절이 전송마다 다른 코드였다. web-advanced는
ADMISSION_REFUSED(503), fileserver는 instance·전송 pool 포화까지
TRANSFER_ADMISSION_REJECTED(429 RATE_LIMIT), gRPC는 RATE_LIMIT_EXCEEDED
(RESOURCE_EXHAUSTED, "요청이 너무 잦습니다")였다. client는 자기 빈도 탓인지 서버
포화인지 가를 수 없었다. web은 상태만 아는 모든 503을 REQUEST_TIMEOUT으로 답했다
(§4는 비동기 시간 초과만). §4에 OPERATION_NOT_FOUND가 없었고, fileserver 오류
code 27개 중 23개가 계약 문서에 없었으며 예시가 "data": null을 실었다. fileserver는
Retry-After를 발생만 보고 붙여, 본문이 retryable=false로 좁힌 code에도 헤더가
나갔다. gRPC Javadoc과 graphql 스냅숏 머리말이 지운 이름(WORKLOG_NOT_FOUND, spi)을
적었다.

고른 방법(리뷰 README 6절 ②):
- 공용 OperationalError.SERVER_BUSY(503, TRANSIENT_DEPENDENCY, retryable,
  Retry-After 1)를 더했다. registry 행, RetryAfterAdvisor, runbook
  server-busy.md를 같이 둔다.
- web-advanced admission 거절, gRPC call admission(UNAVAILABLE), fileserver의
  instance permit·직접 다운로드 permit·전송 pool 포화가 이 코드다. fileserver는
  같은 이름의 FileserverErrorCode.SERVER_BUSY를 두고, 이름이 같은 code의 뜻이
  같은지 FileserverExceptionTest가 이미 본다. 호출자 자기 scope의 동시 전송
  초과만 TRANSFER_ADMISSION_REJECTED 429로 남는다.
- 상태만 아는 503은 SERVER_BUSY, AsyncRequestTimeoutException만
  REQUEST_TIMEOUT이다(HttpStatusErrorCodes.forException).
- fileserver Retry-After는 본문과 같은 규칙(code와 발생이 모두 retryable)으로
  붙인다. 재시도 불가 code인 ATOMIC_PUBLISH_UNSUPPORTED를 표에서 뺐다.
- docs/fileserver/http-contract.md에 오류 code 표 전체를 두고 예시의 data:null을
  지웠다. http-contract §4에 OPERATION_NOT_FOUND와 SERVER_BUSY를 적었다.
- decisions 2.3에 결정 ①②③ 행을 더했다.
- 포화 시험의 sleep(200)을 첫 거절 신호로 바꾸고, timeout 시험이 code까지
  본다(같은 파일이라 RR9-03-20의 그 부분을 여기서 고친다).
- web-advanced 파일의 spotless 정리(앞 커밋분)를 함께 싣는다.
버린 대안: JOB_EXECUTOR_REJECTED를 재사용 — 백그라운드 job 이름이라 요청 허가
거절을 읽는 client와 runbook을 잘못 이끈다.

위험·한계: 포화 거절로 분기하던 client와 재시도 정책, WebAdvancedError
.ADMISSION_REFUSED를 참조하던 코드, 모든 503을 REQUEST_TIMEOUT으로 보던 client가
바뀐다(upgrade 안내 8·9·13).

BREAKING CHANGE: 서버 포화 코드가 SERVER_BUSY 하나다(web-advanced
ADMISSION_REFUSED 삭제, fileserver instance·pool 포화 429 → 503, gRPC
RESOURCE_EXHAUSTED → UNAVAILABLE). 상태만 아는 503은 SERVER_BUSY다. 할 일은
docs/upgrade/2026-11-round9-contracts.md.

Refs: RR9-03-06, RR9-03-07, RR9-03-11, RR9-03-12, RR9-03-15, RR9-03-16, RR9-03-20
This commit is contained in:
DongHyeonka
2026-11-13 04:47:03 +09:00
parent 7b5b4c6983
commit 697e62e8cb
32 changed files with 244 additions and 81 deletions
+6 -6
View File
@@ -139,17 +139,17 @@ strictTestLanes {
// be.
//
// A snapshot does not shrink the surface. It makes each addition visible in review, which is the
// prerequisite for shrinking it: the `api` and `spi` packages are the surface an adopter is meant
// to use, and everything else in this file is a candidate for becoming internal when the leaf is
// split into capability artifacts. Until then the number cannot grow by accident.
// prerequisite for shrinking it: the `api` package is the surface an adopter is meant to use, and
// everything else in this file is a candidate for becoming internal when the leaf is split into
// capability artifacts. Until then the number cannot grow by accident.
apiSurface {
label = 'GraphQl'
baseline = rootProject.file('docs/architecture/graphql-api-surface.txt')
description = 'GraphQL leaf public API surface — every public top-level type in src/main/java.'
rationale = [
'A public type in a single-jar leaf is reachable from every adopter\'s code, so',
'additions are reviewed rather than discovered. `api` and `spi` are the intended',
'external surface; the rest are candidates to become internal when this leaf is',
'split into capability artifacts.',
'additions are reviewed rather than discovered. `api` is the intended external',
'surface; the rest are candidates to become internal when this leaf is split into',
'capability artifacts.',
]
}
+2 -1
View File
@@ -94,7 +94,8 @@ executor, 서버 deadline 없음, 연결 수명 무제한이다. 어댑터 서
health/reflection 에는 적용하지 않는다(`Health/Watch` 는 원래 오래 사는 스트림이다).
- **포화 시 동작(`GrpcCallAdmission`).** feature 호출은 `executor-max-pool-size +
executor-queue-capacity` 개까지만 받고, 넘친 호출은 handler 를 시작하지 않고 즉시
`RESOURCE_EXHAUSTED`(`RATE_LIMIT_EXCEEDED`, retryable)로 닫는다. 구현: io.grpc
`UNAVAILABLE`(`SERVER_BUSY`, retryable)로 닫는다. 서버 포화는 모든 전송이 이 공용 코드 하나다(2026-11-12 전에는
`RESOURCE_EXHAUSTED`/`RATE_LIMIT_EXCEEDED`였다). 구현: io.grpc
`ServerCallExecutorSupplier`(`@ExperimentalApi`, 1.39부터)가 호출 시작 전에 permit 을 확인해
받은 호출만 고정 크기 work pool 로 옮기고, 못 받은 호출은 가장 바깥 인터셉터가 닫는다. 메서드
조회·context 정리·health·reflection 은 작은 dispatch pool 에서 돈다. 버린 대안:
@@ -28,9 +28,11 @@ import java.util.concurrent.atomic.AtomicBoolean;
* dispatch executor, before the call starts. A feature call that gets a permit is moved to the
* bounded work pool; one that does not stays on the dispatch executor and is marked, and this same
* object, installed as the outermost feature interceptor, closes the marked call with {@code
* RATE_LIMIT_EXCEEDED} without starting the handler. The permit is returned once, when the admitted
* call completes or is cancelled. Health and reflection are not feature calls: they take no permit
* and run on the dispatch executor, so a saturated pool cannot make health lie.
* SERVER_BUSY} ({@code UNAVAILABLE}) without starting the handler: the shared code every transport
* answers server saturation with, not the caller-rate {@code RATE_LIMIT_EXCEEDED} it used to be.
* The permit is returned once, when the admitted call completes or is cancelled. Health and
* reflection are not feature calls: they take no permit and run on the dispatch executor, so a
* saturated pool cannot make health lie.
*
* <p>Each admitted call has at most one task in the work pool at a time (io.grpc serializes a
* call's callbacks), so a work queue as deep as the permit count can never reject.
@@ -79,8 +81,8 @@ final class GrpcCallAdmission implements ServerCallExecutorSupplier, ServerInter
ServerCall<Q, R> call, Metadata headers, ServerCallHandler<Q, R> next) {
if (refused.remove(headers)) {
call.close(
statusMapper.statusFor(OperationalError.RATE_LIMIT_EXCEEDED),
statusMapper.trailersFor(OperationalError.RATE_LIMIT_EXCEEDED));
statusMapper.statusFor(OperationalError.SERVER_BUSY),
statusMapper.trailersFor(OperationalError.SERVER_BUSY));
return new ServerCall.Listener<>() {};
}
AtomicBoolean released = new AtomicBoolean();
@@ -17,9 +17,7 @@ import io.grpc.Status;
*/
public class GrpcStatusMapper {
/**
* Trailer key carrying the stable {@link ApiErrorCode#code()} (e.g. {@code WORKLOG_NOT_FOUND}).
*/
/** Trailer key carrying the stable {@link ApiErrorCode#code()} (e.g. {@code SERVER_BUSY}). */
static final Metadata.Key<String> CODE_KEY =
Metadata.Key.of("error-code", Metadata.ASCII_STRING_MARSHALLER);
@@ -329,8 +329,8 @@ class GrpcP1BoundaryWireTest {
}
@Test
@DisplayName("스레드와 대기열이 모두 찬 서버는 넘친 호출을 기다리게 두지 않고 즉시 RESOURCE_EXHAUSTED로 거절한다")
void aCallBeyondTheExecutorAndQueueIsRefusedAtOnceAsResourceExhausted() {
@DisplayName("스레드와 대기열이 모두 찬 서버는 넘친 호출을 기다리게 두지 않고 즉시 UNAVAILABLE(SERVER_BUSY)로 거절한다")
void aCallBeyondTheExecutorAndQueueIsRefusedAtOnceAsServerBusy() {
withChannel(
contextRunner.withPropertyValues(
"app.grpc.server.executor-max-pool-size=1",
@@ -351,12 +351,11 @@ class GrpcP1BoundaryWireTest {
// then
assertThat(outcomes)
.extracting(Status::getCode)
.containsExactlyInAnyOrder(
Status.Code.OK, Status.Code.OK, Status.Code.RESOURCE_EXHAUSTED);
.containsExactlyInAnyOrder(Status.Code.OK, Status.Code.OK, Status.Code.UNAVAILABLE);
assertThat(outcomes)
.filteredOn(status -> !status.isOk())
.extracting(Status::getDescription)
.containsExactly(ClientErrorMessages.of(Category.RATE_LIMIT));
.containsExactly(ClientErrorMessages.of(Category.TRANSIENT_DEPENDENCY));
});
}
+2 -2
View File
@@ -51,12 +51,12 @@ virtual thread 실행기는 `TaskDecorator` bean 을 요구한다(요청 상관
이 leaf 의 예외는 모두 `ApiErrorCarrier` 다. 코드는 `error.WebAdvancedError` 한 enum 이 정하고, web 의
`GlobalExceptionHandler.handleCarrier` 한 분기가 Envelope 로 답한다(별도 `@RestControllerAdvice` 없음).
예전에는 10종 모두 `500 INTERNAL_ERROR` 였다(리뷰 2026-11-01 02-07). 아래 표와 enum 은 같은 값을 말하고,
예전에는 10종 모두 `500 INTERNAL_ERROR` 였다(리뷰 2026-11-01 02-07). admission 거절은 2026-11-12부터 공용 `SERVER_BUSY`다. 아래 표와 enum 은 같은 값을 말하고,
`WebAdvancedErrorMappingTest` 가 실제 응답으로 확인한다.
| 예외 | `error.code` | status | category | retryable | 응답에 더 실리는 것 | 클라이언트 행동 |
| --- | --- | --- | --- | --- | --- | --- |
| `AdmissionRefusedException` | `ADMISSION_REFUSED` | 503 | `TRANSIENT_DEPENDENCY` | true | `Retry-After: 1` | 잠시 뒤 같은 요청 |
| `AdmissionRefusedException` | `SERVER_BUSY`(공용 `OperationalError`) | 503 | `TRANSIENT_DEPENDENCY` | true | `Retry-After: 1` | 잠시 뒤 같은 요청. 서버 포화는 모든 전송이 이 코드 하나다 |
| `JsonPatchTestFailedException` | `PATCH_TEST_FAILED` | 409 | `CONFLICT` | false | `details.pointer`(서버 값은 없음) | 다시 읽고 새 patch |
| `JsonPatchRejectedException` | `PATCH_REJECTED` | 400 | `VALIDATION` | false | — | patch 문서를 고친다. JSON이 아닌 본문, 빈 문서, 한도·깊이 초과, 연산 모양 오류, 결과가 자원 타입으로 읽히지 않는 patch가 모두 이 코드다 |
| `PatchAuthorizationException` | `PATCH_PATH_FORBIDDEN` | 403 | `AUTHZ` | false | `details.refusedPaths` | 해당 경로를 빼거나 권한 확보 |
@@ -11,16 +11,14 @@ import dev.caskeleton.shared.error.Category;
* leaf README's "오류 응답" section; it states the same values.
*
* <p>Before this existed every one of these reached the client as {@code 500 INTERNAL_ERROR} with
* an ERROR log line, including an admission refusal the profile documents as a 503 (review
* 2026-11-01 02-07).
* an ERROR log line (review 2026-11-01 02-07). The admission refusal it used to name is now the
* shared {@code OperationalError.SERVER_BUSY} (review 2026-11-12).
*
* <p>Like {@code RateLimitTransportError}, these are not in {@code
* docs/registries/error-codes.yaml}, which is the {@code OperationalError} SSOT.
*/
public enum WebAdvancedError implements ApiErrorCode, RetryAfterHint {
/** The virtual-thread admission limit was full. A refusal in a millisecond; come back shortly. */
ADMISSION_REFUSED(Category.TRANSIENT_DEPENDENCY, 503, true, 1),
/** A JSON Patch {@code test} did not hold; nothing was applied. Re-read, then retry. */
PATCH_TEST_FAILED(Category.CONFLICT, 409, false, 0),
/** The patch document is malformed or over a limit; nothing was applied. */
@@ -29,8 +29,8 @@ import java.util.Objects;
* or a rewind it would apply.
*
* <p>What this writer does not do: it writes synchronously from an iterator on the request thread,
* so it sends no heartbeat while the source is blocked, enforces no idle timeout or maximum age, and
* keeps no buffer (a slow consumer blocks the source instead). Those bounds are in {@code
* so it sends no heartbeat while the source is blocked, enforces no idle timeout or maximum age,
* and keeps no buffer (a slow consumer blocks the source instead). Those bounds are in {@code
* WebStreamPolicy} and the drain in {@code WebStreamDrainCoordinator} for a service that runs its
* streams through {@code WebStreamRegistry}; {@code docs/web/streaming-contract.md} says which.
*
@@ -4,7 +4,7 @@ import dev.caskeleton.adapter.inbound.web.advanced.virtualthread.AdmissionRefuse
import dev.caskeleton.adapter.inbound.web.advanced.virtualthread.VirtualThreadAdmissionGuard;
import dev.caskeleton.adapter.inbound.web.error.EnvelopeResponseWriter;
import dev.caskeleton.adapter.inbound.web.http.ApiHeaders;
import dev.caskeleton.adapter.inbound.web.observability.RetryAfterHint;
import dev.caskeleton.adapter.inbound.web.observability.RetryAfterAdvisor;
import jakarta.servlet.AsyncEvent;
import jakarta.servlet.AsyncListener;
import jakarta.servlet.FilterChain;
@@ -26,8 +26,8 @@ import tools.jackson.databind.ObjectMapper;
* security chain, and refuses before any of it runs.
*
* <p>A request that goes async keeps its permit until the async work ends: releasing when this
* thread returns would bound only the hand-off, and the work the limit protects the downstream pools
* from happens after it.
* thread returns would bound only the hand-off, and the work the limit protects the downstream
* pools from happens after it.
*/
public final class VirtualThreadAdmissionFilter extends OncePerRequestFilter implements Ordered {
@@ -37,7 +37,8 @@ public final class VirtualThreadAdmissionFilter extends OncePerRequestFilter imp
private final VirtualThreadAdmissionGuard guard;
private final ObjectMapper objectMapper;
public VirtualThreadAdmissionFilter(VirtualThreadAdmissionGuard guard, ObjectMapper objectMapper) {
public VirtualThreadAdmissionFilter(
VirtualThreadAdmissionGuard guard, ObjectMapper objectMapper) {
this.guard = Objects.requireNonNull(guard, "guard");
this.objectMapper = Objects.requireNonNull(objectMapper, "objectMapper");
}
@@ -55,9 +56,9 @@ public final class VirtualThreadAdmissionFilter extends OncePerRequestFilter imp
try {
permit = guard.enter();
} catch (AdmissionRefusedException refused) {
if (refused.errorCode() instanceof RetryAfterHint hint) {
response.setHeader(ApiHeaders.RETRY_AFTER, Integer.toString(hint.retryAfterSeconds()));
}
RetryAfterAdvisor.retryAfterSeconds(refused.errorCode())
.ifPresent(
seconds -> response.setHeader(ApiHeaders.RETRY_AFTER, Integer.toString(seconds)));
EnvelopeResponseWriter.write(response, refused.errorCode(), objectMapper);
return;
} catch (InterruptedException interrupted) {
@@ -1,8 +1,8 @@
package dev.caskeleton.adapter.inbound.web.advanced.virtualthread;
import dev.caskeleton.adapter.inbound.web.advanced.error.WebAdvancedError;
import dev.caskeleton.shared.error.ApiErrorCarrier;
import dev.caskeleton.shared.error.ApiErrorCode;
import dev.caskeleton.shared.error.OperationalError;
/**
* An arrival was refused because the concurrency limit was full.
@@ -27,9 +27,12 @@ public final class AdmissionRefusedException extends RuntimeException implements
return limit;
}
/** {@link WebAdvancedError#ADMISSION_REFUSED}; see the leaf README for the client contract. */
/**
* {@link OperationalError#SERVER_BUSY}, the one code every transport answers server saturation
* with; see the leaf README for the client contract.
*/
@Override
public ApiErrorCode errorCode() {
return WebAdvancedError.ADMISSION_REFUSED;
return OperationalError.SERVER_BUSY;
}
}
@@ -62,7 +62,7 @@ class WebAdvancedErrorMappingTest {
static Stream<Arguments> documentedMappings() {
return Stream.of(
Arguments.of("admission", 503, "ADMISSION_REFUSED", "TRANSIENT_DEPENDENCY", true),
Arguments.of("admission", 503, "SERVER_BUSY", "TRANSIENT_DEPENDENCY", true),
Arguments.of("patch-test", 409, "PATCH_TEST_FAILED", "CONFLICT", false),
Arguments.of("patch-rejected", 400, "PATCH_REJECTED", "VALIDATION", false),
Arguments.of("patch-forbidden", 403, "PATCH_PATH_FORBIDDEN", "AUTHZ", false),
@@ -12,12 +12,17 @@ import org.springframework.mock.web.MockHttpServletRequest;
import org.springframework.mock.web.MockHttpServletResponse;
import tools.jackson.databind.json.JsonMapper;
/** The request path through the admission guard. */
/**
* The request path through the admission guard.
*
* <p>required_test: {@code contract-verification:server-saturation} — a full limit answers the
* shared {@code SERVER_BUSY} 503 with {@code Retry-After}, before any handler runs.
*/
@Tag("web-advanced")
class VirtualThreadAdmissionFilterTest {
@Test
@DisplayName("한도가 차면 요청은 controller에 닿지 않고 503 ADMISSION_REFUSED와 Retry-After를 받는다")
@DisplayName("한도가 차면 요청은 controller에 닿지 않고 503 SERVER_BUSY와 Retry-After를 받는다")
void aFullLimitRefusesWith503AndRetryAfter() throws Exception {
// given
VirtualThreadAdmissionGuard guard = new VirtualThreadAdmissionGuard(1, Duration.ZERO);
@@ -38,7 +43,7 @@ class VirtualThreadAdmissionFilterTest {
assertThat(reached).as("the guard was a bean nothing called").hasValue(0);
assertThat(response.getStatus()).isEqualTo(503);
assertThat(response.getHeader("Retry-After")).isEqualTo("1");
assertThat(response.getContentAsString()).contains("ADMISSION_REFUSED");
assertThat(response.getContentAsString()).contains("SERVER_BUSY");
assertThat(guard.rejectedCount()).isEqualTo(1);
}
@@ -269,8 +269,7 @@ class JsonPatchApplierTest {
// when
Throwable thrown =
catchThrowable(
() -> applier("/displayName").apply(profile(), document, Profile.class));
catchThrowable(() -> applier("/displayName").apply(profile(), document, Profile.class));
// then
assertThat(thrown).isInstanceOf(JsonPatchRejectedException.class);
@@ -285,8 +284,7 @@ class JsonPatchApplierTest {
// when
Throwable thrown =
catchThrowable(
() -> applier("/version").apply(profile(), document, Profile.class));
catchThrowable(() -> applier("/version").apply(profile(), document, Profile.class));
// then
assertThat(thrown).isInstanceOf(JsonPatchRejectedException.class);
@@ -58,7 +58,7 @@ public final class BlockingTransferExecutor {
throw new FileserverException(
"transfer pool is saturated",
rejected,
FileserverFailureContext.of(FileserverErrorCode.TRANSFER_ADMISSION_REJECTED, true));
FileserverFailureContext.of(FileserverErrorCode.SERVER_BUSY, true));
}
try {
return future.get(awaitSeconds, TimeUnit.SECONDS);
@@ -30,8 +30,8 @@ public final class FileserverProblemHeaders {
private static final Map<FileserverErrorCode, Integer> RETRY_AFTER_SECONDS =
Map.of(
FileserverErrorCode.TRANSFER_ADMISSION_REJECTED, 1,
FileserverErrorCode.SERVER_BUSY, 1,
FileserverErrorCode.STORAGE_UNAVAILABLE, 30,
FileserverErrorCode.ATOMIC_PUBLISH_UNSUPPORTED, 30,
FileserverErrorCode.TRANSFER_TIMEOUT, 5,
FileserverErrorCode.FILE_NOT_READY, 2,
FileserverErrorCode.CONCURRENT_MODIFICATION, 1);
@@ -47,8 +47,10 @@ public final class FileserverProblemHeaders {
Map<String, String> headers = new LinkedHashMap<>();
Integer retryAfter = RETRY_AFTER_SECONDS.get(failure.code());
// Only a failure the server itself called retryable advertises a retry: telling a client to
// come back after a permanent rejection is worse than saying nothing.
if (retryAfter != null && failure.context().retryable()) {
// come back after a permanent rejection is worse than saying nothing. The same narrowing as the
// body (ErrorResponseFactory): the occurrence can only narrow its code, so a code that is never
// retryable never carries the header, whatever the occurrence said (review 2026-11-12).
if (retryAfter != null && failure.code().retryable() && failure.context().retryable()) {
headers.put("Retry-After", String.valueOf(retryAfter));
}
if (failure instanceof HttpRangeNotSatisfiableException unsatisfiable) {
@@ -23,6 +23,7 @@ class MvcUploadExecutorSaturationTest {
BlockingTransferExecutor executor = new BlockingTransferExecutor(pool, bounds.awaitSeconds());
CountDownLatch release = new CountDownLatch(1);
AtomicInteger rejections = new AtomicInteger();
CountDownLatch firstRejection = new CountDownLatch(1);
List<Thread> callers = new ArrayList<>();
for (int index = 0; index < 8; index++) {
@@ -34,21 +35,25 @@ class MvcUploadExecutorSaturationTest {
} catch (FileserverException failure) {
// A caller that outlived the budget (TRANSFER_TIMEOUT) is not a saturation
// signal; only an admission rejection is.
if (failure.code() == FileserverErrorCode.TRANSFER_ADMISSION_REJECTED) {
if (failure.code() == FileserverErrorCode.SERVER_BUSY) {
rejections.incrementAndGet();
firstRejection.countDown();
}
}
});
callers.add(caller);
caller.start();
}
Thread.sleep(200);
// Held until the pool has refused someone, rather than for a guessed 200 ms: under load the
// callers may not have arrived yet, and then the release lets every one of them through.
boolean refused = firstRejection.await(10, TimeUnit.SECONDS);
release.countDown();
for (Thread caller : callers) {
caller.join(TimeUnit.SECONDS.toMillis(10));
}
pool.shutdown();
assertThat(refused).as("a pool of 1 with a queue of 1 refused within the wait").isTrue();
assertThat(rejections.get())
.as("a pool of 1 with a queue of 1 cannot absorb 8 concurrent transfers")
.isPositive();
@@ -102,8 +107,7 @@ class MvcUploadExecutorSaturationTest {
.satisfies(
failure -> {
FileserverException rejected = (FileserverException) failure;
assertThat(rejected.code())
.isEqualTo(FileserverErrorCode.TRANSFER_ADMISSION_REJECTED);
assertThat(rejected.code()).isEqualTo(FileserverErrorCode.SERVER_BUSY);
assertThat(rejected.context().retryable()).isTrue();
});
}
@@ -115,7 +119,10 @@ class MvcUploadExecutorSaturationTest {
BlockingTransferExecutor executor = new BlockingTransferExecutor(pool, bounds.awaitSeconds());
assertThatThrownBy(() -> executor.call(() -> await(new CountDownLatch(1))))
.isInstanceOf(FileserverException.class);
.isInstanceOf(FileserverException.class)
.extracting(failure -> ((FileserverException) failure).code())
.as("the overrun is answered as the timeout it is, not as some other transfer failure")
.isEqualTo(FileserverErrorCode.TRANSFER_TIMEOUT);
pool.shutdown();
}
@@ -412,7 +412,7 @@ public class GlobalExceptionHandler extends ResponseEntityExceptionHandler {
return null;
}
}
ApiErrorCode code = HttpStatusErrorCodes.forStatus(statusCode);
ApiErrorCode code = HttpStatusErrorCodes.forException(ex, statusCode);
if (code.httpStatus() >= 500) {
log.error("spring mvc failure classified as {}", code.code(), ex);
spanErrorRecorder.recordException(ex, code.code());
@@ -3,6 +3,7 @@ package dev.caskeleton.adapter.inbound.web.error;
import dev.caskeleton.shared.error.ApiErrorCode;
import dev.caskeleton.shared.error.OperationalError;
import org.springframework.http.HttpStatusCode;
import org.springframework.web.context.request.async.AsyncRequestTimeoutException;
/**
* The operational code a bare HTTP status stands for, when a status is all that is known.
@@ -15,6 +16,17 @@ final class HttpStatusErrorCodes {
private HttpStatusErrorCodes() {}
/**
* The code for a Spring MVC exception that carries only a status. An async request timeout is
* {@code REQUEST_TIMEOUT}; anything else is classified by its status.
*/
static ApiErrorCode forException(Exception failure, HttpStatusCode status) {
if (failure instanceof AsyncRequestTimeoutException) {
return WebTransportError.REQUEST_TIMEOUT;
}
return forStatus(status);
}
static ApiErrorCode forStatus(HttpStatusCode status) {
return switch (status.value()) {
case 401 -> OperationalError.UNAUTHENTICATED;
@@ -27,8 +39,10 @@ final class HttpStatusErrorCodes {
case 414 -> OperationalError.URI_TOO_LONG;
case 415 -> OperationalError.UNSUPPORTED_MEDIA_TYPE;
case 429 -> OperationalError.RATE_LIMIT_EXCEEDED;
// Spring answers its async request timeout with 503: transient, not an internal fault.
case 503 -> WebTransportError.REQUEST_TIMEOUT;
// A bare 503 is the server saying it cannot serve now (RFC 9110 15.6.4). An async request
// timeout is also a 503, and forException tells it apart; every other one used to be
// labelled REQUEST_TIMEOUT whatever it was (review 2026-11-12).
case 503 -> OperationalError.SERVER_BUSY;
default ->
status.is4xxClientError()
? OperationalError.BAD_PARAMETER
@@ -19,6 +19,7 @@ public final class RetryAfterAdvisor {
private static final Map<String, Integer> RETRY_AFTER_SECONDS =
Map.ofEntries(
Map.entry("RATE_LIMIT_EXCEEDED", 1),
Map.entry("SERVER_BUSY", 1),
// The application's value, which the processing-lease check also reads: the resend
// arrives while the first attempt can still finish. docs/web/http-contract.md §2.
Map.entry(
@@ -28,6 +28,7 @@ class EnvelopeErrorControllerTest {
"401, 401, UNAUTHENTICATED",
"404, 404, ROUTE_NOT_FOUND",
"429, 429, RATE_LIMIT_EXCEEDED",
"503, 503, SERVER_BUSY",
"409, 400, BAD_PARAMETER",
"422, 400, BAD_PARAMETER",
"500, 500, INTERNAL_ERROR",
@@ -17,10 +17,10 @@ import dev.caskeleton.shared.error.Category;
* servlet transport, the reactive transport and the Nginx delegation path cannot answer the same
* failure three different ways.
*
* <p>{@code UNAUTHENTICATED}, {@code PRECONDITION_FAILED} and {@code UNSUPPORTED_MEDIA_TYPE} share
* their name with an {@code OperationalError}; they are kept here so the failure context stays one
* enum, and {@code FileserverExceptionTest} pins that each means exactly what the operational code
* means on the wire (category, status, retryability).
* <p>{@code UNAUTHENTICATED}, {@code PRECONDITION_FAILED}, {@code UNSUPPORTED_MEDIA_TYPE} and
* {@code SERVER_BUSY} share their name with an {@code OperationalError}; they are kept here so the
* failure context stays one enum, and {@code FileserverExceptionTest} pins that each means exactly
* what the operational code means on the wire (category, status, retryability).
*/
public enum FileserverErrorCode implements ApiErrorCode {
BAD_REQUEST(400, Category.VALIDATION, false),
@@ -42,6 +42,7 @@ public enum FileserverErrorCode implements ApiErrorCode {
MALWARE_DETECTED(422, Category.DATA_INTEGRITY, false),
INVALID_PATH(422, Category.VALIDATION, false),
PATH_OUTSIDE_NAMESPACE(422, Category.VALIDATION, false),
// The caller's own transfers fill its scope's permits: the caller's rate, so 429.
TRANSFER_ADMISSION_REJECTED(429, Category.RATE_LIMIT, true),
// tus checksum extension status: one request's bytes did not match the checksum the client sent
// for them. The chunk is rolled back before its offset commits, so the upload stays resumable.
@@ -52,6 +53,9 @@ public enum FileserverErrorCode implements ApiErrorCode {
AMBIGUOUS_COMPLETION(500, Category.DATA_INTEGRITY, false),
ATOMIC_PUBLISH_UNSUPPORTED(503, Category.PERMANENT_DEPENDENCY, false),
STORAGE_UNAVAILABLE(503, Category.TRANSIENT_DEPENDENCY, true),
// The instance's permits or its transfer pool are full: server saturation, the shared code every
// transport answers it with (decisions 2.3, 2026-11-12).
SERVER_BUSY(503, Category.TRANSIENT_DEPENDENCY, true),
TRANSFER_TIMEOUT(504, Category.TRANSIENT_DEPENDENCY, true),
STORAGE_FULL(507, Category.TRANSIENT_DEPENDENCY, false);
@@ -15,9 +15,11 @@ import java.util.concurrent.atomic.AtomicBoolean;
* In-process admission control backed by bounded semaphores.
*
* <p>The failure vocabulary is deliberately distinct so a client can tell a permanent policy denial
* from a transient one: an exhausted pool is {@code STORAGE_FULL}, and running out of concurrency
* permits — for the caller's scope or for the whole instance — is a retryable admission rejection.
* Stored bytes over a quota ({@code QUOTA_EXCEEDED}) are decided by the quota service, not here.
* from a transient one: an exhausted pool is {@code STORAGE_FULL}; running out of the caller's own
* scope permits is a retryable {@code TRANSFER_ADMISSION_REJECTED} (429, the caller's rate);
* running out of the whole instance's permits is the shared server-saturation {@code SERVER_BUSY}
* (503). Stored bytes over a quota ({@code QUOTA_EXCEEDED}) are decided by the quota service, not
* here.
*/
public final class TransferAdmissionController {
@@ -74,7 +76,7 @@ public final class TransferAdmissionController {
scopePermits.release();
throw new FileserverException(
"instance upload permits are exhausted",
FileserverFailureContext.of(FileserverErrorCode.TRANSFER_ADMISSION_REJECTED, true));
FileserverFailureContext.of(FileserverErrorCode.SERVER_BUSY, true));
}
return new SemaphorePermit(scopePermits, instanceUploads);
}
@@ -83,7 +85,7 @@ public final class TransferAdmissionController {
if (!directDownloads.tryAcquire()) {
throw new FileserverException(
"instance direct download permits are exhausted",
FileserverFailureContext.of(FileserverErrorCode.TRANSFER_ADMISSION_REJECTED, true));
FileserverFailureContext.of(FileserverErrorCode.SERVER_BUSY, true));
}
return new SemaphorePermit(directDownloads);
}
@@ -119,7 +119,12 @@ class TransferAdmissionControllerTest {
assertThatThrownBy(() -> narrow.acquireUpload(scope("tenant-b"), 10))
.isInstanceOf(FileserverException.class)
.satisfies(
failure -> assertThat(((FileserverException) failure).context().retryable()).isTrue());
failure -> {
assertThat(((FileserverException) failure).context().retryable()).isTrue();
assertThat(((FileserverException) failure).context().code())
.as("the whole instance is full: server saturation, not the caller's rate")
.isEqualTo(FileserverErrorCode.SERVER_BUSY);
});
held.close();
TransferPermit afterRelease = narrow.acquireUpload(scope("tenant-b"), 10);
@@ -197,7 +202,9 @@ class TransferAdmissionControllerTest {
TransferPermit download = narrow.acquireDirectDownload(scope("tenant-a"));
assertThatThrownBy(() -> narrow.acquireDirectDownload(scope("tenant-a")))
.isInstanceOf(FileserverException.class);
.isInstanceOf(FileserverException.class)
.extracting(failure -> ((FileserverException) failure).context().code())
.isEqualTo(FileserverErrorCode.SERVER_BUSY);
assertThat(narrow.acquireUpload(scope("tenant-a"), 10).isHeld()).isTrue();
download.close();
@@ -44,6 +44,12 @@ public enum OperationalError implements ApiErrorCode {
ROUTE_NOT_FOUND(Category.NOT_FOUND, 404, false),
// rate-limit / idempotency
RATE_LIMIT_EXCEEDED(Category.RATE_LIMIT, 429, true),
/**
* This instance has no capacity for another request right now: an admission limit, a full work
* pool. Every transport answers server saturation with this one code; {@link
* #RATE_LIMIT_EXCEEDED} is the caller's own rate (decisions 2.3, 2026-11-12).
*/
SERVER_BUSY(Category.TRANSIENT_DEPENDENCY, 503, true),
IDEMPOTENT_IN_FLIGHT(Category.CONFLICT, 409, true),
IDEMPOTENT_REQUEST_MISMATCH(Category.VALIDATION, 422, false),
IDEMPOTENCY_UNAVAILABLE(Category.TRANSIENT_DEPENDENCY, 503, true),