diff --git a/docs/httpclient/configuration-reference.md b/docs/httpclient/configuration-reference.md index 5456757e..340573a2 100644 --- a/docs/httpclient/configuration-reference.md +++ b/docs/httpclient/configuration-reference.md @@ -167,8 +167,8 @@ profile에는 `connect`와 `response-header`를 가장 짧은 호출자 예산 | `redirect.max-hops` | `0` | redirect를 켜고 hop을 0으로 두면 설정 오류다 | | `redirect.allow-cross-origin` | `false` | 켜면 hop에서 credential을 지운다 | | `request.max-body-bytes` | `0` | 운영에서 필수 | -| `request.compression` | `false` | | -| `response.max-wire-bytes` | `5242880` | wire 위의 바이트 | +| `request.compression` | `false` | 켜면 transport와 상관없이 `Accept-Encoding: gzip, deflate`를 보낸다. 받은 본문은 엔진이 아니라 이 플랫폼이 wire 바이트를 센 뒤 푼다(버퍼 본문과 스트리밍 다운로드 모두) | +| `response.max-wire-bytes` | `5242880` | wire 위의 바이트. 압축을 켜도 압축된 크기로 센다 | | `response.max-decoded-bytes` | `10485760` | 디코딩한 뒤의 바이트. 넘을 수 없는 최대는 64 MiB | | `response.allowed-content-types` | JSON + problem+json | 비어 있으면 「아무것이나」 | @@ -212,7 +212,7 @@ profile에는 `connect`와 `response-header`를 가장 짧은 호출자 예산 | 속성 | 기본값 | 뜻 | |---|---|---| | `proxy.enabled` | `false` | | -| `proxy.host` / `proxy.port` / `proxy.type` | — / `0` / `HTTP` | | +| `proxy.host` / `proxy.port` / `proxy.type` | — / `0` / `HTTP` | 지금 transport는 모두 HTTP forward proxy만 한다. `type`의 값은 `HTTP` 하나이고, `SOCKS`는 바인딩이 거부한다 | | `proxy.credential-provider` | — | proxy 인증은 대상 인증과 따로다 | | `proxy.import-ambient-no-proxy` | `false` | 환경의 `NO_PROXY`는 검증한 profile을 넓히지 않는다 | | `observability.full-url-recording` | `false` | | diff --git a/docs/httpclient/retry-and-ambiguity.md b/docs/httpclient/retry-and-ambiguity.md index 8aa43fe1..e2c44497 100644 --- a/docs/httpclient/retry-and-ambiguity.md +++ b/docs/httpclient/retry-and-ambiguity.md @@ -50,9 +50,9 @@ | 상태 | 판정 | |---|---| -| 408 | 기한과 예산 안에서 재시도 | -| 425 | 첫 시도에서만, 많아야 한 번 재시도 | -| 429 | `Retry-After`, 기한, 예산 안에서 재시도 | +| 408 | 안전하게 멱등인 연산만 기한과 예산 안에서 재시도. 아니면 결과 모름(upstream이 받은 뒤 답한 것일 수 있다) | +| 425 | 안전하게 멱등인 연산만, 첫 시도에서만, 많아야 한 번 재시도. 아니면 결과 모름 | +| 429 | 안전하게 멱등인 연산만 `Retry-After`, 기한, 예산 안에서 재시도. 아니면 거부(`notRetryableStatus`). 429는 일정 신호이지 "아무 일도 없었다"는 말이 아니다 | | 401 | 안전하고 재전송 가능한 연산에서만 갱신 후 재전송 한 번. credential은 재시도를 허락받은 뒤에만 갱신한다. 재전송하지 않을 401(시도 한 번, 예산 없음, 기한 없음)은 token cache를 건드리지 않는다 | | 500 | upstream이 일시적이라고 등록했고 **그리고** 연산이 안전하게 멱등일 때만 허락. 아니면 거부 | | 502, 503, 504 | 안전하게 멱등인 연산은 재시도. 아니면 결과 모름 | diff --git a/docs/httpclient/streaming.md b/docs/httpclient/streaming.md index fd5e0686..4c9ecaa2 100644 --- a/docs/httpclient/streaming.md +++ b/docs/httpclient/streaming.md @@ -7,7 +7,13 @@ 검증한다. 그래서 실패한 다운로드가 호출자가 따져야 하는 반쯤 읽힌 stream이 되지 않는다. wire 바이트와 디코딩한 바이트는 따로 제한한다. 압축한 payload는 wire 검사를 통과한 뒤 늘어나기 때문이다. 한도는 -버퍼링한 뒤가 아니라 읽는 동안 강제한다. +버퍼링한 뒤가 아니라 읽는 동안 강제한다. 압축(`Content-Encoding: gzip`·`deflate`)은 엔진이 아니라 응답 reader가 +푼다. 도착한 바이트를 wire 한도로 세고, 푼 바이트를 조각마다 디코딩 한도로 센다. 엔진이 먼저 풀면 wire 계수가 푼 +크기를 세게 되어, 잘 압축된 응답이 wire 위에는 없던 크기로 거부됐다(round9 리뷰 2026-11-12, RR9-01-30). 부탁하지 않은 +압축(`br` 등)과 겹친 coding은 거부한다. 스트리밍 다운로드도 같은 곳(`ContentDecoding`)을 거친다. 호출자는 푼 바이트를 +읽고, 응답 헤더에는 `Content-Encoding`·`Content-Length`가 없다. 풀기는 호출자가 읽을 때 시작하므로, 본문이 빈 gzip +응답(HEAD·204·304)은 빈 본문이다. 어느 transport도 스스로 풀지 않는다. Apache는 압축 처리를 끄고, 실험 transport인 +Jetty HTTP/3은 시작할 때 classpath에서 등록한 decoder를 지운다. ## 첫 바이트 경계 diff --git a/docs/manual/05-idempotency.md b/docs/manual/05-idempotency.md index fce96ff2..cbdfee67 100644 --- a/docs/manual/05-idempotency.md +++ b/docs/manual/05-idempotency.md @@ -57,7 +57,10 @@ executor 조립과 실제 PostgreSQL 저장소 왕복은 `PostgreSqlIdempotencyP 같은 키 재전송이 무엇을 막는지는 저장소마다 다르다(정본 `docs/web/http-contract.md` 2절). Redis 저장소에서 같은 키 재전송은 효과가 있었을 수 있는 작업을 다시 돌리지 -않는다. PostgreSQL 저장소를 템플릿처럼 쓰기 트랜잭션 하나로 쓰면 실패한 요청의 기록은 롤백되어 다음 같은 키 +않는다. 단, 그 기록이 Redis에 남아 있는 동안만이다. Sentinel·Cluster failover는 비동기 복제라 primary가 받고 +replica에 아직 넘기지 못한 기록을 잃을 수 있고, 그러면 같은 키 요청은 처음 보는 키로 다시 실행된다 +(`docs/runbooks/redis-capability-incident.md` 「Topology and persistence」). 이 창이 업무상 받아들일 수 없는 +작업은 PostgreSQL 저장소를 쓰거나 업무 데이터의 유일 제약으로 따로 막는다. PostgreSQL 저장소를 템플릿처럼 쓰기 트랜잭션 하나로 쓰면 실패한 요청의 기록은 롤백되어 다음 같은 키 요청이 다시 실행한다. 그래서 트랜잭션 밖 효과는 따로 멱등하게 만든다. 첫 시도가 processing lease(기본 30s)보다 오래 걸릴 때 보이는 모습도 저장소마다 다르다. diff --git a/docs/mongodb/advanced/multi-tenancy.md b/docs/mongodb/advanced/multi-tenancy.md index f05e7ff5..419baf46 100644 --- a/docs/mongodb/advanced/multi-tenancy.md +++ b/docs/mongodb/advanced/multi-tenancy.md @@ -41,7 +41,10 @@ email을 모든 tenant에 걸쳐 유일하게 만든다. 그러면 tenant A가 ## 2. tenant마다 database (Experimental) `MongoTenantClientRegistry`는 tenant마다 client를 하나 둔다. 개수 상한이 있고 쓰지 않는 client는 -내보낸다. tenant를 어느 database에 대응시킬지는 도입하는 쪽의 코드가 정한다. +내보낸다. tenant를 어느 database에 대응시킬지는 도입하는 쪽의 코드가 정한다. `acquire`는 `Lease`를 돌려주고, +그 client는 lease를 모두 닫을 때까지 쓰는 중이다. 쓰는 중인 client는 얼마나 오래전에 받았든 내보내지 않는다. +내보낸 client를 닫다가 실패하면 그 tenant 대신 `evictionCloseFailures()`가 센다. 닫기를 부른 다른 tenant의 +`acquire`는 실패하지 않는다(round9 리뷰 2026-11-12, RR9-01-27). Experimental인 이유가 분명하다. 작을 때는 맞고, 클 때는 한계가 없다. tenant database마다 connection, file handle, 모니터링 가짓수가 든다. tenant 20개에서는 잘 돌고 2,000개에서는 쓰러진다. 기능 테스트로는 diff --git a/docs/mongodb/query-aggregation-guide.md b/docs/mongodb/query-aggregation-guide.md index cb3f5805..b6531001 100644 --- a/docs/mongodb/query-aggregation-guide.md +++ b/docs/mongodb/query-aggregation-guide.md @@ -39,9 +39,15 @@ regex는 index를 쓰는 척하는 collection 전체 scan이고, 사용자가 | 상한 | 이유 | |---|---| | `maxTimeMS` | 아무도 기다리지 않는 query를 서버가 멈춘다. | -| 결과 개수 상한 | 상한 없는 결과는 단계가 몇 개 더 붙은 OOM이다. | -| batch 크기 | 왕복 한 번의 메모리를 제한한다. | -| 검사한 문서 수 상한 | 빠른 날에는 시간 상한만으로 숨겨지는 index 회귀를 잡는다. | +| 결과 개수 상한(`maxResults`) | 상한 없는 결과는 단계가 몇 개 더 붙은 OOM이다. | +| 결과 바이트 상한(`maxResultBytes`) | 문서 하나하나가 크면 개수 상한 안에서도 heap이 찬다. | +| batch 크기(`cursorBatchSize`) | 왕복 한 번의 메모리를 제한한다. | + +개수·시간·batch는 서버에 보내는 요청에 실린다(`PolicyAwareMongoQueryBuilder.build`가 만든 `Query`, aggregation의 +`$limit`과 옵션). 바이트는 결과를 읽는 쪽만 셀 수 있다. aggregation은 `PolicyAwareMongoAggregationExecutor`가 +cursor에서 문서를 하나씩 읽으며 encode한 크기를 더하고 넘으면 거부한다(5절). query builder는 `Query`만 돌려주므로 +query의 바이트 상한을 집행하는 템플릿 코드는 없다. 그 `Query`로 결과를 읽는 서비스의 adapter가 `maxResultBytes`를 +직접 센다. 세지 않으면 그 상한은 선언일 뿐이다. `MongoBudgetPolicyRegistry`가 예산을 연산 이름에 묶는다. `MongoBudgetEnforcer`가 그것을 적용한다. 요청이 예산을 넘으면 실행 전에 `MongoOperationRejectedException`을 던지고, 서버가 상한을 집행하면 @@ -86,6 +92,11 @@ regex는 index를 쓰는 척하는 collection 전체 scan이고, 사용자가 | high | `$lookup`, `$graphLookup`, `$facet`, index 없는 `$sort` | profile의 명시적 승인이 필요하다. | | forbidden | admin plane 밖의 `$out`·`$merge`, `$function`, `$accumulator` | 거부한다. | +실행은 예산 둘을 결과가 오는 동안 본다. pipeline 끝에 `$limit(maxResults + 1)`을 붙여 서버가 예산보다 한 개 넘게는 +만들지 않게 하고, 결과를 cursor에서 한 문서씩 읽으며 개수와 encode한 바이트를 더한다. 둘 중 하나라도 넘으면 +`MongoOperationRejectedException`(`aggregation.result`)이다. 예전에는 결과를 전부 메모리에 올린 뒤 개수만 셌고 바이트 +상한은 선언만 있었다(round9 리뷰 2026-11-12, RR9-01-20). + `allowDiskUse`는 실행 중 flag가 아니라 plan에 선언하는 속성이다. 디스크가 필요한 pipeline은 모양을 검토해야 하는 pipeline이다. diff --git a/docs/upgrade/2026-11-round9-outbound.md b/docs/upgrade/2026-11-round9-outbound.md index c9a769a5..702989cf 100644 --- a/docs/upgrade/2026-11-round9-outbound.md +++ b/docs/upgrade/2026-11-round9-outbound.md @@ -1,13 +1,15 @@ # 업그레이드 안내 — 2026-11 outbound 발행·발송·캐시·S3·MongoDB·HTTP client -- 범위: outbox 발행 대기, SMTP 발송 pool과 시간 상한, Redis 캐시 시간·크기 상한을 고친 변경. 기동 조건이 바뀐 곳이 - 있다. 아래 표가 그 목록이다. -- 근거: 2026-11-12 전체 리뷰(`docs/reviews/2026-11-12-round9-review/`)의 발행·발송·캐시(B10a) 발견 +- 범위: outbox 발행 대기, SMTP 발송 pool, Redis 캐시 시간·크기 상한, S3 조건부 쓰기, MongoDB 집계 예산·tenant + client·migration lease, HTTP client 압축·proxy를 고친 변경. 기동 조건과 공개 타입이 바뀐 곳이 있다. 아래 표가 그 + 목록이다. +- 근거: 2026-11-12 전체 리뷰(`docs/reviews/2026-11-12-round9-review/`)의 발행·발송·캐시(B10a)와 S3·Mongo·httpclient + (B10b) 발견 ## 1. 누가 읽나 -이 템플릿을 `upstream` remote로 두고 merge해 받는 파생 서비스가 읽는다. 특히 SMTP 알림, Redis 캐시, outbox -publisher를 직접 구현한 서비스다. +이 템플릿을 `upstream` remote로 두고 merge해 받는 파생 서비스가 읽는다. 특히 SMTP 알림, Redis 캐시, MongoDB +집계·tenant별 database, HTTP client의 압축이나 proxy를 쓰는 서비스다. ## 2. 할 일 @@ -16,7 +18,12 @@ publisher를 직접 구현한 서비스다. | 1 | SMTP provider profile들의 `max-concurrency` 합이 발송 pool이 담는 수(`app.notification.platform.smtp.dispatch-concurrency`개 thread와 그 네 배의 대기열) 이하인지 확인한다. 모든 SMTP profile이 pool 하나를 같이 쓴다. 기본값(profile 하나, 4와 4)은 맞는다 | 넘으면 넘게 만든 profile의 조립이 기동 때 실패한다. 전에는 넘친 발송이 대기열에서 거절되어 결과 불명으로 남았다 | SMTP 알림을 켠 서비스 | | 2 | Redis 캐시의 `command-timeout`이 이제 한 캐시 연산 전체(연결 대기와 응답 전부)의 상한이다. 연결 대기가 길던 배포는 값을 다시 본다 | 상한 안에 못 끝난 연산이 캐시 없음(`DEGRADED_UNAVAILABLE`·`Unavailable`)으로 끝나는 일이 늘 수 있다. 전에는 lane 대기 2초가 더해졌다 | Redis 캐시를 켠 서비스 | | 3 | encode한 값이 1 MiB(`RedisCacheRegionAdapter.DEFAULT_MAXIMUM_VALUE_BYTES`)를 넘는 캐시는 그 값을 기록하지 않는다(`NOT_RECORDED_PROVIDER_POLICY`). 더 큰 값이 필요하면 `app.capabilities.cache.maximum-value-bytes`(`APP_CACHE_REDIS_MAXIMUM_VALUE_BYTES`, 1..16777216)로 상한을 올린다. 2026-08-10에 폐기했던 그 env 키가 다시 묶였다 | 그 값은 매번 source에서 읽힌다 | 큰 값을 캐시하던 서비스 | -| 4 | SMTP provider profile의 `timeout`(기본이 `10s`에서 `30s`로 바뀌었다)이 발송 한 번을 두 번 묶는다. 발송이 send pool에서 그 시간이 지나도록 시작하지 못하면 relay에 보내지 않고 시작 전 실패(`PROVIDER_UNAVAILABLE`, 재시도)로 기록한다. 시작한 뒤 그 시간이 지나면 결과 불명(`PROVIDER_RESPONSE_LOST`)으로 기록하고 자동으로 다시 보내지 않는다. 값은 `mail.smtp.connectiontimeout`에 `mail.smtp.timeout`·`mail.smtp.writetimeout` 중 긴 것을 더한 값보다 길어야 하고, `dispatch.lease-duration`은 그 두 배보다 길어야 한다. `dispatch.shutdown-grace`가 그 두 배보다 짧으면 종료 중에 끝나지 않은 발송이 결과 불명으로 남는다(`docs/notification/configuration-reference.md` 「SMTP relay」). `SmtpNotificationProviderAdapter`를 직접 만들던 코드는 마지막 인자로 이 timeout을 넘긴다. 직접 만든 `ProviderRuntimeAssembler`는 새 default 메서드 `requireTogether(profiles, deliveryLease)`로 한 family의 켜진 profile 전체에 걸린 검사를 할 수 있다(`NotificationProviderAssembly`가 SERVING일 때 family마다 한 번 부른다). `assemble`을 직접 부르는 코드는 그 전에 `requireTogether`를 부른다 | timeout이 socket timeout 합 이하이거나 lease가 timeout 두 배 이하면 SMTP 조립이 기동 때 실패한다. 상한 안에 끝나지 않는 발송이 결과 불명으로 남는다. 전에는 socket timeout이 명령마다만 묶어, 느린 relay에서 발송이 lease를 넘으면 lease 복구가 아직 도는 발송을 운영자 확인으로 넘겼다. | SMTP 알림을 켠 서비스 | +| 4 | MongoDB `PolicyAwareMongoAggregationExecutor`는 등록 예산의 `maxResultBytes`를 집행하고, pipeline 끝에 `$limit(maxResults + 1)`을 붙인다. 결과가 큰 집계의 예산을 다시 본다. 집계 결과는 template의 converter로만 바뀌어 `AfterLoadEvent`·`AfterConvertEvent`와 entity callback이 돌지 않는다. 그것에 기대던 listener는 돌려받은 타입에서 직접 한다 | 바이트 예산을 넘는 집계가 `MongoOperationRejectedException`(`aggregation.result`)으로 실패한다. 전에는 결과를 전부 메모리에 올렸다 | MongoDB 집계를 쓰는 서비스 | +| 5 | experimental `MongoTenantClientRegistry.acquire`가 `Lease`(interface)를 돌려준다. client를 다 쓴 뒤 `lease.close()`(또는 try-with-resources)를 부른다. 쓰는 중인 tenant의 `release`는 거부된다. idle 시간은 마지막 lease를 닫은 때부터 잰다. 열린 lease 수는 `openLeases()`다. 시각을 인자로 받던 `acquire(tenant, Instant)`는 없어졌다. 시계는 생성자의 `Clock` 하나다 | 반환값을 버리는 호출은 컴파일되지만, 닫지 않은 lease의 client는 쓰는 중으로 남아 내보내지지 않고 cap 한 칸을 계속 차지한다. `acquire(tenant, Instant)`를 부른 코드는 컴파일이 실패한다 | tenant별 database를 쓰는 서비스 | +| 6 | MongoDB migration lease를 잃은 뒤의 `refresh`는 `MongoOperationRejectedException`(`migration.lock`)을 던진다. `IllegalStateException`을 잡던 코드를 바꾼다. collection을 직접 넘겨 `MongoCollectionMigrationLedger`를 만들던 코드는 셋째 인자로 lock collection(`MongoCollectionMigrationLock.DEFAULT_COLLECTION`)을 넘긴다. 옛 runner의 `recordApplied`는 새 runner가 lease를 잡은 뒤면(checkpoint를 쓰기 전이어도) 거부된다 | 그 예외를 잡지 못한다. 두 인자 생성자는 컴파일이 실패한다 | MongoDB migration runner를 직접 쓰는 서비스 | +| 7 | HTTP client profile의 `proxy.type: SOCKS`를 지우거나 HTTP proxy로 바꾼다. `ProxyType`에는 이제 `HTTP`만 있다 | 설정 바인딩이 실패해 기동하지 않는다. 전에는 SOCKS port에 HTTP proxy로 붙어 첫 요청에서 실패했다 | SOCKS proxy를 설정한 서비스 | +| 8 | `request.compression: true`인 HTTP client는 transport와 상관없이 `Accept-Encoding: gzip, deflate`를 보내고, 버퍼 본문과 스트리밍 다운로드 모두 이 플랫폼이 푼다(JDK transport는 전에 이 설정을 읽지 않았다). 압축을 켰든 껐든, 그 밖의 `Content-Encoding`(`br` 등)이나 겹친 coding으로 온 응답은 `HttpTargetRejectedException`(`TARGET_REJECTED`, 증거 `RESPONSE_RECEIVED`)으로 거부된다. 본문이 빈 gzip 응답(HEAD·204·304)은 빈 본문이다 | — 동작 변경. `response.max-wire-bytes`가 이제 압축된 크기로 센다. 스트리밍 다운로드도 identity 본문까지 `max-decoded-bytes`로 센다. 실험 transport인 Jetty HTTP/3도 이제 스스로 풀지 않는다 | HTTP client를 쓰는 서비스 | +| 9 | SMTP provider profile의 `timeout`(기본이 `10s`에서 `30s`로 바뀌었다)이 발송 한 번을 두 번 묶는다. 발송이 send pool에서 그 시간이 지나도록 시작하지 못하면 relay에 보내지 않고 시작 전 실패(`PROVIDER_UNAVAILABLE`, 재시도)로 기록한다. 시작한 뒤 그 시간이 지나면 결과 불명(`PROVIDER_RESPONSE_LOST`)으로 기록하고 자동으로 다시 보내지 않는다. 값은 `mail.smtp.connectiontimeout`에 `mail.smtp.timeout`·`mail.smtp.writetimeout` 중 긴 것을 더한 값보다 길어야 하고, `dispatch.lease-duration`은 그 두 배보다 길어야 한다. `dispatch.shutdown-grace`가 그 두 배보다 짧으면 종료 중에 끝나지 않은 발송이 결과 불명으로 남는다(`docs/notification/configuration-reference.md` 「SMTP relay」). `SmtpNotificationProviderAdapter`를 직접 만들던 코드는 마지막 인자로 이 timeout을 넘긴다. 직접 만든 `ProviderRuntimeAssembler`는 새 default 메서드 `requireTogether(profiles, deliveryLease)`로 한 family의 켜진 profile 전체에 걸린 검사를 할 수 있다(`NotificationProviderAssembly`가 SERVING일 때 family마다 한 번 부른다). `assemble`을 직접 부르는 코드는 그 전에 `requireTogether`를 부른다 | timeout이 socket timeout 합 이하이거나 lease가 timeout 두 배 이하면 SMTP 조립이 기동 때 실패한다. 상한 안에 끝나지 않는 발송이 결과 불명으로 남는다. 전에는 socket timeout이 명령마다만 묶어, 느린 relay에서 발송이 lease를 넘으면 lease 복구가 아직 도는 발송을 운영자 확인으로 넘겼다. | SMTP 알림을 켠 서비스 | ## 3. 바뀐 동작 (따라 할 일 없음) @@ -27,3 +34,8 @@ publisher를 직접 구현한 서비스다. relay가 그 행을 backoff 뒤로 미뤘다가 발행한다. 기다리는 중의 interrupt는 전처럼 결과 불명(`PUBLISH_WAIT_INTERRUPTED`)이다. - SMTP 발송 pool은 첫 발송부터 `dispatch-concurrency`개 thread로 돈다(전에는 대기열이 찰 때까지 thread 하나였다). pool이 거절한 발송은 relay에 닿지 않은 것(`PROVIDER_UNAVAILABLE`, `CAPACITY_REJECTED`)으로 기록되어 재시도된다. +- S3 control 저장소의 조건부 `PUT`이 412를 받으면 key를 다시 읽는다. 이 호출이 쓴 record 그대로면 성공이다(SDK가 응답을 + 잃은 첫 쓰기를 다시 보낸 경우). 다른 내용이나 없음이면 전처럼 충돌이다. 「그대로」는 같은 값이라는 뜻이라, 두 writer가 + 같은 버전에서 같은 record를 쓰면 둘 다 성공으로 본다. 그럴 수 있는 호출자는 record에 자기 토큰을 싣는다. +- MongoDB tenant client를 내보내다 닫기에 실패하면 그 tenant 대신 `evictionCloseFailures()`가 센다. 닫기를 부른 다른 + tenant의 `acquire`는 실패하지 않는다. diff --git a/docs/web/http-contract.md b/docs/web/http-contract.md index 1462840f..c5df1788 100644 --- a/docs/web/http-contract.md +++ b/docs/web/http-contract.md @@ -125,7 +125,9 @@ Envelope가 아닌 응답은 두 곳뿐이다. fileserver의 프로토콜 예외 `IDEMPOTENT_IN_FLIGHT`, lease가 지났으면 500 `IDEMPOTENCY_RECOVERY_REQUIRED`다. 이것은 Redis 저장소의 판정이다. PostgreSQL 저장소는 아래처럼 첫 요청이 끝날 때까지 기다린 뒤 판정한다. 새 키로 보내면 서버는 다른 요청으로 보고 막지 못한다(재리뷰 2026-11-03 README 4절 결정 RR-02-03, 2026-11-04 9.1절 결정 - RR-01-11). + RR-01-11). Redis 저장소의 판정은 그 기록이 Redis에 남아 있는 동안만이다. Sentinel·Cluster failover는 비동기 + 복제라 replica에 아직 넘기지 못한 기록을 잃을 수 있고, 그러면 같은 키 요청은 처음 보는 키로 다시 실행된다 + (`docs/manual/05-idempotency.md` 「클라이언트에게 무엇을 약속하나」). - **첫 시도가 lease보다 오래 걸릴 때 — 저장소마다 다르다.** executor는 lease를 늘리지 않으므로 30s를 넘는 정상 요청도 lease가 지난다(재리뷰 2026-11-04 RR5-01-05). - Redis: lease가 지난 실행 기록을 다음 요청이 만나면 500으로 답하고 기록은 건드리지 않는다. 첫 시도가 diff --git a/src/adapter/outbound/httpclient/src/main/java/dev/caskeleton/adapter/outbound/httpclient/apache/ApacheClientFactory.java b/src/adapter/outbound/httpclient/src/main/java/dev/caskeleton/adapter/outbound/httpclient/apache/ApacheClientFactory.java index 8a12e87a..6478893a 100644 --- a/src/adapter/outbound/httpclient/src/main/java/dev/caskeleton/adapter/outbound/httpclient/apache/ApacheClientFactory.java +++ b/src/adapter/outbound/httpclient/src/main/java/dev/caskeleton/adapter/outbound/httpclient/apache/ApacheClientFactory.java @@ -111,15 +111,17 @@ public final class ApacheClientFactory { .evictIdleConnections(TimeValue.ofMilliseconds(profile.pool().maxIdleTime().toMillis())) .setDefaultRequestConfig(requestConfig(profile)); - if (!profile.request().compression()) { - clientBuilder.disableContentCompression(); - } + // Apache never decodes the body. With its content compression on, the stream the response + // reader counted as "wire" bytes was already decompressed, so a profile's wire limit was a + // second decoded limit and a well-compressed response under it was refused for its decoded size + // (round9 review 2026-11-12, RR9-01-30). The reader counts the bytes as they arrive and decodes + // them itself. Whether to ask for compression is the profile's, sent by the RestClient for + // every transport. + clientBuilder.disableContentCompression(); if (profile.proxy().enabled()) { - clientBuilder.setProxy( - new HttpHost( - profile.proxy().type().name().toLowerCase(java.util.Locale.ROOT), - profile.proxy().host(), - profile.proxy().port())); + // An HTTP forward proxy, the only kind ProxyType has: SOCKS used to bind and then become an + // HTTP proxy at the SOCKS port. + clientBuilder.setProxy(new HttpHost("http", profile.proxy().host(), profile.proxy().port())); } return new ApacheRuntime(clientBuilder.build(), pool); } @@ -131,7 +133,8 @@ public final class ApacheClientFactory { .setRedirectsEnabled(false) .setMaxRedirects(0) .setCircularRedirectsAllowed(false) - .setContentCompressionEnabled(profile.request().compression()) + // Decoding is the response reader's, after it has counted the wire bytes; see create(). + .setContentCompressionEnabled(false) // Hard cancellation lets a deadline abort actually abort the socket instead of leaving a // request in flight after the caller has given up. .setHardCancellationEnabled(true) diff --git a/src/adapter/outbound/httpclient/src/main/java/dev/caskeleton/adapter/outbound/httpclient/http3/JettyHttp3TransportProvider.java b/src/adapter/outbound/httpclient/src/main/java/dev/caskeleton/adapter/outbound/httpclient/http3/JettyHttp3TransportProvider.java index 8bf18d5b..3332b1ea 100644 --- a/src/adapter/outbound/httpclient/src/main/java/dev/caskeleton/adapter/outbound/httpclient/http3/JettyHttp3TransportProvider.java +++ b/src/adapter/outbound/httpclient/src/main/java/dev/caskeleton/adapter/outbound/httpclient/http3/JettyHttp3TransportProvider.java @@ -22,6 +22,7 @@ import org.eclipse.jetty.http3.client.HTTP3Client; import org.eclipse.jetty.http3.client.transport.HttpClientTransportOverHTTP3; import org.eclipse.jetty.io.Transport; import org.eclipse.jetty.quic.client.ClientQuicConfiguration; +import org.eclipse.jetty.util.component.LifeCycle; import org.springframework.http.client.ClientHttpRequestFactory; import org.springframework.http.client.JettyClientHttpRequestFactory; @@ -132,6 +133,18 @@ public final class JettyHttp3TransportProvider implements BlockingTransportProvi .getRequestListeners() .addQueuedListener( request -> AttemptDeadline.abortOnExpiry(() -> abortAtDeadline(request))); + // Jetty registers its gzip decoder from the classpath when it starts, then asks for gzip on + // every request, decodes the response itself and drops Content-Encoding — so the reader's wire + // count saw the decoded size, whatever the profile said (round9 review 2026-11-12, + // RR9-01-30). Cleared once started, because an empty list before start is what tells Jetty to + // load them; the coded bytes then reach the reader, which counts and decodes them. + client.addEventListener( + new LifeCycle.Listener() { + @Override + public void lifeCycleStarted(LifeCycle started) { + client.getContentDecoderFactories().clear(); + } + }); return client; } diff --git a/src/adapter/outbound/httpclient/src/main/java/dev/caskeleton/adapter/outbound/httpclient/profile/ProxyType.java b/src/adapter/outbound/httpclient/src/main/java/dev/caskeleton/adapter/outbound/httpclient/profile/ProxyType.java index cbdba40b..10b2d93b 100644 --- a/src/adapter/outbound/httpclient/src/main/java/dev/caskeleton/adapter/outbound/httpclient/profile/ProxyType.java +++ b/src/adapter/outbound/httpclient/src/main/java/dev/caskeleton/adapter/outbound/httpclient/profile/ProxyType.java @@ -1,7 +1,13 @@ package dev.caskeleton.adapter.outbound.httpclient.profile; -/** Forward proxy kind (design §24.3). */ +/** + * Forward proxy kind (design §24.3). + * + *

HTTP only. Every shipped transport speaks an HTTP forward proxy and none speaks SOCKS; a + * {@code SOCKS} value used to bind and then be wired as an HTTP proxy at the SOCKS port, and a + * value whose only possible effect is a failure is not kept as an option ({@code + * docs/manual/10-capabilities.md} standard 3). {@code proxy.type: SOCKS} now fails to bind. + */ public enum ProxyType { - HTTP, - SOCKS + HTTP } diff --git a/src/adapter/outbound/httpclient/src/main/java/dev/caskeleton/adapter/outbound/httpclient/restclient/BlockingStreamingGateway.java b/src/adapter/outbound/httpclient/src/main/java/dev/caskeleton/adapter/outbound/httpclient/restclient/BlockingStreamingGateway.java index 4f21342e..4bcfe560 100644 --- a/src/adapter/outbound/httpclient/src/main/java/dev/caskeleton/adapter/outbound/httpclient/restclient/BlockingStreamingGateway.java +++ b/src/adapter/outbound/httpclient/src/main/java/dev/caskeleton/adapter/outbound/httpclient/restclient/BlockingStreamingGateway.java @@ -5,6 +5,7 @@ import dev.caskeleton.adapter.outbound.httpclient.api.HttpStatus; import dev.caskeleton.adapter.outbound.httpclient.api.error.HttpDeadlineExceededException; import dev.caskeleton.adapter.outbound.httpclient.api.error.HttpFailureMetadata; import dev.caskeleton.adapter.outbound.httpclient.api.error.HttpRemoteErrorException; +import dev.caskeleton.adapter.outbound.httpclient.api.error.HttpTargetRejectedException; import dev.caskeleton.adapter.outbound.httpclient.api.operation.AttemptStage; import dev.caskeleton.adapter.outbound.httpclient.api.operation.ExecutionEvidence; import dev.caskeleton.adapter.outbound.httpclient.api.operation.HttpOperation; @@ -109,10 +110,38 @@ public final class BlockingStreamingGateway { // exceptions a buffered call throws, classified by the transport's own // classifier at the body stage — not as the engine's bare IOException, which // carried no category, no evidence and nothing to log (RR-04-03). + // Decoded here as the buffered reader decodes: the transport hands over the coded + // bytes, counted on the wire as they arrive, and what they decode to is counted + // against the decoded budget as the caller reads it. A streamed download used to + // reach the caller still compressed once the transport stopped decoding. + String coding; + try { + coding = ContentDecoding.codingOf(rawResponse.getHeaders()); + } catch (ContentDecoding.RefusedCodingException refused) { + // A coding this client did not ask for: nothing is handed over, and the + // connection is not left open behind the refusal. + rawResponse.close(); + throw new HttpTargetRejectedException( + "response content coding is not one this client asked for", + metadata + .withStatus(new HttpStatus(status)) + .withEvidence(ExecutionEvidence.RESPONSE_RECEIVED), + refused); + } + InputStream wire = + new CountingBoundedInputStream( + rawResponse.getBody(), limiter::recordWireBytes); + // Counted against the decoded budget whatever the coding, identity included, as + // the buffered reader counts it: an identity body is its own decoding. + InputStream body = + new CountingBoundedInputStream( + ContentDecoding.decoding(coding, wire), limiter::recordDecodedBytes); + if (!coding.isEmpty()) { + ContentDecoding.dropCodingHeaders(responseHeaders); + } InputStream bounded = new ClassifiedBodyInputStream( - new CountingBoundedInputStream( - rawResponse.getBody(), limiter::recordWireBytes), + body, bodyFailure -> runtime .support() @@ -129,11 +158,7 @@ public final class BlockingStreamingGateway { false); } catch (RuntimeException failure) { closeQuietly(admission); - if (setupDeadline.expired() && !(failure instanceof HttpRemoteErrorException)) { - throw new HttpDeadlineExceededException( - "effective deadline reached before the stream was set up", metadata, failure); - } - throw failure; + throw setupFailure(failure, setupDeadline.expired(), metadata); } BlockingStreamingResponse wrapped = @@ -155,6 +180,29 @@ public final class BlockingStreamingGateway { } } + /** + * What a failure while setting up the stream reaches the caller as. + * + *

Once the setup budget is spent the failure is the deadline, whatever the transport said. A + * response that arrived and was refused, or a remote error, stays what it is: its evidence is + * that the server answered. + * + * @param failure what setting up the stream threw + * @param budgetSpent whether the setup deadline had expired when it did + * @param metadata the call's failure metadata + * @return the exception to throw + */ + static RuntimeException setupFailure( + RuntimeException failure, boolean budgetSpent, HttpFailureMetadata metadata) { + if (budgetSpent + && !(failure instanceof HttpRemoteErrorException) + && !(failure instanceof HttpTargetRejectedException)) { + return new HttpDeadlineExceededException( + "effective deadline reached before the stream was set up", metadata, failure); + } + return failure; + } + /** A permit release must not mask the failure that is already propagating. */ private static void closeQuietly(AutoCloseable closeable) { try { diff --git a/src/adapter/outbound/httpclient/src/main/java/dev/caskeleton/adapter/outbound/httpclient/restclient/ContentDecoding.java b/src/adapter/outbound/httpclient/src/main/java/dev/caskeleton/adapter/outbound/httpclient/restclient/ContentDecoding.java new file mode 100644 index 00000000..2f5b2ba2 --- /dev/null +++ b/src/adapter/outbound/httpclient/src/main/java/dev/caskeleton/adapter/outbound/httpclient/restclient/ContentDecoding.java @@ -0,0 +1,162 @@ +package dev.caskeleton.adapter.outbound.httpclient.restclient; + +import java.io.IOException; +import java.io.InputStream; +import java.io.PushbackInputStream; +import java.util.ArrayList; +import java.util.List; +import java.util.Locale; +import java.util.Map; +import java.util.Set; +import java.util.zip.GZIPInputStream; +import java.util.zip.InflaterInputStream; +import org.springframework.http.HttpHeaders; + +/** + * Undoes a response's content coding, for the buffered reader and the streaming download alike. + * + *

The transport does not decode (round9 review 2026-11-12, RR9-01-30): the bytes that arrive are + * counted against the wire budget, and what they decode to against the decoded budget. Both read + * paths come through here, so a streamed download is decoded the way a buffered body is rather than + * handed over compressed. + */ +final class ContentDecoding { + + /** + * The {@code Accept-Encoding} value a profile with compression on sends: the codings {@link + * #decoding} undoes. Kept beside the decoder so the request never asks for a coding the reader + * would hand over still compressed. + */ + static final String ACCEPTED = "gzip, deflate"; + + /** How much a decoder reads from the wire at a time. */ + static final int CHUNK = 8 * 1024; + + private ContentDecoding() {} + + /** The codings {@link #decoding} undoes; {@code x-gzip} is the old name of {@code gzip}. */ + private static final Set DECODABLE = Set.of("gzip", "x-gzip", "deflate"); + + /** + * A response coded in a way this client did not ask for, refused before its body is read. + * + *

Its own type so the gateways can report it as what it is — a response that arrived and was + * refused — rather than as the bare {@link IOException} a transport classifier reads as a call + * that got no response. + */ + static final class RefusedCodingException extends IOException { + + private static final long serialVersionUID = 1L; + + private RefusedCodingException(String message) { + super(message); + } + } + + /** + * The response's content coding, lower case; empty for none or {@code identity}. + * + * @throws RefusedCodingException when the response names more than one coding — stacked codings + * are not something this client asks for, and decoding only the first would hand the rest + * over as the body — or a coding {@link #decoding} cannot undo + */ + static String codingOf(HttpHeaders headers) throws RefusedCodingException { + List codings = new ArrayList<>(); + for (String value : headers.getOrEmpty(HttpHeaders.CONTENT_ENCODING)) { + // Limit -1: every element, an empty trailing one included, which the check below skips. + for (String part : value.split(",", -1)) { + String coding = part.trim().toLowerCase(Locale.ROOT); + if (!coding.isEmpty() && !coding.equals("identity")) { + codings.add(coding); + } + } + } + if (codings.size() > 1) { + throw new RefusedCodingException( + "the response stacked content codings this client did not ask for: " + codings); + } + String coding = codings.isEmpty() ? "" : codings.get(0); + if (!coding.isEmpty() && !DECODABLE.contains(coding)) { + throw new RefusedCodingException( + "the response used a content coding this client did not ask for: " + coding); + } + return coding; + } + + /** + * The decoded body, read lazily from {@code wire}. + * + *

Lazily, because a gzip stream reads its header when it is built: built eagerly, an empty + * body under {@code Content-Encoding: gzip} — a {@code HEAD}, a 204 or a 304 — failed the whole + * response, and a streamed download read from the network before it was handed over. Here nothing + * is read until the caller reads, and a body that turns out empty decodes to nothing. + * + * @param coding a coding {@link #codingOf} returned + * @param wire the counted wire stream + */ + static InputStream decoding(String coding, InputStream wire) { + return coding.isEmpty() ? wire : new LazilyDecoded(coding, wire); + } + + /** {@code headers} without the two that describe the coded body rather than the decoded one. */ + static void dropCodingHeaders(Map> headers) { + headers + .keySet() + .removeIf( + name -> + name.equalsIgnoreCase(HttpHeaders.CONTENT_ENCODING) + || name.equalsIgnoreCase(HttpHeaders.CONTENT_LENGTH)); + } + + private static final class LazilyDecoded extends InputStream { + + private final String coding; + private final InputStream wire; + private InputStream decoded; + + private LazilyDecoded(String coding, InputStream wire) { + this.coding = coding; + this.wire = wire; + } + + @Override + public int read() throws IOException { + return open().read(); + } + + @Override + public int read(byte[] buffer, int offset, int length) throws IOException { + return open().read(buffer, offset, length); + } + + @Override + public void close() throws IOException { + try { + if (decoded != null) { + decoded.close(); + } + } finally { + wire.close(); + } + } + + private InputStream open() throws IOException { + if (decoded == null) { + PushbackInputStream peeked = new PushbackInputStream(wire, 1); + int first = peeked.read(); + if (first == -1) { + decoded = InputStream.nullInputStream(); + } else { + peeked.unread(first); + // deflate is the zlib-wrapped form RFC 9110 names; a server that sends raw deflate under + // this name fails to decode, which is a failed response rather than a guessed one. + decoded = + coding.equals("deflate") + ? new InflaterInputStream(peeked) + : new GZIPInputStream(peeked, CHUNK); + } + } + return decoded; + } + } +} diff --git a/src/adapter/outbound/httpclient/src/main/java/dev/caskeleton/adapter/outbound/httpclient/restclient/ResponseSizeLimiter.java b/src/adapter/outbound/httpclient/src/main/java/dev/caskeleton/adapter/outbound/httpclient/restclient/ResponseSizeLimiter.java index e88961bc..c4f1aa31 100644 --- a/src/adapter/outbound/httpclient/src/main/java/dev/caskeleton/adapter/outbound/httpclient/restclient/ResponseSizeLimiter.java +++ b/src/adapter/outbound/httpclient/src/main/java/dev/caskeleton/adapter/outbound/httpclient/restclient/ResponseSizeLimiter.java @@ -36,6 +36,11 @@ public final class ResponseSizeLimiter { this.metadata = metadata; } + /** What a failure of the response this limiter counts says about the call. */ + HttpFailureMetadata failureMetadata() { + return metadata; + } + public void recordWireBytes(long additional) { wireBytes += additional; if (wireBytes > maxWireBytes) { diff --git a/src/adapter/outbound/httpclient/src/main/java/dev/caskeleton/adapter/outbound/httpclient/restclient/RestClientResponseReader.java b/src/adapter/outbound/httpclient/src/main/java/dev/caskeleton/adapter/outbound/httpclient/restclient/RestClientResponseReader.java index ef841c2b..37be1c0b 100644 --- a/src/adapter/outbound/httpclient/src/main/java/dev/caskeleton/adapter/outbound/httpclient/restclient/RestClientResponseReader.java +++ b/src/adapter/outbound/httpclient/src/main/java/dev/caskeleton/adapter/outbound/httpclient/restclient/RestClientResponseReader.java @@ -1,8 +1,10 @@ package dev.caskeleton.adapter.outbound.httpclient.restclient; +import dev.caskeleton.adapter.outbound.httpclient.api.HttpStatus; import dev.caskeleton.adapter.outbound.httpclient.api.error.HttpFailureMetadata; import dev.caskeleton.adapter.outbound.httpclient.api.error.HttpSerializationException; import dev.caskeleton.adapter.outbound.httpclient.api.error.HttpTargetRejectedException; +import dev.caskeleton.adapter.outbound.httpclient.api.operation.ExecutionEvidence; import dev.caskeleton.adapter.outbound.httpclient.api.result.ByteArrayResponseType; import dev.caskeleton.adapter.outbound.httpclient.api.result.ClassResponseType; import dev.caskeleton.adapter.outbound.httpclient.api.result.EmptyResponseType; @@ -10,6 +12,7 @@ import dev.caskeleton.adapter.outbound.httpclient.api.result.GenericResponseType import dev.caskeleton.adapter.outbound.httpclient.api.result.ResponseType; import dev.caskeleton.adapter.outbound.httpclient.profile.ResponseLimits; import java.io.ByteArrayInputStream; +import java.io.ByteArrayOutputStream; import java.io.IOException; import java.io.InputStream; import java.lang.reflect.Type; @@ -34,6 +37,11 @@ import org.springframework.http.converter.json.JacksonJsonHttpMessageConverter; *

The body is bounded while it is read and then decoded from memory by a Spring message * converter. Doing the bounding first is what makes a decompression bomb or an unexpectedly huge * payload a rejected request instead of an out-of-memory error. + * + *

Content coding is undone here, not by the transport: the bytes that arrive are counted against + * the wire budget as they arrive, and the bytes they decode to are counted against the decoded + * budget as they are produced — a chunk at a time, so a decompression bomb is refused before it is + * held in memory (round9 review 2026-11-12, RR9-01-30). */ public final class RestClientResponseReader { @@ -112,16 +120,42 @@ public final class RestClientResponseReader { java.util.function.LongConsumer responseBytes) throws IOException { Objects.requireNonNull(responseBytes, "response byte observer"); + String coding; + try { + coding = ContentDecoding.codingOf(headers); + } catch (ContentDecoding.RefusedCodingException refused) { + // A response arrived and was refused, which the transport classifier would otherwise read + // from a bare IOException as a call that got no answer. + throw new HttpTargetRejectedException( + "response content coding is not one this client asked for", + limiter + .failureMetadata() + .withStatus(new HttpStatus(status)) + .withEvidence(ExecutionEvidence.RESPONSE_RECEIVED), + refused); + } try (InputStream counted = - new CountingBoundedInputStream( - body, - read -> { - limiter.recordWireBytes(read); - responseBytes.accept(read); - })) { - byte[] bytes = counted.readAllBytes(); - limiter.recordDecodedBytes(bytes.length); - return new RawResponse(status, toMap(headers), bytes); + new CountingBoundedInputStream( + body, + read -> { + limiter.recordWireBytes(read); + responseBytes.accept(read); + }); + InputStream decoded = ContentDecoding.decoding(coding, counted)) { + ByteArrayOutputStream bytes = new ByteArrayOutputStream(); + byte[] chunk = new byte[ContentDecoding.CHUNK]; + int read; + while ((read = decoded.read(chunk)) != -1) { + limiter.recordDecodedBytes(read); + bytes.write(chunk, 0, read); + } + Map> kept = toMap(headers); + if (!coding.isEmpty()) { + // The body handed on is the decoded one; keeping these would describe bytes it no longer + // is. + ContentDecoding.dropCodingHeaders(kept); + } + return new RawResponse(status, kept, bytes.toByteArray()); } } diff --git a/src/adapter/outbound/httpclient/src/main/java/dev/caskeleton/adapter/outbound/httpclient/restclient/RestClientRuntimeFactory.java b/src/adapter/outbound/httpclient/src/main/java/dev/caskeleton/adapter/outbound/httpclient/restclient/RestClientRuntimeFactory.java index be59f5ff..14eb9824 100644 --- a/src/adapter/outbound/httpclient/src/main/java/dev/caskeleton/adapter/outbound/httpclient/restclient/RestClientRuntimeFactory.java +++ b/src/adapter/outbound/httpclient/src/main/java/dev/caskeleton/adapter/outbound/httpclient/restclient/RestClientRuntimeFactory.java @@ -22,6 +22,7 @@ import java.util.Map; import java.util.Objects; import java.util.function.Supplier; import java.util.random.RandomGenerator; +import org.springframework.http.HttpHeaders; import org.springframework.http.client.ClientHttpRequestFactory; import org.springframework.web.client.RestClient; @@ -77,11 +78,15 @@ public final class RestClientRuntimeFactory implements ClientRuntimeFactory { ClientHttpRequestFactory requestFactory = provider.create(profile, generation, lifecycleListener); - RestClient restClient = - RestClient.builder() - .requestFactory(requestFactory) - .baseUrl(profile.baseUrl().toString()) - .build(); + RestClient.Builder builder = + RestClient.builder().requestFactory(requestFactory).baseUrl(profile.baseUrl().toString()); + if (profile.request().compression()) { + // Asked for here, once, for every transport: none of them decodes any more, so the reader + // counts the coded bytes on the wire and decodes them itself. The JDK transport used to read + // the setting nowhere, and Apache and Jetty each asked on their own. + builder.defaultHeader(HttpHeaders.ACCEPT_ENCODING, ContentDecoding.ACCEPTED); + } + RestClient restClient = builder.build(); AttemptResiliencePipeline pipeline = newResiliencePipeline(profile); diff --git a/src/adapter/outbound/httpclient/src/main/java/dev/caskeleton/adapter/outbound/httpclient/transport/BlockingTransportCapabilities.java b/src/adapter/outbound/httpclient/src/main/java/dev/caskeleton/adapter/outbound/httpclient/transport/BlockingTransportCapabilities.java index f813a537..a0e6a053 100644 --- a/src/adapter/outbound/httpclient/src/main/java/dev/caskeleton/adapter/outbound/httpclient/transport/BlockingTransportCapabilities.java +++ b/src/adapter/outbound/httpclient/src/main/java/dev/caskeleton/adapter/outbound/httpclient/transport/BlockingTransportCapabilities.java @@ -8,7 +8,8 @@ import java.util.Set; * What a blocking transport can actually guarantee (design §13.3). * *

A capability that is weaker than the profile requires fails at startup rather than degrading - * silently at runtime. + * silently at runtime. {@code proxySupport} is an HTTP forward proxy, the only kind {@code + * ProxyType} has. */ public record BlockingTransportCapabilities( Set protocols, diff --git a/src/adapter/outbound/httpclient/src/test/java/dev/caskeleton/adapter/outbound/httpclient/autoconfigure/HttpClientPlatformSettingsTest.java b/src/adapter/outbound/httpclient/src/test/java/dev/caskeleton/adapter/outbound/httpclient/autoconfigure/HttpClientPlatformSettingsTest.java index afa721b1..04ce86e6 100644 --- a/src/adapter/outbound/httpclient/src/test/java/dev/caskeleton/adapter/outbound/httpclient/autoconfigure/HttpClientPlatformSettingsTest.java +++ b/src/adapter/outbound/httpclient/src/test/java/dev/caskeleton/adapter/outbound/httpclient/autoconfigure/HttpClientPlatformSettingsTest.java @@ -2,13 +2,18 @@ package dev.caskeleton.adapter.outbound.httpclient.autoconfigure; import static org.assertj.core.api.Assertions.assertThat; import static org.assertj.core.api.Assertions.assertThatThrownBy; +import static org.assertj.core.api.Assertions.catchThrowable; import dev.caskeleton.adapter.outbound.httpclient.profile.ClientMode; import dev.caskeleton.adapter.outbound.httpclient.profile.HttpProtocol; import dev.caskeleton.adapter.outbound.httpclient.profile.TransportType; import java.util.List; +import java.util.Map; import org.junit.jupiter.api.DisplayName; import org.junit.jupiter.api.Test; +import org.springframework.boot.context.properties.bind.BindException; +import org.springframework.boot.context.properties.bind.Binder; +import org.springframework.boot.context.properties.source.MapConfigurationPropertySource; /** * The aggregate rules a single client's own fields cannot express. @@ -96,6 +101,26 @@ class HttpClientPlatformSettingsTest { .hasMessageContaining("dynamic-targets"); } + @Test + void aSocksProxyTypeFailsToBind() { + // given + Binder binder = + new Binder( + new MapConfigurationPropertySource( + Map.of("proxy.enabled", "true", "proxy.type", "SOCKS"))); + + // when + Throwable refused = + catchThrowable(() -> binder.bind("proxy", HttpClientPlatformSettings.Proxy.class)); + + // then + assertThat(refused) + .as("no shipped transport speaks SOCKS; the value used to bind and then misroute") + .isInstanceOf(BindException.class) + .rootCause() + .hasMessageContaining("SOCKS"); + } + /** * A client whose only meaningful field here is its name. * diff --git a/src/adapter/outbound/httpclient/src/test/java/dev/caskeleton/adapter/outbound/httpclient/http3/JettyContentDecodingTest.java b/src/adapter/outbound/httpclient/src/test/java/dev/caskeleton/adapter/outbound/httpclient/http3/JettyContentDecodingTest.java new file mode 100644 index 00000000..43e1cbaf --- /dev/null +++ b/src/adapter/outbound/httpclient/src/test/java/dev/caskeleton/adapter/outbound/httpclient/http3/JettyContentDecodingTest.java @@ -0,0 +1,38 @@ +package dev.caskeleton.adapter.outbound.httpclient.http3; + +import static org.assertj.core.api.Assertions.assertThat; + +import dev.caskeleton.adapter.outbound.httpclient.testkit.ClientProfiles; +import org.eclipse.jetty.client.HttpClient; +import org.junit.jupiter.api.Test; + +/** + * The experimental Jetty client hands the response reader the coded bytes (round9 review + * 2026-11-12, RR9-01-30). + * + *

Jetty loads its gzip decoder from the classpath on start and then decodes every response + * itself, which made the reader's wire count the decoded size. Over a plain TCP transport, as the + * deadline test does: what is checked is the client's configuration, not the wire. + */ +class JettyContentDecodingTest { + + @Test + void aStartedClientDecodesNoContentCodingItself() throws Exception { + // given + HttpClient client = + JettyHttp3TransportProvider.configure( + new HttpClient(), ClientProfiles.http3Experimental("edge")); + + // when + client.start(); + + // then + try { + assertThat(client.getContentDecoderFactories().isEmpty()) + .as("a decoder left registered decodes the body before the reader can count it") + .isTrue(); + } finally { + client.stop(); + } + } +} diff --git a/src/adapter/outbound/httpclient/src/test/java/dev/caskeleton/adapter/outbound/httpclient/restclient/RestClientResponseReaderEncodingTest.java b/src/adapter/outbound/httpclient/src/test/java/dev/caskeleton/adapter/outbound/httpclient/restclient/RestClientResponseReaderEncodingTest.java new file mode 100644 index 00000000..9706ac6f --- /dev/null +++ b/src/adapter/outbound/httpclient/src/test/java/dev/caskeleton/adapter/outbound/httpclient/restclient/RestClientResponseReaderEncodingTest.java @@ -0,0 +1,156 @@ +package dev.caskeleton.adapter.outbound.httpclient.restclient; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.catchThrowable; + +import dev.caskeleton.adapter.outbound.httpclient.api.error.HttpResponseTooLargeException; +import dev.caskeleton.adapter.outbound.httpclient.api.error.HttpTargetRejectedException; +import dev.caskeleton.adapter.outbound.httpclient.api.operation.ExecutionEvidence; +import java.io.ByteArrayInputStream; +import java.io.ByteArrayOutputStream; +import java.io.IOException; +import java.nio.charset.StandardCharsets; +import java.util.List; +import java.util.zip.GZIPOutputStream; +import org.junit.jupiter.api.Test; +import org.springframework.http.HttpHeaders; + +/** + * The response reader counts what arrives against the wire budget and what it decodes to against + * the decoded budget, a chunk at a time. + * + *

The transport used to decode before the reader saw the body, so the "wire" count was the + * decoded size and a well-compressed response was refused for a size it never had on the wire + * (round9 review 2026-11-12, RR9-01-30). + */ +class RestClientResponseReaderEncodingTest { + + private final RestClientResponseReader reader = new RestClientResponseReader(); + + @Test + void aGzipBodyIsCountedCompressedOnTheWireAndDecodedForTheCaller() throws IOException { + // given + // Six thousand bytes that compress to a few dozen: inside a 1,000-byte wire budget only when + // the wire count is the compressed size. + byte[] decoded = "a".repeat(6_000).getBytes(StandardCharsets.US_ASCII); + byte[] wire = gzip(decoded); + ResponseSizeLimiter limiter = new ResponseSizeLimiter(1_000, 10_000); + + // when + RestClientResponseReader.RawResponse response = + reader.readBounded(200, gzipHeaders(), new ByteArrayInputStream(wire), limiter); + + // then + assertThat( + List.of( + response.body().length, + limiter.wireBytes(), + limiter.decodedBytes(), + response.headers().containsKey(HttpHeaders.CONTENT_ENCODING))) + .containsExactly(6_000, (long) wire.length, 6_000L, false); + } + + @Test + void aBodyThatDecodesPastTheBudgetIsRefusedWhileItIsDecoded() throws IOException { + // given + // A small wire payload that would expand to a megabyte: refused at the decoded budget, before + // the whole expansion is held in memory. + byte[] wire = gzip(new byte[1_048_576]); + ResponseSizeLimiter limiter = new ResponseSizeLimiter(1_048_576, 64 * 1024); + + // when + Throwable refused = + catchThrowable( + () -> reader.readBounded(200, gzipHeaders(), new ByteArrayInputStream(wire), limiter)); + + // then + assertThat( + List.of( + refused instanceof HttpResponseTooLargeException, + limiter.decodedBytes() <= 64L * 1024 + 8 * 1024)) + .as("decoding stopped within one chunk past the budget: %s", refused) + .containsExactly(true, true); + } + + @Test + void anUnaskedForContentCodingIsRefused() { + // given + HttpHeaders brotli = new HttpHeaders(); + brotli.add(HttpHeaders.CONTENT_ENCODING, "br"); + ResponseSizeLimiter limiter = new ResponseSizeLimiter(1_000, 1_000); + + // when + Throwable refused = + catchThrowable( + () -> + reader.readBounded( + 200, brotli, new ByteArrayInputStream(new byte[] {1, 2, 3}), limiter)); + + // then + assertThat(refused) + .as("a response arrived and was refused; it is not a call that got no answer") + .isInstanceOfSatisfying( + HttpTargetRejectedException.class, + rejected -> + assertThat( + List.of( + rejected.getMessage(), + rejected.getCause().getMessage(), + rejected.metadata().evidence())) + .containsExactly( + "response content coding is not one this client asked for", + "the response used a content coding this client did not ask for: br", + ExecutionEvidence.RESPONSE_RECEIVED)); + } + + @Test + void anEmptyBodyUnderGzipIsAnEmptyBody() throws IOException { + // given + // What a HEAD, a 204 or a 304 sends with the coding of the representation it describes. + ResponseSizeLimiter limiter = new ResponseSizeLimiter(1_000, 1_000); + + // when + RestClientResponseReader.RawResponse response = + reader.readBounded(204, gzipHeaders(), new ByteArrayInputStream(new byte[0]), limiter); + + // then + assertThat(response.body()).isEmpty(); + } + + @Test + void stackedContentCodingsAreRefused() { + // given + HttpHeaders stacked = new HttpHeaders(); + stacked.add(HttpHeaders.CONTENT_ENCODING, "gzip"); + stacked.add(HttpHeaders.CONTENT_ENCODING, "br"); + ResponseSizeLimiter limiter = new ResponseSizeLimiter(1_000, 1_000); + + // when + Throwable refused = + catchThrowable( + () -> + reader.readBounded( + 200, stacked, new ByteArrayInputStream(new byte[] {1, 2, 3}), limiter)); + + // then + assertThat(refused) + .isInstanceOf(HttpTargetRejectedException.class) + .hasMessage("response content coding is not one this client asked for") + .cause() + .hasMessageContaining("stacked content codings"); + } + + private static HttpHeaders gzipHeaders() { + HttpHeaders headers = new HttpHeaders(); + headers.add(HttpHeaders.CONTENT_ENCODING, "gzip"); + return headers; + } + + private static byte[] gzip(byte[] plain) throws IOException { + ByteArrayOutputStream compressed = new ByteArrayOutputStream(); + try (GZIPOutputStream out = new GZIPOutputStream(compressed)) { + out.write(plain); + } + return compressed.toByteArray(); + } +} diff --git a/src/adapter/outbound/httpclient/src/test/java/dev/caskeleton/adapter/outbound/httpclient/restclient/StreamingContentCodingTest.java b/src/adapter/outbound/httpclient/src/test/java/dev/caskeleton/adapter/outbound/httpclient/restclient/StreamingContentCodingTest.java new file mode 100644 index 00000000..20c31874 --- /dev/null +++ b/src/adapter/outbound/httpclient/src/test/java/dev/caskeleton/adapter/outbound/httpclient/restclient/StreamingContentCodingTest.java @@ -0,0 +1,149 @@ +package dev.caskeleton.adapter.outbound.httpclient.restclient; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.catchThrowable; + +import dev.caskeleton.adapter.outbound.httpclient.api.OperationName; +import dev.caskeleton.adapter.outbound.httpclient.api.error.HttpResponseTooLargeException; +import dev.caskeleton.adapter.outbound.httpclient.api.error.HttpTargetRejectedException; +import dev.caskeleton.adapter.outbound.httpclient.api.operation.ExecutionEvidence; +import dev.caskeleton.adapter.outbound.httpclient.api.operation.HttpOperation; +import dev.caskeleton.adapter.outbound.httpclient.api.result.BlockingStreamingResponse; +import dev.caskeleton.adapter.outbound.httpclient.profile.ClientProfile; +import dev.caskeleton.adapter.outbound.httpclient.profile.RequestLimits; +import dev.caskeleton.adapter.outbound.httpclient.profile.ResponseLimits; +import dev.caskeleton.adapter.outbound.httpclient.profile.TransportType; +import dev.caskeleton.adapter.outbound.httpclient.testkit.ClientProfiles; +import dev.caskeleton.adapter.outbound.httpclient.testkit.MockHttpServer; +import dev.caskeleton.adapter.outbound.httpclient.testkit.TestGateways; +import java.io.ByteArrayOutputStream; +import java.io.IOException; +import java.nio.charset.StandardCharsets; +import java.util.List; +import java.util.Map; +import java.util.Set; +import java.util.zip.GZIPOutputStream; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.EnumSource; + +/** + * A streamed download is decoded the way a buffered body is, on every blocking transport. + * + *

Once the transports stopped decoding — so the wire budget counts what arrives — the streaming + * path still handed the body straight on, and a compressed response reached the caller as gzip + * bytes (round9 review 2026-11-12, RR9-01-30). The wire budget here is smaller than the decoded + * body and larger than the compressed one: the download passes only when the wire is counted + * compressed, which is also what fails if a transport goes back to decoding on its own. + */ +class StreamingContentCodingTest { + + private static final byte[] DECODED = "a".repeat(6_000).getBytes(StandardCharsets.US_ASCII); + + @ParameterizedTest(name = "{0}") + @EnumSource( + value = TransportType.class, + names = {"APACHE", "JDK"}) + void aGzipDownloadReachesTheCallerDecodedAndCountedCompressedOnTheWire(TransportType transport) + throws Exception { + try (MockHttpServer server = MockHttpServer.start(); + TestGateways.Harness harness = TestGateways.forProfile(profile(server, transport))) { + // given + server.enqueueEncodedBody(200, "text/plain", "gzip", gzip(DECODED)); + BlockingStreamingGateway gateway = new BlockingStreamingGateway(harness.registry()); + + // when + List received; + try (BlockingStreamingResponse response = gateway.download(harness.profile().name(), get())) { + received = + List.of( + response.body().readAllBytes().length, + response.headers().keySet().stream() + .anyMatch(name -> name.equalsIgnoreCase("Content-Encoding"))); + } + + // then + assertThat(received).containsExactly(DECODED.length, false); + } + } + + @ParameterizedTest(name = "{0}") + @EnumSource( + value = TransportType.class, + names = {"APACHE", "JDK"}) + void anIdentityDownloadPastTheDecodedBudgetFailsAsTheBufferedReadDoes(TransportType transport) + throws Exception { + try (MockHttpServer server = MockHttpServer.start(); + TestGateways.Harness harness = + TestGateways.forProfile( + profile( + server, transport, new ResponseLimits(10_000, 1_000, Set.of("text/plain"))))) { + // given + // Within the wire budget and past the decoded one: an identity body is its own decoding. + server.enqueueBody(200, "text/plain", DECODED); + BlockingStreamingGateway gateway = new BlockingStreamingGateway(harness.registry()); + + // when + Throwable failure; + try (BlockingStreamingResponse response = gateway.download(harness.profile().name(), get())) { + failure = catchThrowable(() -> response.body().readAllBytes()); + } + + // then + assertThat(failure) + .isInstanceOf(HttpResponseTooLargeException.class) + .hasMessageContaining("decoded bytes exceeded the profile limit of 1000"); + } + } + + @ParameterizedTest(name = "{0}") + @EnumSource( + value = TransportType.class, + names = {"APACHE", "JDK"}) + void aStreamedResponseInACodingThisClientDidNotAskForIsRefusedAsReceived(TransportType transport) + throws Exception { + try (MockHttpServer server = MockHttpServer.start(); + TestGateways.Harness harness = TestGateways.forProfile(profile(server, transport))) { + // given + server.enqueueEncodedBody(200, "text/plain", "br", new byte[] {1, 2, 3}); + BlockingStreamingGateway gateway = new BlockingStreamingGateway(harness.registry()); + + // when + Throwable refused = catchThrowable(() -> gateway.download(harness.profile().name(), get())); + + // then + assertThat(refused) + .isInstanceOfSatisfying( + HttpTargetRejectedException.class, + rejected -> + assertThat(rejected.metadata().evidence()) + .as("the server answered; the client refused what it sent") + .isEqualTo(ExecutionEvidence.RESPONSE_RECEIVED)); + } + } + + private static HttpOperation get() { + return HttpOperation.get(new OperationName("download"), "/download", Map.of()); + } + + private static ClientProfile profile(MockHttpServer server, TransportType transport) { + return profile(server, transport, new ResponseLimits(1_000, 10_000, Set.of("text/plain"))); + } + + private static ClientProfile profile( + MockHttpServer server, TransportType transport, ResponseLimits limits) { + return ClientProfiles.builder("streaming-coding") + .baseUrl(server.uri("/")) + .transport(transport) + .request(new RequestLimits(1024, true)) + .response(limits) + .build(); + } + + private static byte[] gzip(byte[] plain) throws IOException { + ByteArrayOutputStream compressed = new ByteArrayOutputStream(); + try (GZIPOutputStream out = new GZIPOutputStream(compressed)) { + out.write(plain); + } + return compressed.toByteArray(); + } +} diff --git a/src/adapter/outbound/httpclient/src/test/java/dev/caskeleton/adapter/outbound/httpclient/restclient/StreamingSetupFailureTest.java b/src/adapter/outbound/httpclient/src/test/java/dev/caskeleton/adapter/outbound/httpclient/restclient/StreamingSetupFailureTest.java new file mode 100644 index 00000000..588b5f78 --- /dev/null +++ b/src/adapter/outbound/httpclient/src/test/java/dev/caskeleton/adapter/outbound/httpclient/restclient/StreamingSetupFailureTest.java @@ -0,0 +1,59 @@ +package dev.caskeleton.adapter.outbound.httpclient.restclient; + +import static org.assertj.core.api.Assertions.assertThat; + +import dev.caskeleton.adapter.outbound.httpclient.api.error.HttpDeadlineExceededException; +import dev.caskeleton.adapter.outbound.httpclient.api.error.HttpFailureMetadata; +import dev.caskeleton.adapter.outbound.httpclient.api.error.HttpRemoteErrorException; +import dev.caskeleton.adapter.outbound.httpclient.api.error.HttpTargetRejectedException; +import dev.caskeleton.adapter.outbound.httpclient.testkit.CoreFixtures; +import java.util.stream.Stream; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.Arguments; +import org.junit.jupiter.params.provider.MethodSource; + +/** + * What a streaming setup failure reaches the caller as, decided apart from the race that produces + * it: a test against a server cannot make the budget run out between the headers arriving and the + * coding being refused, so the decision is held here directly. + */ +class StreamingSetupFailureTest { + + private static final HttpFailureMetadata METADATA = CoreFixtures.ambiguousMetadata(); + + static Stream failuresThatKeepWhatTheyAre() { + return Stream.of( + Arguments.of( + "예산을 다 쓴 뒤 거절한 응답", new HttpTargetRejectedException("refused coding", METADATA), true), + Arguments.of( + "예산을 다 쓴 뒤 원격 오류", new HttpRemoteErrorException("remote error", METADATA), true), + Arguments.of("예산이 남은 transport 실패", new IllegalStateException("transport"), false)); + } + + @ParameterizedTest(name = "{0}") + @MethodSource("failuresThatKeepWhatTheyAre") + void aFailureTheDeadlineDidNotCauseKeepsWhatItIs( + String name, RuntimeException failure, boolean budgetSpent) { + // when + RuntimeException thrown = BlockingStreamingGateway.setupFailure(failure, budgetSpent, METADATA); + + // then + assertThat(thrown).isSameAs(failure); + } + + @Test + void aTransportFailureAfterTheBudgetIsSpentIsTheDeadline() { + // given + IllegalStateException failure = new IllegalStateException("transport"); + + // when + RuntimeException thrown = BlockingStreamingGateway.setupFailure(failure, true, METADATA); + + // then + assertThat(thrown) + .isInstanceOf(HttpDeadlineExceededException.class) + .hasMessage("effective deadline reached before the stream was set up") + .hasCause(failure); + } +} diff --git a/src/adapter/outbound/httpclient/src/test/java/dev/caskeleton/adapter/outbound/httpclient/transport/TransportCapabilityValidatorTest.java b/src/adapter/outbound/httpclient/src/test/java/dev/caskeleton/adapter/outbound/httpclient/transport/TransportCapabilityValidatorTest.java index 40075617..84894c77 100644 --- a/src/adapter/outbound/httpclient/src/test/java/dev/caskeleton/adapter/outbound/httpclient/transport/TransportCapabilityValidatorTest.java +++ b/src/adapter/outbound/httpclient/src/test/java/dev/caskeleton/adapter/outbound/httpclient/transport/TransportCapabilityValidatorTest.java @@ -6,7 +6,10 @@ import static org.assertj.core.api.Assertions.assertThatThrownBy; import dev.caskeleton.adapter.outbound.httpclient.api.error.HttpConfigurationException; import dev.caskeleton.adapter.outbound.httpclient.profile.ClientMode; import dev.caskeleton.adapter.outbound.httpclient.profile.ClientProfile; +import dev.caskeleton.adapter.outbound.httpclient.profile.ProxySettings; +import dev.caskeleton.adapter.outbound.httpclient.profile.ProxyType; import dev.caskeleton.adapter.outbound.httpclient.testkit.ClientProfiles; +import java.util.Optional; import org.junit.jupiter.api.Test; class TransportCapabilityValidatorTest { @@ -46,6 +49,19 @@ class TransportCapabilityValidatorTest { .hasMessageContaining("validated DNS pinning"); } + @Test + void acceptsAnHttpProxyOnATransportThatHasOne() { + ClientProfile http = + ClientProfiles.builder("egress") + .proxy( + new ProxySettings( + true, "proxy.internal", 3128, ProxyType.HTTP, Optional.empty(), false)) + .build(); + + assertThatCode(() -> validator.validate(http, BlockingTransportCapabilities.apacheClassic())) + .doesNotThrowAnyException(); + } + @Test void acceptsAMatchingCapabilitySet() { assertThatCode( diff --git a/src/adapter/outbound/httpclient/src/testFixtures/java/dev/caskeleton/adapter/outbound/httpclient/testkit/MockHttpServer.java b/src/adapter/outbound/httpclient/src/testFixtures/java/dev/caskeleton/adapter/outbound/httpclient/testkit/MockHttpServer.java index c249c86b..168c07c9 100644 --- a/src/adapter/outbound/httpclient/src/testFixtures/java/dev/caskeleton/adapter/outbound/httpclient/testkit/MockHttpServer.java +++ b/src/adapter/outbound/httpclient/src/testFixtures/java/dev/caskeleton/adapter/outbound/httpclient/testkit/MockHttpServer.java @@ -121,6 +121,22 @@ public final class MockHttpServer implements AutoCloseable { .setBody(new okio.Buffer().write(body))); } + /** + * Answers with a body already in {@code contentEncoding}, as a server that compresses would. + * + * @param body the coded bytes, sent as they are + */ + @SuppressWarnings("resource") // the okio Buffer, as in enqueueBody + public void enqueueEncodedBody( + int status, String contentType, String contentEncoding, byte[] body) { + server.enqueue( + new MockResponse() + .setResponseCode(status) + .setHeader("Content-Type", contentType) + .setHeader("Content-Encoding", contentEncoding) + .setBody(new okio.Buffer().write(body))); + } + /** Accepts the request and never answers; drives response-header timeout tests. */ public void enqueueNoResponse() { server.enqueue(new MockResponse().setSocketPolicy(SocketPolicy.NO_RESPONSE)); diff --git a/src/adapter/outbound/objectstorage/src/main/java/dev/caskeleton/adapter/outbound/objectstorage/s3/S3ConditionalObjectControlStore.java b/src/adapter/outbound/objectstorage/src/main/java/dev/caskeleton/adapter/outbound/objectstorage/s3/S3ConditionalObjectControlStore.java index f7a57a87..f38d16d7 100644 --- a/src/adapter/outbound/objectstorage/src/main/java/dev/caskeleton/adapter/outbound/objectstorage/s3/S3ConditionalObjectControlStore.java +++ b/src/adapter/outbound/objectstorage/src/main/java/dev/caskeleton/adapter/outbound/objectstorage/s3/S3ConditionalObjectControlStore.java @@ -29,6 +29,8 @@ import software.amazon.awssdk.services.s3.model.ServerSideEncryption; /** Exact-key S3 control store using native conditional PUT rather than check-then-act emulation. */ public final class S3ConditionalObjectControlStore implements ObjectControlStore { + private static final String CONFLICT = "S3 control conditional mutation conflicted"; + private static final int MAXIMUM_CONTROL_BYTES = 64 * 1024; private final S3AsyncClient client; @@ -107,9 +109,11 @@ public final class S3ConditionalObjectControlStore implements ObjectControlStore true); return stored(response.eTag(), record); } catch (ObjectStorageProviderException exception) { - if (exception.failure() == ObjectStorageProviderException.Failure.CONFLICT - || exception.failure() == ObjectStorageProviderException.Failure.PRECONDITION_FAILED) { - throw new ObjectControlConflictException("S3 control conditional mutation conflicted"); + if (exception.failure() == ObjectStorageProviderException.Failure.PRECONDITION_FAILED) { + return reconcilePreconditionFailure(key, record); + } + if (exception.failure() == ObjectStorageProviderException.Failure.CONFLICT) { + throw new ObjectControlConflictException(CONFLICT); } if (exception.failure() == ObjectStorageProviderException.Failure.INDETERMINATE) { return reconcileLostResponse(key, record); @@ -118,20 +122,52 @@ public final class S3ConditionalObjectControlStore implements ObjectControlStore } } + /** + * Decides a failed precondition by what the key holds now. + * + *

A 412 is not always somebody else's write. The SDK resends a conditional {@code PUT} whose + * response it lost — the body is replayable — and when the first send landed, the resend fails + * the very precondition the first one satisfied: {@code If-None-Match: *} now finds the object, + * {@code If-Match} now finds a new ETag. Reported as a conflict, this store's own write was + * called stale and the CAS above it gave up on a mutation that had succeeded (round9 review + * 2026-11-12, RR9-01-19). The key holding exactly the record this call wrote is that case; any + * other content, or none, is a real conflict. + * + *

"Exactly the record" is equality, not authorship: two writers that CAS the same version to + * equal records — same revision, state and timestamp — are indistinguishable here, and both are + * told they succeeded. The record's revision is what normally tells them apart; a caller that can + * produce equal records from two places carries its own token in the record. + */ + private StoredControlRecord reconcilePreconditionFailure( + String key, ObjectControlRecord expected) { + return holdingExactly(key, expected) + .orElseThrow(() -> new ObjectControlConflictException(CONFLICT)); + } + private StoredControlRecord reconcileLostResponse(String key, ObjectControlRecord expected) { Optional current = read(key); - if (current.isPresent() && current.orElseThrow().record().equals(expected)) { - return current.orElseThrow(); - } if (current.isPresent()) { - throw new ObjectControlConflictException( - "S3 control response loss resolved to different exact record evidence"); + return holding(current.orElseThrow(), expected) + .orElseThrow( + () -> + new ObjectControlConflictException( + "S3 control response loss resolved to different exact record evidence")); } throw new ObjectStorageProviderException( ObjectStorageProviderException.Failure.INDETERMINATE, "S3 control mutation response was lost and no authoritative applied evidence exists"); } + /** What the key holds now, when it is exactly {@code expected}. */ + private Optional holdingExactly(String key, ObjectControlRecord expected) { + return read(key).flatMap(current -> holding(current, expected)); + } + + private static Optional holding( + StoredControlRecord current, ObjectControlRecord expected) { + return current.record().equals(expected) ? Optional.of(current) : Optional.empty(); + } + private static StoredControlRecord stored(String eTag, ObjectControlRecord record) { if (eTag == null || eTag.isBlank() || eTag.length() > 256) { throw new ObjectStorageProviderException( diff --git a/src/adapter/outbound/objectstorage/src/test/java/dev/caskeleton/adapter/outbound/objectstorage/s3/S3ConditionalObjectControlStoreTest.java b/src/adapter/outbound/objectstorage/src/test/java/dev/caskeleton/adapter/outbound/objectstorage/s3/S3ConditionalObjectControlStoreTest.java index 68fa30eb..28a5d50e 100644 --- a/src/adapter/outbound/objectstorage/src/test/java/dev/caskeleton/adapter/outbound/objectstorage/s3/S3ConditionalObjectControlStoreTest.java +++ b/src/adapter/outbound/objectstorage/src/test/java/dev/caskeleton/adapter/outbound/objectstorage/s3/S3ConditionalObjectControlStoreTest.java @@ -2,6 +2,7 @@ package dev.caskeleton.adapter.outbound.objectstorage.s3; import static org.assertj.core.api.Assertions.assertThat; import static org.assertj.core.api.Assertions.assertThatThrownBy; +import static org.assertj.core.api.Assertions.catchThrowable; import static org.mockito.ArgumentMatchers.any; import static org.mockito.Mockito.mock; import static org.mockito.Mockito.never; @@ -26,6 +27,7 @@ import java.util.List; import java.util.concurrent.CompletableFuture; import org.junit.jupiter.api.Test; import org.mockito.ArgumentCaptor; +import org.mockito.ArgumentMatchers; import software.amazon.awssdk.core.async.AsyncRequestBody; import software.amazon.awssdk.core.async.AsyncResponseTransformer; import software.amazon.awssdk.core.async.SdkPublisher; @@ -99,6 +101,9 @@ class S3ConditionalObjectControlStoreTest { S3AsyncClient client = mock(S3AsyncClient.class); when(client.putObject(any(PutObjectRequest.class), any(AsyncRequestBody.class))) .thenReturn(CompletableFuture.failedFuture(S3Exception.builder().statusCode(412).build())); + // Another writer's record holds the key: the precondition failed on somebody else's write. + respondToGet( + client, CODEC.encode(record(3, PublishedReferenceState.PUBLISHED)), "\"other-writer\""); S3ConditionalObjectControlStore store = new S3ConditionalObjectControlStore(client, awsBinding(), CODEC); @@ -118,12 +123,64 @@ class S3ConditionalObjectControlStoreTest { .class); } + @Test + void aPreconditionFailedByTheResendOfOurOwnLandedWriteIsThatWrite() { + // given + // The SDK lost the first response and resent the conditional PUT; the first one had landed, so + // the resend fails If-Match against the ETag it produced. + S3AsyncClient client = mock(S3AsyncClient.class); + ObjectReferenceRecord written = record(2, PublishedReferenceState.RETIRED); + when(client.putObject(any(PutObjectRequest.class), any(AsyncRequestBody.class))) + .thenReturn(CompletableFuture.failedFuture(S3Exception.builder().statusCode(412).build())); + respondToGet(client, CODEC.encode(written), "\"after-our-write\""); + S3ConditionalObjectControlStore store = + new S3ConditionalObjectControlStore(client, awsBinding(), CODEC); + + // when + ObjectControlStore.StoredControlRecord stored = + store.compareAndSet( + KEY, new ObjectControlMutation(ObjectControlVersion.of("\"before\""), written)); + + // then + assertThat(List.of(stored.record(), stored.version())) + .containsExactly(written, ObjectControlVersion.of("\"after-our-write\"")); + } + + @Test + void aPreconditionFailureOnAnAbsentKeyIsAConflict() { + // given + S3AsyncClient client = mock(S3AsyncClient.class); + when(client.putObject(any(PutObjectRequest.class), any(AsyncRequestBody.class))) + .thenReturn(CompletableFuture.failedFuture(S3Exception.builder().statusCode(412).build())); + when(client.getObject( + any(GetObjectRequest.class), + ArgumentMatchers.>any())) + .thenReturn(CompletableFuture.failedFuture(S3Exception.builder().statusCode(404).build())); + S3ConditionalObjectControlStore store = + new S3ConditionalObjectControlStore(client, awsBinding(), CODEC); + + // when + Throwable refused = + catchThrowable( + () -> + store.compareAndSet( + KEY, + new ObjectControlMutation( + ObjectControlVersion.of("\"stale\""), + record(2, PublishedReferenceState.RETIRED)))); + + // then + assertThat(refused) + .isInstanceOf(ObjectControlConflictException.class) + .hasMessageContaining("conflicted"); + } + @Test void exact404IsTheOnlyAbsentRead() { S3AsyncClient client = mock(S3AsyncClient.class); when(client.getObject( any(GetObjectRequest.class), - org.mockito.ArgumentMatchers.>any())) + ArgumentMatchers.>any())) .thenReturn(CompletableFuture.failedFuture(S3Exception.builder().statusCode(404).build())); S3ConditionalObjectControlStore store = new S3ConditionalObjectControlStore(client, awsBinding(), CODEC); @@ -154,7 +211,7 @@ class S3ConditionalObjectControlStoreTest { private static void respondToGet(S3AsyncClient client, byte[] encoded, String eTag) { when(client.getObject( any(GetObjectRequest.class), - org.mockito.ArgumentMatchers.>any())) + ArgumentMatchers.>any())) .thenAnswer( invocation -> { AsyncResponseTransformer transformer = diff --git a/src/adapter/outbound/persistence-mongo/src/main/java/dev/caskeleton/adapter/outbound/mongo/advanced/tenancy/database/MongoTenantClientRegistry.java b/src/adapter/outbound/persistence-mongo/src/main/java/dev/caskeleton/adapter/outbound/mongo/advanced/tenancy/database/MongoTenantClientRegistry.java index ef80fdb1..a435e5b5 100644 --- a/src/adapter/outbound/persistence-mongo/src/main/java/dev/caskeleton/adapter/outbound/mongo/advanced/tenancy/database/MongoTenantClientRegistry.java +++ b/src/adapter/outbound/persistence-mongo/src/main/java/dev/caskeleton/adapter/outbound/mongo/advanced/tenancy/database/MongoTenantClientRegistry.java @@ -7,9 +7,14 @@ import dev.caskeleton.adapter.outbound.mongo.api.error.MongoOperationRejectedExc import java.time.Clock; import java.time.Duration; import java.time.Instant; +import java.util.HashMap; +import java.util.List; import java.util.Map; import java.util.Objects; import java.util.Set; +import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.atomic.AtomicLong; +import java.util.function.Function; /** * Bounds how many tenant clients exist at once (advanced plan Task 11). @@ -18,6 +23,13 @@ import java.util.Set; * connection pool and its own monitoring threads, so a thousand tenants is a thousand pools. The * registry caps the number and evicts idle ones, and refuses rather than exceeding the cap — an * unbounded registry fails later, as connection exhaustion on an unrelated request. + * + *

{@link #acquire} hands back a {@link Lease}; the client is in use until every lease on it is + * closed, and only a client nobody holds is idle. Acquire used to return nothing, so idle meant + * "acquired longest ago" and eviction could close a client a long-running caller was still using. + * Closing an evicted client is this registry's own clean-up: if it fails, the tenant whose acquire + * triggered the eviction is not told about another tenant's client — the failure is counted in + * {@link #evictionCloseFailures()} (round9 review 2026-11-12, RR9-01-27). */ @MongoAdvancedEntryPoint(MongoCapability.DATABASE_PER_TENANT) public final class MongoTenantClientRegistry { @@ -26,11 +38,21 @@ public final class MongoTenantClientRegistry { private final Duration idleEviction; - private final Map lastUsed = new java.util.concurrent.ConcurrentHashMap<>(); + private final Map lastUsed = new ConcurrentHashMap<>(); - private final Map clients = new java.util.concurrent.ConcurrentHashMap<>(); + private final Map clients = new ConcurrentHashMap<>(); - private final java.util.function.Function clientFactory; + private final Map leases = new HashMap<>(); + + /** + * Guards the cap, the lease counts and the eviction as one decision. A private monitor rather + * than the registry itself, so code outside cannot take it and stall every acquire. + */ + private final Object lock = new Object(); + + private final AtomicLong evictionCloseFailures = new AtomicLong(); + + private final Function clientFactory; private final Clock clock; @@ -47,15 +69,15 @@ public final class MongoTenantClientRegistry { MongoAdvancedCapabilityGuard guard, int maximumActiveClients, Duration idleEviction, - java.util.function.Function clientFactory) { + Function clientFactory) { this(guard, maximumActiveClients, idleEviction, clientFactory, Clock.systemUTC()); } /** * A registry that owns real clients. * - *

Idle eviction reads {@code clock}; {@link #acquire(String)} used to read {@code - * Instant.now()} beside it. + *

Acquiring, closing a lease and idle eviction all read {@code clock}; {@link + * #acquire(String)} used to read {@code Instant.now()} beside it. * *

The factory is called at most once per tenant, inside the map's own computation, so two * threads asking for the same tenant get one client rather than two — a per-tenant client carries @@ -65,7 +87,7 @@ public final class MongoTenantClientRegistry { MongoAdvancedCapabilityGuard guard, int maximumActiveClients, Duration idleEviction, - java.util.function.Function clientFactory, + Function clientFactory, Clock clock) { Objects.requireNonNull(guard, "guard").require(MongoTenantClientRegistry.class); this.clientFactory = Objects.requireNonNull(clientFactory, "clientFactory"); @@ -80,41 +102,128 @@ public final class MongoTenantClientRegistry { /** * Acquires a client for a tenant. * + * @return the lease to close when the work with the client is done * @throws MongoOperationRejectedException when the cap is reached and nothing can be evicted */ - public void acquire(String tenantKey) { - acquire(tenantKey, clock.instant()); + public Lease acquire(String tenantKey) { + Objects.requireNonNull(tenantKey, "tenantKey"); + synchronized (lock) { + return acquireHeld(tenantKey); + } } - /** Acquires a client for a tenant at an explicit instant, so eviction is testable. */ - public synchronized void acquire(String tenantKey, Instant now) { - Objects.requireNonNull(tenantKey, "tenantKey"); - Objects.requireNonNull(now, "now"); + private Lease acquireHeld(String tenantKey) { + // One time source for the acquire, the close and the eviction: an overload that took the + // instant as an argument let a caller acquire in one timeline and close in the clock's. + Instant now = clock.instant(); // Synchronized rather than a bare ConcurrentHashMap: the cap, the eviction and the create are // one decision. Checking the size on one thread while another inserts is how a cap of ten ends // up holding twelve connection pools. - if (lastUsed.containsKey(tenantKey)) { - lastUsed.put(tenantKey, now); - return; + if (!lastUsed.containsKey(tenantKey)) { + evictIdle(now); + if (lastUsed.size() >= maximumActiveClients) { + throw MongoOperationRejectedException.of( + "tenancy.client", + "the tenant client cap of " + + maximumActiveClients + + " is reached and no client is idle; each tenant client carries its own" + + " connection pool and monitoring threads, so the cap is what stops one process" + + " from exhausting the cluster's connections"); + } + clients.computeIfAbsent(tenantKey, clientFactory); } - evictIdle(now); - if (lastUsed.size() >= maximumActiveClients) { - throw MongoOperationRejectedException.of( - "tenancy.client", - "the tenant client cap of " - + maximumActiveClients - + " is reached and no client is idle; each tenant client carries its own connection " - + "pool and monitoring threads, so the cap is what stops one process from exhausting " - + "the cluster's connections"); - } - clients.computeIfAbsent(tenantKey, clientFactory); lastUsed.put(tenantKey, now); + leases.merge(tenantKey, 1, Integer::sum); + return new HeldLease(tenantKey, clients.get(tenantKey)); } - /** Releases a tenant's client and closes it. */ - public synchronized void release(String tenantKey) { - lastUsed.remove(Objects.requireNonNull(tenantKey, "tenantKey")); - close(tenantKey); + /** + * Releases a tenant's client and closes it. + * + * @throws IllegalStateException when a lease on the client is still open, or the close fails + */ + public void release(String tenantKey) { + Objects.requireNonNull(tenantKey, "tenantKey"); + synchronized (lock) { + if (leases.containsKey(tenantKey)) { + throw new IllegalStateException( + "the client for tenant '" + tenantKey + "' is still in use and cannot be released"); + } + lastUsed.remove(tenantKey); + close(tenantKey); + } + } + + /** + * How many evicted clients failed to close. + * + *

The registry has dropped each of them and freed its slot; the count is how an operator + * learns that clients — and their pools — may still be open behind it. + */ + public long evictionCloseFailures() { + return evictionCloseFailures.get(); + } + + /** + * One use of a tenant client; close it when the work with that client is done. + * + *

An interface so the registry's own implementation stays private: only the registry makes a + * lease, and only through {@link #acquire}. + */ + public interface Lease extends AutoCloseable { + + /** The tenant's client, valid until this lease is closed. */ + AutoCloseable client(); + + /** Ends this use. Closing twice is the same as closing once. */ + @Override + void close(); + } + + private final class HeldLease implements Lease { + + private final String tenantKey; + + private final AutoCloseable client; + + private boolean closed; + + private HeldLease(String tenantKey, AutoCloseable client) { + this.tenantKey = tenantKey; + this.client = client; + } + + @Override + public AutoCloseable client() { + return client; + } + + @Override + public void close() { + synchronized (lock) { + if (closed) { + return; + } + closed = true; + leases.computeIfPresent(tenantKey, (tenant, open) -> open == 1 ? null : open - 1); + // Idle from the end of the last use, not its start: a client held for half an hour was in + // use all that time, and measuring from the acquire evicted it the moment it was let go. + Instant released = clock.instant(); + lastUsed.computeIfPresent( + tenantKey, (tenant, used) -> released.isAfter(used) ? released : used); + } + } + } + + /** + * How many leases are open now, across all tenants. + * + *

A lease nobody closed keeps its client's slot for good; this is how that becomes visible. + */ + public int openLeases() { + synchronized (lock) { + return leases.values().stream().mapToInt(Integer::intValue).sum(); + } } /** The tenants with an active client. */ @@ -128,8 +237,9 @@ public final class MongoTenantClientRegistry { } private void evictIdle(Instant now) { - java.util.List idle = + List idle = lastUsed.entrySet().stream() + .filter(entry -> !leases.containsKey(entry.getKey())) .filter(entry -> entry.getValue().plus(idleEviction).isBefore(now)) .map(Map.Entry::getKey) .toList(); @@ -138,7 +248,11 @@ public final class MongoTenantClientRegistry { lastUsed.remove(tenant); // Evicting the bookkeeping entry without closing the client frees the slot and keeps the // connection pool, which is the exhaustion this cap exists to prevent, delayed. - close(tenant); + try { + close(tenant); + } catch (IllegalStateException closeFailed) { + evictionCloseFailures.incrementAndGet(); + } }); } diff --git a/src/adapter/outbound/persistence-mongo/src/main/java/dev/caskeleton/adapter/outbound/mongo/aggregation/PolicyAwareMongoAggregationExecutor.java b/src/adapter/outbound/persistence-mongo/src/main/java/dev/caskeleton/adapter/outbound/mongo/aggregation/PolicyAwareMongoAggregationExecutor.java index 8ed663b7..0f0a91d4 100644 --- a/src/adapter/outbound/persistence-mongo/src/main/java/dev/caskeleton/adapter/outbound/mongo/aggregation/PolicyAwareMongoAggregationExecutor.java +++ b/src/adapter/outbound/persistence-mongo/src/main/java/dev/caskeleton/adapter/outbound/mongo/aggregation/PolicyAwareMongoAggregationExecutor.java @@ -6,20 +6,30 @@ import dev.caskeleton.adapter.outbound.mongo.query.budget.MongoBudgetEnforcer; import dev.caskeleton.adapter.outbound.mongo.query.budget.MongoBudgetPolicyRegistry; import dev.caskeleton.adapter.outbound.mongo.query.budget.MongoOperationBudget; import java.time.Duration; +import java.util.ArrayList; +import java.util.Iterator; import java.util.List; import java.util.Objects; +import java.util.stream.Stream; +import org.bson.Document; +import org.bson.RawBsonDocument; +import org.bson.codecs.Codec; import org.springframework.data.mongodb.core.MongoOperations; import org.springframework.data.mongodb.core.aggregation.Aggregation; +import org.springframework.data.mongodb.core.aggregation.AggregationOperation; import org.springframework.data.mongodb.core.aggregation.AggregationOptions; -import org.springframework.data.mongodb.core.aggregation.AggregationResults; /** * Runs an aggregation only after its stages, budget and lookups have been checked (design §17). * *

Every guard runs before the first stage reaches the server, because an aggregation that is - * going to be refused should cost nothing. The one guard that runs afterwards is the result-count - * check: {@code $limit} bounds what the pipeline emits, but a pipeline whose own shape produced - * more than the budget allows has still told the caller something worth failing over. + * going to be refused should cost nothing. Two run while the results arrive: the result count and + * the result bytes. The pipeline gets a final {@code $limit} of one document past the count budget, + * so the server never produces more than that, and the results are read from the cursor one + * document at a time, the encoded size of each added up as it arrives. They used to be read whole — + * every document in memory before anything was counted — and the byte budget was declared and never + * checked (round9 review 2026-11-12, RR9-01-20; {@code docs/mongodb/query-aggregation-guide.md} + * 「집계에 예산을 둔다」). */ public final class PolicyAwareMongoAggregationExecutor { @@ -53,6 +63,14 @@ public final class PolicyAwareMongoAggregationExecutor { /** * Executes a typed aggregation. * + *

Results are read as {@link Document}, sized, then mapped with the template's converter. Two + * costs follow, both deliberate. Each document is encoded once more to be measured: the template + * hands back decoded documents, so the bytes the server sent are not available, and the budget + * has to be in bytes. And the mapping is the converter's alone: the template's typed aggregate + * would also publish {@code AfterLoadEvent}/{@code AfterConvertEvent} and run entity callbacks, + * and those do not run for an aggregation result read here. A listener that needs them maps from + * the returned type itself. + * * @throws MongoOperationRejectedException when a stage, lookup, budget or result size is not * permitted */ @@ -71,19 +89,38 @@ public final class PolicyAwareMongoAggregationExecutor { MongoOperationBudget effective = enforcer.narrow(budget, budget.narrowedTo(budgetFor(context, budget))); + List bounded = new ArrayList<>(plan.operations()); + // One past the budget: enough to know the budget was exceeded, never more than that. + bounded.add(Aggregation.limit(effective.maxResults() + 1L)); Aggregation aggregation = - Aggregation.newAggregation(plan.operations()).withOptions(optionsFor(profile, effective)); - AggregationResults results = operations.aggregate(aggregation, collection, outputType); - List mapped = results.getMappedResults(); - if (mapped.size() > effective.maxResults()) { - throw MongoOperationRejectedException.of( - "aggregation.result", - "the aggregation returned " - + mapped.size() - + " documents, above the budget of " - + effective.maxResults()); + Aggregation.newAggregation(bounded).withOptions(optionsFor(profile, effective)); + Codec sizing = operations.getConverter().getCodecRegistry().get(Document.class); + List mapped = new ArrayList<>(); + long bytes = 0; + try (Stream documents = + operations.aggregateStream(aggregation, collection, Document.class)) { + Iterator cursor = documents.iterator(); + while (cursor.hasNext()) { + Document document = cursor.next(); + if (mapped.size() == effective.maxResults()) { + throw MongoOperationRejectedException.of( + "aggregation.result", + "the aggregation returned more than the budget of " + + effective.maxResults() + + " documents"); + } + bytes += new RawBsonDocument(document, sizing).getByteBuffer().remaining(); + if (bytes > effective.maxResultBytes()) { + throw MongoOperationRejectedException.of( + "aggregation.result", + "the aggregation returned more than the budget of " + + effective.maxResultBytes() + + " bytes"); + } + mapped.add(operations.getConverter().read(outputType, document)); + } } - return mapped; + return List.copyOf(mapped); } private static MongoOperationBudget budgetFor( diff --git a/src/adapter/outbound/persistence-mongo/src/main/java/dev/caskeleton/adapter/outbound/mongo/migration/MongoCollectionMigrationLedger.java b/src/adapter/outbound/persistence-mongo/src/main/java/dev/caskeleton/adapter/outbound/mongo/migration/MongoCollectionMigrationLedger.java index 0466ca15..855db8ec 100644 --- a/src/adapter/outbound/persistence-mongo/src/main/java/dev/caskeleton/adapter/outbound/mongo/migration/MongoCollectionMigrationLedger.java +++ b/src/adapter/outbound/persistence-mongo/src/main/java/dev/caskeleton/adapter/outbound/mongo/migration/MongoCollectionMigrationLedger.java @@ -36,20 +36,46 @@ public final class MongoCollectionMigrationLedger implements MongoMigrationLedge private static final String MIGRATION_ID = "migrationId"; + private static final String FENCE = "fence"; + private final MongoCollection ledger; private final MongoCollection checkpoints; + private final MongoCollection locks; + + /** + * A ledger over the default collections of {@code database}, the lock's included. + * + *

The lock collection is {@link MongoCollectionMigrationLock#DEFAULT_COLLECTION}. A deployment + * that keeps its runner lease in a collection of another name uses the three-collection + * constructor with that collection; with this one the ledger would read a lock that is never + * taken and refuse nothing on its account. + * + * @param database the database being migrated + */ public MongoCollectionMigrationLedger(MongoDatabase database) { this( Objects.requireNonNull(database, "database").getCollection(DEFAULT_COLLECTION), - database.getCollection(DEFAULT_CHECKPOINT_COLLECTION)); + database.getCollection(DEFAULT_CHECKPOINT_COLLECTION), + database.getCollection(MongoCollectionMigrationLock.DEFAULT_COLLECTION)); } + /** + * A ledger over explicit collections. + * + * @param ledger the applied migrations + * @param checkpoints the running migrations' positions + * @param locks the collection {@link MongoCollectionMigrationLock} keeps the runner lease in, + * read to refuse a replaced runner's ledger entry + */ public MongoCollectionMigrationLedger( - MongoCollection ledger, MongoCollection checkpoints) { + MongoCollection ledger, + MongoCollection checkpoints, + MongoCollection locks) { this.ledger = Objects.requireNonNull(ledger, "ledger"); this.checkpoints = Objects.requireNonNull(checkpoints, "checkpoints"); + this.locks = Objects.requireNonNull(locks, "locks"); } /** @@ -90,13 +116,34 @@ public final class MongoCollectionMigrationLedger implements MongoMigrationLedge Objects.requireNonNull(checksum, "checksum"); Objects.requireNonNull(operator, "operator"); Objects.requireNonNull(appliedAt, "appliedAt"); - requireCurrentFence(fence, "ledger entry for " + migrationId.value()); + requireFenced(fence, "ledger entry for " + migrationId.value()); + // A superseded runner that wakes up after its replacement took the lease must not record the + // migration as applied under it (round9 review 2026-11-12, RR9-01-28). Two pieces of evidence: + // the lock's newest acquisition, which a replacement leaves the moment it takes the lease, and + // a checkpoint a newer runner wrote. + // + // A narrowing, not a guarantee: the reads and the insert are separate operations, so a runner + // replaced between them still records. What stops a superseded runner first is the lock's + // refresh, which refuses it; this check is the second line for a runner that skipped a + // refresh. Making it atomic needs the lock and the ledger in one transaction, which a + // standalone deployment does not have. + if (MongoCollectionMigrationLock.newestFence(locks) > fence + || checkpoints + .find( + Filters.and( + Filters.eq(MIGRATION_ID, migrationId.value()), Filters.gt(FENCE, fence))) + .first() + != null) { + throw MongoOperationRejectedException.of( + "migration.fence", + "a newer migration runner owns the lease; this runner's ledger entry was refused"); + } ledger.insertOne( new Document(MIGRATION_ID, migrationId.value()) .append("checksum", checksum.value()) .append("operator", operator) .append("appliedAt", Date.from(appliedAt)) - .append("fence", fence)); + .append(FENCE, fence)); } @Override @@ -117,7 +164,7 @@ public final class MongoCollectionMigrationLedger implements MongoMigrationLedge @Override public void saveCheckpoint(MongoMigrationCheckpoint checkpoint, long fence) { Objects.requireNonNull(checkpoint, "checkpoint"); - requireCurrentFence(fence, "checkpoint for " + checkpoint.migrationId().value()); + requireFenced(fence, "checkpoint for " + checkpoint.migrationId().value()); // Conditioned on the stored fence as well as matched by id: a superseded runner that wakes up // mid-batch must not move the checkpoint of the runner that replaced it, which would send the // live migration back over rows it has already done or forward past rows it has not. @@ -126,7 +173,7 @@ public final class MongoCollectionMigrationLedger implements MongoMigrationLedge .append("resumePosition", checkpoint.resumePosition()) .append("processedCount", checkpoint.processedCount()) .append("updatedAt", Date.from(checkpoint.updatedAt())) - .append("fence", fence); + .append(FENCE, fence); // Replace-then-insert rather than one upsert, and the two cases this separates are the whole // behaviour of the fence. @@ -147,7 +194,7 @@ public final class MongoCollectionMigrationLedger implements MongoMigrationLedge .replaceOne( Filters.and( Filters.eq(MIGRATION_ID, checkpoint.migrationId().value()), - Filters.or(Filters.exists("fence", false), Filters.lte("fence", fence))), + Filters.or(Filters.exists(FENCE, false), Filters.lte(FENCE, fence))), document, new ReplaceOptions().upsert(false)) .getMatchedCount(); @@ -172,9 +219,13 @@ public final class MongoCollectionMigrationLedger implements MongoMigrationLedge * Refuses a write from an unfenced lease. * *

An unfenced lease cannot prove the writer is still the owner, and the whole point of the - * ledger is that it says what actually happened. + * ledger is that it says what actually happened. Whether a fenced writer is still the current one + * is decided in two places, not here: the runner refreshes its lease, conditioned on the fence, + * right before it writes, and each write is conditioned on the newest fence the ledger has seen + * for the migration. The method was called {@code requireCurrentFence} and checked neither + * (round9 review 2026-11-12, RR9-01-28). */ - private static void requireCurrentFence(long fence, String what) { + private static void requireFenced(long fence, String what) { if (fence == MongoMigrationLock.UNFENCED) { throw MongoOperationRejectedException.of( "migration.fence", diff --git a/src/adapter/outbound/persistence-mongo/src/main/java/dev/caskeleton/adapter/outbound/mongo/migration/MongoCollectionMigrationLock.java b/src/adapter/outbound/persistence-mongo/src/main/java/dev/caskeleton/adapter/outbound/mongo/migration/MongoCollectionMigrationLock.java index cca219af..94c4ac26 100644 --- a/src/adapter/outbound/persistence-mongo/src/main/java/dev/caskeleton/adapter/outbound/mongo/migration/MongoCollectionMigrationLock.java +++ b/src/adapter/outbound/persistence-mongo/src/main/java/dev/caskeleton/adapter/outbound/mongo/migration/MongoCollectionMigrationLock.java @@ -6,6 +6,7 @@ import com.mongodb.client.MongoCollection; import com.mongodb.client.MongoDatabase; import com.mongodb.client.model.Filters; import com.mongodb.client.model.Updates; +import dev.caskeleton.adapter.outbound.mongo.api.error.MongoOperationRejectedException; import java.time.Clock; import java.time.Duration; import java.time.Instant; @@ -125,6 +126,19 @@ public final class MongoCollectionMigrationLock implements MongoMigrationLock { return fence; } + /** + * The fence of the newest acquisition of the lock in {@code locks}, or {@link #UNFENCED} when the + * lock was never taken there. + * + *

For the ledger: a write whose fence is older than this one comes from a runner that has been + * replaced, whether or not its replacement has written anything yet. + */ + static long newestFence(MongoCollection locks) { + Document current = + Objects.requireNonNull(locks, "locks").find(Filters.eq("_id", LOCK_ID)).first(); + return current == null ? UNFENCED : fenceOf(current); + } + private static long fenceOf(Document lock) { Object stored = lock.get("fence"); return stored instanceof Number number ? number.longValue() : 1L; @@ -165,7 +179,11 @@ public final class MongoCollectionMigrationLock implements MongoMigrationLock { .getMatchedCount(); if (stillOwned != 1) { held = false; - throw new IllegalStateException( + // The refusal the heartbeat promises (MongoMigrationHeartbeat#beat), so a migration that + // stops on it stops for the same reason the heartbeat's own ownership check gives. It was an + // IllegalStateException, which read as a server defect (round9 review 2026-11-12, RR9-01-28). + throw MongoOperationRejectedException.of( + "migration.lock", "the migration lease was lost before it could be refreshed; another runner may have " + "taken it after this one appeared to stall"); } diff --git a/src/adapter/outbound/persistence-mongo/src/mongoIntegrationTest/java/dev/caskeleton/adapter/outbound/mongo/migration/MongoMigrationLaneTest.java b/src/adapter/outbound/persistence-mongo/src/mongoIntegrationTest/java/dev/caskeleton/adapter/outbound/mongo/migration/MongoMigrationLaneTest.java index abbcad9c..edb28966 100644 --- a/src/adapter/outbound/persistence-mongo/src/mongoIntegrationTest/java/dev/caskeleton/adapter/outbound/mongo/migration/MongoMigrationLaneTest.java +++ b/src/adapter/outbound/persistence-mongo/src/mongoIntegrationTest/java/dev/caskeleton/adapter/outbound/mongo/migration/MongoMigrationLaneTest.java @@ -224,6 +224,95 @@ class MongoMigrationLaneTest { assertThat(ledger.findCheckpoint(id).orElseThrow().processedCount()).isEqualTo(200); } + @Test + void aSupersededRunnerCannotRecordTheMigrationAppliedUnderItsReplacement() { + // given + MongoDatabase database = freshDatabase(); + MongoCollectionMigrationLedger ledger = new MongoCollectionMigrationLedger(database); + ledger.ensureIndexes(); + MongoMigrationId id = new MongoMigrationId("20260811-011"); + ledger.saveCheckpoint( + MongoMigrationCheckpoint.start(id, FIXED.instant()) + .advancedTo("o-300", 300, FIXED.instant()), + 2L); + + // when + Throwable refused = + catchThrowable( + () -> + ledger.recordApplied( + id, new MongoMigrationChecksum("abc"), "operator", FIXED.instant(), 1L)); + + // then + assertThat(refused) + .isInstanceOf(MongoOperationRejectedException.class) + .hasMessageContaining("a newer migration runner owns the lease"); + } + + @Test + void aReplacedRunnerCannotRecordTheMigrationBeforeItsReplacementWroteAnything() { + // given + MongoDatabase database = freshDatabase(); + MongoCollectionMigrationLedger ledger = new MongoCollectionMigrationLedger(database); + ledger.ensureIndexes(); + MongoMigrationId id = new MongoMigrationId("20260811-012"); + Instant early = Instant.parse("2026-08-11T00:00:00Z"); + MongoMigrationLock stalled = + MongoCollectionMigrationLock.tryAcquire( + database, "stalled-runner", LEASE, Clock.fixed(early, ZoneOffset.UTC)) + .orElseThrow(); + var replacement = + MongoCollectionMigrationLock.tryAcquire( + database, + "next-runner", + LEASE, + Clock.fixed(early.plus(LEASE).plusSeconds(1), ZoneOffset.UTC)) + .orElseThrow(); + + // when + Throwable refused = + catchThrowable( + () -> + ledger.recordApplied( + id, + new MongoMigrationChecksum("abc"), + "operator", + FIXED.instant(), + stalled.fence())); + + // then + assertThat(refused) + .as("%s holds the lease with a newer fence and has not checkpointed yet", replacement) + .isInstanceOf(MongoOperationRejectedException.class) + .hasMessageContaining("a newer migration runner owns the lease"); + } + + @Test + void aLeaseLostBeforeItsRefreshIsTheRefusalTheHeartbeatPromises() { + // given + MongoDatabase database = freshDatabase(); + Instant early = Instant.parse("2026-08-11T00:00:00Z"); + MongoMigrationLock stalled = + MongoCollectionMigrationLock.tryAcquire( + database, "stalled-runner", LEASE, Clock.fixed(early, ZoneOffset.UTC)) + .orElseThrow(); + var unused = + MongoCollectionMigrationLock.tryAcquire( + database, + "next-runner", + LEASE, + Clock.fixed(early.plus(LEASE).plusSeconds(1), ZoneOffset.UTC)) + .orElseThrow(); + + // when + Throwable refused = catchThrowable(() -> stalled.refresh(LEASE)); + + // then + assertThat(refused) + .isInstanceOf(MongoOperationRejectedException.class) + .hasMessageContaining("the migration lease was lost"); + } + @Test void theLedgerRefusesADoubleApplyOfTheSameId() { MongoDatabase database = freshDatabase(); diff --git a/src/adapter/outbound/persistence-mongo/src/test/java/dev/caskeleton/adapter/outbound/mongo/advanced/tenancy/MongoTenancyContractTest.java b/src/adapter/outbound/persistence-mongo/src/test/java/dev/caskeleton/adapter/outbound/mongo/advanced/tenancy/MongoTenancyContractTest.java index 97309f29..06f1316e 100644 --- a/src/adapter/outbound/persistence-mongo/src/test/java/dev/caskeleton/adapter/outbound/mongo/advanced/tenancy/MongoTenancyContractTest.java +++ b/src/adapter/outbound/persistence-mongo/src/test/java/dev/caskeleton/adapter/outbound/mongo/advanced/tenancy/MongoTenancyContractTest.java @@ -3,6 +3,7 @@ package dev.caskeleton.adapter.outbound.mongo.advanced.tenancy; import static org.assertj.core.api.Assertions.assertThat; import static org.assertj.core.api.Assertions.assertThatCode; import static org.assertj.core.api.Assertions.assertThatThrownBy; +import static org.assertj.core.api.Assertions.catchThrowable; import dev.caskeleton.adapter.outbound.mongo.advanced.MongoAdvancedCapabilityFlags; import dev.caskeleton.adapter.outbound.mongo.advanced.MongoAdvancedCapabilityGuard; @@ -19,10 +20,25 @@ import dev.caskeleton.adapter.outbound.mongo.imperative.atomic.AtomicFilter; import dev.caskeleton.adapter.outbound.mongo.migration.MongoMigrationCheckpoint; import dev.caskeleton.adapter.outbound.mongo.schema.manifest.MongoCollectionManifest; import dev.caskeleton.adapter.outbound.mongo.schema.manifest.MongoIndexManifest; +import java.io.IOException; +import java.time.Clock; import java.time.Duration; import java.time.Instant; +import java.time.ZoneId; +import java.time.ZoneOffset; +import java.util.ArrayList; import java.util.List; import java.util.Optional; +import java.util.Set; +import java.util.concurrent.Callable; +import java.util.concurrent.ExecutorService; +import java.util.concurrent.Executors; +import java.util.concurrent.Future; +import java.util.concurrent.atomic.AtomicInteger; +import java.util.concurrent.atomic.AtomicReference; +import java.util.function.Function; +import java.util.stream.IntStream; +import org.junit.jupiter.api.DisplayName; import org.junit.jupiter.api.Tag; import org.junit.jupiter.api.Test; @@ -37,6 +53,8 @@ class MongoTenancyContractTest { MongoAdvancedCapabilityFlags.allDisabled() .withEnabled(MongoCapability.DATABASE_PER_TENANT)); + private static final Instant START = Instant.parse("2026-08-13T00:00:00Z"); + @Test void missingTenantContextFailsClosed() { assertThatThrownBy(() -> injector.apply(Optional.empty(), AtomicFilter.id("o-1"))) @@ -116,29 +134,99 @@ class MongoTenancyContractTest { @Test void registryEnforcesMaximumActiveClients() { + // given MongoTenantClientRegistry registry = new MongoTenantClientRegistry(TENANCY, 2); - Instant now = Instant.parse("2026-08-13T00:00:00Z"); + registry.acquire("t1"); + registry.acquire("t2"); - registry.acquire("t1", now); - registry.acquire("t2", now); + // when + Throwable refused = catchThrowable(() -> registry.acquire("t3")); - assertThatThrownBy(() -> registry.acquire("t3", now)) - .isInstanceOf(MongoOperationRejectedException.class); - assertThat(registry.activeCount()).isEqualTo(2); + // then + assertThat(List.of(refused instanceof MongoOperationRejectedException, registry.activeCount())) + .containsExactly(true, 2); } @Test void anIdleClientIsEvictedToMakeRoom() { - MongoTenantClientRegistry registry = - new MongoTenantClientRegistry(TENANCY, 1, Duration.ofMinutes(5)); - Instant start = Instant.parse("2026-08-13T00:00:00Z"); + // given + MovingClock clock = new MovingClock(START); + MongoTenantClientRegistry registry = registry(1, tenant -> () -> {}, clock); + registry.acquire("t1").close(); + clock.advance(Duration.ofMinutes(10)); - registry.acquire("t1", start); - registry.acquire("t2", start.plus(Duration.ofMinutes(10))); + // when + registry.acquire("t2"); + // then assertThat(registry.activeTenants()).containsExactly("t2"); } + @Test + void aClientIsIdleFromTheEndOfItsLastUseNotItsStart() { + // given + MovingClock clock = new MovingClock(START); + MongoTenantClientRegistry registry = registry(1, tenant -> () -> {}, clock); + MongoTenantClientRegistry.Lease held = registry.acquire("t1"); + clock.advance(Duration.ofMinutes(30)); + held.close(); + clock.advance(Duration.ofMinutes(1)); + + // when + Throwable refused = catchThrowable(() -> registry.acquire("t2")); + + // then + assertThat(List.of(refused instanceof MongoOperationRejectedException, registry.openLeases())) + .as("held for thirty minutes, it was in use all that time and is not idle yet") + .containsExactly(true, 0); + } + + @Test + void anUnclosedLeaseIsCounted() { + // given + MongoTenantClientRegistry registry = new MongoTenantClientRegistry(TENANCY, 2); + + // when + var unused = registry.acquire("t1"); + + // then + assertThat(registry.openLeases()).isEqualTo(1); + } + + private static MongoTenantClientRegistry registry( + int cap, Function clients, Clock clock) { + return new MongoTenantClientRegistry(TENANCY, cap, Duration.ofMinutes(5), clients, clock); + } + + /** A clock a test moves by hand: one time source for acquiring, closing and eviction. */ + private static final class MovingClock extends Clock { + + private final AtomicReference now; + + private MovingClock(Instant start) { + this.now = new AtomicReference<>(start); + } + + private void advance(Duration by) { + now.updateAndGet(instant -> instant.plus(by)); + } + + @Override + public ZoneId getZone() { + return ZoneOffset.UTC; + } + + @Override + public Clock withZone(ZoneId zone) { + return this; + } + + @Override + public Instant instant() { + return now.get(); + } + } + @Test void aTenantDatabaseCannotServeBeforeItsSchemaIsValidated() { MongoTenantLifecyclePolicy policy = MongoTenantLifecyclePolicy.standard(); @@ -172,100 +260,112 @@ class MongoTenancyContractTest { } @Test - @org.junit.jupiter.api.DisplayName("the same tenant is created once under concurrency") - void theSameTenantIsCreatedOnce() throws Exception { - java.util.concurrent.atomic.AtomicInteger created = - new java.util.concurrent.atomic.AtomicInteger(); + void theSameTenantIsCreatedOnceUnderConcurrency() throws Exception { + // given + AtomicInteger created = new AtomicInteger(); MongoTenantClientRegistry registry = - new MongoTenantClientRegistry( - TENANCY, + registry( 8, - Duration.ofMinutes(10), tenant -> { created.incrementAndGet(); return () -> {}; - }); - java.util.concurrent.ExecutorService pool = - java.util.concurrent.Executors.newFixedThreadPool(8); + }, + new MovingClock(START)); + List> work = + IntStream.range(0, 50) + .>mapToObj( + index -> + () -> { + registry.acquire("t1").close(); + return null; + }) + .toList(); + ExecutorService pool = Executors.newFixedThreadPool(8); + + // when try { - java.util.List> work = - java.util.stream.IntStream.range(0, 50) - .>mapToObj( - index -> - () -> { - registry.acquire("t1", Instant.parse("2026-08-13T00:00:00Z")); - return null; - }) - .toList(); - for (java.util.concurrent.Future future : pool.invokeAll(work)) { + for (Future future : pool.invokeAll(work)) { future.get(); } } finally { pool.shutdownNow(); } - assertThat(created) + // then + assertThat(List.of(created.get(), registry.activeCount())) .as("a per-tenant client carries a pool and monitoring threads; a second one doubles both") - .hasValue(1); - assertThat(registry.activeCount()).isEqualTo(1); + .containsExactly(1, 1); } @Test - @org.junit.jupiter.api.DisplayName("acquire(tenant)는 주입된 clock으로 idle 여부를 판단한다") - void anAcquireWithoutAnInstantReadsTheInjectedClock() { + void aClientStillHeldIsNotEvictedHoweverLongAgoItWasAcquired() { // given - java.util.List closed = new java.util.ArrayList<>(); - java.util.concurrent.atomic.AtomicReference now = - new java.util.concurrent.atomic.AtomicReference<>(Instant.parse("2026-08-13T00:00:00Z")); - java.time.Clock clock = - new java.time.Clock() { - @Override - public java.time.ZoneId getZone() { - return java.time.ZoneOffset.UTC; - } + List closed = new ArrayList<>(); + MovingClock clock = new MovingClock(START); + MongoTenantClientRegistry registry = registry(1, tenant -> () -> closed.add(tenant), clock); + MongoTenantClientRegistry.Lease longRunning = registry.acquire("t1"); + clock.advance(Duration.ofMinutes(10)); - @Override - public java.time.Clock withZone(java.time.ZoneId zone) { - return this; - } + // when + Throwable refused = catchThrowable(() -> registry.acquire("t2")); - @Override - public Instant instant() { - return now.get(); - } - }; + // then + assertThat(List.of(refused instanceof MongoOperationRejectedException, closed)) + .as( + "t1 is in use by %s; closing it would pull the pool out from under that work", + longRunning) + .containsExactly(true, List.of()); + } + + @Test + void anotherTenantsClientThatFailsToCloseDoesNotFailThisAcquire() { + // given + MovingClock clock = new MovingClock(START); MongoTenantClientRegistry registry = - new MongoTenantClientRegistry( - TENANCY, 1, Duration.ofMinutes(5), tenant -> () -> closed.add(tenant), clock); - registry.acquire("t1"); - now.set(now.get().plus(Duration.ofMinutes(10))); + registry( + 1, + tenant -> + () -> { + if (tenant.equals("t1")) { + throw new IOException("t1's pool would not shut down"); + } + }, + clock); + registry.acquire("t1").close(); + clock.advance(Duration.ofMinutes(10)); + + // when + MongoTenantClientRegistry.Lease admitted = registry.acquire("t2"); + + // then + assertThat( + List.of( + admitted.client() != null, + registry.activeTenants(), + registry.evictionCloseFailures())) + .containsExactly(true, Set.of("t2"), 1L); + } + + @Test + void evictionClosesTheClientItEvicts() { + // given + List closed = new ArrayList<>(); + MovingClock clock = new MovingClock(START); + MongoTenantClientRegistry registry = registry(1, tenant -> () -> closed.add(tenant), clock); + registry.acquire("t1").close(); + clock.advance(Duration.ofMinutes(10)); // when registry.acquire("t2"); // then - assertThat(closed).containsExactly("t1"); - } - - @Test - @org.junit.jupiter.api.DisplayName("eviction closes the client it evicts") - void evictionClosesTheClient() { - java.util.List closed = new java.util.ArrayList<>(); - MongoTenantClientRegistry registry = - new MongoTenantClientRegistry( - TENANCY, 1, Duration.ofMinutes(5), tenant -> () -> closed.add(tenant)); - Instant start = Instant.parse("2026-08-13T00:00:00Z"); - - registry.acquire("t1", start); - registry.acquire("t2", start.plus(Duration.ofMinutes(10))); - assertThat(closed) .as("freeing the slot without closing the client keeps the connection pool it counted") .containsExactly("t1"); } @Test - @org.junit.jupiter.api.DisplayName("a coordinator resumes from its durable store after a restart") + @DisplayName("a coordinator resumes from its durable store after a restart") void aCoordinatorResumesFromItsDurableStore() { RecordingCheckpointStore store = new RecordingCheckpointStore(); MongoTenantMigrationCoordinator first = diff --git a/src/adapter/outbound/persistence-mongo/src/test/java/dev/caskeleton/adapter/outbound/mongo/aggregation/PolicyAwareMongoAggregationExecutorTest.java b/src/adapter/outbound/persistence-mongo/src/test/java/dev/caskeleton/adapter/outbound/mongo/aggregation/PolicyAwareMongoAggregationExecutorTest.java index 89fd5ff0..bcf90f54 100644 --- a/src/adapter/outbound/persistence-mongo/src/test/java/dev/caskeleton/adapter/outbound/mongo/aggregation/PolicyAwareMongoAggregationExecutorTest.java +++ b/src/adapter/outbound/persistence-mongo/src/test/java/dev/caskeleton/adapter/outbound/mongo/aggregation/PolicyAwareMongoAggregationExecutorTest.java @@ -3,11 +3,31 @@ package dev.caskeleton.adapter.outbound.mongo.aggregation; import static org.assertj.core.api.Assertions.assertThat; import static org.assertj.core.api.Assertions.assertThatCode; import static org.assertj.core.api.Assertions.assertThatThrownBy; +import static org.assertj.core.api.Assertions.catchThrowable; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.ArgumentMatchers.eq; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.when; +import com.mongodb.MongoClientSettings; +import dev.caskeleton.adapter.outbound.mongo.api.MongoOperationContext; +import dev.caskeleton.adapter.outbound.mongo.api.consistency.MongoConsistencyProfile; import dev.caskeleton.adapter.outbound.mongo.api.error.MongoOperationRejectedException; +import dev.caskeleton.adapter.outbound.mongo.query.budget.MongoBudgetEnforcer; +import dev.caskeleton.adapter.outbound.mongo.query.budget.MongoBudgetPolicyRegistry; +import dev.caskeleton.adapter.outbound.mongo.query.budget.MongoOperationBudget; +import java.time.Duration; +import java.util.List; import java.util.Set; +import org.bson.Document; +import org.bson.conversions.Bson; import org.junit.jupiter.api.Tag; import org.junit.jupiter.api.Test; +import org.mockito.ArgumentCaptor; +import org.springframework.data.mongodb.core.MongoOperations; +import org.springframework.data.mongodb.core.aggregation.Aggregation; +import org.springframework.data.mongodb.core.convert.MongoConverter; +import org.springframework.data.mongodb.core.query.Criteria; /** Design §17 — write stages are D4, and stage cost is graded rather than assumed. */ @Tag("mongodb-contract") @@ -105,4 +125,141 @@ class PolicyAwareMongoAggregationExecutorTest { assertThat(MongoAggregationProfile.stableRead().allowDiskUse()).isFalse(); assertThat(MongoAggregationProfile.stableRead().strictMapping()).isTrue(); } + + private static final MongoOperationContext CONTEXT = + MongoOperationContext.of( + "orders.summary", + "primary", + "orders", + MongoConsistencyProfile.PRIMARY_MAJORITY, + Duration.ofSeconds(2)); + + private static final MongoAggregationPlan OPEN_ORDERS = + MongoAggregationPlan.builder() + .stage("$match", Aggregation.match(Criteria.where("status").is("OPEN"))) + .build(); + + @Test + void thePipelineEndsWithALimitOnePastTheCountBudget() { + // given + ArgumentCaptor sent = ArgumentCaptor.forClass(Aggregation.class); + MongoOperations operations = returning(List.of(), sent); + + // when + var unused = + executorOver(operations, budget(3, 1_048_576L)) + .execute( + CONTEXT, + MongoAggregationProfile.stableRead(), + OPEN_ORDERS, + "orders", + Document.class); + + // then + List pipeline = sent.getValue().toPipeline(Aggregation.DEFAULT_CONTEXT); + assertThat(pipeline.get(pipeline.size() - 1)).isEqualTo(new Document("$limit", 4L)); + } + + @Test + void moreDocumentsThanTheCountBudgetAreRefused() { + // given + MongoOperations operations = + returning( + List.of(order(1), order(2), order(3), order(4)), + ArgumentCaptor.forClass(Aggregation.class)); + + // when + Throwable refused = + catchThrowable( + () -> + executorOver(operations, budget(3, 1_048_576L)) + .execute( + CONTEXT, + MongoAggregationProfile.stableRead(), + OPEN_ORDERS, + "orders", + Document.class)); + + // then + assertThat(refused) + .isInstanceOf(MongoOperationRejectedException.class) + .hasMessageContaining("more than the budget of 3 documents"); + } + + @Test + void documentsLargerTogetherThanTheByteBudgetAreRefused() { + // given + // Each order encodes to well over 40 bytes: two of them pass any 80-byte budget. + MongoOperations operations = + returning(List.of(order(1), order(2)), ArgumentCaptor.forClass(Aggregation.class)); + + // when + Throwable refused = + catchThrowable( + () -> + executorOver(operations, budget(10, 80L)) + .execute( + CONTEXT, + MongoAggregationProfile.stableRead(), + OPEN_ORDERS, + "orders", + Document.class)); + + // then + assertThat(refused) + .isInstanceOf(MongoOperationRejectedException.class) + .hasMessageContaining("more than the budget of 80 bytes"); + } + + @Test + void resultsInsideBothBudgetsAreReturnedInCursorOrder() { + // given + MongoOperations operations = + returning(List.of(order(1), order(2)), ArgumentCaptor.forClass(Aggregation.class)); + + // when + List results = + executorOver(operations, budget(10, 1_048_576L)) + .execute( + CONTEXT, + MongoAggregationProfile.stableRead(), + OPEN_ORDERS, + "orders", + Document.class); + + // then + assertThat(results).containsExactly(order(1), order(2)); + } + + private static MongoOperationBudget budget(int maxResults, long maxResultBytes) { + return new MongoOperationBudget(maxResults, maxResultBytes, 500L, 50); + } + + private static PolicyAwareMongoAggregationExecutor executorOver( + MongoOperations operations, MongoOperationBudget budget) { + return new PolicyAwareMongoAggregationExecutor( + operations, + MongoBudgetPolicyRegistry.builder().register("orders.summary", budget).build(), + new MongoBudgetEnforcer()); + } + + /** The driver boundary: a cursor over {@code documents}, mapped as the converter would. */ + private static MongoOperations returning( + List documents, ArgumentCaptor sent) { + MongoOperations operations = mock(MongoOperations.class); + MongoConverter converter = mock(MongoConverter.class); + when(operations.getConverter()).thenReturn(converter); + when(converter.getCodecRegistry()).thenReturn(MongoClientSettings.getDefaultCodecRegistry()); + when(converter.read(eq(Document.class), any(Bson.class))) + .thenAnswer(invocation -> invocation.getArgument(1)); + when(operations.aggregateStream(sent.capture(), eq("orders"), eq(Document.class))) + .thenAnswer(invocation -> documents.stream()); + return operations; + } + + private static Document order(int number) { + return new Document("_id", "order-" + number) + .append("status", "OPEN") + .append("customer", "customer-" + number); + } }