S3 control 저장소는 SDK가 응답을 잃고 다시 보낸 조건부 PUT의 412를 충돌로 본다. MongoDB 집계는 결과 바이트 예산을 보지 않고 전부 메모리에 올린다. tenant client registry는 쓰는 중인 client를 내보내고, 내보내다 닫기에 실패하면 다른 tenant의 acquire가 실패한다. migration lease를 잃은 runner가 그 뒤에도 checkpoint를 쓴다. HTTP client는 SOCKS proxy를 HTTP로 붙이고, 압축을 켜면 Apache는 스스로 풀어 max-wire-bytes가 푼 크기를 세며, JDK transport는 그 설정을 읽지 않는다. 고른 방법: S3는 412에서 다시 읽어 이 호출이 쓴 record 그대로면 성공으로 본다. 집계는 $limit(maxResults + 1)과 maxResultBytes를 집행한다. registry acquire는 Lease를 돌려주고 쓰는 중에는 내보내지 않으며 idle은 마지막 close부터 잰다. 상한·lease 수·내보내기는 registry 밖에서 잡을 수 없는 전용 lock 하나로 묶는다. 닫기 실패는 evictionCloseFailures가 센다. migration ledger는 lock 문서의 현재 fence로 쓴다. ProxyType은 HTTP만 남긴다. 압축은 RestClient 기본 헤더로 보내고 ContentDecoding이 버퍼·스트리밍 본문을 모두 푼다(겹친 coding과 그 밖의 coding은 거부, 빈 본문은 빈 본문). Jetty HTTP/3는 기동 뒤 decoder를 비운다. 버린 대안: SOCKS 지원 (transport 셋 중 하나만 된다), transport마다 자동 풀기(크기 상한이 transport마다 달라진다), 집계를 RawBsonDocument로 재기(converter를 거치지 않으면 반환 타입 mapping이 갈라진다). 위험·한계: MongoTenantClientRegistry.acquire 반환형과 ProxyType이 바뀐다. S3에서 두 writer가 같은 버전에서 같은 record를 쓰면 둘 다 성공으로 본다 (Javadoc과 안내 3절). 집계 결과는 converter로만 바뀌어 entity callback이 돌지 않는다. evictionCloseFailures 지표는 지표 묶음에 넘긴다. 검증: desktop RED(pr2-red2). GREEN은 PR 본문의 verify.sh pr 결과. Refs: RR9-01-08, RR9-01-11, RR9-01-19, RR9-01-20, RR9-01-27, RR9-01-28, RR9-01-30, RR9-01-31 BREAKING CHANGE: MongoTenantClientRegistry.acquire가 Lease를 돌려주고, migration lease를 잃은 refresh가 MongoOperationRejectedException을 던진다. HTTP client ProxyType.SOCKS가 없어져 그 설정은 바인딩이 실패한다. request.compression이 모든 transport에서 같은 coding만 받는다. docs/upgrade/2026-11-round9-outbound.md
This commit is contained in:
@@ -167,8 +167,8 @@ profile에는 `connect`와 `response-header`를 가장 짧은 호출자 예산
|
||||
| `redirect.max-hops` | `0` | redirect를 켜고 hop을 0으로 두면 설정 오류다 |
|
||||
| `redirect.allow-cross-origin` | `false` | 켜면 hop에서 credential을 지운다 |
|
||||
| `request.max-body-bytes` | `0` | 운영에서 필수 |
|
||||
| `request.compression` | `false` | |
|
||||
| `response.max-wire-bytes` | `5242880` | wire 위의 바이트 |
|
||||
| `request.compression` | `false` | 켜면 transport와 상관없이 `Accept-Encoding: gzip, deflate`를 보낸다. 받은 본문은 엔진이 아니라 이 플랫폼이 wire 바이트를 센 뒤 푼다(버퍼 본문과 스트리밍 다운로드 모두) |
|
||||
| `response.max-wire-bytes` | `5242880` | wire 위의 바이트. 압축을 켜도 압축된 크기로 센다 |
|
||||
| `response.max-decoded-bytes` | `10485760` | 디코딩한 뒤의 바이트. 넘을 수 없는 최대는 64 MiB |
|
||||
| `response.allowed-content-types` | JSON + problem+json | 비어 있으면 「아무것이나」 |
|
||||
|
||||
@@ -212,7 +212,7 @@ profile에는 `connect`와 `response-header`를 가장 짧은 호출자 예산
|
||||
| 속성 | 기본값 | 뜻 |
|
||||
|---|---|---|
|
||||
| `proxy.enabled` | `false` | |
|
||||
| `proxy.host` / `proxy.port` / `proxy.type` | — / `0` / `HTTP` | |
|
||||
| `proxy.host` / `proxy.port` / `proxy.type` | — / `0` / `HTTP` | 지금 transport는 모두 HTTP forward proxy만 한다. `type`의 값은 `HTTP` 하나이고, `SOCKS`는 바인딩이 거부한다 |
|
||||
| `proxy.credential-provider` | — | proxy 인증은 대상 인증과 따로다 |
|
||||
| `proxy.import-ambient-no-proxy` | `false` | 환경의 `NO_PROXY`는 검증한 profile을 넓히지 않는다 |
|
||||
| `observability.full-url-recording` | `false` | |
|
||||
|
||||
@@ -50,9 +50,9 @@
|
||||
|
||||
| 상태 | 판정 |
|
||||
|---|---|
|
||||
| 408 | 기한과 예산 안에서 재시도 |
|
||||
| 425 | 첫 시도에서만, 많아야 한 번 재시도 |
|
||||
| 429 | `Retry-After`, 기한, 예산 안에서 재시도 |
|
||||
| 408 | 안전하게 멱등인 연산만 기한과 예산 안에서 재시도. 아니면 결과 모름(upstream이 받은 뒤 답한 것일 수 있다) |
|
||||
| 425 | 안전하게 멱등인 연산만, 첫 시도에서만, 많아야 한 번 재시도. 아니면 결과 모름 |
|
||||
| 429 | 안전하게 멱등인 연산만 `Retry-After`, 기한, 예산 안에서 재시도. 아니면 거부(`notRetryableStatus`). 429는 일정 신호이지 "아무 일도 없었다"는 말이 아니다 |
|
||||
| 401 | 안전하고 재전송 가능한 연산에서만 갱신 후 재전송 한 번. credential은 재시도를 허락받은 뒤에만 갱신한다. 재전송하지 않을 401(시도 한 번, 예산 없음, 기한 없음)은 token cache를 건드리지 않는다 |
|
||||
| 500 | upstream이 일시적이라고 등록했고 **그리고** 연산이 안전하게 멱등일 때만 허락. 아니면 거부 |
|
||||
| 502, 503, 504 | 안전하게 멱등인 연산은 재시도. 아니면 결과 모름 |
|
||||
|
||||
@@ -7,7 +7,13 @@
|
||||
검증한다. 그래서 실패한 다운로드가 호출자가 따져야 하는 반쯤 읽힌 stream이 되지 않는다.
|
||||
|
||||
wire 바이트와 디코딩한 바이트는 따로 제한한다. 압축한 payload는 wire 검사를 통과한 뒤 늘어나기 때문이다. 한도는
|
||||
버퍼링한 뒤가 아니라 읽는 동안 강제한다.
|
||||
버퍼링한 뒤가 아니라 읽는 동안 강제한다. 압축(`Content-Encoding: gzip`·`deflate`)은 엔진이 아니라 응답 reader가
|
||||
푼다. 도착한 바이트를 wire 한도로 세고, 푼 바이트를 조각마다 디코딩 한도로 센다. 엔진이 먼저 풀면 wire 계수가 푼
|
||||
크기를 세게 되어, 잘 압축된 응답이 wire 위에는 없던 크기로 거부됐다(round9 리뷰 2026-11-12, RR9-01-30). 부탁하지 않은
|
||||
압축(`br` 등)과 겹친 coding은 거부한다. 스트리밍 다운로드도 같은 곳(`ContentDecoding`)을 거친다. 호출자는 푼 바이트를
|
||||
읽고, 응답 헤더에는 `Content-Encoding`·`Content-Length`가 없다. 풀기는 호출자가 읽을 때 시작하므로, 본문이 빈 gzip
|
||||
응답(HEAD·204·304)은 빈 본문이다. 어느 transport도 스스로 풀지 않는다. Apache는 압축 처리를 끄고, 실험 transport인
|
||||
Jetty HTTP/3은 시작할 때 classpath에서 등록한 decoder를 지운다.
|
||||
|
||||
## 첫 바이트 경계
|
||||
|
||||
|
||||
@@ -57,7 +57,10 @@ executor 조립과 실제 PostgreSQL 저장소 왕복은 `PostgreSqlIdempotencyP
|
||||
|
||||
같은 키 재전송이 무엇을 막는지는 저장소마다 다르다(정본
|
||||
`docs/web/http-contract.md` 2절). Redis 저장소에서 같은 키 재전송은 효과가 있었을 수 있는 작업을 다시 돌리지
|
||||
않는다. PostgreSQL 저장소를 템플릿처럼 쓰기 트랜잭션 하나로 쓰면 실패한 요청의 기록은 롤백되어 다음 같은 키
|
||||
않는다. 단, 그 기록이 Redis에 남아 있는 동안만이다. Sentinel·Cluster failover는 비동기 복제라 primary가 받고
|
||||
replica에 아직 넘기지 못한 기록을 잃을 수 있고, 그러면 같은 키 요청은 처음 보는 키로 다시 실행된다
|
||||
(`docs/runbooks/redis-capability-incident.md` 「Topology and persistence」). 이 창이 업무상 받아들일 수 없는
|
||||
작업은 PostgreSQL 저장소를 쓰거나 업무 데이터의 유일 제약으로 따로 막는다. PostgreSQL 저장소를 템플릿처럼 쓰기 트랜잭션 하나로 쓰면 실패한 요청의 기록은 롤백되어 다음 같은 키
|
||||
요청이 다시 실행한다. 그래서 트랜잭션 밖 효과는 따로 멱등하게 만든다. 첫 시도가 processing lease(기본 30s)보다
|
||||
오래 걸릴 때 보이는 모습도 저장소마다 다르다.
|
||||
|
||||
|
||||
@@ -41,7 +41,10 @@ email을 모든 tenant에 걸쳐 유일하게 만든다. 그러면 tenant A가
|
||||
## 2. tenant마다 database (Experimental)
|
||||
|
||||
`MongoTenantClientRegistry`는 tenant마다 client를 하나 둔다. 개수 상한이 있고 쓰지 않는 client는
|
||||
내보낸다. tenant를 어느 database에 대응시킬지는 도입하는 쪽의 코드가 정한다.
|
||||
내보낸다. tenant를 어느 database에 대응시킬지는 도입하는 쪽의 코드가 정한다. `acquire`는 `Lease`를 돌려주고,
|
||||
그 client는 lease를 모두 닫을 때까지 쓰는 중이다. 쓰는 중인 client는 얼마나 오래전에 받았든 내보내지 않는다.
|
||||
내보낸 client를 닫다가 실패하면 그 tenant 대신 `evictionCloseFailures()`가 센다. 닫기를 부른 다른 tenant의
|
||||
`acquire`는 실패하지 않는다(round9 리뷰 2026-11-12, RR9-01-27).
|
||||
|
||||
Experimental인 이유가 분명하다. 작을 때는 맞고, 클 때는 한계가 없다. tenant database마다 connection,
|
||||
file handle, 모니터링 가짓수가 든다. tenant 20개에서는 잘 돌고 2,000개에서는 쓰러진다. 기능 테스트로는
|
||||
|
||||
@@ -39,9 +39,15 @@ regex는 index를 쓰는 척하는 collection 전체 scan이고, 사용자가
|
||||
| 상한 | 이유 |
|
||||
|---|---|
|
||||
| `maxTimeMS` | 아무도 기다리지 않는 query를 서버가 멈춘다. |
|
||||
| 결과 개수 상한 | 상한 없는 결과는 단계가 몇 개 더 붙은 OOM이다. |
|
||||
| batch 크기 | 왕복 한 번의 메모리를 제한한다. |
|
||||
| 검사한 문서 수 상한 | 빠른 날에는 시간 상한만으로 숨겨지는 index 회귀를 잡는다. |
|
||||
| 결과 개수 상한(`maxResults`) | 상한 없는 결과는 단계가 몇 개 더 붙은 OOM이다. |
|
||||
| 결과 바이트 상한(`maxResultBytes`) | 문서 하나하나가 크면 개수 상한 안에서도 heap이 찬다. |
|
||||
| batch 크기(`cursorBatchSize`) | 왕복 한 번의 메모리를 제한한다. |
|
||||
|
||||
개수·시간·batch는 서버에 보내는 요청에 실린다(`PolicyAwareMongoQueryBuilder.build`가 만든 `Query`, aggregation의
|
||||
`$limit`과 옵션). 바이트는 결과를 읽는 쪽만 셀 수 있다. aggregation은 `PolicyAwareMongoAggregationExecutor`가
|
||||
cursor에서 문서를 하나씩 읽으며 encode한 크기를 더하고 넘으면 거부한다(5절). query builder는 `Query`만 돌려주므로
|
||||
query의 바이트 상한을 집행하는 템플릿 코드는 없다. 그 `Query`로 결과를 읽는 서비스의 adapter가 `maxResultBytes`를
|
||||
직접 센다. 세지 않으면 그 상한은 선언일 뿐이다.
|
||||
|
||||
`MongoBudgetPolicyRegistry`가 예산을 연산 이름에 묶는다. `MongoBudgetEnforcer`가 그것을 적용한다. 요청이
|
||||
예산을 넘으면 실행 전에 `MongoOperationRejectedException`을 던지고, 서버가 상한을 집행하면
|
||||
@@ -86,6 +92,11 @@ regex는 index를 쓰는 척하는 collection 전체 scan이고, 사용자가
|
||||
| high | `$lookup`, `$graphLookup`, `$facet`, index 없는 `$sort` | profile의 명시적 승인이 필요하다. |
|
||||
| forbidden | admin plane 밖의 `$out`·`$merge`, `$function`, `$accumulator` | 거부한다. |
|
||||
|
||||
실행은 예산 둘을 결과가 오는 동안 본다. pipeline 끝에 `$limit(maxResults + 1)`을 붙여 서버가 예산보다 한 개 넘게는
|
||||
만들지 않게 하고, 결과를 cursor에서 한 문서씩 읽으며 개수와 encode한 바이트를 더한다. 둘 중 하나라도 넘으면
|
||||
`MongoOperationRejectedException`(`aggregation.result`)이다. 예전에는 결과를 전부 메모리에 올린 뒤 개수만 셌고 바이트
|
||||
상한은 선언만 있었다(round9 리뷰 2026-11-12, RR9-01-20).
|
||||
|
||||
`allowDiskUse`는 실행 중 flag가 아니라 plan에 선언하는 속성이다. 디스크가 필요한 pipeline은 모양을 검토해야
|
||||
하는 pipeline이다.
|
||||
|
||||
|
||||
@@ -1,13 +1,15 @@
|
||||
# 업그레이드 안내 — 2026-11 outbound 발행·발송·캐시·S3·MongoDB·HTTP client
|
||||
|
||||
- 범위: outbox 발행 대기, SMTP 발송 pool과 시간 상한, Redis 캐시 시간·크기 상한을 고친 변경. 기동 조건이 바뀐 곳이
|
||||
있다. 아래 표가 그 목록이다.
|
||||
- 근거: 2026-11-12 전체 리뷰(`docs/reviews/2026-11-12-round9-review/`)의 발행·발송·캐시(B10a) 발견
|
||||
- 범위: outbox 발행 대기, SMTP 발송 pool, Redis 캐시 시간·크기 상한, S3 조건부 쓰기, MongoDB 집계 예산·tenant
|
||||
client·migration lease, HTTP client 압축·proxy를 고친 변경. 기동 조건과 공개 타입이 바뀐 곳이 있다. 아래 표가 그
|
||||
목록이다.
|
||||
- 근거: 2026-11-12 전체 리뷰(`docs/reviews/2026-11-12-round9-review/`)의 발행·발송·캐시(B10a)와 S3·Mongo·httpclient
|
||||
(B10b) 발견
|
||||
|
||||
## 1. 누가 읽나
|
||||
|
||||
이 템플릿을 `upstream` remote로 두고 merge해 받는 파생 서비스가 읽는다. 특히 SMTP 알림, Redis 캐시, outbox
|
||||
publisher를 직접 구현한 서비스다.
|
||||
이 템플릿을 `upstream` remote로 두고 merge해 받는 파생 서비스가 읽는다. 특히 SMTP 알림, Redis 캐시, MongoDB
|
||||
집계·tenant별 database, HTTP client의 압축이나 proxy를 쓰는 서비스다.
|
||||
|
||||
## 2. 할 일
|
||||
|
||||
@@ -16,7 +18,12 @@ publisher를 직접 구현한 서비스다.
|
||||
| 1 | SMTP provider profile들의 `max-concurrency` 합이 발송 pool이 담는 수(`app.notification.platform.smtp.dispatch-concurrency`개 thread와 그 네 배의 대기열) 이하인지 확인한다. 모든 SMTP profile이 pool 하나를 같이 쓴다. 기본값(profile 하나, 4와 4)은 맞는다 | 넘으면 넘게 만든 profile의 조립이 기동 때 실패한다. 전에는 넘친 발송이 대기열에서 거절되어 결과 불명으로 남았다 | SMTP 알림을 켠 서비스 |
|
||||
| 2 | Redis 캐시의 `command-timeout`이 이제 한 캐시 연산 전체(연결 대기와 응답 전부)의 상한이다. 연결 대기가 길던 배포는 값을 다시 본다 | 상한 안에 못 끝난 연산이 캐시 없음(`DEGRADED_UNAVAILABLE`·`Unavailable`)으로 끝나는 일이 늘 수 있다. 전에는 lane 대기 2초가 더해졌다 | Redis 캐시를 켠 서비스 |
|
||||
| 3 | encode한 값이 1 MiB(`RedisCacheRegionAdapter.DEFAULT_MAXIMUM_VALUE_BYTES`)를 넘는 캐시는 그 값을 기록하지 않는다(`NOT_RECORDED_PROVIDER_POLICY`). 더 큰 값이 필요하면 `app.capabilities.cache.maximum-value-bytes`(`APP_CACHE_REDIS_MAXIMUM_VALUE_BYTES`, 1..16777216)로 상한을 올린다. 2026-08-10에 폐기했던 그 env 키가 다시 묶였다 | 그 값은 매번 source에서 읽힌다 | 큰 값을 캐시하던 서비스 |
|
||||
| 4 | SMTP provider profile의 `timeout`(기본이 `10s`에서 `30s`로 바뀌었다)이 발송 한 번을 두 번 묶는다. 발송이 send pool에서 그 시간이 지나도록 시작하지 못하면 relay에 보내지 않고 시작 전 실패(`PROVIDER_UNAVAILABLE`, 재시도)로 기록한다. 시작한 뒤 그 시간이 지나면 결과 불명(`PROVIDER_RESPONSE_LOST`)으로 기록하고 자동으로 다시 보내지 않는다. 값은 `mail.smtp.connectiontimeout`에 `mail.smtp.timeout`·`mail.smtp.writetimeout` 중 긴 것을 더한 값보다 길어야 하고, `dispatch.lease-duration`은 그 두 배보다 길어야 한다. `dispatch.shutdown-grace`가 그 두 배보다 짧으면 종료 중에 끝나지 않은 발송이 결과 불명으로 남는다(`docs/notification/configuration-reference.md` 「SMTP relay」). `SmtpNotificationProviderAdapter`를 직접 만들던 코드는 마지막 인자로 이 timeout을 넘긴다. 직접 만든 `ProviderRuntimeAssembler`는 새 default 메서드 `requireTogether(profiles, deliveryLease)`로 한 family의 켜진 profile 전체에 걸린 검사를 할 수 있다(`NotificationProviderAssembly`가 SERVING일 때 family마다 한 번 부른다). `assemble`을 직접 부르는 코드는 그 전에 `requireTogether`를 부른다 | timeout이 socket timeout 합 이하이거나 lease가 timeout 두 배 이하면 SMTP 조립이 기동 때 실패한다. 상한 안에 끝나지 않는 발송이 결과 불명으로 남는다. 전에는 socket timeout이 명령마다만 묶어, 느린 relay에서 발송이 lease를 넘으면 lease 복구가 아직 도는 발송을 운영자 확인으로 넘겼다. | SMTP 알림을 켠 서비스 |
|
||||
| 4 | MongoDB `PolicyAwareMongoAggregationExecutor`는 등록 예산의 `maxResultBytes`를 집행하고, pipeline 끝에 `$limit(maxResults + 1)`을 붙인다. 결과가 큰 집계의 예산을 다시 본다. 집계 결과는 template의 converter로만 바뀌어 `AfterLoadEvent`·`AfterConvertEvent`와 entity callback이 돌지 않는다. 그것에 기대던 listener는 돌려받은 타입에서 직접 한다 | 바이트 예산을 넘는 집계가 `MongoOperationRejectedException`(`aggregation.result`)으로 실패한다. 전에는 결과를 전부 메모리에 올렸다 | MongoDB 집계를 쓰는 서비스 |
|
||||
| 5 | experimental `MongoTenantClientRegistry.acquire`가 `Lease`(interface)를 돌려준다. client를 다 쓴 뒤 `lease.close()`(또는 try-with-resources)를 부른다. 쓰는 중인 tenant의 `release`는 거부된다. idle 시간은 마지막 lease를 닫은 때부터 잰다. 열린 lease 수는 `openLeases()`다. 시각을 인자로 받던 `acquire(tenant, Instant)`는 없어졌다. 시계는 생성자의 `Clock` 하나다 | 반환값을 버리는 호출은 컴파일되지만, 닫지 않은 lease의 client는 쓰는 중으로 남아 내보내지지 않고 cap 한 칸을 계속 차지한다. `acquire(tenant, Instant)`를 부른 코드는 컴파일이 실패한다 | tenant별 database를 쓰는 서비스 |
|
||||
| 6 | MongoDB migration lease를 잃은 뒤의 `refresh`는 `MongoOperationRejectedException`(`migration.lock`)을 던진다. `IllegalStateException`을 잡던 코드를 바꾼다. collection을 직접 넘겨 `MongoCollectionMigrationLedger`를 만들던 코드는 셋째 인자로 lock collection(`MongoCollectionMigrationLock.DEFAULT_COLLECTION`)을 넘긴다. 옛 runner의 `recordApplied`는 새 runner가 lease를 잡은 뒤면(checkpoint를 쓰기 전이어도) 거부된다 | 그 예외를 잡지 못한다. 두 인자 생성자는 컴파일이 실패한다 | MongoDB migration runner를 직접 쓰는 서비스 |
|
||||
| 7 | HTTP client profile의 `proxy.type: SOCKS`를 지우거나 HTTP proxy로 바꾼다. `ProxyType`에는 이제 `HTTP`만 있다 | 설정 바인딩이 실패해 기동하지 않는다. 전에는 SOCKS port에 HTTP proxy로 붙어 첫 요청에서 실패했다 | SOCKS proxy를 설정한 서비스 |
|
||||
| 8 | `request.compression: true`인 HTTP client는 transport와 상관없이 `Accept-Encoding: gzip, deflate`를 보내고, 버퍼 본문과 스트리밍 다운로드 모두 이 플랫폼이 푼다(JDK transport는 전에 이 설정을 읽지 않았다). 압축을 켰든 껐든, 그 밖의 `Content-Encoding`(`br` 등)이나 겹친 coding으로 온 응답은 `HttpTargetRejectedException`(`TARGET_REJECTED`, 증거 `RESPONSE_RECEIVED`)으로 거부된다. 본문이 빈 gzip 응답(HEAD·204·304)은 빈 본문이다 | — 동작 변경. `response.max-wire-bytes`가 이제 압축된 크기로 센다. 스트리밍 다운로드도 identity 본문까지 `max-decoded-bytes`로 센다. 실험 transport인 Jetty HTTP/3도 이제 스스로 풀지 않는다 | HTTP client를 쓰는 서비스 |
|
||||
| 9 | SMTP provider profile의 `timeout`(기본이 `10s`에서 `30s`로 바뀌었다)이 발송 한 번을 두 번 묶는다. 발송이 send pool에서 그 시간이 지나도록 시작하지 못하면 relay에 보내지 않고 시작 전 실패(`PROVIDER_UNAVAILABLE`, 재시도)로 기록한다. 시작한 뒤 그 시간이 지나면 결과 불명(`PROVIDER_RESPONSE_LOST`)으로 기록하고 자동으로 다시 보내지 않는다. 값은 `mail.smtp.connectiontimeout`에 `mail.smtp.timeout`·`mail.smtp.writetimeout` 중 긴 것을 더한 값보다 길어야 하고, `dispatch.lease-duration`은 그 두 배보다 길어야 한다. `dispatch.shutdown-grace`가 그 두 배보다 짧으면 종료 중에 끝나지 않은 발송이 결과 불명으로 남는다(`docs/notification/configuration-reference.md` 「SMTP relay」). `SmtpNotificationProviderAdapter`를 직접 만들던 코드는 마지막 인자로 이 timeout을 넘긴다. 직접 만든 `ProviderRuntimeAssembler`는 새 default 메서드 `requireTogether(profiles, deliveryLease)`로 한 family의 켜진 profile 전체에 걸린 검사를 할 수 있다(`NotificationProviderAssembly`가 SERVING일 때 family마다 한 번 부른다). `assemble`을 직접 부르는 코드는 그 전에 `requireTogether`를 부른다 | timeout이 socket timeout 합 이하이거나 lease가 timeout 두 배 이하면 SMTP 조립이 기동 때 실패한다. 상한 안에 끝나지 않는 발송이 결과 불명으로 남는다. 전에는 socket timeout이 명령마다만 묶어, 느린 relay에서 발송이 lease를 넘으면 lease 복구가 아직 도는 발송을 운영자 확인으로 넘겼다. | SMTP 알림을 켠 서비스 |
|
||||
|
||||
## 3. 바뀐 동작 (따라 할 일 없음)
|
||||
|
||||
@@ -27,3 +34,8 @@ publisher를 직접 구현한 서비스다.
|
||||
relay가 그 행을 backoff 뒤로 미뤘다가 발행한다. 기다리는 중의 interrupt는 전처럼 결과 불명(`PUBLISH_WAIT_INTERRUPTED`)이다.
|
||||
- SMTP 발송 pool은 첫 발송부터 `dispatch-concurrency`개 thread로 돈다(전에는 대기열이 찰 때까지 thread 하나였다).
|
||||
pool이 거절한 발송은 relay에 닿지 않은 것(`PROVIDER_UNAVAILABLE`, `CAPACITY_REJECTED`)으로 기록되어 재시도된다.
|
||||
- S3 control 저장소의 조건부 `PUT`이 412를 받으면 key를 다시 읽는다. 이 호출이 쓴 record 그대로면 성공이다(SDK가 응답을
|
||||
잃은 첫 쓰기를 다시 보낸 경우). 다른 내용이나 없음이면 전처럼 충돌이다. 「그대로」는 같은 값이라는 뜻이라, 두 writer가
|
||||
같은 버전에서 같은 record를 쓰면 둘 다 성공으로 본다. 그럴 수 있는 호출자는 record에 자기 토큰을 싣는다.
|
||||
- MongoDB tenant client를 내보내다 닫기에 실패하면 그 tenant 대신 `evictionCloseFailures()`가 센다. 닫기를 부른 다른
|
||||
tenant의 `acquire`는 실패하지 않는다.
|
||||
|
||||
@@ -125,7 +125,9 @@ Envelope가 아닌 응답은 두 곳뿐이다. fileserver의 프로토콜 예외
|
||||
`IDEMPOTENT_IN_FLIGHT`, lease가 지났으면 500 `IDEMPOTENCY_RECOVERY_REQUIRED`다. 이것은 Redis 저장소의
|
||||
판정이다. PostgreSQL 저장소는 아래처럼 첫 요청이 끝날 때까지 기다린 뒤 판정한다. 새 키로 보내면 서버는
|
||||
다른 요청으로 보고 막지 못한다(재리뷰 2026-11-03 README 4절 결정 RR-02-03, 2026-11-04 9.1절 결정
|
||||
RR-01-11).
|
||||
RR-01-11). Redis 저장소의 판정은 그 기록이 Redis에 남아 있는 동안만이다. Sentinel·Cluster failover는 비동기
|
||||
복제라 replica에 아직 넘기지 못한 기록을 잃을 수 있고, 그러면 같은 키 요청은 처음 보는 키로 다시 실행된다
|
||||
(`docs/manual/05-idempotency.md` 「클라이언트에게 무엇을 약속하나」).
|
||||
- **첫 시도가 lease보다 오래 걸릴 때 — 저장소마다 다르다.** executor는 lease를 늘리지 않으므로 30s를 넘는
|
||||
정상 요청도 lease가 지난다(재리뷰 2026-11-04 RR5-01-05).
|
||||
- Redis: lease가 지난 실행 기록을 다음 요청이 만나면 500으로 답하고 기록은 건드리지 않는다. 첫 시도가
|
||||
|
||||
+12
-9
@@ -111,15 +111,17 @@ public final class ApacheClientFactory {
|
||||
.evictIdleConnections(TimeValue.ofMilliseconds(profile.pool().maxIdleTime().toMillis()))
|
||||
.setDefaultRequestConfig(requestConfig(profile));
|
||||
|
||||
if (!profile.request().compression()) {
|
||||
clientBuilder.disableContentCompression();
|
||||
}
|
||||
// Apache never decodes the body. With its content compression on, the stream the response
|
||||
// reader counted as "wire" bytes was already decompressed, so a profile's wire limit was a
|
||||
// second decoded limit and a well-compressed response under it was refused for its decoded size
|
||||
// (round9 review 2026-11-12, RR9-01-30). The reader counts the bytes as they arrive and decodes
|
||||
// them itself. Whether to ask for compression is the profile's, sent by the RestClient for
|
||||
// every transport.
|
||||
clientBuilder.disableContentCompression();
|
||||
if (profile.proxy().enabled()) {
|
||||
clientBuilder.setProxy(
|
||||
new HttpHost(
|
||||
profile.proxy().type().name().toLowerCase(java.util.Locale.ROOT),
|
||||
profile.proxy().host(),
|
||||
profile.proxy().port()));
|
||||
// An HTTP forward proxy, the only kind ProxyType has: SOCKS used to bind and then become an
|
||||
// HTTP proxy at the SOCKS port.
|
||||
clientBuilder.setProxy(new HttpHost("http", profile.proxy().host(), profile.proxy().port()));
|
||||
}
|
||||
return new ApacheRuntime(clientBuilder.build(), pool);
|
||||
}
|
||||
@@ -131,7 +133,8 @@ public final class ApacheClientFactory {
|
||||
.setRedirectsEnabled(false)
|
||||
.setMaxRedirects(0)
|
||||
.setCircularRedirectsAllowed(false)
|
||||
.setContentCompressionEnabled(profile.request().compression())
|
||||
// Decoding is the response reader's, after it has counted the wire bytes; see create().
|
||||
.setContentCompressionEnabled(false)
|
||||
// Hard cancellation lets a deadline abort actually abort the socket instead of leaving a
|
||||
// request in flight after the caller has given up.
|
||||
.setHardCancellationEnabled(true)
|
||||
|
||||
+13
@@ -22,6 +22,7 @@ import org.eclipse.jetty.http3.client.HTTP3Client;
|
||||
import org.eclipse.jetty.http3.client.transport.HttpClientTransportOverHTTP3;
|
||||
import org.eclipse.jetty.io.Transport;
|
||||
import org.eclipse.jetty.quic.client.ClientQuicConfiguration;
|
||||
import org.eclipse.jetty.util.component.LifeCycle;
|
||||
import org.springframework.http.client.ClientHttpRequestFactory;
|
||||
import org.springframework.http.client.JettyClientHttpRequestFactory;
|
||||
|
||||
@@ -132,6 +133,18 @@ public final class JettyHttp3TransportProvider implements BlockingTransportProvi
|
||||
.getRequestListeners()
|
||||
.addQueuedListener(
|
||||
request -> AttemptDeadline.abortOnExpiry(() -> abortAtDeadline(request)));
|
||||
// Jetty registers its gzip decoder from the classpath when it starts, then asks for gzip on
|
||||
// every request, decodes the response itself and drops Content-Encoding — so the reader's wire
|
||||
// count saw the decoded size, whatever the profile said (round9 review 2026-11-12,
|
||||
// RR9-01-30). Cleared once started, because an empty list before start is what tells Jetty to
|
||||
// load them; the coded bytes then reach the reader, which counts and decodes them.
|
||||
client.addEventListener(
|
||||
new LifeCycle.Listener() {
|
||||
@Override
|
||||
public void lifeCycleStarted(LifeCycle started) {
|
||||
client.getContentDecoderFactories().clear();
|
||||
}
|
||||
});
|
||||
return client;
|
||||
}
|
||||
|
||||
|
||||
+9
-3
@@ -1,7 +1,13 @@
|
||||
package dev.caskeleton.adapter.outbound.httpclient.profile;
|
||||
|
||||
/** Forward proxy kind (design §24.3). */
|
||||
/**
|
||||
* Forward proxy kind (design §24.3).
|
||||
*
|
||||
* <p>HTTP only. Every shipped transport speaks an HTTP forward proxy and none speaks SOCKS; a
|
||||
* {@code SOCKS} value used to bind and then be wired as an HTTP proxy at the SOCKS port, and a
|
||||
* value whose only possible effect is a failure is not kept as an option ({@code
|
||||
* docs/manual/10-capabilities.md} standard 3). {@code proxy.type: SOCKS} now fails to bind.
|
||||
*/
|
||||
public enum ProxyType {
|
||||
HTTP,
|
||||
SOCKS
|
||||
HTTP
|
||||
}
|
||||
|
||||
+55
-7
@@ -5,6 +5,7 @@ import dev.caskeleton.adapter.outbound.httpclient.api.HttpStatus;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.error.HttpDeadlineExceededException;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.error.HttpFailureMetadata;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.error.HttpRemoteErrorException;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.error.HttpTargetRejectedException;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.operation.AttemptStage;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.operation.ExecutionEvidence;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.operation.HttpOperation;
|
||||
@@ -109,10 +110,38 @@ public final class BlockingStreamingGateway {
|
||||
// exceptions a buffered call throws, classified by the transport's own
|
||||
// classifier at the body stage — not as the engine's bare IOException, which
|
||||
// carried no category, no evidence and nothing to log (RR-04-03).
|
||||
// Decoded here as the buffered reader decodes: the transport hands over the coded
|
||||
// bytes, counted on the wire as they arrive, and what they decode to is counted
|
||||
// against the decoded budget as the caller reads it. A streamed download used to
|
||||
// reach the caller still compressed once the transport stopped decoding.
|
||||
String coding;
|
||||
try {
|
||||
coding = ContentDecoding.codingOf(rawResponse.getHeaders());
|
||||
} catch (ContentDecoding.RefusedCodingException refused) {
|
||||
// A coding this client did not ask for: nothing is handed over, and the
|
||||
// connection is not left open behind the refusal.
|
||||
rawResponse.close();
|
||||
throw new HttpTargetRejectedException(
|
||||
"response content coding is not one this client asked for",
|
||||
metadata
|
||||
.withStatus(new HttpStatus(status))
|
||||
.withEvidence(ExecutionEvidence.RESPONSE_RECEIVED),
|
||||
refused);
|
||||
}
|
||||
InputStream wire =
|
||||
new CountingBoundedInputStream(
|
||||
rawResponse.getBody(), limiter::recordWireBytes);
|
||||
// Counted against the decoded budget whatever the coding, identity included, as
|
||||
// the buffered reader counts it: an identity body is its own decoding.
|
||||
InputStream body =
|
||||
new CountingBoundedInputStream(
|
||||
ContentDecoding.decoding(coding, wire), limiter::recordDecodedBytes);
|
||||
if (!coding.isEmpty()) {
|
||||
ContentDecoding.dropCodingHeaders(responseHeaders);
|
||||
}
|
||||
InputStream bounded =
|
||||
new ClassifiedBodyInputStream(
|
||||
new CountingBoundedInputStream(
|
||||
rawResponse.getBody(), limiter::recordWireBytes),
|
||||
body,
|
||||
bodyFailure ->
|
||||
runtime
|
||||
.support()
|
||||
@@ -129,11 +158,7 @@ public final class BlockingStreamingGateway {
|
||||
false);
|
||||
} catch (RuntimeException failure) {
|
||||
closeQuietly(admission);
|
||||
if (setupDeadline.expired() && !(failure instanceof HttpRemoteErrorException)) {
|
||||
throw new HttpDeadlineExceededException(
|
||||
"effective deadline reached before the stream was set up", metadata, failure);
|
||||
}
|
||||
throw failure;
|
||||
throw setupFailure(failure, setupDeadline.expired(), metadata);
|
||||
}
|
||||
|
||||
BlockingStreamingResponse wrapped =
|
||||
@@ -155,6 +180,29 @@ public final class BlockingStreamingGateway {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* What a failure while setting up the stream reaches the caller as.
|
||||
*
|
||||
* <p>Once the setup budget is spent the failure is the deadline, whatever the transport said. A
|
||||
* response that arrived and was refused, or a remote error, stays what it is: its evidence is
|
||||
* that the server answered.
|
||||
*
|
||||
* @param failure what setting up the stream threw
|
||||
* @param budgetSpent whether the setup deadline had expired when it did
|
||||
* @param metadata the call's failure metadata
|
||||
* @return the exception to throw
|
||||
*/
|
||||
static RuntimeException setupFailure(
|
||||
RuntimeException failure, boolean budgetSpent, HttpFailureMetadata metadata) {
|
||||
if (budgetSpent
|
||||
&& !(failure instanceof HttpRemoteErrorException)
|
||||
&& !(failure instanceof HttpTargetRejectedException)) {
|
||||
return new HttpDeadlineExceededException(
|
||||
"effective deadline reached before the stream was set up", metadata, failure);
|
||||
}
|
||||
return failure;
|
||||
}
|
||||
|
||||
/** A permit release must not mask the failure that is already propagating. */
|
||||
private static void closeQuietly(AutoCloseable closeable) {
|
||||
try {
|
||||
|
||||
+162
@@ -0,0 +1,162 @@
|
||||
package dev.caskeleton.adapter.outbound.httpclient.restclient;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.io.InputStream;
|
||||
import java.io.PushbackInputStream;
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
import java.util.Locale;
|
||||
import java.util.Map;
|
||||
import java.util.Set;
|
||||
import java.util.zip.GZIPInputStream;
|
||||
import java.util.zip.InflaterInputStream;
|
||||
import org.springframework.http.HttpHeaders;
|
||||
|
||||
/**
|
||||
* Undoes a response's content coding, for the buffered reader and the streaming download alike.
|
||||
*
|
||||
* <p>The transport does not decode (round9 review 2026-11-12, RR9-01-30): the bytes that arrive are
|
||||
* counted against the wire budget, and what they decode to against the decoded budget. Both read
|
||||
* paths come through here, so a streamed download is decoded the way a buffered body is rather than
|
||||
* handed over compressed.
|
||||
*/
|
||||
final class ContentDecoding {
|
||||
|
||||
/**
|
||||
* The {@code Accept-Encoding} value a profile with compression on sends: the codings {@link
|
||||
* #decoding} undoes. Kept beside the decoder so the request never asks for a coding the reader
|
||||
* would hand over still compressed.
|
||||
*/
|
||||
static final String ACCEPTED = "gzip, deflate";
|
||||
|
||||
/** How much a decoder reads from the wire at a time. */
|
||||
static final int CHUNK = 8 * 1024;
|
||||
|
||||
private ContentDecoding() {}
|
||||
|
||||
/** The codings {@link #decoding} undoes; {@code x-gzip} is the old name of {@code gzip}. */
|
||||
private static final Set<String> DECODABLE = Set.of("gzip", "x-gzip", "deflate");
|
||||
|
||||
/**
|
||||
* A response coded in a way this client did not ask for, refused before its body is read.
|
||||
*
|
||||
* <p>Its own type so the gateways can report it as what it is — a response that arrived and was
|
||||
* refused — rather than as the bare {@link IOException} a transport classifier reads as a call
|
||||
* that got no response.
|
||||
*/
|
||||
static final class RefusedCodingException extends IOException {
|
||||
|
||||
private static final long serialVersionUID = 1L;
|
||||
|
||||
private RefusedCodingException(String message) {
|
||||
super(message);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* The response's content coding, lower case; empty for none or {@code identity}.
|
||||
*
|
||||
* @throws RefusedCodingException when the response names more than one coding — stacked codings
|
||||
* are not something this client asks for, and decoding only the first would hand the rest
|
||||
* over as the body — or a coding {@link #decoding} cannot undo
|
||||
*/
|
||||
static String codingOf(HttpHeaders headers) throws RefusedCodingException {
|
||||
List<String> codings = new ArrayList<>();
|
||||
for (String value : headers.getOrEmpty(HttpHeaders.CONTENT_ENCODING)) {
|
||||
// Limit -1: every element, an empty trailing one included, which the check below skips.
|
||||
for (String part : value.split(",", -1)) {
|
||||
String coding = part.trim().toLowerCase(Locale.ROOT);
|
||||
if (!coding.isEmpty() && !coding.equals("identity")) {
|
||||
codings.add(coding);
|
||||
}
|
||||
}
|
||||
}
|
||||
if (codings.size() > 1) {
|
||||
throw new RefusedCodingException(
|
||||
"the response stacked content codings this client did not ask for: " + codings);
|
||||
}
|
||||
String coding = codings.isEmpty() ? "" : codings.get(0);
|
||||
if (!coding.isEmpty() && !DECODABLE.contains(coding)) {
|
||||
throw new RefusedCodingException(
|
||||
"the response used a content coding this client did not ask for: " + coding);
|
||||
}
|
||||
return coding;
|
||||
}
|
||||
|
||||
/**
|
||||
* The decoded body, read lazily from {@code wire}.
|
||||
*
|
||||
* <p>Lazily, because a gzip stream reads its header when it is built: built eagerly, an empty
|
||||
* body under {@code Content-Encoding: gzip} — a {@code HEAD}, a 204 or a 304 — failed the whole
|
||||
* response, and a streamed download read from the network before it was handed over. Here nothing
|
||||
* is read until the caller reads, and a body that turns out empty decodes to nothing.
|
||||
*
|
||||
* @param coding a coding {@link #codingOf} returned
|
||||
* @param wire the counted wire stream
|
||||
*/
|
||||
static InputStream decoding(String coding, InputStream wire) {
|
||||
return coding.isEmpty() ? wire : new LazilyDecoded(coding, wire);
|
||||
}
|
||||
|
||||
/** {@code headers} without the two that describe the coded body rather than the decoded one. */
|
||||
static void dropCodingHeaders(Map<String, List<String>> headers) {
|
||||
headers
|
||||
.keySet()
|
||||
.removeIf(
|
||||
name ->
|
||||
name.equalsIgnoreCase(HttpHeaders.CONTENT_ENCODING)
|
||||
|| name.equalsIgnoreCase(HttpHeaders.CONTENT_LENGTH));
|
||||
}
|
||||
|
||||
private static final class LazilyDecoded extends InputStream {
|
||||
|
||||
private final String coding;
|
||||
private final InputStream wire;
|
||||
private InputStream decoded;
|
||||
|
||||
private LazilyDecoded(String coding, InputStream wire) {
|
||||
this.coding = coding;
|
||||
this.wire = wire;
|
||||
}
|
||||
|
||||
@Override
|
||||
public int read() throws IOException {
|
||||
return open().read();
|
||||
}
|
||||
|
||||
@Override
|
||||
public int read(byte[] buffer, int offset, int length) throws IOException {
|
||||
return open().read(buffer, offset, length);
|
||||
}
|
||||
|
||||
@Override
|
||||
public void close() throws IOException {
|
||||
try {
|
||||
if (decoded != null) {
|
||||
decoded.close();
|
||||
}
|
||||
} finally {
|
||||
wire.close();
|
||||
}
|
||||
}
|
||||
|
||||
private InputStream open() throws IOException {
|
||||
if (decoded == null) {
|
||||
PushbackInputStream peeked = new PushbackInputStream(wire, 1);
|
||||
int first = peeked.read();
|
||||
if (first == -1) {
|
||||
decoded = InputStream.nullInputStream();
|
||||
} else {
|
||||
peeked.unread(first);
|
||||
// deflate is the zlib-wrapped form RFC 9110 names; a server that sends raw deflate under
|
||||
// this name fails to decode, which is a failed response rather than a guessed one.
|
||||
decoded =
|
||||
coding.equals("deflate")
|
||||
? new InflaterInputStream(peeked)
|
||||
: new GZIPInputStream(peeked, CHUNK);
|
||||
}
|
||||
}
|
||||
return decoded;
|
||||
}
|
||||
}
|
||||
}
|
||||
+5
@@ -36,6 +36,11 @@ public final class ResponseSizeLimiter {
|
||||
this.metadata = metadata;
|
||||
}
|
||||
|
||||
/** What a failure of the response this limiter counts says about the call. */
|
||||
HttpFailureMetadata failureMetadata() {
|
||||
return metadata;
|
||||
}
|
||||
|
||||
public void recordWireBytes(long additional) {
|
||||
wireBytes += additional;
|
||||
if (wireBytes > maxWireBytes) {
|
||||
|
||||
+43
-9
@@ -1,8 +1,10 @@
|
||||
package dev.caskeleton.adapter.outbound.httpclient.restclient;
|
||||
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.HttpStatus;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.error.HttpFailureMetadata;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.error.HttpSerializationException;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.error.HttpTargetRejectedException;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.operation.ExecutionEvidence;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.result.ByteArrayResponseType;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.result.ClassResponseType;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.result.EmptyResponseType;
|
||||
@@ -10,6 +12,7 @@ import dev.caskeleton.adapter.outbound.httpclient.api.result.GenericResponseType
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.result.ResponseType;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.profile.ResponseLimits;
|
||||
import java.io.ByteArrayInputStream;
|
||||
import java.io.ByteArrayOutputStream;
|
||||
import java.io.IOException;
|
||||
import java.io.InputStream;
|
||||
import java.lang.reflect.Type;
|
||||
@@ -34,6 +37,11 @@ import org.springframework.http.converter.json.JacksonJsonHttpMessageConverter;
|
||||
* <p>The body is bounded while it is read and then decoded from memory by a Spring message
|
||||
* converter. Doing the bounding first is what makes a decompression bomb or an unexpectedly huge
|
||||
* payload a rejected request instead of an out-of-memory error.
|
||||
*
|
||||
* <p>Content coding is undone here, not by the transport: the bytes that arrive are counted against
|
||||
* the wire budget as they arrive, and the bytes they decode to are counted against the decoded
|
||||
* budget as they are produced — a chunk at a time, so a decompression bomb is refused before it is
|
||||
* held in memory (round9 review 2026-11-12, RR9-01-30).
|
||||
*/
|
||||
public final class RestClientResponseReader {
|
||||
|
||||
@@ -112,16 +120,42 @@ public final class RestClientResponseReader {
|
||||
java.util.function.LongConsumer responseBytes)
|
||||
throws IOException {
|
||||
Objects.requireNonNull(responseBytes, "response byte observer");
|
||||
String coding;
|
||||
try {
|
||||
coding = ContentDecoding.codingOf(headers);
|
||||
} catch (ContentDecoding.RefusedCodingException refused) {
|
||||
// A response arrived and was refused, which the transport classifier would otherwise read
|
||||
// from a bare IOException as a call that got no answer.
|
||||
throw new HttpTargetRejectedException(
|
||||
"response content coding is not one this client asked for",
|
||||
limiter
|
||||
.failureMetadata()
|
||||
.withStatus(new HttpStatus(status))
|
||||
.withEvidence(ExecutionEvidence.RESPONSE_RECEIVED),
|
||||
refused);
|
||||
}
|
||||
try (InputStream counted =
|
||||
new CountingBoundedInputStream(
|
||||
body,
|
||||
read -> {
|
||||
limiter.recordWireBytes(read);
|
||||
responseBytes.accept(read);
|
||||
})) {
|
||||
byte[] bytes = counted.readAllBytes();
|
||||
limiter.recordDecodedBytes(bytes.length);
|
||||
return new RawResponse(status, toMap(headers), bytes);
|
||||
new CountingBoundedInputStream(
|
||||
body,
|
||||
read -> {
|
||||
limiter.recordWireBytes(read);
|
||||
responseBytes.accept(read);
|
||||
});
|
||||
InputStream decoded = ContentDecoding.decoding(coding, counted)) {
|
||||
ByteArrayOutputStream bytes = new ByteArrayOutputStream();
|
||||
byte[] chunk = new byte[ContentDecoding.CHUNK];
|
||||
int read;
|
||||
while ((read = decoded.read(chunk)) != -1) {
|
||||
limiter.recordDecodedBytes(read);
|
||||
bytes.write(chunk, 0, read);
|
||||
}
|
||||
Map<String, List<String>> kept = toMap(headers);
|
||||
if (!coding.isEmpty()) {
|
||||
// The body handed on is the decoded one; keeping these would describe bytes it no longer
|
||||
// is.
|
||||
ContentDecoding.dropCodingHeaders(kept);
|
||||
}
|
||||
return new RawResponse(status, kept, bytes.toByteArray());
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
+10
-5
@@ -22,6 +22,7 @@ import java.util.Map;
|
||||
import java.util.Objects;
|
||||
import java.util.function.Supplier;
|
||||
import java.util.random.RandomGenerator;
|
||||
import org.springframework.http.HttpHeaders;
|
||||
import org.springframework.http.client.ClientHttpRequestFactory;
|
||||
import org.springframework.web.client.RestClient;
|
||||
|
||||
@@ -77,11 +78,15 @@ public final class RestClientRuntimeFactory implements ClientRuntimeFactory {
|
||||
|
||||
ClientHttpRequestFactory requestFactory =
|
||||
provider.create(profile, generation, lifecycleListener);
|
||||
RestClient restClient =
|
||||
RestClient.builder()
|
||||
.requestFactory(requestFactory)
|
||||
.baseUrl(profile.baseUrl().toString())
|
||||
.build();
|
||||
RestClient.Builder builder =
|
||||
RestClient.builder().requestFactory(requestFactory).baseUrl(profile.baseUrl().toString());
|
||||
if (profile.request().compression()) {
|
||||
// Asked for here, once, for every transport: none of them decodes any more, so the reader
|
||||
// counts the coded bytes on the wire and decodes them itself. The JDK transport used to read
|
||||
// the setting nowhere, and Apache and Jetty each asked on their own.
|
||||
builder.defaultHeader(HttpHeaders.ACCEPT_ENCODING, ContentDecoding.ACCEPTED);
|
||||
}
|
||||
RestClient restClient = builder.build();
|
||||
|
||||
AttemptResiliencePipeline pipeline = newResiliencePipeline(profile);
|
||||
|
||||
|
||||
+2
-1
@@ -8,7 +8,8 @@ import java.util.Set;
|
||||
* What a blocking transport can actually guarantee (design §13.3).
|
||||
*
|
||||
* <p>A capability that is weaker than the profile requires fails at startup rather than degrading
|
||||
* silently at runtime.
|
||||
* silently at runtime. {@code proxySupport} is an HTTP forward proxy, the only kind {@code
|
||||
* ProxyType} has.
|
||||
*/
|
||||
public record BlockingTransportCapabilities(
|
||||
Set<HttpProtocol> protocols,
|
||||
|
||||
+25
@@ -2,13 +2,18 @@ package dev.caskeleton.adapter.outbound.httpclient.autoconfigure;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
import static org.assertj.core.api.Assertions.assertThatThrownBy;
|
||||
import static org.assertj.core.api.Assertions.catchThrowable;
|
||||
|
||||
import dev.caskeleton.adapter.outbound.httpclient.profile.ClientMode;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.profile.HttpProtocol;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.profile.TransportType;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import org.junit.jupiter.api.DisplayName;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.springframework.boot.context.properties.bind.BindException;
|
||||
import org.springframework.boot.context.properties.bind.Binder;
|
||||
import org.springframework.boot.context.properties.source.MapConfigurationPropertySource;
|
||||
|
||||
/**
|
||||
* The aggregate rules a single client's own fields cannot express.
|
||||
@@ -96,6 +101,26 @@ class HttpClientPlatformSettingsTest {
|
||||
.hasMessageContaining("dynamic-targets");
|
||||
}
|
||||
|
||||
@Test
|
||||
void aSocksProxyTypeFailsToBind() {
|
||||
// given
|
||||
Binder binder =
|
||||
new Binder(
|
||||
new MapConfigurationPropertySource(
|
||||
Map.of("proxy.enabled", "true", "proxy.type", "SOCKS")));
|
||||
|
||||
// when
|
||||
Throwable refused =
|
||||
catchThrowable(() -> binder.bind("proxy", HttpClientPlatformSettings.Proxy.class));
|
||||
|
||||
// then
|
||||
assertThat(refused)
|
||||
.as("no shipped transport speaks SOCKS; the value used to bind and then misroute")
|
||||
.isInstanceOf(BindException.class)
|
||||
.rootCause()
|
||||
.hasMessageContaining("SOCKS");
|
||||
}
|
||||
|
||||
/**
|
||||
* A client whose only meaningful field here is its name.
|
||||
*
|
||||
|
||||
+38
@@ -0,0 +1,38 @@
|
||||
package dev.caskeleton.adapter.outbound.httpclient.http3;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
|
||||
import dev.caskeleton.adapter.outbound.httpclient.testkit.ClientProfiles;
|
||||
import org.eclipse.jetty.client.HttpClient;
|
||||
import org.junit.jupiter.api.Test;
|
||||
|
||||
/**
|
||||
* The experimental Jetty client hands the response reader the coded bytes (round9 review
|
||||
* 2026-11-12, RR9-01-30).
|
||||
*
|
||||
* <p>Jetty loads its gzip decoder from the classpath on start and then decodes every response
|
||||
* itself, which made the reader's wire count the decoded size. Over a plain TCP transport, as the
|
||||
* deadline test does: what is checked is the client's configuration, not the wire.
|
||||
*/
|
||||
class JettyContentDecodingTest {
|
||||
|
||||
@Test
|
||||
void aStartedClientDecodesNoContentCodingItself() throws Exception {
|
||||
// given
|
||||
HttpClient client =
|
||||
JettyHttp3TransportProvider.configure(
|
||||
new HttpClient(), ClientProfiles.http3Experimental("edge"));
|
||||
|
||||
// when
|
||||
client.start();
|
||||
|
||||
// then
|
||||
try {
|
||||
assertThat(client.getContentDecoderFactories().isEmpty())
|
||||
.as("a decoder left registered decodes the body before the reader can count it")
|
||||
.isTrue();
|
||||
} finally {
|
||||
client.stop();
|
||||
}
|
||||
}
|
||||
}
|
||||
+156
@@ -0,0 +1,156 @@
|
||||
package dev.caskeleton.adapter.outbound.httpclient.restclient;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
import static org.assertj.core.api.Assertions.catchThrowable;
|
||||
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.error.HttpResponseTooLargeException;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.error.HttpTargetRejectedException;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.operation.ExecutionEvidence;
|
||||
import java.io.ByteArrayInputStream;
|
||||
import java.io.ByteArrayOutputStream;
|
||||
import java.io.IOException;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.util.List;
|
||||
import java.util.zip.GZIPOutputStream;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.springframework.http.HttpHeaders;
|
||||
|
||||
/**
|
||||
* The response reader counts what arrives against the wire budget and what it decodes to against
|
||||
* the decoded budget, a chunk at a time.
|
||||
*
|
||||
* <p>The transport used to decode before the reader saw the body, so the "wire" count was the
|
||||
* decoded size and a well-compressed response was refused for a size it never had on the wire
|
||||
* (round9 review 2026-11-12, RR9-01-30).
|
||||
*/
|
||||
class RestClientResponseReaderEncodingTest {
|
||||
|
||||
private final RestClientResponseReader reader = new RestClientResponseReader();
|
||||
|
||||
@Test
|
||||
void aGzipBodyIsCountedCompressedOnTheWireAndDecodedForTheCaller() throws IOException {
|
||||
// given
|
||||
// Six thousand bytes that compress to a few dozen: inside a 1,000-byte wire budget only when
|
||||
// the wire count is the compressed size.
|
||||
byte[] decoded = "a".repeat(6_000).getBytes(StandardCharsets.US_ASCII);
|
||||
byte[] wire = gzip(decoded);
|
||||
ResponseSizeLimiter limiter = new ResponseSizeLimiter(1_000, 10_000);
|
||||
|
||||
// when
|
||||
RestClientResponseReader.RawResponse response =
|
||||
reader.readBounded(200, gzipHeaders(), new ByteArrayInputStream(wire), limiter);
|
||||
|
||||
// then
|
||||
assertThat(
|
||||
List.of(
|
||||
response.body().length,
|
||||
limiter.wireBytes(),
|
||||
limiter.decodedBytes(),
|
||||
response.headers().containsKey(HttpHeaders.CONTENT_ENCODING)))
|
||||
.containsExactly(6_000, (long) wire.length, 6_000L, false);
|
||||
}
|
||||
|
||||
@Test
|
||||
void aBodyThatDecodesPastTheBudgetIsRefusedWhileItIsDecoded() throws IOException {
|
||||
// given
|
||||
// A small wire payload that would expand to a megabyte: refused at the decoded budget, before
|
||||
// the whole expansion is held in memory.
|
||||
byte[] wire = gzip(new byte[1_048_576]);
|
||||
ResponseSizeLimiter limiter = new ResponseSizeLimiter(1_048_576, 64 * 1024);
|
||||
|
||||
// when
|
||||
Throwable refused =
|
||||
catchThrowable(
|
||||
() -> reader.readBounded(200, gzipHeaders(), new ByteArrayInputStream(wire), limiter));
|
||||
|
||||
// then
|
||||
assertThat(
|
||||
List.of(
|
||||
refused instanceof HttpResponseTooLargeException,
|
||||
limiter.decodedBytes() <= 64L * 1024 + 8 * 1024))
|
||||
.as("decoding stopped within one chunk past the budget: %s", refused)
|
||||
.containsExactly(true, true);
|
||||
}
|
||||
|
||||
@Test
|
||||
void anUnaskedForContentCodingIsRefused() {
|
||||
// given
|
||||
HttpHeaders brotli = new HttpHeaders();
|
||||
brotli.add(HttpHeaders.CONTENT_ENCODING, "br");
|
||||
ResponseSizeLimiter limiter = new ResponseSizeLimiter(1_000, 1_000);
|
||||
|
||||
// when
|
||||
Throwable refused =
|
||||
catchThrowable(
|
||||
() ->
|
||||
reader.readBounded(
|
||||
200, brotli, new ByteArrayInputStream(new byte[] {1, 2, 3}), limiter));
|
||||
|
||||
// then
|
||||
assertThat(refused)
|
||||
.as("a response arrived and was refused; it is not a call that got no answer")
|
||||
.isInstanceOfSatisfying(
|
||||
HttpTargetRejectedException.class,
|
||||
rejected ->
|
||||
assertThat(
|
||||
List.of(
|
||||
rejected.getMessage(),
|
||||
rejected.getCause().getMessage(),
|
||||
rejected.metadata().evidence()))
|
||||
.containsExactly(
|
||||
"response content coding is not one this client asked for",
|
||||
"the response used a content coding this client did not ask for: br",
|
||||
ExecutionEvidence.RESPONSE_RECEIVED));
|
||||
}
|
||||
|
||||
@Test
|
||||
void anEmptyBodyUnderGzipIsAnEmptyBody() throws IOException {
|
||||
// given
|
||||
// What a HEAD, a 204 or a 304 sends with the coding of the representation it describes.
|
||||
ResponseSizeLimiter limiter = new ResponseSizeLimiter(1_000, 1_000);
|
||||
|
||||
// when
|
||||
RestClientResponseReader.RawResponse response =
|
||||
reader.readBounded(204, gzipHeaders(), new ByteArrayInputStream(new byte[0]), limiter);
|
||||
|
||||
// then
|
||||
assertThat(response.body()).isEmpty();
|
||||
}
|
||||
|
||||
@Test
|
||||
void stackedContentCodingsAreRefused() {
|
||||
// given
|
||||
HttpHeaders stacked = new HttpHeaders();
|
||||
stacked.add(HttpHeaders.CONTENT_ENCODING, "gzip");
|
||||
stacked.add(HttpHeaders.CONTENT_ENCODING, "br");
|
||||
ResponseSizeLimiter limiter = new ResponseSizeLimiter(1_000, 1_000);
|
||||
|
||||
// when
|
||||
Throwable refused =
|
||||
catchThrowable(
|
||||
() ->
|
||||
reader.readBounded(
|
||||
200, stacked, new ByteArrayInputStream(new byte[] {1, 2, 3}), limiter));
|
||||
|
||||
// then
|
||||
assertThat(refused)
|
||||
.isInstanceOf(HttpTargetRejectedException.class)
|
||||
.hasMessage("response content coding is not one this client asked for")
|
||||
.cause()
|
||||
.hasMessageContaining("stacked content codings");
|
||||
}
|
||||
|
||||
private static HttpHeaders gzipHeaders() {
|
||||
HttpHeaders headers = new HttpHeaders();
|
||||
headers.add(HttpHeaders.CONTENT_ENCODING, "gzip");
|
||||
return headers;
|
||||
}
|
||||
|
||||
private static byte[] gzip(byte[] plain) throws IOException {
|
||||
ByteArrayOutputStream compressed = new ByteArrayOutputStream();
|
||||
try (GZIPOutputStream out = new GZIPOutputStream(compressed)) {
|
||||
out.write(plain);
|
||||
}
|
||||
return compressed.toByteArray();
|
||||
}
|
||||
}
|
||||
+149
@@ -0,0 +1,149 @@
|
||||
package dev.caskeleton.adapter.outbound.httpclient.restclient;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
import static org.assertj.core.api.Assertions.catchThrowable;
|
||||
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.OperationName;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.error.HttpResponseTooLargeException;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.error.HttpTargetRejectedException;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.operation.ExecutionEvidence;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.operation.HttpOperation;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.result.BlockingStreamingResponse;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.profile.ClientProfile;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.profile.RequestLimits;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.profile.ResponseLimits;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.profile.TransportType;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.testkit.ClientProfiles;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.testkit.MockHttpServer;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.testkit.TestGateways;
|
||||
import java.io.ByteArrayOutputStream;
|
||||
import java.io.IOException;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.Set;
|
||||
import java.util.zip.GZIPOutputStream;
|
||||
import org.junit.jupiter.params.ParameterizedTest;
|
||||
import org.junit.jupiter.params.provider.EnumSource;
|
||||
|
||||
/**
|
||||
* A streamed download is decoded the way a buffered body is, on every blocking transport.
|
||||
*
|
||||
* <p>Once the transports stopped decoding — so the wire budget counts what arrives — the streaming
|
||||
* path still handed the body straight on, and a compressed response reached the caller as gzip
|
||||
* bytes (round9 review 2026-11-12, RR9-01-30). The wire budget here is smaller than the decoded
|
||||
* body and larger than the compressed one: the download passes only when the wire is counted
|
||||
* compressed, which is also what fails if a transport goes back to decoding on its own.
|
||||
*/
|
||||
class StreamingContentCodingTest {
|
||||
|
||||
private static final byte[] DECODED = "a".repeat(6_000).getBytes(StandardCharsets.US_ASCII);
|
||||
|
||||
@ParameterizedTest(name = "{0}")
|
||||
@EnumSource(
|
||||
value = TransportType.class,
|
||||
names = {"APACHE", "JDK"})
|
||||
void aGzipDownloadReachesTheCallerDecodedAndCountedCompressedOnTheWire(TransportType transport)
|
||||
throws Exception {
|
||||
try (MockHttpServer server = MockHttpServer.start();
|
||||
TestGateways.Harness harness = TestGateways.forProfile(profile(server, transport))) {
|
||||
// given
|
||||
server.enqueueEncodedBody(200, "text/plain", "gzip", gzip(DECODED));
|
||||
BlockingStreamingGateway gateway = new BlockingStreamingGateway(harness.registry());
|
||||
|
||||
// when
|
||||
List<Object> received;
|
||||
try (BlockingStreamingResponse response = gateway.download(harness.profile().name(), get())) {
|
||||
received =
|
||||
List.of(
|
||||
response.body().readAllBytes().length,
|
||||
response.headers().keySet().stream()
|
||||
.anyMatch(name -> name.equalsIgnoreCase("Content-Encoding")));
|
||||
}
|
||||
|
||||
// then
|
||||
assertThat(received).containsExactly(DECODED.length, false);
|
||||
}
|
||||
}
|
||||
|
||||
@ParameterizedTest(name = "{0}")
|
||||
@EnumSource(
|
||||
value = TransportType.class,
|
||||
names = {"APACHE", "JDK"})
|
||||
void anIdentityDownloadPastTheDecodedBudgetFailsAsTheBufferedReadDoes(TransportType transport)
|
||||
throws Exception {
|
||||
try (MockHttpServer server = MockHttpServer.start();
|
||||
TestGateways.Harness harness =
|
||||
TestGateways.forProfile(
|
||||
profile(
|
||||
server, transport, new ResponseLimits(10_000, 1_000, Set.of("text/plain"))))) {
|
||||
// given
|
||||
// Within the wire budget and past the decoded one: an identity body is its own decoding.
|
||||
server.enqueueBody(200, "text/plain", DECODED);
|
||||
BlockingStreamingGateway gateway = new BlockingStreamingGateway(harness.registry());
|
||||
|
||||
// when
|
||||
Throwable failure;
|
||||
try (BlockingStreamingResponse response = gateway.download(harness.profile().name(), get())) {
|
||||
failure = catchThrowable(() -> response.body().readAllBytes());
|
||||
}
|
||||
|
||||
// then
|
||||
assertThat(failure)
|
||||
.isInstanceOf(HttpResponseTooLargeException.class)
|
||||
.hasMessageContaining("decoded bytes exceeded the profile limit of 1000");
|
||||
}
|
||||
}
|
||||
|
||||
@ParameterizedTest(name = "{0}")
|
||||
@EnumSource(
|
||||
value = TransportType.class,
|
||||
names = {"APACHE", "JDK"})
|
||||
void aStreamedResponseInACodingThisClientDidNotAskForIsRefusedAsReceived(TransportType transport)
|
||||
throws Exception {
|
||||
try (MockHttpServer server = MockHttpServer.start();
|
||||
TestGateways.Harness harness = TestGateways.forProfile(profile(server, transport))) {
|
||||
// given
|
||||
server.enqueueEncodedBody(200, "text/plain", "br", new byte[] {1, 2, 3});
|
||||
BlockingStreamingGateway gateway = new BlockingStreamingGateway(harness.registry());
|
||||
|
||||
// when
|
||||
Throwable refused = catchThrowable(() -> gateway.download(harness.profile().name(), get()));
|
||||
|
||||
// then
|
||||
assertThat(refused)
|
||||
.isInstanceOfSatisfying(
|
||||
HttpTargetRejectedException.class,
|
||||
rejected ->
|
||||
assertThat(rejected.metadata().evidence())
|
||||
.as("the server answered; the client refused what it sent")
|
||||
.isEqualTo(ExecutionEvidence.RESPONSE_RECEIVED));
|
||||
}
|
||||
}
|
||||
|
||||
private static HttpOperation get() {
|
||||
return HttpOperation.get(new OperationName("download"), "/download", Map.of());
|
||||
}
|
||||
|
||||
private static ClientProfile profile(MockHttpServer server, TransportType transport) {
|
||||
return profile(server, transport, new ResponseLimits(1_000, 10_000, Set.of("text/plain")));
|
||||
}
|
||||
|
||||
private static ClientProfile profile(
|
||||
MockHttpServer server, TransportType transport, ResponseLimits limits) {
|
||||
return ClientProfiles.builder("streaming-coding")
|
||||
.baseUrl(server.uri("/"))
|
||||
.transport(transport)
|
||||
.request(new RequestLimits(1024, true))
|
||||
.response(limits)
|
||||
.build();
|
||||
}
|
||||
|
||||
private static byte[] gzip(byte[] plain) throws IOException {
|
||||
ByteArrayOutputStream compressed = new ByteArrayOutputStream();
|
||||
try (GZIPOutputStream out = new GZIPOutputStream(compressed)) {
|
||||
out.write(plain);
|
||||
}
|
||||
return compressed.toByteArray();
|
||||
}
|
||||
}
|
||||
+59
@@ -0,0 +1,59 @@
|
||||
package dev.caskeleton.adapter.outbound.httpclient.restclient;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.error.HttpDeadlineExceededException;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.error.HttpFailureMetadata;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.error.HttpRemoteErrorException;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.error.HttpTargetRejectedException;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.testkit.CoreFixtures;
|
||||
import java.util.stream.Stream;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.junit.jupiter.params.ParameterizedTest;
|
||||
import org.junit.jupiter.params.provider.Arguments;
|
||||
import org.junit.jupiter.params.provider.MethodSource;
|
||||
|
||||
/**
|
||||
* What a streaming setup failure reaches the caller as, decided apart from the race that produces
|
||||
* it: a test against a server cannot make the budget run out between the headers arriving and the
|
||||
* coding being refused, so the decision is held here directly.
|
||||
*/
|
||||
class StreamingSetupFailureTest {
|
||||
|
||||
private static final HttpFailureMetadata METADATA = CoreFixtures.ambiguousMetadata();
|
||||
|
||||
static Stream<Arguments> failuresThatKeepWhatTheyAre() {
|
||||
return Stream.of(
|
||||
Arguments.of(
|
||||
"예산을 다 쓴 뒤 거절한 응답", new HttpTargetRejectedException("refused coding", METADATA), true),
|
||||
Arguments.of(
|
||||
"예산을 다 쓴 뒤 원격 오류", new HttpRemoteErrorException("remote error", METADATA), true),
|
||||
Arguments.of("예산이 남은 transport 실패", new IllegalStateException("transport"), false));
|
||||
}
|
||||
|
||||
@ParameterizedTest(name = "{0}")
|
||||
@MethodSource("failuresThatKeepWhatTheyAre")
|
||||
void aFailureTheDeadlineDidNotCauseKeepsWhatItIs(
|
||||
String name, RuntimeException failure, boolean budgetSpent) {
|
||||
// when
|
||||
RuntimeException thrown = BlockingStreamingGateway.setupFailure(failure, budgetSpent, METADATA);
|
||||
|
||||
// then
|
||||
assertThat(thrown).isSameAs(failure);
|
||||
}
|
||||
|
||||
@Test
|
||||
void aTransportFailureAfterTheBudgetIsSpentIsTheDeadline() {
|
||||
// given
|
||||
IllegalStateException failure = new IllegalStateException("transport");
|
||||
|
||||
// when
|
||||
RuntimeException thrown = BlockingStreamingGateway.setupFailure(failure, true, METADATA);
|
||||
|
||||
// then
|
||||
assertThat(thrown)
|
||||
.isInstanceOf(HttpDeadlineExceededException.class)
|
||||
.hasMessage("effective deadline reached before the stream was set up")
|
||||
.hasCause(failure);
|
||||
}
|
||||
}
|
||||
+16
@@ -6,7 +6,10 @@ import static org.assertj.core.api.Assertions.assertThatThrownBy;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.error.HttpConfigurationException;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.profile.ClientMode;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.profile.ClientProfile;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.profile.ProxySettings;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.profile.ProxyType;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.testkit.ClientProfiles;
|
||||
import java.util.Optional;
|
||||
import org.junit.jupiter.api.Test;
|
||||
|
||||
class TransportCapabilityValidatorTest {
|
||||
@@ -46,6 +49,19 @@ class TransportCapabilityValidatorTest {
|
||||
.hasMessageContaining("validated DNS pinning");
|
||||
}
|
||||
|
||||
@Test
|
||||
void acceptsAnHttpProxyOnATransportThatHasOne() {
|
||||
ClientProfile http =
|
||||
ClientProfiles.builder("egress")
|
||||
.proxy(
|
||||
new ProxySettings(
|
||||
true, "proxy.internal", 3128, ProxyType.HTTP, Optional.empty(), false))
|
||||
.build();
|
||||
|
||||
assertThatCode(() -> validator.validate(http, BlockingTransportCapabilities.apacheClassic()))
|
||||
.doesNotThrowAnyException();
|
||||
}
|
||||
|
||||
@Test
|
||||
void acceptsAMatchingCapabilitySet() {
|
||||
assertThatCode(
|
||||
|
||||
+16
@@ -121,6 +121,22 @@ public final class MockHttpServer implements AutoCloseable {
|
||||
.setBody(new okio.Buffer().write(body)));
|
||||
}
|
||||
|
||||
/**
|
||||
* Answers with a body already in {@code contentEncoding}, as a server that compresses would.
|
||||
*
|
||||
* @param body the coded bytes, sent as they are
|
||||
*/
|
||||
@SuppressWarnings("resource") // the okio Buffer, as in enqueueBody
|
||||
public void enqueueEncodedBody(
|
||||
int status, String contentType, String contentEncoding, byte[] body) {
|
||||
server.enqueue(
|
||||
new MockResponse()
|
||||
.setResponseCode(status)
|
||||
.setHeader("Content-Type", contentType)
|
||||
.setHeader("Content-Encoding", contentEncoding)
|
||||
.setBody(new okio.Buffer().write(body)));
|
||||
}
|
||||
|
||||
/** Accepts the request and never answers; drives response-header timeout tests. */
|
||||
public void enqueueNoResponse() {
|
||||
server.enqueue(new MockResponse().setSocketPolicy(SocketPolicy.NO_RESPONSE));
|
||||
|
||||
+44
-8
@@ -29,6 +29,8 @@ import software.amazon.awssdk.services.s3.model.ServerSideEncryption;
|
||||
/** Exact-key S3 control store using native conditional PUT rather than check-then-act emulation. */
|
||||
public final class S3ConditionalObjectControlStore implements ObjectControlStore {
|
||||
|
||||
private static final String CONFLICT = "S3 control conditional mutation conflicted";
|
||||
|
||||
private static final int MAXIMUM_CONTROL_BYTES = 64 * 1024;
|
||||
|
||||
private final S3AsyncClient client;
|
||||
@@ -107,9 +109,11 @@ public final class S3ConditionalObjectControlStore implements ObjectControlStore
|
||||
true);
|
||||
return stored(response.eTag(), record);
|
||||
} catch (ObjectStorageProviderException exception) {
|
||||
if (exception.failure() == ObjectStorageProviderException.Failure.CONFLICT
|
||||
|| exception.failure() == ObjectStorageProviderException.Failure.PRECONDITION_FAILED) {
|
||||
throw new ObjectControlConflictException("S3 control conditional mutation conflicted");
|
||||
if (exception.failure() == ObjectStorageProviderException.Failure.PRECONDITION_FAILED) {
|
||||
return reconcilePreconditionFailure(key, record);
|
||||
}
|
||||
if (exception.failure() == ObjectStorageProviderException.Failure.CONFLICT) {
|
||||
throw new ObjectControlConflictException(CONFLICT);
|
||||
}
|
||||
if (exception.failure() == ObjectStorageProviderException.Failure.INDETERMINATE) {
|
||||
return reconcileLostResponse(key, record);
|
||||
@@ -118,20 +122,52 @@ public final class S3ConditionalObjectControlStore implements ObjectControlStore
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Decides a failed precondition by what the key holds now.
|
||||
*
|
||||
* <p>A 412 is not always somebody else's write. The SDK resends a conditional {@code PUT} whose
|
||||
* response it lost — the body is replayable — and when the first send landed, the resend fails
|
||||
* the very precondition the first one satisfied: {@code If-None-Match: *} now finds the object,
|
||||
* {@code If-Match} now finds a new ETag. Reported as a conflict, this store's own write was
|
||||
* called stale and the CAS above it gave up on a mutation that had succeeded (round9 review
|
||||
* 2026-11-12, RR9-01-19). The key holding exactly the record this call wrote is that case; any
|
||||
* other content, or none, is a real conflict.
|
||||
*
|
||||
* <p>"Exactly the record" is equality, not authorship: two writers that CAS the same version to
|
||||
* equal records — same revision, state and timestamp — are indistinguishable here, and both are
|
||||
* told they succeeded. The record's revision is what normally tells them apart; a caller that can
|
||||
* produce equal records from two places carries its own token in the record.
|
||||
*/
|
||||
private StoredControlRecord reconcilePreconditionFailure(
|
||||
String key, ObjectControlRecord expected) {
|
||||
return holdingExactly(key, expected)
|
||||
.orElseThrow(() -> new ObjectControlConflictException(CONFLICT));
|
||||
}
|
||||
|
||||
private StoredControlRecord reconcileLostResponse(String key, ObjectControlRecord expected) {
|
||||
Optional<StoredControlRecord> current = read(key);
|
||||
if (current.isPresent() && current.orElseThrow().record().equals(expected)) {
|
||||
return current.orElseThrow();
|
||||
}
|
||||
if (current.isPresent()) {
|
||||
throw new ObjectControlConflictException(
|
||||
"S3 control response loss resolved to different exact record evidence");
|
||||
return holding(current.orElseThrow(), expected)
|
||||
.orElseThrow(
|
||||
() ->
|
||||
new ObjectControlConflictException(
|
||||
"S3 control response loss resolved to different exact record evidence"));
|
||||
}
|
||||
throw new ObjectStorageProviderException(
|
||||
ObjectStorageProviderException.Failure.INDETERMINATE,
|
||||
"S3 control mutation response was lost and no authoritative applied evidence exists");
|
||||
}
|
||||
|
||||
/** What the key holds now, when it is exactly {@code expected}. */
|
||||
private Optional<StoredControlRecord> holdingExactly(String key, ObjectControlRecord expected) {
|
||||
return read(key).flatMap(current -> holding(current, expected));
|
||||
}
|
||||
|
||||
private static Optional<StoredControlRecord> holding(
|
||||
StoredControlRecord current, ObjectControlRecord expected) {
|
||||
return current.record().equals(expected) ? Optional.of(current) : Optional.empty();
|
||||
}
|
||||
|
||||
private static StoredControlRecord stored(String eTag, ObjectControlRecord record) {
|
||||
if (eTag == null || eTag.isBlank() || eTag.length() > 256) {
|
||||
throw new ObjectStorageProviderException(
|
||||
|
||||
+59
-2
@@ -2,6 +2,7 @@ package dev.caskeleton.adapter.outbound.objectstorage.s3;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
import static org.assertj.core.api.Assertions.assertThatThrownBy;
|
||||
import static org.assertj.core.api.Assertions.catchThrowable;
|
||||
import static org.mockito.ArgumentMatchers.any;
|
||||
import static org.mockito.Mockito.mock;
|
||||
import static org.mockito.Mockito.never;
|
||||
@@ -26,6 +27,7 @@ import java.util.List;
|
||||
import java.util.concurrent.CompletableFuture;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.mockito.ArgumentCaptor;
|
||||
import org.mockito.ArgumentMatchers;
|
||||
import software.amazon.awssdk.core.async.AsyncRequestBody;
|
||||
import software.amazon.awssdk.core.async.AsyncResponseTransformer;
|
||||
import software.amazon.awssdk.core.async.SdkPublisher;
|
||||
@@ -99,6 +101,9 @@ class S3ConditionalObjectControlStoreTest {
|
||||
S3AsyncClient client = mock(S3AsyncClient.class);
|
||||
when(client.putObject(any(PutObjectRequest.class), any(AsyncRequestBody.class)))
|
||||
.thenReturn(CompletableFuture.failedFuture(S3Exception.builder().statusCode(412).build()));
|
||||
// Another writer's record holds the key: the precondition failed on somebody else's write.
|
||||
respondToGet(
|
||||
client, CODEC.encode(record(3, PublishedReferenceState.PUBLISHED)), "\"other-writer\"");
|
||||
S3ConditionalObjectControlStore store =
|
||||
new S3ConditionalObjectControlStore(client, awsBinding(), CODEC);
|
||||
|
||||
@@ -118,12 +123,64 @@ class S3ConditionalObjectControlStoreTest {
|
||||
.class);
|
||||
}
|
||||
|
||||
@Test
|
||||
void aPreconditionFailedByTheResendOfOurOwnLandedWriteIsThatWrite() {
|
||||
// given
|
||||
// The SDK lost the first response and resent the conditional PUT; the first one had landed, so
|
||||
// the resend fails If-Match against the ETag it produced.
|
||||
S3AsyncClient client = mock(S3AsyncClient.class);
|
||||
ObjectReferenceRecord written = record(2, PublishedReferenceState.RETIRED);
|
||||
when(client.putObject(any(PutObjectRequest.class), any(AsyncRequestBody.class)))
|
||||
.thenReturn(CompletableFuture.failedFuture(S3Exception.builder().statusCode(412).build()));
|
||||
respondToGet(client, CODEC.encode(written), "\"after-our-write\"");
|
||||
S3ConditionalObjectControlStore store =
|
||||
new S3ConditionalObjectControlStore(client, awsBinding(), CODEC);
|
||||
|
||||
// when
|
||||
ObjectControlStore.StoredControlRecord stored =
|
||||
store.compareAndSet(
|
||||
KEY, new ObjectControlMutation(ObjectControlVersion.of("\"before\""), written));
|
||||
|
||||
// then
|
||||
assertThat(List.of(stored.record(), stored.version()))
|
||||
.containsExactly(written, ObjectControlVersion.of("\"after-our-write\""));
|
||||
}
|
||||
|
||||
@Test
|
||||
void aPreconditionFailureOnAnAbsentKeyIsAConflict() {
|
||||
// given
|
||||
S3AsyncClient client = mock(S3AsyncClient.class);
|
||||
when(client.putObject(any(PutObjectRequest.class), any(AsyncRequestBody.class)))
|
||||
.thenReturn(CompletableFuture.failedFuture(S3Exception.builder().statusCode(412).build()));
|
||||
when(client.getObject(
|
||||
any(GetObjectRequest.class),
|
||||
ArgumentMatchers.<AsyncResponseTransformer<GetObjectResponse, ?>>any()))
|
||||
.thenReturn(CompletableFuture.failedFuture(S3Exception.builder().statusCode(404).build()));
|
||||
S3ConditionalObjectControlStore store =
|
||||
new S3ConditionalObjectControlStore(client, awsBinding(), CODEC);
|
||||
|
||||
// when
|
||||
Throwable refused =
|
||||
catchThrowable(
|
||||
() ->
|
||||
store.compareAndSet(
|
||||
KEY,
|
||||
new ObjectControlMutation(
|
||||
ObjectControlVersion.of("\"stale\""),
|
||||
record(2, PublishedReferenceState.RETIRED))));
|
||||
|
||||
// then
|
||||
assertThat(refused)
|
||||
.isInstanceOf(ObjectControlConflictException.class)
|
||||
.hasMessageContaining("conflicted");
|
||||
}
|
||||
|
||||
@Test
|
||||
void exact404IsTheOnlyAbsentRead() {
|
||||
S3AsyncClient client = mock(S3AsyncClient.class);
|
||||
when(client.getObject(
|
||||
any(GetObjectRequest.class),
|
||||
org.mockito.ArgumentMatchers.<AsyncResponseTransformer<GetObjectResponse, ?>>any()))
|
||||
ArgumentMatchers.<AsyncResponseTransformer<GetObjectResponse, ?>>any()))
|
||||
.thenReturn(CompletableFuture.failedFuture(S3Exception.builder().statusCode(404).build()));
|
||||
S3ConditionalObjectControlStore store =
|
||||
new S3ConditionalObjectControlStore(client, awsBinding(), CODEC);
|
||||
@@ -154,7 +211,7 @@ class S3ConditionalObjectControlStoreTest {
|
||||
private static void respondToGet(S3AsyncClient client, byte[] encoded, String eTag) {
|
||||
when(client.getObject(
|
||||
any(GetObjectRequest.class),
|
||||
org.mockito.ArgumentMatchers.<AsyncResponseTransformer<GetObjectResponse, ?>>any()))
|
||||
ArgumentMatchers.<AsyncResponseTransformer<GetObjectResponse, ?>>any()))
|
||||
.thenAnswer(
|
||||
invocation -> {
|
||||
AsyncResponseTransformer<GetObjectResponse, Object> transformer =
|
||||
|
||||
+147
-33
@@ -7,9 +7,14 @@ import dev.caskeleton.adapter.outbound.mongo.api.error.MongoOperationRejectedExc
|
||||
import java.time.Clock;
|
||||
import java.time.Duration;
|
||||
import java.time.Instant;
|
||||
import java.util.HashMap;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.Objects;
|
||||
import java.util.Set;
|
||||
import java.util.concurrent.ConcurrentHashMap;
|
||||
import java.util.concurrent.atomic.AtomicLong;
|
||||
import java.util.function.Function;
|
||||
|
||||
/**
|
||||
* Bounds how many tenant clients exist at once (advanced plan Task 11).
|
||||
@@ -18,6 +23,13 @@ import java.util.Set;
|
||||
* connection pool and its own monitoring threads, so a thousand tenants is a thousand pools. The
|
||||
* registry caps the number and evicts idle ones, and refuses rather than exceeding the cap — an
|
||||
* unbounded registry fails later, as connection exhaustion on an unrelated request.
|
||||
*
|
||||
* <p>{@link #acquire} hands back a {@link Lease}; the client is in use until every lease on it is
|
||||
* closed, and only a client nobody holds is idle. Acquire used to return nothing, so idle meant
|
||||
* "acquired longest ago" and eviction could close a client a long-running caller was still using.
|
||||
* Closing an evicted client is this registry's own clean-up: if it fails, the tenant whose acquire
|
||||
* triggered the eviction is not told about another tenant's client — the failure is counted in
|
||||
* {@link #evictionCloseFailures()} (round9 review 2026-11-12, RR9-01-27).
|
||||
*/
|
||||
@MongoAdvancedEntryPoint(MongoCapability.DATABASE_PER_TENANT)
|
||||
public final class MongoTenantClientRegistry {
|
||||
@@ -26,11 +38,21 @@ public final class MongoTenantClientRegistry {
|
||||
|
||||
private final Duration idleEviction;
|
||||
|
||||
private final Map<String, Instant> lastUsed = new java.util.concurrent.ConcurrentHashMap<>();
|
||||
private final Map<String, Instant> lastUsed = new ConcurrentHashMap<>();
|
||||
|
||||
private final Map<String, AutoCloseable> clients = new java.util.concurrent.ConcurrentHashMap<>();
|
||||
private final Map<String, AutoCloseable> clients = new ConcurrentHashMap<>();
|
||||
|
||||
private final java.util.function.Function<String, AutoCloseable> clientFactory;
|
||||
private final Map<String, Integer> leases = new HashMap<>();
|
||||
|
||||
/**
|
||||
* Guards the cap, the lease counts and the eviction as one decision. A private monitor rather
|
||||
* than the registry itself, so code outside cannot take it and stall every acquire.
|
||||
*/
|
||||
private final Object lock = new Object();
|
||||
|
||||
private final AtomicLong evictionCloseFailures = new AtomicLong();
|
||||
|
||||
private final Function<String, AutoCloseable> clientFactory;
|
||||
|
||||
private final Clock clock;
|
||||
|
||||
@@ -47,15 +69,15 @@ public final class MongoTenantClientRegistry {
|
||||
MongoAdvancedCapabilityGuard guard,
|
||||
int maximumActiveClients,
|
||||
Duration idleEviction,
|
||||
java.util.function.Function<String, AutoCloseable> clientFactory) {
|
||||
Function<String, AutoCloseable> clientFactory) {
|
||||
this(guard, maximumActiveClients, idleEviction, clientFactory, Clock.systemUTC());
|
||||
}
|
||||
|
||||
/**
|
||||
* A registry that owns real clients.
|
||||
*
|
||||
* <p>Idle eviction reads {@code clock}; {@link #acquire(String)} used to read {@code
|
||||
* Instant.now()} beside it.
|
||||
* <p>Acquiring, closing a lease and idle eviction all read {@code clock}; {@link
|
||||
* #acquire(String)} used to read {@code Instant.now()} beside it.
|
||||
*
|
||||
* <p>The factory is called at most once per tenant, inside the map's own computation, so two
|
||||
* threads asking for the same tenant get one client rather than two — a per-tenant client carries
|
||||
@@ -65,7 +87,7 @@ public final class MongoTenantClientRegistry {
|
||||
MongoAdvancedCapabilityGuard guard,
|
||||
int maximumActiveClients,
|
||||
Duration idleEviction,
|
||||
java.util.function.Function<String, AutoCloseable> clientFactory,
|
||||
Function<String, AutoCloseable> clientFactory,
|
||||
Clock clock) {
|
||||
Objects.requireNonNull(guard, "guard").require(MongoTenantClientRegistry.class);
|
||||
this.clientFactory = Objects.requireNonNull(clientFactory, "clientFactory");
|
||||
@@ -80,41 +102,128 @@ public final class MongoTenantClientRegistry {
|
||||
/**
|
||||
* Acquires a client for a tenant.
|
||||
*
|
||||
* @return the lease to close when the work with the client is done
|
||||
* @throws MongoOperationRejectedException when the cap is reached and nothing can be evicted
|
||||
*/
|
||||
public void acquire(String tenantKey) {
|
||||
acquire(tenantKey, clock.instant());
|
||||
public Lease acquire(String tenantKey) {
|
||||
Objects.requireNonNull(tenantKey, "tenantKey");
|
||||
synchronized (lock) {
|
||||
return acquireHeld(tenantKey);
|
||||
}
|
||||
}
|
||||
|
||||
/** Acquires a client for a tenant at an explicit instant, so eviction is testable. */
|
||||
public synchronized void acquire(String tenantKey, Instant now) {
|
||||
Objects.requireNonNull(tenantKey, "tenantKey");
|
||||
Objects.requireNonNull(now, "now");
|
||||
private Lease acquireHeld(String tenantKey) {
|
||||
// One time source for the acquire, the close and the eviction: an overload that took the
|
||||
// instant as an argument let a caller acquire in one timeline and close in the clock's.
|
||||
Instant now = clock.instant();
|
||||
// Synchronized rather than a bare ConcurrentHashMap: the cap, the eviction and the create are
|
||||
// one decision. Checking the size on one thread while another inserts is how a cap of ten ends
|
||||
// up holding twelve connection pools.
|
||||
if (lastUsed.containsKey(tenantKey)) {
|
||||
lastUsed.put(tenantKey, now);
|
||||
return;
|
||||
if (!lastUsed.containsKey(tenantKey)) {
|
||||
evictIdle(now);
|
||||
if (lastUsed.size() >= maximumActiveClients) {
|
||||
throw MongoOperationRejectedException.of(
|
||||
"tenancy.client",
|
||||
"the tenant client cap of "
|
||||
+ maximumActiveClients
|
||||
+ " is reached and no client is idle; each tenant client carries its own"
|
||||
+ " connection pool and monitoring threads, so the cap is what stops one process"
|
||||
+ " from exhausting the cluster's connections");
|
||||
}
|
||||
clients.computeIfAbsent(tenantKey, clientFactory);
|
||||
}
|
||||
evictIdle(now);
|
||||
if (lastUsed.size() >= maximumActiveClients) {
|
||||
throw MongoOperationRejectedException.of(
|
||||
"tenancy.client",
|
||||
"the tenant client cap of "
|
||||
+ maximumActiveClients
|
||||
+ " is reached and no client is idle; each tenant client carries its own connection "
|
||||
+ "pool and monitoring threads, so the cap is what stops one process from exhausting "
|
||||
+ "the cluster's connections");
|
||||
}
|
||||
clients.computeIfAbsent(tenantKey, clientFactory);
|
||||
lastUsed.put(tenantKey, now);
|
||||
leases.merge(tenantKey, 1, Integer::sum);
|
||||
return new HeldLease(tenantKey, clients.get(tenantKey));
|
||||
}
|
||||
|
||||
/** Releases a tenant's client and closes it. */
|
||||
public synchronized void release(String tenantKey) {
|
||||
lastUsed.remove(Objects.requireNonNull(tenantKey, "tenantKey"));
|
||||
close(tenantKey);
|
||||
/**
|
||||
* Releases a tenant's client and closes it.
|
||||
*
|
||||
* @throws IllegalStateException when a lease on the client is still open, or the close fails
|
||||
*/
|
||||
public void release(String tenantKey) {
|
||||
Objects.requireNonNull(tenantKey, "tenantKey");
|
||||
synchronized (lock) {
|
||||
if (leases.containsKey(tenantKey)) {
|
||||
throw new IllegalStateException(
|
||||
"the client for tenant '" + tenantKey + "' is still in use and cannot be released");
|
||||
}
|
||||
lastUsed.remove(tenantKey);
|
||||
close(tenantKey);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* How many evicted clients failed to close.
|
||||
*
|
||||
* <p>The registry has dropped each of them and freed its slot; the count is how an operator
|
||||
* learns that clients — and their pools — may still be open behind it.
|
||||
*/
|
||||
public long evictionCloseFailures() {
|
||||
return evictionCloseFailures.get();
|
||||
}
|
||||
|
||||
/**
|
||||
* One use of a tenant client; close it when the work with that client is done.
|
||||
*
|
||||
* <p>An interface so the registry's own implementation stays private: only the registry makes a
|
||||
* lease, and only through {@link #acquire}.
|
||||
*/
|
||||
public interface Lease extends AutoCloseable {
|
||||
|
||||
/** The tenant's client, valid until this lease is closed. */
|
||||
AutoCloseable client();
|
||||
|
||||
/** Ends this use. Closing twice is the same as closing once. */
|
||||
@Override
|
||||
void close();
|
||||
}
|
||||
|
||||
private final class HeldLease implements Lease {
|
||||
|
||||
private final String tenantKey;
|
||||
|
||||
private final AutoCloseable client;
|
||||
|
||||
private boolean closed;
|
||||
|
||||
private HeldLease(String tenantKey, AutoCloseable client) {
|
||||
this.tenantKey = tenantKey;
|
||||
this.client = client;
|
||||
}
|
||||
|
||||
@Override
|
||||
public AutoCloseable client() {
|
||||
return client;
|
||||
}
|
||||
|
||||
@Override
|
||||
public void close() {
|
||||
synchronized (lock) {
|
||||
if (closed) {
|
||||
return;
|
||||
}
|
||||
closed = true;
|
||||
leases.computeIfPresent(tenantKey, (tenant, open) -> open == 1 ? null : open - 1);
|
||||
// Idle from the end of the last use, not its start: a client held for half an hour was in
|
||||
// use all that time, and measuring from the acquire evicted it the moment it was let go.
|
||||
Instant released = clock.instant();
|
||||
lastUsed.computeIfPresent(
|
||||
tenantKey, (tenant, used) -> released.isAfter(used) ? released : used);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* How many leases are open now, across all tenants.
|
||||
*
|
||||
* <p>A lease nobody closed keeps its client's slot for good; this is how that becomes visible.
|
||||
*/
|
||||
public int openLeases() {
|
||||
synchronized (lock) {
|
||||
return leases.values().stream().mapToInt(Integer::intValue).sum();
|
||||
}
|
||||
}
|
||||
|
||||
/** The tenants with an active client. */
|
||||
@@ -128,8 +237,9 @@ public final class MongoTenantClientRegistry {
|
||||
}
|
||||
|
||||
private void evictIdle(Instant now) {
|
||||
java.util.List<String> idle =
|
||||
List<String> idle =
|
||||
lastUsed.entrySet().stream()
|
||||
.filter(entry -> !leases.containsKey(entry.getKey()))
|
||||
.filter(entry -> entry.getValue().plus(idleEviction).isBefore(now))
|
||||
.map(Map.Entry::getKey)
|
||||
.toList();
|
||||
@@ -138,7 +248,11 @@ public final class MongoTenantClientRegistry {
|
||||
lastUsed.remove(tenant);
|
||||
// Evicting the bookkeeping entry without closing the client frees the slot and keeps the
|
||||
// connection pool, which is the exhaustion this cap exists to prevent, delayed.
|
||||
close(tenant);
|
||||
try {
|
||||
close(tenant);
|
||||
} catch (IllegalStateException closeFailed) {
|
||||
evictionCloseFailures.incrementAndGet();
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
|
||||
+52
-15
@@ -6,20 +6,30 @@ import dev.caskeleton.adapter.outbound.mongo.query.budget.MongoBudgetEnforcer;
|
||||
import dev.caskeleton.adapter.outbound.mongo.query.budget.MongoBudgetPolicyRegistry;
|
||||
import dev.caskeleton.adapter.outbound.mongo.query.budget.MongoOperationBudget;
|
||||
import java.time.Duration;
|
||||
import java.util.ArrayList;
|
||||
import java.util.Iterator;
|
||||
import java.util.List;
|
||||
import java.util.Objects;
|
||||
import java.util.stream.Stream;
|
||||
import org.bson.Document;
|
||||
import org.bson.RawBsonDocument;
|
||||
import org.bson.codecs.Codec;
|
||||
import org.springframework.data.mongodb.core.MongoOperations;
|
||||
import org.springframework.data.mongodb.core.aggregation.Aggregation;
|
||||
import org.springframework.data.mongodb.core.aggregation.AggregationOperation;
|
||||
import org.springframework.data.mongodb.core.aggregation.AggregationOptions;
|
||||
import org.springframework.data.mongodb.core.aggregation.AggregationResults;
|
||||
|
||||
/**
|
||||
* Runs an aggregation only after its stages, budget and lookups have been checked (design §17).
|
||||
*
|
||||
* <p>Every guard runs before the first stage reaches the server, because an aggregation that is
|
||||
* going to be refused should cost nothing. The one guard that runs afterwards is the result-count
|
||||
* check: {@code $limit} bounds what the pipeline emits, but a pipeline whose own shape produced
|
||||
* more than the budget allows has still told the caller something worth failing over.
|
||||
* going to be refused should cost nothing. Two run while the results arrive: the result count and
|
||||
* the result bytes. The pipeline gets a final {@code $limit} of one document past the count budget,
|
||||
* so the server never produces more than that, and the results are read from the cursor one
|
||||
* document at a time, the encoded size of each added up as it arrives. They used to be read whole —
|
||||
* every document in memory before anything was counted — and the byte budget was declared and never
|
||||
* checked (round9 review 2026-11-12, RR9-01-20; {@code docs/mongodb/query-aggregation-guide.md}
|
||||
* 「집계에 예산을 둔다」).
|
||||
*/
|
||||
public final class PolicyAwareMongoAggregationExecutor {
|
||||
|
||||
@@ -53,6 +63,14 @@ public final class PolicyAwareMongoAggregationExecutor {
|
||||
/**
|
||||
* Executes a typed aggregation.
|
||||
*
|
||||
* <p>Results are read as {@link Document}, sized, then mapped with the template's converter. Two
|
||||
* costs follow, both deliberate. Each document is encoded once more to be measured: the template
|
||||
* hands back decoded documents, so the bytes the server sent are not available, and the budget
|
||||
* has to be in bytes. And the mapping is the converter's alone: the template's typed aggregate
|
||||
* would also publish {@code AfterLoadEvent}/{@code AfterConvertEvent} and run entity callbacks,
|
||||
* and those do not run for an aggregation result read here. A listener that needs them maps from
|
||||
* the returned type itself.
|
||||
*
|
||||
* @throws MongoOperationRejectedException when a stage, lookup, budget or result size is not
|
||||
* permitted
|
||||
*/
|
||||
@@ -71,19 +89,38 @@ public final class PolicyAwareMongoAggregationExecutor {
|
||||
MongoOperationBudget effective =
|
||||
enforcer.narrow(budget, budget.narrowedTo(budgetFor(context, budget)));
|
||||
|
||||
List<AggregationOperation> bounded = new ArrayList<>(plan.operations());
|
||||
// One past the budget: enough to know the budget was exceeded, never more than that.
|
||||
bounded.add(Aggregation.limit(effective.maxResults() + 1L));
|
||||
Aggregation aggregation =
|
||||
Aggregation.newAggregation(plan.operations()).withOptions(optionsFor(profile, effective));
|
||||
AggregationResults<T> results = operations.aggregate(aggregation, collection, outputType);
|
||||
List<T> mapped = results.getMappedResults();
|
||||
if (mapped.size() > effective.maxResults()) {
|
||||
throw MongoOperationRejectedException.of(
|
||||
"aggregation.result",
|
||||
"the aggregation returned "
|
||||
+ mapped.size()
|
||||
+ " documents, above the budget of "
|
||||
+ effective.maxResults());
|
||||
Aggregation.newAggregation(bounded).withOptions(optionsFor(profile, effective));
|
||||
Codec<Document> sizing = operations.getConverter().getCodecRegistry().get(Document.class);
|
||||
List<T> mapped = new ArrayList<>();
|
||||
long bytes = 0;
|
||||
try (Stream<Document> documents =
|
||||
operations.aggregateStream(aggregation, collection, Document.class)) {
|
||||
Iterator<Document> cursor = documents.iterator();
|
||||
while (cursor.hasNext()) {
|
||||
Document document = cursor.next();
|
||||
if (mapped.size() == effective.maxResults()) {
|
||||
throw MongoOperationRejectedException.of(
|
||||
"aggregation.result",
|
||||
"the aggregation returned more than the budget of "
|
||||
+ effective.maxResults()
|
||||
+ " documents");
|
||||
}
|
||||
bytes += new RawBsonDocument(document, sizing).getByteBuffer().remaining();
|
||||
if (bytes > effective.maxResultBytes()) {
|
||||
throw MongoOperationRejectedException.of(
|
||||
"aggregation.result",
|
||||
"the aggregation returned more than the budget of "
|
||||
+ effective.maxResultBytes()
|
||||
+ " bytes");
|
||||
}
|
||||
mapped.add(operations.getConverter().read(outputType, document));
|
||||
}
|
||||
}
|
||||
return mapped;
|
||||
return List.copyOf(mapped);
|
||||
}
|
||||
|
||||
private static MongoOperationBudget budgetFor(
|
||||
|
||||
+60
-9
@@ -36,20 +36,46 @@ public final class MongoCollectionMigrationLedger implements MongoMigrationLedge
|
||||
|
||||
private static final String MIGRATION_ID = "migrationId";
|
||||
|
||||
private static final String FENCE = "fence";
|
||||
|
||||
private final MongoCollection<Document> ledger;
|
||||
|
||||
private final MongoCollection<Document> checkpoints;
|
||||
|
||||
private final MongoCollection<Document> locks;
|
||||
|
||||
/**
|
||||
* A ledger over the default collections of {@code database}, the lock's included.
|
||||
*
|
||||
* <p>The lock collection is {@link MongoCollectionMigrationLock#DEFAULT_COLLECTION}. A deployment
|
||||
* that keeps its runner lease in a collection of another name uses the three-collection
|
||||
* constructor with that collection; with this one the ledger would read a lock that is never
|
||||
* taken and refuse nothing on its account.
|
||||
*
|
||||
* @param database the database being migrated
|
||||
*/
|
||||
public MongoCollectionMigrationLedger(MongoDatabase database) {
|
||||
this(
|
||||
Objects.requireNonNull(database, "database").getCollection(DEFAULT_COLLECTION),
|
||||
database.getCollection(DEFAULT_CHECKPOINT_COLLECTION));
|
||||
database.getCollection(DEFAULT_CHECKPOINT_COLLECTION),
|
||||
database.getCollection(MongoCollectionMigrationLock.DEFAULT_COLLECTION));
|
||||
}
|
||||
|
||||
/**
|
||||
* A ledger over explicit collections.
|
||||
*
|
||||
* @param ledger the applied migrations
|
||||
* @param checkpoints the running migrations' positions
|
||||
* @param locks the collection {@link MongoCollectionMigrationLock} keeps the runner lease in,
|
||||
* read to refuse a replaced runner's ledger entry
|
||||
*/
|
||||
public MongoCollectionMigrationLedger(
|
||||
MongoCollection<Document> ledger, MongoCollection<Document> checkpoints) {
|
||||
MongoCollection<Document> ledger,
|
||||
MongoCollection<Document> checkpoints,
|
||||
MongoCollection<Document> locks) {
|
||||
this.ledger = Objects.requireNonNull(ledger, "ledger");
|
||||
this.checkpoints = Objects.requireNonNull(checkpoints, "checkpoints");
|
||||
this.locks = Objects.requireNonNull(locks, "locks");
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -90,13 +116,34 @@ public final class MongoCollectionMigrationLedger implements MongoMigrationLedge
|
||||
Objects.requireNonNull(checksum, "checksum");
|
||||
Objects.requireNonNull(operator, "operator");
|
||||
Objects.requireNonNull(appliedAt, "appliedAt");
|
||||
requireCurrentFence(fence, "ledger entry for " + migrationId.value());
|
||||
requireFenced(fence, "ledger entry for " + migrationId.value());
|
||||
// A superseded runner that wakes up after its replacement took the lease must not record the
|
||||
// migration as applied under it (round9 review 2026-11-12, RR9-01-28). Two pieces of evidence:
|
||||
// the lock's newest acquisition, which a replacement leaves the moment it takes the lease, and
|
||||
// a checkpoint a newer runner wrote.
|
||||
//
|
||||
// A narrowing, not a guarantee: the reads and the insert are separate operations, so a runner
|
||||
// replaced between them still records. What stops a superseded runner first is the lock's
|
||||
// refresh, which refuses it; this check is the second line for a runner that skipped a
|
||||
// refresh. Making it atomic needs the lock and the ledger in one transaction, which a
|
||||
// standalone deployment does not have.
|
||||
if (MongoCollectionMigrationLock.newestFence(locks) > fence
|
||||
|| checkpoints
|
||||
.find(
|
||||
Filters.and(
|
||||
Filters.eq(MIGRATION_ID, migrationId.value()), Filters.gt(FENCE, fence)))
|
||||
.first()
|
||||
!= null) {
|
||||
throw MongoOperationRejectedException.of(
|
||||
"migration.fence",
|
||||
"a newer migration runner owns the lease; this runner's ledger entry was refused");
|
||||
}
|
||||
ledger.insertOne(
|
||||
new Document(MIGRATION_ID, migrationId.value())
|
||||
.append("checksum", checksum.value())
|
||||
.append("operator", operator)
|
||||
.append("appliedAt", Date.from(appliedAt))
|
||||
.append("fence", fence));
|
||||
.append(FENCE, fence));
|
||||
}
|
||||
|
||||
@Override
|
||||
@@ -117,7 +164,7 @@ public final class MongoCollectionMigrationLedger implements MongoMigrationLedge
|
||||
@Override
|
||||
public void saveCheckpoint(MongoMigrationCheckpoint checkpoint, long fence) {
|
||||
Objects.requireNonNull(checkpoint, "checkpoint");
|
||||
requireCurrentFence(fence, "checkpoint for " + checkpoint.migrationId().value());
|
||||
requireFenced(fence, "checkpoint for " + checkpoint.migrationId().value());
|
||||
// Conditioned on the stored fence as well as matched by id: a superseded runner that wakes up
|
||||
// mid-batch must not move the checkpoint of the runner that replaced it, which would send the
|
||||
// live migration back over rows it has already done or forward past rows it has not.
|
||||
@@ -126,7 +173,7 @@ public final class MongoCollectionMigrationLedger implements MongoMigrationLedge
|
||||
.append("resumePosition", checkpoint.resumePosition())
|
||||
.append("processedCount", checkpoint.processedCount())
|
||||
.append("updatedAt", Date.from(checkpoint.updatedAt()))
|
||||
.append("fence", fence);
|
||||
.append(FENCE, fence);
|
||||
|
||||
// Replace-then-insert rather than one upsert, and the two cases this separates are the whole
|
||||
// behaviour of the fence.
|
||||
@@ -147,7 +194,7 @@ public final class MongoCollectionMigrationLedger implements MongoMigrationLedge
|
||||
.replaceOne(
|
||||
Filters.and(
|
||||
Filters.eq(MIGRATION_ID, checkpoint.migrationId().value()),
|
||||
Filters.or(Filters.exists("fence", false), Filters.lte("fence", fence))),
|
||||
Filters.or(Filters.exists(FENCE, false), Filters.lte(FENCE, fence))),
|
||||
document,
|
||||
new ReplaceOptions().upsert(false))
|
||||
.getMatchedCount();
|
||||
@@ -172,9 +219,13 @@ public final class MongoCollectionMigrationLedger implements MongoMigrationLedge
|
||||
* Refuses a write from an unfenced lease.
|
||||
*
|
||||
* <p>An unfenced lease cannot prove the writer is still the owner, and the whole point of the
|
||||
* ledger is that it says what actually happened.
|
||||
* ledger is that it says what actually happened. Whether a fenced writer is still the current one
|
||||
* is decided in two places, not here: the runner refreshes its lease, conditioned on the fence,
|
||||
* right before it writes, and each write is conditioned on the newest fence the ledger has seen
|
||||
* for the migration. The method was called {@code requireCurrentFence} and checked neither
|
||||
* (round9 review 2026-11-12, RR9-01-28).
|
||||
*/
|
||||
private static void requireCurrentFence(long fence, String what) {
|
||||
private static void requireFenced(long fence, String what) {
|
||||
if (fence == MongoMigrationLock.UNFENCED) {
|
||||
throw MongoOperationRejectedException.of(
|
||||
"migration.fence",
|
||||
|
||||
+19
-1
@@ -6,6 +6,7 @@ import com.mongodb.client.MongoCollection;
|
||||
import com.mongodb.client.MongoDatabase;
|
||||
import com.mongodb.client.model.Filters;
|
||||
import com.mongodb.client.model.Updates;
|
||||
import dev.caskeleton.adapter.outbound.mongo.api.error.MongoOperationRejectedException;
|
||||
import java.time.Clock;
|
||||
import java.time.Duration;
|
||||
import java.time.Instant;
|
||||
@@ -125,6 +126,19 @@ public final class MongoCollectionMigrationLock implements MongoMigrationLock {
|
||||
return fence;
|
||||
}
|
||||
|
||||
/**
|
||||
* The fence of the newest acquisition of the lock in {@code locks}, or {@link #UNFENCED} when the
|
||||
* lock was never taken there.
|
||||
*
|
||||
* <p>For the ledger: a write whose fence is older than this one comes from a runner that has been
|
||||
* replaced, whether or not its replacement has written anything yet.
|
||||
*/
|
||||
static long newestFence(MongoCollection<Document> locks) {
|
||||
Document current =
|
||||
Objects.requireNonNull(locks, "locks").find(Filters.eq("_id", LOCK_ID)).first();
|
||||
return current == null ? UNFENCED : fenceOf(current);
|
||||
}
|
||||
|
||||
private static long fenceOf(Document lock) {
|
||||
Object stored = lock.get("fence");
|
||||
return stored instanceof Number number ? number.longValue() : 1L;
|
||||
@@ -165,7 +179,11 @@ public final class MongoCollectionMigrationLock implements MongoMigrationLock {
|
||||
.getMatchedCount();
|
||||
if (stillOwned != 1) {
|
||||
held = false;
|
||||
throw new IllegalStateException(
|
||||
// The refusal the heartbeat promises (MongoMigrationHeartbeat#beat), so a migration that
|
||||
// stops on it stops for the same reason the heartbeat's own ownership check gives. It was an
|
||||
// IllegalStateException, which read as a server defect (round9 review 2026-11-12, RR9-01-28).
|
||||
throw MongoOperationRejectedException.of(
|
||||
"migration.lock",
|
||||
"the migration lease was lost before it could be refreshed; another runner may have "
|
||||
+ "taken it after this one appeared to stall");
|
||||
}
|
||||
|
||||
+89
@@ -224,6 +224,95 @@ class MongoMigrationLaneTest {
|
||||
assertThat(ledger.findCheckpoint(id).orElseThrow().processedCount()).isEqualTo(200);
|
||||
}
|
||||
|
||||
@Test
|
||||
void aSupersededRunnerCannotRecordTheMigrationAppliedUnderItsReplacement() {
|
||||
// given
|
||||
MongoDatabase database = freshDatabase();
|
||||
MongoCollectionMigrationLedger ledger = new MongoCollectionMigrationLedger(database);
|
||||
ledger.ensureIndexes();
|
||||
MongoMigrationId id = new MongoMigrationId("20260811-011");
|
||||
ledger.saveCheckpoint(
|
||||
MongoMigrationCheckpoint.start(id, FIXED.instant())
|
||||
.advancedTo("o-300", 300, FIXED.instant()),
|
||||
2L);
|
||||
|
||||
// when
|
||||
Throwable refused =
|
||||
catchThrowable(
|
||||
() ->
|
||||
ledger.recordApplied(
|
||||
id, new MongoMigrationChecksum("abc"), "operator", FIXED.instant(), 1L));
|
||||
|
||||
// then
|
||||
assertThat(refused)
|
||||
.isInstanceOf(MongoOperationRejectedException.class)
|
||||
.hasMessageContaining("a newer migration runner owns the lease");
|
||||
}
|
||||
|
||||
@Test
|
||||
void aReplacedRunnerCannotRecordTheMigrationBeforeItsReplacementWroteAnything() {
|
||||
// given
|
||||
MongoDatabase database = freshDatabase();
|
||||
MongoCollectionMigrationLedger ledger = new MongoCollectionMigrationLedger(database);
|
||||
ledger.ensureIndexes();
|
||||
MongoMigrationId id = new MongoMigrationId("20260811-012");
|
||||
Instant early = Instant.parse("2026-08-11T00:00:00Z");
|
||||
MongoMigrationLock stalled =
|
||||
MongoCollectionMigrationLock.tryAcquire(
|
||||
database, "stalled-runner", LEASE, Clock.fixed(early, ZoneOffset.UTC))
|
||||
.orElseThrow();
|
||||
var replacement =
|
||||
MongoCollectionMigrationLock.tryAcquire(
|
||||
database,
|
||||
"next-runner",
|
||||
LEASE,
|
||||
Clock.fixed(early.plus(LEASE).plusSeconds(1), ZoneOffset.UTC))
|
||||
.orElseThrow();
|
||||
|
||||
// when
|
||||
Throwable refused =
|
||||
catchThrowable(
|
||||
() ->
|
||||
ledger.recordApplied(
|
||||
id,
|
||||
new MongoMigrationChecksum("abc"),
|
||||
"operator",
|
||||
FIXED.instant(),
|
||||
stalled.fence()));
|
||||
|
||||
// then
|
||||
assertThat(refused)
|
||||
.as("%s holds the lease with a newer fence and has not checkpointed yet", replacement)
|
||||
.isInstanceOf(MongoOperationRejectedException.class)
|
||||
.hasMessageContaining("a newer migration runner owns the lease");
|
||||
}
|
||||
|
||||
@Test
|
||||
void aLeaseLostBeforeItsRefreshIsTheRefusalTheHeartbeatPromises() {
|
||||
// given
|
||||
MongoDatabase database = freshDatabase();
|
||||
Instant early = Instant.parse("2026-08-11T00:00:00Z");
|
||||
MongoMigrationLock stalled =
|
||||
MongoCollectionMigrationLock.tryAcquire(
|
||||
database, "stalled-runner", LEASE, Clock.fixed(early, ZoneOffset.UTC))
|
||||
.orElseThrow();
|
||||
var unused =
|
||||
MongoCollectionMigrationLock.tryAcquire(
|
||||
database,
|
||||
"next-runner",
|
||||
LEASE,
|
||||
Clock.fixed(early.plus(LEASE).plusSeconds(1), ZoneOffset.UTC))
|
||||
.orElseThrow();
|
||||
|
||||
// when
|
||||
Throwable refused = catchThrowable(() -> stalled.refresh(LEASE));
|
||||
|
||||
// then
|
||||
assertThat(refused)
|
||||
.isInstanceOf(MongoOperationRejectedException.class)
|
||||
.hasMessageContaining("the migration lease was lost");
|
||||
}
|
||||
|
||||
@Test
|
||||
void theLedgerRefusesADoubleApplyOfTheSameId() {
|
||||
MongoDatabase database = freshDatabase();
|
||||
|
||||
+174
-74
@@ -3,6 +3,7 @@ package dev.caskeleton.adapter.outbound.mongo.advanced.tenancy;
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
import static org.assertj.core.api.Assertions.assertThatCode;
|
||||
import static org.assertj.core.api.Assertions.assertThatThrownBy;
|
||||
import static org.assertj.core.api.Assertions.catchThrowable;
|
||||
|
||||
import dev.caskeleton.adapter.outbound.mongo.advanced.MongoAdvancedCapabilityFlags;
|
||||
import dev.caskeleton.adapter.outbound.mongo.advanced.MongoAdvancedCapabilityGuard;
|
||||
@@ -19,10 +20,25 @@ import dev.caskeleton.adapter.outbound.mongo.imperative.atomic.AtomicFilter;
|
||||
import dev.caskeleton.adapter.outbound.mongo.migration.MongoMigrationCheckpoint;
|
||||
import dev.caskeleton.adapter.outbound.mongo.schema.manifest.MongoCollectionManifest;
|
||||
import dev.caskeleton.adapter.outbound.mongo.schema.manifest.MongoIndexManifest;
|
||||
import java.io.IOException;
|
||||
import java.time.Clock;
|
||||
import java.time.Duration;
|
||||
import java.time.Instant;
|
||||
import java.time.ZoneId;
|
||||
import java.time.ZoneOffset;
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
import java.util.Optional;
|
||||
import java.util.Set;
|
||||
import java.util.concurrent.Callable;
|
||||
import java.util.concurrent.ExecutorService;
|
||||
import java.util.concurrent.Executors;
|
||||
import java.util.concurrent.Future;
|
||||
import java.util.concurrent.atomic.AtomicInteger;
|
||||
import java.util.concurrent.atomic.AtomicReference;
|
||||
import java.util.function.Function;
|
||||
import java.util.stream.IntStream;
|
||||
import org.junit.jupiter.api.DisplayName;
|
||||
import org.junit.jupiter.api.Tag;
|
||||
import org.junit.jupiter.api.Test;
|
||||
|
||||
@@ -37,6 +53,8 @@ class MongoTenancyContractTest {
|
||||
MongoAdvancedCapabilityFlags.allDisabled()
|
||||
.withEnabled(MongoCapability.DATABASE_PER_TENANT));
|
||||
|
||||
private static final Instant START = Instant.parse("2026-08-13T00:00:00Z");
|
||||
|
||||
@Test
|
||||
void missingTenantContextFailsClosed() {
|
||||
assertThatThrownBy(() -> injector.apply(Optional.empty(), AtomicFilter.id("o-1")))
|
||||
@@ -116,29 +134,99 @@ class MongoTenancyContractTest {
|
||||
|
||||
@Test
|
||||
void registryEnforcesMaximumActiveClients() {
|
||||
// given
|
||||
MongoTenantClientRegistry registry = new MongoTenantClientRegistry(TENANCY, 2);
|
||||
Instant now = Instant.parse("2026-08-13T00:00:00Z");
|
||||
registry.acquire("t1");
|
||||
registry.acquire("t2");
|
||||
|
||||
registry.acquire("t1", now);
|
||||
registry.acquire("t2", now);
|
||||
// when
|
||||
Throwable refused = catchThrowable(() -> registry.acquire("t3"));
|
||||
|
||||
assertThatThrownBy(() -> registry.acquire("t3", now))
|
||||
.isInstanceOf(MongoOperationRejectedException.class);
|
||||
assertThat(registry.activeCount()).isEqualTo(2);
|
||||
// then
|
||||
assertThat(List.of(refused instanceof MongoOperationRejectedException, registry.activeCount()))
|
||||
.containsExactly(true, 2);
|
||||
}
|
||||
|
||||
@Test
|
||||
void anIdleClientIsEvictedToMakeRoom() {
|
||||
MongoTenantClientRegistry registry =
|
||||
new MongoTenantClientRegistry(TENANCY, 1, Duration.ofMinutes(5));
|
||||
Instant start = Instant.parse("2026-08-13T00:00:00Z");
|
||||
// given
|
||||
MovingClock clock = new MovingClock(START);
|
||||
MongoTenantClientRegistry registry = registry(1, tenant -> () -> {}, clock);
|
||||
registry.acquire("t1").close();
|
||||
clock.advance(Duration.ofMinutes(10));
|
||||
|
||||
registry.acquire("t1", start);
|
||||
registry.acquire("t2", start.plus(Duration.ofMinutes(10)));
|
||||
// when
|
||||
registry.acquire("t2");
|
||||
|
||||
// then
|
||||
assertThat(registry.activeTenants()).containsExactly("t2");
|
||||
}
|
||||
|
||||
@Test
|
||||
void aClientIsIdleFromTheEndOfItsLastUseNotItsStart() {
|
||||
// given
|
||||
MovingClock clock = new MovingClock(START);
|
||||
MongoTenantClientRegistry registry = registry(1, tenant -> () -> {}, clock);
|
||||
MongoTenantClientRegistry.Lease held = registry.acquire("t1");
|
||||
clock.advance(Duration.ofMinutes(30));
|
||||
held.close();
|
||||
clock.advance(Duration.ofMinutes(1));
|
||||
|
||||
// when
|
||||
Throwable refused = catchThrowable(() -> registry.acquire("t2"));
|
||||
|
||||
// then
|
||||
assertThat(List.of(refused instanceof MongoOperationRejectedException, registry.openLeases()))
|
||||
.as("held for thirty minutes, it was in use all that time and is not idle yet")
|
||||
.containsExactly(true, 0);
|
||||
}
|
||||
|
||||
@Test
|
||||
void anUnclosedLeaseIsCounted() {
|
||||
// given
|
||||
MongoTenantClientRegistry registry = new MongoTenantClientRegistry(TENANCY, 2);
|
||||
|
||||
// when
|
||||
var unused = registry.acquire("t1");
|
||||
|
||||
// then
|
||||
assertThat(registry.openLeases()).isEqualTo(1);
|
||||
}
|
||||
|
||||
private static MongoTenantClientRegistry registry(
|
||||
int cap, Function<String, AutoCloseable> clients, Clock clock) {
|
||||
return new MongoTenantClientRegistry(TENANCY, cap, Duration.ofMinutes(5), clients, clock);
|
||||
}
|
||||
|
||||
/** A clock a test moves by hand: one time source for acquiring, closing and eviction. */
|
||||
private static final class MovingClock extends Clock {
|
||||
|
||||
private final AtomicReference<Instant> now;
|
||||
|
||||
private MovingClock(Instant start) {
|
||||
this.now = new AtomicReference<>(start);
|
||||
}
|
||||
|
||||
private void advance(Duration by) {
|
||||
now.updateAndGet(instant -> instant.plus(by));
|
||||
}
|
||||
|
||||
@Override
|
||||
public ZoneId getZone() {
|
||||
return ZoneOffset.UTC;
|
||||
}
|
||||
|
||||
@Override
|
||||
public Clock withZone(ZoneId zone) {
|
||||
return this;
|
||||
}
|
||||
|
||||
@Override
|
||||
public Instant instant() {
|
||||
return now.get();
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
void aTenantDatabaseCannotServeBeforeItsSchemaIsValidated() {
|
||||
MongoTenantLifecyclePolicy policy = MongoTenantLifecyclePolicy.standard();
|
||||
@@ -172,100 +260,112 @@ class MongoTenancyContractTest {
|
||||
}
|
||||
|
||||
@Test
|
||||
@org.junit.jupiter.api.DisplayName("the same tenant is created once under concurrency")
|
||||
void theSameTenantIsCreatedOnce() throws Exception {
|
||||
java.util.concurrent.atomic.AtomicInteger created =
|
||||
new java.util.concurrent.atomic.AtomicInteger();
|
||||
void theSameTenantIsCreatedOnceUnderConcurrency() throws Exception {
|
||||
// given
|
||||
AtomicInteger created = new AtomicInteger();
|
||||
MongoTenantClientRegistry registry =
|
||||
new MongoTenantClientRegistry(
|
||||
TENANCY,
|
||||
registry(
|
||||
8,
|
||||
Duration.ofMinutes(10),
|
||||
tenant -> {
|
||||
created.incrementAndGet();
|
||||
return () -> {};
|
||||
});
|
||||
java.util.concurrent.ExecutorService pool =
|
||||
java.util.concurrent.Executors.newFixedThreadPool(8);
|
||||
},
|
||||
new MovingClock(START));
|
||||
List<Callable<Void>> work =
|
||||
IntStream.range(0, 50)
|
||||
.<Callable<Void>>mapToObj(
|
||||
index ->
|
||||
() -> {
|
||||
registry.acquire("t1").close();
|
||||
return null;
|
||||
})
|
||||
.toList();
|
||||
ExecutorService pool = Executors.newFixedThreadPool(8);
|
||||
|
||||
// when
|
||||
try {
|
||||
java.util.List<java.util.concurrent.Callable<Void>> work =
|
||||
java.util.stream.IntStream.range(0, 50)
|
||||
.<java.util.concurrent.Callable<Void>>mapToObj(
|
||||
index ->
|
||||
() -> {
|
||||
registry.acquire("t1", Instant.parse("2026-08-13T00:00:00Z"));
|
||||
return null;
|
||||
})
|
||||
.toList();
|
||||
for (java.util.concurrent.Future<Void> future : pool.invokeAll(work)) {
|
||||
for (Future<Void> future : pool.invokeAll(work)) {
|
||||
future.get();
|
||||
}
|
||||
} finally {
|
||||
pool.shutdownNow();
|
||||
}
|
||||
|
||||
assertThat(created)
|
||||
// then
|
||||
assertThat(List.of(created.get(), registry.activeCount()))
|
||||
.as("a per-tenant client carries a pool and monitoring threads; a second one doubles both")
|
||||
.hasValue(1);
|
||||
assertThat(registry.activeCount()).isEqualTo(1);
|
||||
.containsExactly(1, 1);
|
||||
}
|
||||
|
||||
@Test
|
||||
@org.junit.jupiter.api.DisplayName("acquire(tenant)는 주입된 clock으로 idle 여부를 판단한다")
|
||||
void anAcquireWithoutAnInstantReadsTheInjectedClock() {
|
||||
void aClientStillHeldIsNotEvictedHoweverLongAgoItWasAcquired() {
|
||||
// given
|
||||
java.util.List<String> closed = new java.util.ArrayList<>();
|
||||
java.util.concurrent.atomic.AtomicReference<Instant> now =
|
||||
new java.util.concurrent.atomic.AtomicReference<>(Instant.parse("2026-08-13T00:00:00Z"));
|
||||
java.time.Clock clock =
|
||||
new java.time.Clock() {
|
||||
@Override
|
||||
public java.time.ZoneId getZone() {
|
||||
return java.time.ZoneOffset.UTC;
|
||||
}
|
||||
List<String> closed = new ArrayList<>();
|
||||
MovingClock clock = new MovingClock(START);
|
||||
MongoTenantClientRegistry registry = registry(1, tenant -> () -> closed.add(tenant), clock);
|
||||
MongoTenantClientRegistry.Lease longRunning = registry.acquire("t1");
|
||||
clock.advance(Duration.ofMinutes(10));
|
||||
|
||||
@Override
|
||||
public java.time.Clock withZone(java.time.ZoneId zone) {
|
||||
return this;
|
||||
}
|
||||
// when
|
||||
Throwable refused = catchThrowable(() -> registry.acquire("t2"));
|
||||
|
||||
@Override
|
||||
public Instant instant() {
|
||||
return now.get();
|
||||
}
|
||||
};
|
||||
// then
|
||||
assertThat(List.of(refused instanceof MongoOperationRejectedException, closed))
|
||||
.as(
|
||||
"t1 is in use by %s; closing it would pull the pool out from under that work",
|
||||
longRunning)
|
||||
.containsExactly(true, List.of());
|
||||
}
|
||||
|
||||
@Test
|
||||
void anotherTenantsClientThatFailsToCloseDoesNotFailThisAcquire() {
|
||||
// given
|
||||
MovingClock clock = new MovingClock(START);
|
||||
MongoTenantClientRegistry registry =
|
||||
new MongoTenantClientRegistry(
|
||||
TENANCY, 1, Duration.ofMinutes(5), tenant -> () -> closed.add(tenant), clock);
|
||||
registry.acquire("t1");
|
||||
now.set(now.get().plus(Duration.ofMinutes(10)));
|
||||
registry(
|
||||
1,
|
||||
tenant ->
|
||||
() -> {
|
||||
if (tenant.equals("t1")) {
|
||||
throw new IOException("t1's pool would not shut down");
|
||||
}
|
||||
},
|
||||
clock);
|
||||
registry.acquire("t1").close();
|
||||
clock.advance(Duration.ofMinutes(10));
|
||||
|
||||
// when
|
||||
MongoTenantClientRegistry.Lease admitted = registry.acquire("t2");
|
||||
|
||||
// then
|
||||
assertThat(
|
||||
List.of(
|
||||
admitted.client() != null,
|
||||
registry.activeTenants(),
|
||||
registry.evictionCloseFailures()))
|
||||
.containsExactly(true, Set.of("t2"), 1L);
|
||||
}
|
||||
|
||||
@Test
|
||||
void evictionClosesTheClientItEvicts() {
|
||||
// given
|
||||
List<String> closed = new ArrayList<>();
|
||||
MovingClock clock = new MovingClock(START);
|
||||
MongoTenantClientRegistry registry = registry(1, tenant -> () -> closed.add(tenant), clock);
|
||||
registry.acquire("t1").close();
|
||||
clock.advance(Duration.ofMinutes(10));
|
||||
|
||||
// when
|
||||
registry.acquire("t2");
|
||||
|
||||
// then
|
||||
assertThat(closed).containsExactly("t1");
|
||||
}
|
||||
|
||||
@Test
|
||||
@org.junit.jupiter.api.DisplayName("eviction closes the client it evicts")
|
||||
void evictionClosesTheClient() {
|
||||
java.util.List<String> closed = new java.util.ArrayList<>();
|
||||
MongoTenantClientRegistry registry =
|
||||
new MongoTenantClientRegistry(
|
||||
TENANCY, 1, Duration.ofMinutes(5), tenant -> () -> closed.add(tenant));
|
||||
Instant start = Instant.parse("2026-08-13T00:00:00Z");
|
||||
|
||||
registry.acquire("t1", start);
|
||||
registry.acquire("t2", start.plus(Duration.ofMinutes(10)));
|
||||
|
||||
assertThat(closed)
|
||||
.as("freeing the slot without closing the client keeps the connection pool it counted")
|
||||
.containsExactly("t1");
|
||||
}
|
||||
|
||||
@Test
|
||||
@org.junit.jupiter.api.DisplayName("a coordinator resumes from its durable store after a restart")
|
||||
@DisplayName("a coordinator resumes from its durable store after a restart")
|
||||
void aCoordinatorResumesFromItsDurableStore() {
|
||||
RecordingCheckpointStore store = new RecordingCheckpointStore();
|
||||
MongoTenantMigrationCoordinator first =
|
||||
|
||||
+157
@@ -3,11 +3,31 @@ package dev.caskeleton.adapter.outbound.mongo.aggregation;
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
import static org.assertj.core.api.Assertions.assertThatCode;
|
||||
import static org.assertj.core.api.Assertions.assertThatThrownBy;
|
||||
import static org.assertj.core.api.Assertions.catchThrowable;
|
||||
import static org.mockito.ArgumentMatchers.any;
|
||||
import static org.mockito.ArgumentMatchers.eq;
|
||||
import static org.mockito.Mockito.mock;
|
||||
import static org.mockito.Mockito.when;
|
||||
|
||||
import com.mongodb.MongoClientSettings;
|
||||
import dev.caskeleton.adapter.outbound.mongo.api.MongoOperationContext;
|
||||
import dev.caskeleton.adapter.outbound.mongo.api.consistency.MongoConsistencyProfile;
|
||||
import dev.caskeleton.adapter.outbound.mongo.api.error.MongoOperationRejectedException;
|
||||
import dev.caskeleton.adapter.outbound.mongo.query.budget.MongoBudgetEnforcer;
|
||||
import dev.caskeleton.adapter.outbound.mongo.query.budget.MongoBudgetPolicyRegistry;
|
||||
import dev.caskeleton.adapter.outbound.mongo.query.budget.MongoOperationBudget;
|
||||
import java.time.Duration;
|
||||
import java.util.List;
|
||||
import java.util.Set;
|
||||
import org.bson.Document;
|
||||
import org.bson.conversions.Bson;
|
||||
import org.junit.jupiter.api.Tag;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.mockito.ArgumentCaptor;
|
||||
import org.springframework.data.mongodb.core.MongoOperations;
|
||||
import org.springframework.data.mongodb.core.aggregation.Aggregation;
|
||||
import org.springframework.data.mongodb.core.convert.MongoConverter;
|
||||
import org.springframework.data.mongodb.core.query.Criteria;
|
||||
|
||||
/** Design §17 — write stages are D4, and stage cost is graded rather than assumed. */
|
||||
@Tag("mongodb-contract")
|
||||
@@ -105,4 +125,141 @@ class PolicyAwareMongoAggregationExecutorTest {
|
||||
assertThat(MongoAggregationProfile.stableRead().allowDiskUse()).isFalse();
|
||||
assertThat(MongoAggregationProfile.stableRead().strictMapping()).isTrue();
|
||||
}
|
||||
|
||||
private static final MongoOperationContext CONTEXT =
|
||||
MongoOperationContext.of(
|
||||
"orders.summary",
|
||||
"primary",
|
||||
"orders",
|
||||
MongoConsistencyProfile.PRIMARY_MAJORITY,
|
||||
Duration.ofSeconds(2));
|
||||
|
||||
private static final MongoAggregationPlan OPEN_ORDERS =
|
||||
MongoAggregationPlan.builder()
|
||||
.stage("$match", Aggregation.match(Criteria.where("status").is("OPEN")))
|
||||
.build();
|
||||
|
||||
@Test
|
||||
void thePipelineEndsWithALimitOnePastTheCountBudget() {
|
||||
// given
|
||||
ArgumentCaptor<Aggregation> sent = ArgumentCaptor.forClass(Aggregation.class);
|
||||
MongoOperations operations = returning(List.of(), sent);
|
||||
|
||||
// when
|
||||
var unused =
|
||||
executorOver(operations, budget(3, 1_048_576L))
|
||||
.execute(
|
||||
CONTEXT,
|
||||
MongoAggregationProfile.stableRead(),
|
||||
OPEN_ORDERS,
|
||||
"orders",
|
||||
Document.class);
|
||||
|
||||
// then
|
||||
List<Document> pipeline = sent.getValue().toPipeline(Aggregation.DEFAULT_CONTEXT);
|
||||
assertThat(pipeline.get(pipeline.size() - 1)).isEqualTo(new Document("$limit", 4L));
|
||||
}
|
||||
|
||||
@Test
|
||||
void moreDocumentsThanTheCountBudgetAreRefused() {
|
||||
// given
|
||||
MongoOperations operations =
|
||||
returning(
|
||||
List.of(order(1), order(2), order(3), order(4)),
|
||||
ArgumentCaptor.forClass(Aggregation.class));
|
||||
|
||||
// when
|
||||
Throwable refused =
|
||||
catchThrowable(
|
||||
() ->
|
||||
executorOver(operations, budget(3, 1_048_576L))
|
||||
.execute(
|
||||
CONTEXT,
|
||||
MongoAggregationProfile.stableRead(),
|
||||
OPEN_ORDERS,
|
||||
"orders",
|
||||
Document.class));
|
||||
|
||||
// then
|
||||
assertThat(refused)
|
||||
.isInstanceOf(MongoOperationRejectedException.class)
|
||||
.hasMessageContaining("more than the budget of 3 documents");
|
||||
}
|
||||
|
||||
@Test
|
||||
void documentsLargerTogetherThanTheByteBudgetAreRefused() {
|
||||
// given
|
||||
// Each order encodes to well over 40 bytes: two of them pass any 80-byte budget.
|
||||
MongoOperations operations =
|
||||
returning(List.of(order(1), order(2)), ArgumentCaptor.forClass(Aggregation.class));
|
||||
|
||||
// when
|
||||
Throwable refused =
|
||||
catchThrowable(
|
||||
() ->
|
||||
executorOver(operations, budget(10, 80L))
|
||||
.execute(
|
||||
CONTEXT,
|
||||
MongoAggregationProfile.stableRead(),
|
||||
OPEN_ORDERS,
|
||||
"orders",
|
||||
Document.class));
|
||||
|
||||
// then
|
||||
assertThat(refused)
|
||||
.isInstanceOf(MongoOperationRejectedException.class)
|
||||
.hasMessageContaining("more than the budget of 80 bytes");
|
||||
}
|
||||
|
||||
@Test
|
||||
void resultsInsideBothBudgetsAreReturnedInCursorOrder() {
|
||||
// given
|
||||
MongoOperations operations =
|
||||
returning(List.of(order(1), order(2)), ArgumentCaptor.forClass(Aggregation.class));
|
||||
|
||||
// when
|
||||
List<Document> results =
|
||||
executorOver(operations, budget(10, 1_048_576L))
|
||||
.execute(
|
||||
CONTEXT,
|
||||
MongoAggregationProfile.stableRead(),
|
||||
OPEN_ORDERS,
|
||||
"orders",
|
||||
Document.class);
|
||||
|
||||
// then
|
||||
assertThat(results).containsExactly(order(1), order(2));
|
||||
}
|
||||
|
||||
private static MongoOperationBudget budget(int maxResults, long maxResultBytes) {
|
||||
return new MongoOperationBudget(maxResults, maxResultBytes, 500L, 50);
|
||||
}
|
||||
|
||||
private static PolicyAwareMongoAggregationExecutor executorOver(
|
||||
MongoOperations operations, MongoOperationBudget budget) {
|
||||
return new PolicyAwareMongoAggregationExecutor(
|
||||
operations,
|
||||
MongoBudgetPolicyRegistry.builder().register("orders.summary", budget).build(),
|
||||
new MongoBudgetEnforcer());
|
||||
}
|
||||
|
||||
/** The driver boundary: a cursor over {@code documents}, mapped as the converter would. */
|
||||
private static MongoOperations returning(
|
||||
List<Document> documents, ArgumentCaptor<Aggregation> sent) {
|
||||
MongoOperations operations = mock(MongoOperations.class);
|
||||
MongoConverter converter = mock(MongoConverter.class);
|
||||
when(operations.getConverter()).thenReturn(converter);
|
||||
when(converter.getCodecRegistry()).thenReturn(MongoClientSettings.getDefaultCodecRegistry());
|
||||
when(converter.read(eq(Document.class), any(Bson.class)))
|
||||
.thenAnswer(invocation -> invocation.getArgument(1));
|
||||
when(operations.aggregateStream(sent.capture(), eq("orders"), eq(Document.class)))
|
||||
.thenAnswer(invocation -> documents.stream());
|
||||
return operations;
|
||||
}
|
||||
|
||||
private static Document order(int number) {
|
||||
return new Document("_id", "order-" + number)
|
||||
.append("status", "OPEN")
|
||||
.append("customer", "customer-" + number);
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user