This commit is contained in:
+2
-1
@@ -171,6 +171,7 @@ APP_CACHE_REDIS_COMMAND_TIMEOUT=200ms
|
||||
APP_CACHE_REDIS_POSITIVE_SOFT_TTL=30s
|
||||
APP_CACHE_REDIS_MINIMUM_HARD_TTL=1s
|
||||
APP_CACHE_REDIS_SEMANTIC_REGION=default
|
||||
APP_CACHE_REDIS_MAXIMUM_VALUE_BYTES=1048576
|
||||
APP_MESSAGING_BROKER=kafka
|
||||
APP_MESSAGING_PRODUCER_ID=
|
||||
APP_MESSAGING_KAFKA_BROKERS=
|
||||
@@ -293,7 +294,7 @@ APP_NOTIFICATION_PLATFORM_SMTP_ENABLED=false
|
||||
APP_NOTIFICATION_PLATFORM_SMTP_PRIMARY=true
|
||||
APP_NOTIFICATION_PLATFORM_SMTP_ENVIRONMENT=local
|
||||
APP_NOTIFICATION_PLATFORM_SMTP_CREDENTIAL_PROFILE=default
|
||||
APP_NOTIFICATION_PLATFORM_SMTP_TIMEOUT=10s
|
||||
APP_NOTIFICATION_PLATFORM_SMTP_TIMEOUT=30s
|
||||
APP_NOTIFICATION_PLATFORM_SMTP_MAX_CONCURRENCY=4
|
||||
APP_NOTIFICATION_PLATFORM_SMTP_RATE_PER_SECOND=10
|
||||
APP_NOTIFICATION_PLATFORM_SMTP_SENDER_IDENTITY=no-reply@example.invalid
|
||||
|
||||
@@ -366,6 +366,15 @@
|
||||
<Method name="abortAtDeadline"/>
|
||||
</Match>
|
||||
|
||||
<!-- DLS_DEAD_LOCAL_STORE, the same disagreement: the SMTP send's queue deadline discards the
|
||||
stage `thenRun` returns into `unused`, because an abandoned send reports through its own
|
||||
result, not through that stage. -->
|
||||
<Match>
|
||||
<Bug pattern="DLS_DEAD_LOCAL_STORE"/>
|
||||
<Class name="dev.caskeleton.adapter.outbound.notification.platform.provider.smtp.SmtpNotificationProviderAdapter$PooledSend"/>
|
||||
<Method name="armQueueDeadline"/>
|
||||
</Match>
|
||||
|
||||
<!-- The servlet error-dispatch controller (re-review 2026-11-03 RR-02-02) must answer for any
|
||||
HTTP method: the original request's method is whatever failed, and the container forwards it
|
||||
unchanged to /error. CSRF protection guards a state change made with ambient credentials;
|
||||
|
||||
@@ -167,8 +167,8 @@ profile에는 `connect`와 `response-header`를 가장 짧은 호출자 예산
|
||||
| `redirect.max-hops` | `0` | redirect를 켜고 hop을 0으로 두면 설정 오류다 |
|
||||
| `redirect.allow-cross-origin` | `false` | 켜면 hop에서 credential을 지운다 |
|
||||
| `request.max-body-bytes` | `0` | 운영에서 필수 |
|
||||
| `request.compression` | `false` | |
|
||||
| `response.max-wire-bytes` | `5242880` | wire 위의 바이트 |
|
||||
| `request.compression` | `false` | 켜면 transport와 상관없이 `Accept-Encoding: gzip, deflate`를 보낸다. 받은 본문은 엔진이 아니라 이 플랫폼이 wire 바이트를 센 뒤 푼다(버퍼 본문과 스트리밍 다운로드 모두) |
|
||||
| `response.max-wire-bytes` | `5242880` | wire 위의 바이트. 압축을 켜도 압축된 크기로 센다 |
|
||||
| `response.max-decoded-bytes` | `10485760` | 디코딩한 뒤의 바이트. 넘을 수 없는 최대는 64 MiB |
|
||||
| `response.allowed-content-types` | JSON + problem+json | 비어 있으면 「아무것이나」 |
|
||||
|
||||
@@ -212,7 +212,7 @@ profile에는 `connect`와 `response-header`를 가장 짧은 호출자 예산
|
||||
| 속성 | 기본값 | 뜻 |
|
||||
|---|---|---|
|
||||
| `proxy.enabled` | `false` | |
|
||||
| `proxy.host` / `proxy.port` / `proxy.type` | — / `0` / `HTTP` | |
|
||||
| `proxy.host` / `proxy.port` / `proxy.type` | — / `0` / `HTTP` | 지금 transport는 모두 HTTP forward proxy만 한다. `type`의 값은 `HTTP` 하나이고, `SOCKS`는 바인딩이 거부한다 |
|
||||
| `proxy.credential-provider` | — | proxy 인증은 대상 인증과 따로다 |
|
||||
| `proxy.import-ambient-no-proxy` | `false` | 환경의 `NO_PROXY`는 검증한 profile을 넓히지 않는다 |
|
||||
| `observability.full-url-recording` | `false` | |
|
||||
|
||||
@@ -50,9 +50,9 @@
|
||||
|
||||
| 상태 | 판정 |
|
||||
|---|---|
|
||||
| 408 | 기한과 예산 안에서 재시도 |
|
||||
| 425 | 첫 시도에서만, 많아야 한 번 재시도 |
|
||||
| 429 | `Retry-After`, 기한, 예산 안에서 재시도 |
|
||||
| 408 | 안전하게 멱등인 연산만 기한과 예산 안에서 재시도. 아니면 결과 모름(upstream이 받은 뒤 답한 것일 수 있다) |
|
||||
| 425 | 안전하게 멱등인 연산만, 첫 시도에서만, 많아야 한 번 재시도. 아니면 결과 모름 |
|
||||
| 429 | 안전하게 멱등인 연산만 `Retry-After`, 기한, 예산 안에서 재시도. 아니면 거부(`notRetryableStatus`). 429는 일정 신호이지 "아무 일도 없었다"는 말이 아니다 |
|
||||
| 401 | 안전하고 재전송 가능한 연산에서만 갱신 후 재전송 한 번. credential은 재시도를 허락받은 뒤에만 갱신한다. 재전송하지 않을 401(시도 한 번, 예산 없음, 기한 없음)은 token cache를 건드리지 않는다 |
|
||||
| 500 | upstream이 일시적이라고 등록했고 **그리고** 연산이 안전하게 멱등일 때만 허락. 아니면 거부 |
|
||||
| 502, 503, 504 | 안전하게 멱등인 연산은 재시도. 아니면 결과 모름 |
|
||||
|
||||
@@ -7,7 +7,13 @@
|
||||
검증한다. 그래서 실패한 다운로드가 호출자가 따져야 하는 반쯤 읽힌 stream이 되지 않는다.
|
||||
|
||||
wire 바이트와 디코딩한 바이트는 따로 제한한다. 압축한 payload는 wire 검사를 통과한 뒤 늘어나기 때문이다. 한도는
|
||||
버퍼링한 뒤가 아니라 읽는 동안 강제한다.
|
||||
버퍼링한 뒤가 아니라 읽는 동안 강제한다. 압축(`Content-Encoding: gzip`·`deflate`)은 엔진이 아니라 응답 reader가
|
||||
푼다. 도착한 바이트를 wire 한도로 세고, 푼 바이트를 조각마다 디코딩 한도로 센다. 엔진이 먼저 풀면 wire 계수가 푼
|
||||
크기를 세게 되어, 잘 압축된 응답이 wire 위에는 없던 크기로 거부됐다(round9 리뷰 2026-11-12, RR9-01-30). 부탁하지 않은
|
||||
압축(`br` 등)과 겹친 coding은 거부한다. 스트리밍 다운로드도 같은 곳(`ContentDecoding`)을 거친다. 호출자는 푼 바이트를
|
||||
읽고, 응답 헤더에는 `Content-Encoding`·`Content-Length`가 없다. 풀기는 호출자가 읽을 때 시작하므로, 본문이 빈 gzip
|
||||
응답(HEAD·204·304)은 빈 본문이다. 어느 transport도 스스로 풀지 않는다. Apache는 압축 처리를 끄고, 실험 transport인
|
||||
Jetty HTTP/3은 시작할 때 classpath에서 등록한 decoder를 지운다.
|
||||
|
||||
## 첫 바이트 경계
|
||||
|
||||
|
||||
@@ -78,7 +78,10 @@ executor는 **응답을 다시 돌려주는 기록**이다: HTTP `Idempotency-Ke
|
||||
|
||||
같은 키 재전송이 무엇을 막는지는 저장소마다 다르다(정본
|
||||
`docs/web/http-contract.md` 2절). Redis 저장소에서 같은 키 재전송은 효과가 있었을 수 있는 작업을 다시 돌리지
|
||||
않는다. PostgreSQL 저장소를 템플릿처럼 쓰기 트랜잭션 하나로 쓰면 실패한 요청의 기록은 롤백되어 다음 같은 키
|
||||
않는다. 단, 그 기록이 Redis에 남아 있는 동안만이다. Sentinel·Cluster failover는 비동기 복제라 primary가 받고
|
||||
replica에 아직 넘기지 못한 기록을 잃을 수 있고, 그러면 같은 키 요청은 처음 보는 키로 다시 실행된다
|
||||
(`docs/runbooks/redis-capability-incident.md` 「Topology and persistence」). 이 창이 업무상 받아들일 수 없는
|
||||
작업은 PostgreSQL 저장소를 쓰거나 업무 데이터의 유일 제약으로 따로 막는다. PostgreSQL 저장소를 템플릿처럼 쓰기 트랜잭션 하나로 쓰면 실패한 요청의 기록은 롤백되어 다음 같은 키
|
||||
요청이 다시 실행한다. 그래서 트랜잭션 밖 효과는 따로 멱등하게 만든다. 첫 시도가 processing lease(기본 30s)보다
|
||||
오래 걸릴 때 보이는 모습도 저장소마다 다르다.
|
||||
|
||||
|
||||
@@ -41,7 +41,10 @@ email을 모든 tenant에 걸쳐 유일하게 만든다. 그러면 tenant A가
|
||||
## 2. tenant마다 database (Experimental)
|
||||
|
||||
`MongoTenantClientRegistry`는 tenant마다 client를 하나 둔다. 개수 상한이 있고 쓰지 않는 client는
|
||||
내보낸다. tenant를 어느 database에 대응시킬지는 도입하는 쪽의 코드가 정한다.
|
||||
내보낸다. tenant를 어느 database에 대응시킬지는 도입하는 쪽의 코드가 정한다. `acquire`는 `Lease`를 돌려주고,
|
||||
그 client는 lease를 모두 닫을 때까지 쓰는 중이다. 쓰는 중인 client는 얼마나 오래전에 받았든 내보내지 않는다.
|
||||
내보낸 client를 닫다가 실패하면 그 tenant 대신 `evictionCloseFailures()`가 센다. 닫기를 부른 다른 tenant의
|
||||
`acquire`는 실패하지 않는다(round9 리뷰 2026-11-12, RR9-01-27).
|
||||
|
||||
Experimental인 이유가 분명하다. 작을 때는 맞고, 클 때는 한계가 없다. tenant database마다 connection,
|
||||
file handle, 모니터링 가짓수가 든다. tenant 20개에서는 잘 돌고 2,000개에서는 쓰러진다. 기능 테스트로는
|
||||
|
||||
@@ -39,9 +39,15 @@ regex는 index를 쓰는 척하는 collection 전체 scan이고, 사용자가
|
||||
| 상한 | 이유 |
|
||||
|---|---|
|
||||
| `maxTimeMS` | 아무도 기다리지 않는 query를 서버가 멈춘다. |
|
||||
| 결과 개수 상한 | 상한 없는 결과는 단계가 몇 개 더 붙은 OOM이다. |
|
||||
| batch 크기 | 왕복 한 번의 메모리를 제한한다. |
|
||||
| 검사한 문서 수 상한 | 빠른 날에는 시간 상한만으로 숨겨지는 index 회귀를 잡는다. |
|
||||
| 결과 개수 상한(`maxResults`) | 상한 없는 결과는 단계가 몇 개 더 붙은 OOM이다. |
|
||||
| 결과 바이트 상한(`maxResultBytes`) | 문서 하나하나가 크면 개수 상한 안에서도 heap이 찬다. |
|
||||
| batch 크기(`cursorBatchSize`) | 왕복 한 번의 메모리를 제한한다. |
|
||||
|
||||
개수·시간·batch는 서버에 보내는 요청에 실린다(`PolicyAwareMongoQueryBuilder.build`가 만든 `Query`, aggregation의
|
||||
`$limit`과 옵션). 바이트는 결과를 읽는 쪽만 셀 수 있다. aggregation은 `PolicyAwareMongoAggregationExecutor`가
|
||||
cursor에서 문서를 하나씩 읽으며 encode한 크기를 더하고 넘으면 거부한다(5절). query builder는 `Query`만 돌려주므로
|
||||
query의 바이트 상한을 집행하는 템플릿 코드는 없다. 그 `Query`로 결과를 읽는 서비스의 adapter가 `maxResultBytes`를
|
||||
직접 센다. 세지 않으면 그 상한은 선언일 뿐이다.
|
||||
|
||||
`MongoBudgetPolicyRegistry`가 예산을 연산 이름에 묶는다. `MongoBudgetEnforcer`가 그것을 적용한다. 요청이
|
||||
예산을 넘으면 실행 전에 `MongoOperationRejectedException`을 던지고, 서버가 상한을 집행하면
|
||||
@@ -86,6 +92,11 @@ regex는 index를 쓰는 척하는 collection 전체 scan이고, 사용자가
|
||||
| high | `$lookup`, `$graphLookup`, `$facet`, index 없는 `$sort` | profile의 명시적 승인이 필요하다. |
|
||||
| forbidden | admin plane 밖의 `$out`·`$merge`, `$function`, `$accumulator` | 거부한다. |
|
||||
|
||||
실행은 예산 둘을 결과가 오는 동안 본다. pipeline 끝에 `$limit(maxResults + 1)`을 붙여 서버가 예산보다 한 개 넘게는
|
||||
만들지 않게 하고, 결과를 cursor에서 한 문서씩 읽으며 개수와 encode한 바이트를 더한다. 둘 중 하나라도 넘으면
|
||||
`MongoOperationRejectedException`(`aggregation.result`)이다. 예전에는 결과를 전부 메모리에 올린 뒤 개수만 셌고 바이트
|
||||
상한은 선언만 있었다(round9 리뷰 2026-11-12, RR9-01-20).
|
||||
|
||||
`allowDiskUse`는 실행 중 flag가 아니라 plan에 선언하는 속성이다. 디스크가 필요한 pipeline은 모양을 검토해야
|
||||
하는 pipeline이다.
|
||||
|
||||
|
||||
@@ -26,7 +26,7 @@ support leaf의 `ModuleDefaultsEnvironmentPostProcessor`가 가장 낮은 우선
|
||||
| 속성 | 환경 변수 | 기본값 | 범위 |
|
||||
|---|---|---|---|
|
||||
| `dispatch.claim-batch-size` | `APP_NOTIFICATION_PLATFORM_CLAIM_BATCH_SIZE` | `50` | 1..1000 |
|
||||
| `dispatch.lease-duration` | `APP_NOTIFICATION_PLATFORM_LEASE_DURATION` | `2m` | 양수, 유한. `poll-interval`과 켜진 provider profile의 `timeout`보다 길어야 기동한다 |
|
||||
| `dispatch.lease-duration` | `APP_NOTIFICATION_PLATFORM_LEASE_DURATION` | `2m` | 양수, 유한. `poll-interval`과 켜진 provider profile의 `timeout`보다 길어야 기동한다. SMTP profile은 `timeout`의 두 배보다 길어야 한다(「SMTP relay」) |
|
||||
| `dispatch.poll-interval` | `APP_NOTIFICATION_PLATFORM_POLL_INTERVAL` | `1s` | 양수, 유한 |
|
||||
| `dispatch.max-global-concurrency` | `APP_NOTIFICATION_PLATFORM_MAX_CONCURRENCY` | `64` | 양수 |
|
||||
| `dispatch.max-additional-attempts` | `APP_NOTIFICATION_PLATFORM_MAX_ADDITIONAL_ATTEMPTS` | `4` | 0 이상 |
|
||||
@@ -146,7 +146,7 @@ Spring이 이미 그것을 가지고 있고, 두 번째 표기는 맞춰 둘 것
|
||||
| `providers.smtp.primary-for-channel` | `APP_NOTIFICATION_PLATFORM_SMTP_PRIMARY` | `true` | boolean. 채널마다 primary는 정확히 하나 |
|
||||
| `providers.smtp.environment` | `APP_NOTIFICATION_PLATFORM_SMTP_ENVIRONMENT` | `local` | 켜면 필요하다 |
|
||||
| `providers.smtp.credential-profile` | `APP_NOTIFICATION_PLATFORM_SMTP_CREDENTIAL_PROFILE` | `default` | `SecretMaterialProvider`로 푼다. 값을 직접 적지 않는다 |
|
||||
| `providers.smtp.timeout` | `APP_NOTIFICATION_PLATFORM_SMTP_TIMEOUT` | `10s` | 양수, 유한 |
|
||||
| `providers.smtp.timeout` | `APP_NOTIFICATION_PLATFORM_SMTP_TIMEOUT` | `30s` | 양수, 유한. `mail.smtp.connectiontimeout`에 `mail.smtp.timeout`·`mail.smtp.writetimeout` 중 긴 것을 더한 값보다 길어야 기동한다 |
|
||||
| `providers.smtp.max-concurrency` | `APP_NOTIFICATION_PLATFORM_SMTP_MAX_CONCURRENCY` | `4` | 양수 |
|
||||
| `providers.smtp.rate-per-second` | `APP_NOTIFICATION_PLATFORM_SMTP_RATE_PER_SECOND` | `10` | 양수 |
|
||||
|
||||
@@ -155,11 +155,40 @@ Spring이 이미 그것을 가지고 있고, 두 번째 표기는 맞춰 둘 것
|
||||
| `smtp.sender-identity` | `APP_NOTIFICATION_PLATFORM_SMTP_SENDER_IDENTITY` | `no-reply@example.invalid` | 주소 |
|
||||
| `smtp.dispatch-concurrency` | `APP_NOTIFICATION_PLATFORM_SMTP_DISPATCH_CONCURRENCY` | `4` | 양수 |
|
||||
|
||||
`smtp.dispatch-concurrency`는 SMTP 발송 pool의 thread 수다. 첫 발송부터 그만큼 함께 돌고, 대기열은 그 네 배다. 모든 SMTP
|
||||
profile이 이 pool 하나를 같이 쓴다. 켜진 SMTP profile들의 `max-concurrency` 합이 thread와 대기열을 합친 것보다 크면 SMTP
|
||||
조립이 기동 때 실패한다. pool이 가득 차서 거절한 발송은 relay에 닿지 않은 것으로 기록되어 다시 시도된다(결과 불명이
|
||||
아니다).
|
||||
|
||||
SMTP 전송 설정의 소유자는 Spring Mail 하나다. 실제 `JavaMailSenderImpl`은 필수 STARTTLS
|
||||
(`mail.smtp.starttls.enable=true`와 `mail.smtp.starttls.required=true`)나 암묵적 TLS(`mail.smtp.ssl.enable=true`)를
|
||||
써야 한다. `mail.smtp.connectiontimeout`, `mail.smtp.timeout`, `mail.smtp.writetimeout`도 양수여야 한다. 이 속성이
|
||||
없거나 틀리면 SMTP 조립이 트래픽을 받기 전에 실패한다.
|
||||
|
||||
이 socket timeout은 SMTP 명령 하나씩을 묶을 뿐 대화 전체를 묶지 않는다. 발송 한 번 전체는 provider profile의 `timeout`이
|
||||
두 번 묶는다(2026-11-13).
|
||||
|
||||
- pool 대기: 발송이 send pool에서 `timeout`이 지나도록 시작하지 못하면 버려지고 relay에 가지 않는다. relay에 닿지 않은
|
||||
것(`PROVIDER_UNAVAILABLE`, `CAPACITY_REJECTED`)으로 기록되어 다시 시도된다.
|
||||
- 발송: pool thread가 잡은 때부터 `timeout`이 지나면 결과 불명(`PROVIDER_RESPONSE_LOST`)으로 기록되고 자동으로 다시 보내지
|
||||
않는다. DATA가 이미 넘어갔을 수 있기 때문이다.
|
||||
|
||||
그래서 한 발송이 delivery lease를 쓰는 시간은 많아야 `timeout`의 두 배다. 기동 때 두 가지를 본다.
|
||||
|
||||
- `dispatch.lease-duration`은 SMTP `timeout`의 두 배보다 길어야 한다. 모든 provider에 거는 조건(lease가 `timeout`보다
|
||||
길다)보다 엄격하다. 짧으면 생기는 일은 위 「발송」 절의 lease 문단과 같다. 기본값(lease 2분, timeout 30초)은 맞는다.
|
||||
- profile `timeout`은 connect timeout에 명령 timeout 하나를 더한 값보다 길어야 한다. 그보다 짧으면 인사말을 주지 않는
|
||||
relay처럼 아무것도 보내지 않은 실패가, socket timeout이 알려 주기 전에 결과 불명으로 기록된다. 기본값(30초, connect 5초 +
|
||||
명령 10초)은 맞는다.
|
||||
|
||||
`dispatch.shutdown-grace`(기본 20초)가 `timeout`의 두 배보다 짧으면, 종료하는 동안 끝나지 않은 발송은 attempt가
|
||||
`DISPATCHING`으로 남고, 위 「발송」 절의 lease 문단대로 운영자 확인으로 넘어간다(종료된 발송은 결과를 기록하지 못한다).
|
||||
느린 relay에서 이것 없이 끝내려면 pod 종료 유예와 함께 grace를 늘린다.
|
||||
|
||||
send pool(`notificationSmtpExecutor` bean)을 바꿔 끼우면 발송을 부른 thread에서 돌리지 않는 executor여야 한다
|
||||
(`CallerRunsPolicy`나 직접 실행 executor는 안 된다). 부른 thread에서 돌면 두 시한이 dispatch thread를 풀지 못한다.
|
||||
`ThreadPoolExecutor`가 아니면 위의 pool 합 검사도 하지 않는다.
|
||||
|
||||
환경 변수 계약은 Spring 그대로다. `SPRING_MAIL_HOST`, `SPRING_MAIL_PORT`, 선택인 `SPRING_MAIL_USERNAME`과 secret
|
||||
`SPRING_MAIL_PASSWORD`, 그리고 `.env.example`에 있는 `SPRING_MAIL_PROPERTIES_MAIL_SMTP_*` TLS·시간 상한 키다. Spring
|
||||
Boot 자신이 `spring.mail.*`를 가지고 읽으므로 이름을 Spring 그대로 둔다. 알림 모듈은 같은 relay에 두 번째 별칭을
|
||||
|
||||
@@ -2688,10 +2688,8 @@ env_keys:
|
||||
required_test: cache-contract:redis-semantic-region-bound
|
||||
|
||||
- name: APP_CACHE_REDIS_MAXIMUM_VALUE_BYTES
|
||||
# 폐기 2026-08-10: 이 속성을 바인딩하던 settings class를 옛 Redis 세대와 함께 지워서 이 키는 어디에도
|
||||
# 닿지 않는다. 아직 이 키를 쓰는 배포에 무엇이 대신하는지 알려 주려고 행을 지우지 않고 둔다.
|
||||
deprecated_orphaned: true
|
||||
removal_deadline: 2026-11-30
|
||||
# 2026-08-10에 옛 Redis 세대와 함께 폐기했다가, 2026-11-13 cache-redis 모듈 기본 설정이
|
||||
# app.capabilities.cache.maximum-value-bytes에 다시 묶었다. 상한을 넘는 값은 기록하지 않는다(round9 리뷰 RR9-01-18).
|
||||
type: int
|
||||
default: 1048576
|
||||
allowed_values: null
|
||||
@@ -4780,9 +4778,10 @@ env_keys:
|
||||
required_test: adapter-contract:notification-smtp-provider-assembled
|
||||
|
||||
- name: APP_NOTIFICATION_PLATFORM_SMTP_TIMEOUT
|
||||
# source: NTF-INT-001 — 이 profile의 시도마다 provider timeout.
|
||||
# source: NTF-INT-001 — 이 profile의 발송 한 번 상한: pool 대기와 발송이 각각 이 시간을 넘지 않는다.
|
||||
# 2026-11-13 10s에서 30s로 올렸다. 기동 때 mail.smtp.connectiontimeout + 명령 timeout보다 길어야 한다.
|
||||
type: duration
|
||||
default: 10s
|
||||
default: 30s
|
||||
allowed_values: null
|
||||
classification: public-config
|
||||
required: false
|
||||
|
||||
@@ -19,7 +19,7 @@ tenant pool 예산을 쓰는 서비스다.
|
||||
| 3 | persistence-jpa의 `V23__durable_operation_lease_fence.sql`(`durable_operation.lease_fence BIGINT NOT NULL DEFAULT 0`)이 기본 stream에 더해진다. 같은 번호의 서비스 마이그레이션이 있으면 번호를 옮긴다. `durable_operation`을 건드린 트랜잭션이 열려 있으면 5초 뒤 실패하고 V22에 남는다(다시 배포) | Flyway가 번호 충돌로 기동하지 않는다 | persistence-jpa를 쓰는 모든 서비스 |
|
||||
| 4 | 알림 stream에 `V14__notification_provider_event_open_index.sql`(부분 인덱스 `ix_notification_provider_event_open`)이 더해진다. `notification_provider_event`가 이미 큰 운영 DB는 배포 전에 같은 인덱스를 `CREATE INDEX CONCURRENTLY IF NOT EXISTS ix_notification_provider_event_open ON notification_provider_event (received_at) WHERE projection_status IN ('PENDING', 'FAILED') OR attempt_id IS NULL;`로 먼저 만든다. 미리 만든 인덱스가 실패해 남았으면(`SELECT indisvalid FROM pg_index WHERE indexrelid = 'ix_notification_provider_event_open'::regclass`가 false) `DROP INDEX CONCURRENTLY ix_notification_provider_event_open` 뒤 다시 만든다. `IF NOT EXISTS`는 쓸 수 없는 인덱스도 있는 것으로 보고 넘어간다 | 마이그레이션이 인덱스를 만드는 동안 provider callback 저장이 기다린다(provider가 callback을 다시 보낸다) | 알림 capability를 켠 서비스 |
|
||||
| 5 | `RecipientLeaseStorePort.expiredDispatching`이 `List<RecipientLease>`(버려질 때의 owner·fence·만료 시각)를 돌려주고, `RecipientDeliveryStorePort`에 `requeueAbandoned(lease, nextDispatchAt)`가 생겼다. 직접 구현했으면 상태가 `DISPATCHING`이고 owner·fence가 같고 lease가 늘지 않았을 때만 `READY_TO_DISPATCH`로 바꾼다(`RecipientDeliveryStoreContract` 새 사례). owner가 없거나 fence가 0인 `DISPATCHING` 행(notification V4 전에 남은 행)은 복구가 더는 고르지 않는다. 그런 행은 `SELECT id FROM notification_recipient_delivery WHERE delivery_state = 'DISPATCHING' AND (lease_owner IS NULL OR lease_fence = 0)`로 찾는다. 배포 전에 `UPDATE notification_recipient_delivery SET lease_owner = COALESCE(lease_owner, 'pre-v4'), lease_fence = lease_fence + 1, lease_until = now() - interval '1 day', version = version + 1, updated_at = now() WHERE delivery_state = 'DISPATCHING' AND (lease_owner IS NULL OR lease_fence = 0)`로 owner를 채우고 fence를 올려 복구 패스에 넘긴다. 복구는 attempt를 보고 정한다: attempt가 없으면 다시 보낼 차례로 돌리고, 끝나지 않은 attempt가 있으면 맞추기로 넘긴다(그 뒤는 `docs/notification/configuration-reference.md` 「발송」의 lease 문단). 손으로 `READY_TO_DISPATCH`로 바꾸지 않는다. 발송 중에 멈춘 행이면 같은 알림이 한 번 더 나간다. owner가 없는 행은 stuck lease 지표에도 잡히지 않는다 | 컴파일이 실패한다. 조건 없이 바꾸면 끝났거나 다시 claim된 delivery가 다시 발송된다 | 알림 store를 직접 구현한 서비스 |
|
||||
| 6 | 알림 provider profile의 `timeout`이 `app.notification.platform.dispatch.lease-duration`보다 짧은지 확인한다. 짧은 것은 필요조건일 뿐이다: lease는 claim 때 시작해서 attempt 기록·렌더링·provider 허가 대기도 lease를 쓴다. lease는 timeout보다 넉넉히 길게 둔다. 이 timeout이 provider 호출 전체를 묶는 것은 SES(API 호출 시간 상한)다. SMTP는 `spring.mail` socket timeout이 명령마다 묶을 뿐이라 이 검사만으로는 호출 전체가 lease 안에 끝난다고 말할 수 없다 | 짧지 않으면 기동이 실패한다(`notification provider profile '…': timeout (…) must be shorter than …lease-duration`). 전에는 provider 호출 하나가 lease보다 길어도 기동했다. 그때 생기는 일은 `docs/notification/configuration-reference.md` 「발송」의 lease 문단이다 | 알림 capability를 켠 서비스 |
|
||||
| 6 | 알림 provider profile의 `timeout`이 `app.notification.platform.dispatch.lease-duration`보다 짧은지 확인한다. 짧은 것은 필요조건일 뿐이다: lease는 claim 때 시작해서 attempt 기록·렌더링·provider 허가 대기도 lease를 쓴다. lease는 timeout보다 넉넉히 길게 둔다. 이 timeout은 SES에서는 API 호출 하나의 상한이다. SMTP의 추가 조건은 `docs/upgrade/2026-11-round9-outbound.md` 9행이다 | 짧지 않으면 기동이 실패한다(`notification provider profile '…': timeout (…) must be shorter than …lease-duration`). 전에는 provider 호출 하나가 lease보다 길어도 기동했다. 그때 생기는 일은 `docs/notification/configuration-reference.md` 「발송」의 lease 문단이다 | 알림 capability를 켠 서비스 |
|
||||
| 7 | `app.lock.wait-time`을 설정에서 지운다. 기다릴 시간은 `tryAcquire(key, waitTime, leaseTtl)`의 인자다 | `app.lock` 아래 모르는 키라 기동이 실패한다. 전에도 읽는 곳이 없었다 | `app.lock.wait-time`을 설정한 서비스 |
|
||||
| 8 | `JpaLiveEventReplay.sweepExpired()`를 부르던 코드를 지운다. replay를 켜면 `LiveEventReaper`가 `app.realtime.replay.reaper-interval`(기본 `PT10M`)마다 `retained_until`이 지난 행을 1000행씩 지운다. `app.realtime.replay.*`는 이제 `LiveEventReplaySettings`가 엄격하게 묶는다: 그 아래 모르는 키, 0 이하의 `retention`·`reaper-interval`은 기동을 실패시킨다 | 컴파일이 실패한다. 잘못 쓴 키는 기동이 실패한다. 전에는 지우는 곳이 없어 `live_event_log`가 끝없이 자랐다 | `app.realtime.replay.enabled=true`인 서비스 |
|
||||
| 9 | `IdempotencyReaper`를 직접 만들던 코드는 `new IdempotencyReaper(repository)`로 바꾼다(`Clock` 인자 삭제). 만료 비교는 DB 시계(`now()`)이고 1000행씩 지운다 | 컴파일이 실패한다 | reaper를 직접 조립한 서비스 |
|
||||
|
||||
@@ -0,0 +1,41 @@
|
||||
# 업그레이드 안내 — 2026-11 outbound 발행·발송·캐시·S3·MongoDB·HTTP client
|
||||
|
||||
- 범위: outbox 발행 대기, SMTP 발송 pool, Redis 캐시 시간·크기 상한, S3 조건부 쓰기, MongoDB 집계 예산·tenant
|
||||
client·migration lease, HTTP client 압축·proxy를 고친 변경. 기동 조건과 공개 타입이 바뀐 곳이 있다. 아래 표가 그
|
||||
목록이다.
|
||||
- 근거: 2026-11-12 전체 리뷰(`docs/reviews/2026-11-12-round9-review/`)의 발행·발송·캐시(B10a)와 S3·Mongo·httpclient
|
||||
(B10b) 발견
|
||||
|
||||
## 1. 누가 읽나
|
||||
|
||||
이 템플릿을 `upstream` remote로 두고 merge해 받는 파생 서비스가 읽는다. 특히 SMTP 알림, Redis 캐시, MongoDB
|
||||
집계·tenant별 database, HTTP client의 압축이나 proxy를 쓰는 서비스다.
|
||||
|
||||
## 2. 할 일
|
||||
|
||||
| # | 할 일 | 안 하면 | 해당 서비스 |
|
||||
| --- | --- | --- | --- |
|
||||
| 1 | SMTP provider profile들의 `max-concurrency` 합이 발송 pool이 담는 수(`app.notification.platform.smtp.dispatch-concurrency`개 thread와 그 네 배의 대기열) 이하인지 확인한다. 모든 SMTP profile이 pool 하나를 같이 쓴다. 기본값(profile 하나, 4와 4)은 맞는다 | 넘으면 넘게 만든 profile의 조립이 기동 때 실패한다. 전에는 넘친 발송이 대기열에서 거절되어 결과 불명으로 남았다 | SMTP 알림을 켠 서비스 |
|
||||
| 2 | Redis 캐시의 `command-timeout`이 이제 한 캐시 연산 전체(연결 대기와 응답 전부)의 상한이다. 연결 대기가 길던 배포는 값을 다시 본다 | 상한 안에 못 끝난 연산이 캐시 없음(`DEGRADED_UNAVAILABLE`·`Unavailable`)으로 끝나는 일이 늘 수 있다. 전에는 lane 대기 2초가 더해졌다 | Redis 캐시를 켠 서비스 |
|
||||
| 3 | encode한 값이 1 MiB(`RedisCacheRegionAdapter.DEFAULT_MAXIMUM_VALUE_BYTES`)를 넘는 캐시는 그 값을 기록하지 않는다(`NOT_RECORDED_PROVIDER_POLICY`). 더 큰 값이 필요하면 `app.capabilities.cache.maximum-value-bytes`(`APP_CACHE_REDIS_MAXIMUM_VALUE_BYTES`, 1..16777216)로 상한을 올린다. 2026-08-10에 폐기했던 그 env 키가 다시 묶였다 | 그 값은 매번 source에서 읽힌다 | 큰 값을 캐시하던 서비스 |
|
||||
| 4 | MongoDB `PolicyAwareMongoAggregationExecutor`는 등록 예산의 `maxResultBytes`를 집행하고, pipeline 끝에 `$limit(maxResults + 1)`을 붙인다. 결과가 큰 집계의 예산을 다시 본다. 집계 결과는 template의 converter로만 바뀌어 `AfterLoadEvent`·`AfterConvertEvent`와 entity callback이 돌지 않는다. 그것에 기대던 listener는 돌려받은 타입에서 직접 한다 | 바이트 예산을 넘는 집계가 `MongoOperationRejectedException`(`aggregation.result`)으로 실패한다. 전에는 결과를 전부 메모리에 올렸다 | MongoDB 집계를 쓰는 서비스 |
|
||||
| 5 | experimental `MongoTenantClientRegistry.acquire`가 `Lease`(interface)를 돌려준다. client를 다 쓴 뒤 `lease.close()`(또는 try-with-resources)를 부른다. 쓰는 중인 tenant의 `release`는 거부된다. idle 시간은 마지막 lease를 닫은 때부터 잰다. 열린 lease 수는 `openLeases()`다. 시각을 인자로 받던 `acquire(tenant, Instant)`는 없어졌다. 시계는 생성자의 `Clock` 하나다 | 반환값을 버리는 호출은 컴파일되지만, 닫지 않은 lease의 client는 쓰는 중으로 남아 내보내지지 않고 cap 한 칸을 계속 차지한다. `acquire(tenant, Instant)`를 부른 코드는 컴파일이 실패한다 | tenant별 database를 쓰는 서비스 |
|
||||
| 6 | MongoDB migration lease를 잃은 뒤의 `refresh`는 `MongoOperationRejectedException`(`migration.lock`)을 던진다. `IllegalStateException`을 잡던 코드를 바꾼다. collection을 직접 넘겨 `MongoCollectionMigrationLedger`를 만들던 코드는 셋째 인자로 lock collection(`MongoCollectionMigrationLock.DEFAULT_COLLECTION`)을 넘긴다. 옛 runner의 `recordApplied`는 새 runner가 lease를 잡은 뒤면(checkpoint를 쓰기 전이어도) 거부된다 | 그 예외를 잡지 못한다. 두 인자 생성자는 컴파일이 실패한다 | MongoDB migration runner를 직접 쓰는 서비스 |
|
||||
| 7 | HTTP client profile의 `proxy.type: SOCKS`를 지우거나 HTTP proxy로 바꾼다. `ProxyType`에는 이제 `HTTP`만 있다 | 설정 바인딩이 실패해 기동하지 않는다. 전에는 SOCKS port에 HTTP proxy로 붙어 첫 요청에서 실패했다 | SOCKS proxy를 설정한 서비스 |
|
||||
| 8 | `request.compression: true`인 HTTP client는 transport와 상관없이 `Accept-Encoding: gzip, deflate`를 보내고, 버퍼 본문과 스트리밍 다운로드 모두 이 플랫폼이 푼다(JDK transport는 전에 이 설정을 읽지 않았다). 압축을 켰든 껐든, 그 밖의 `Content-Encoding`(`br` 등)이나 겹친 coding으로 온 응답은 `HttpTargetRejectedException`(`TARGET_REJECTED`, 증거 `RESPONSE_RECEIVED`)으로 거부된다. 본문이 빈 gzip 응답(HEAD·204·304)은 빈 본문이다 | — 동작 변경. `response.max-wire-bytes`가 이제 압축된 크기로 센다. 스트리밍 다운로드도 identity 본문까지 `max-decoded-bytes`로 센다. 실험 transport인 Jetty HTTP/3도 이제 스스로 풀지 않는다 | HTTP client를 쓰는 서비스 |
|
||||
| 9 | SMTP provider profile의 `timeout`(기본이 `10s`에서 `30s`로 바뀌었다)이 발송 한 번을 두 번 묶는다. 발송이 send pool에서 그 시간이 지나도록 시작하지 못하면 relay에 보내지 않고 시작 전 실패(`PROVIDER_UNAVAILABLE`, 재시도)로 기록한다. 시작한 뒤 그 시간이 지나면 결과 불명(`PROVIDER_RESPONSE_LOST`)으로 기록하고 자동으로 다시 보내지 않는다. 값은 `mail.smtp.connectiontimeout`에 `mail.smtp.timeout`·`mail.smtp.writetimeout` 중 긴 것을 더한 값보다 길어야 하고, `dispatch.lease-duration`은 그 두 배보다 길어야 한다. `dispatch.shutdown-grace`가 그 두 배보다 짧으면 종료 중에 끝나지 않은 발송이 결과 불명으로 남는다(`docs/notification/configuration-reference.md` 「SMTP relay」). `SmtpNotificationProviderAdapter`를 직접 만들던 코드는 마지막 인자로 이 timeout을 넘긴다. 직접 만든 `ProviderRuntimeAssembler`는 새 default 메서드 `requireTogether(profiles, deliveryLease)`로 한 family의 켜진 profile 전체에 걸린 검사를 할 수 있다(`NotificationProviderAssembly`가 SERVING일 때 family마다 한 번 부른다). `assemble`을 직접 부르는 코드는 그 전에 `requireTogether`를 부른다 | timeout이 socket timeout 합 이하이거나 lease가 timeout 두 배 이하면 SMTP 조립이 기동 때 실패한다. 상한 안에 끝나지 않는 발송이 결과 불명으로 남는다. 전에는 socket timeout이 명령마다만 묶어, 느린 relay에서 발송이 lease를 넘으면 lease 복구가 아직 도는 발송을 운영자 확인으로 넘겼다. | SMTP 알림을 켠 서비스 |
|
||||
|
||||
## 3. 바뀐 동작 (따라 할 일 없음)
|
||||
|
||||
- outbox relay의 발행 대기는 `publish()`를 부른 때부터 publisher의 `longestAttempt()`까지다. 넘으면 결과 불명(`PUBLISH_WAIT_EXCEEDED`)으로 기록하고
|
||||
다음 claim에서 다시 본다. 직접 만든 `IntegrationEventPublishPort`는 `longestAttempt()`를 실제 최장 시간으로 답한다. 0이나
|
||||
음수를 답하면 그 행은 보내지 않고 `REJECTED_RETRYABLE`(`PUBLISH_BOUND_NOT_POSITIVE`)로 backoff 뒤에 다시 본다.
|
||||
- 이미 interrupt된 relay thread는 그 행을 발행하지 않고 `REJECTED_RETRYABLE`(`PUBLISH_INTERRUPTED_BEFORE_SEND`)로 돌려준다.
|
||||
relay가 그 행을 backoff 뒤로 미뤘다가 발행한다. 기다리는 중의 interrupt는 전처럼 결과 불명(`PUBLISH_WAIT_INTERRUPTED`)이다.
|
||||
- SMTP 발송 pool은 첫 발송부터 `dispatch-concurrency`개 thread로 돈다(전에는 대기열이 찰 때까지 thread 하나였다).
|
||||
pool이 거절한 발송은 relay에 닿지 않은 것(`PROVIDER_UNAVAILABLE`, `CAPACITY_REJECTED`)으로 기록되어 재시도된다.
|
||||
- S3 control 저장소의 조건부 `PUT`이 412를 받으면 key를 다시 읽는다. 이 호출이 쓴 record 그대로면 성공이다(SDK가 응답을
|
||||
잃은 첫 쓰기를 다시 보낸 경우). 다른 내용이나 없음이면 전처럼 충돌이다. 「그대로」는 같은 값이라는 뜻이라, 두 writer가
|
||||
같은 버전에서 같은 record를 쓰면 둘 다 성공으로 본다. 그럴 수 있는 호출자는 record에 자기 토큰을 싣는다.
|
||||
- MongoDB tenant client를 내보내다 닫기에 실패하면 그 tenant 대신 `evictionCloseFailures()`가 센다. 닫기를 부른 다른
|
||||
tenant의 `acquire`는 실패하지 않는다.
|
||||
@@ -125,7 +125,9 @@ Envelope가 아닌 응답은 두 곳뿐이다. fileserver의 프로토콜 예외
|
||||
`IDEMPOTENT_IN_FLIGHT`, lease가 지났으면 500 `IDEMPOTENCY_RECOVERY_REQUIRED`다. 이것은 Redis 저장소의
|
||||
판정이다. PostgreSQL 저장소는 아래처럼 첫 요청이 끝날 때까지 기다린 뒤 판정한다. 새 키로 보내면 서버는
|
||||
다른 요청으로 보고 막지 못한다(재리뷰 2026-11-03 README 4절 결정 RR-02-03, 2026-11-04 9.1절 결정
|
||||
RR-01-11).
|
||||
RR-01-11). Redis 저장소의 판정은 그 기록이 Redis에 남아 있는 동안만이다. Sentinel·Cluster failover는 비동기
|
||||
복제라 replica에 아직 넘기지 못한 기록을 잃을 수 있고, 그러면 같은 키 요청은 처음 보는 키로 다시 실행된다
|
||||
(`docs/manual/05-idempotency.md` 「클라이언트에게 무엇을 약속하나」).
|
||||
- **첫 시도가 lease보다 오래 걸릴 때 — 저장소마다 다르다.** executor는 lease를 늘리지 않으므로 30s를 넘는
|
||||
정상 요청도 lease가 지난다(재리뷰 2026-11-04 RR5-01-05).
|
||||
- Redis: lease가 지난 실행 기록을 다음 요청이 만나면 500으로 답하고 기록은 건드리지 않는다. 첫 시도가
|
||||
|
||||
+8
-1
@@ -91,6 +91,10 @@ final class RuntimeLimitsTestSupport {
|
||||
|
||||
static final AtomicInteger SLOW_STARTED = new AtomicInteger();
|
||||
static final AtomicInteger DETAIL_INVOCATIONS = new AtomicInteger();
|
||||
|
||||
/** How many {@code slowNode} calls ran to their {@code millis} cap without being released. */
|
||||
static final AtomicInteger SLOW_CAPPED = new AtomicInteger();
|
||||
|
||||
static volatile CountDownLatch slowFinished = new CountDownLatch(1);
|
||||
|
||||
/**
|
||||
@@ -114,6 +118,7 @@ final class RuntimeLimitsTestSupport {
|
||||
static void reset() {
|
||||
SLOW_STARTED.set(0);
|
||||
DETAIL_INVOCATIONS.set(0);
|
||||
SLOW_CAPPED.set(0);
|
||||
slowFinished = new CountDownLatch(1);
|
||||
slowExecutionEnded = new CountDownLatch(1);
|
||||
release = new CountDownLatch(1);
|
||||
@@ -153,7 +158,9 @@ final class RuntimeLimitsTestSupport {
|
||||
CountDownLatch release = SlowWork.release;
|
||||
try {
|
||||
// Released by the test, or the cap ran out; either way the resolver returns normally.
|
||||
var unused = release.await(millis, TimeUnit.MILLISECONDS);
|
||||
if (!release.await(millis, TimeUnit.MILLISECONDS)) {
|
||||
SlowWork.SLOW_CAPPED.incrementAndGet();
|
||||
}
|
||||
} finally {
|
||||
SlowWork.slowFinished.countDown();
|
||||
}
|
||||
|
||||
@@ -132,7 +132,12 @@ credential이 깨지면 regular lane PING이 UP을 알리게 두지 않고 pod
|
||||
- 부정 항목은 더 짧은 부정 TTL을 쓴다;
|
||||
- Redis 물리 TTL은 hard 만료다;
|
||||
- cache 장애는 애플리케이션의 타입 있는 unavailable·degraded 결과로 떨어진다;
|
||||
- 식별자는 keyspace에 들어가기 전에 키 있는 digest를 거친다.
|
||||
- 식별자는 keyspace에 들어가기 전에 키 있는 digest를 거친다;
|
||||
- `app.capabilities.cache.command-timeout`은 한 cache 연산 전체(lane 연결 대기와 응답 전부)의 상한이다. rate limit과
|
||||
같은 방식이다. 2026-11-13 전에는 lane 대기 2초가 그 위에 더해졌다. 상한 안에 못 끝난 연산은 `DEGRADED_UNAVAILABLE`이나
|
||||
`Unavailable`이다;
|
||||
- encode한 값이 `app.capabilities.cache.maximum-value-bytes`(`APP_CACHE_REDIS_MAXIMUM_VALUE_BYTES`, 기본 1 MiB,
|
||||
1..16777216)를 넘으면 기록하지 않는다(`NOT_RECORDED_PROVIDER_POLICY`, 보낸 것 없음). 그 값은 매번 source에서 읽힌다.
|
||||
|
||||
### 조건부 바꾸기
|
||||
|
||||
|
||||
+24
@@ -116,6 +116,30 @@ public final class SemanticRedisCall {
|
||||
return call(operation, null, body, notSent, unknown, false);
|
||||
}
|
||||
|
||||
/**
|
||||
* Runs a call that decides for itself when its effect was handed to the driver, and must finish
|
||||
* within {@code budget}.
|
||||
*
|
||||
* <p>The budget bounds the wait for a lane connection and every awaited reply together, as {@link
|
||||
* #effectWithin} does. Without it each await gets the frame's timeout of its own and the lane
|
||||
* wait comes on top, so a caller that called its timeout the ceiling of one operation waited the
|
||||
* lane's acquire timeout plus one timeout per reply (round9 review 2026-11-12, RR9-01-17).
|
||||
*
|
||||
* @param operation the semantic operation, for the failure log
|
||||
* @param budget the ceiling on the whole call
|
||||
* @param body what to send, and how to read the reply; calls {@link Session#markSent()} before
|
||||
* the command whose loss would leave the outcome unknown
|
||||
* @param notSent the answer when nothing can have happened
|
||||
* @param unknown the answer when the command may have run
|
||||
* @param <R> the answer type
|
||||
* @return the body's answer, or one of the two failure answers
|
||||
*/
|
||||
public <R> R runWithin(
|
||||
String operation, Duration budget, Body<R> body, Supplier<R> notSent, Supplier<R> unknown) {
|
||||
Objects.requireNonNull(budget, "budget must be non-null");
|
||||
return call(operation, budget, body, notSent, unknown, false);
|
||||
}
|
||||
|
||||
private <R> R call(
|
||||
String operation,
|
||||
Duration budget,
|
||||
|
||||
+67
-6
@@ -91,6 +91,20 @@ public final class RedisCacheRegionAdapter<K, V> implements CacheRegionPort<K, V
|
||||
|
||||
private final Duration negativeTtl;
|
||||
|
||||
private final Duration commandTimeout;
|
||||
|
||||
private final int maximumValueBytes;
|
||||
|
||||
/**
|
||||
* The largest encoded value one entry carries by default: 1 MiB, the default the old {@code
|
||||
* APP_CACHE_REDIS_MAXIMUM_VALUE_BYTES} had before that key stopped reaching anything.
|
||||
*
|
||||
* <p>Without a ceiling a single oversized value — a list that grew, a payload nobody sized — was
|
||||
* written whole: one {@code SET} that holds the connection and the server's single thread while
|
||||
* it is copied, then a key every reader fetches whole (round9 review 2026-11-12, RR9-01-18).
|
||||
*/
|
||||
public static final int DEFAULT_MAXIMUM_VALUE_BYTES = 1_048_576;
|
||||
|
||||
private final CacheScripts scripts = new CacheScripts();
|
||||
|
||||
private static final String ABSENT = "absent";
|
||||
@@ -115,7 +129,8 @@ public final class RedisCacheRegionAdapter<K, V> implements CacheRegionPort<K, V
|
||||
* @param softTtl how long an entry stays fresh
|
||||
* @param hardTtl how long an entry stays usable
|
||||
* @param negativeTtl how long an authoritative absence is cached
|
||||
* @param commandTimeout the ceiling on one cache operation
|
||||
* @param commandTimeout the ceiling on one cache operation, the lane wait and every reply
|
||||
* together
|
||||
*/
|
||||
public RedisCacheRegionAdapter(
|
||||
RedisRuntimeOwner owner,
|
||||
@@ -128,6 +143,38 @@ public final class RedisCacheRegionAdapter<K, V> implements CacheRegionPort<K, V
|
||||
Duration hardTtl,
|
||||
Duration negativeTtl,
|
||||
Duration commandTimeout) {
|
||||
this(
|
||||
owner,
|
||||
keys,
|
||||
keyDigest,
|
||||
encoder,
|
||||
decoder,
|
||||
clock,
|
||||
softTtl,
|
||||
hardTtl,
|
||||
negativeTtl,
|
||||
commandTimeout,
|
||||
DEFAULT_MAXIMUM_VALUE_BYTES);
|
||||
}
|
||||
|
||||
/**
|
||||
* Creates the adapter with its own ceiling on an encoded value.
|
||||
*
|
||||
* @param maximumValueBytes the largest encoded value one entry may carry; a larger one is not
|
||||
* recorded ({@link CacheRecordOutcome#NOT_RECORDED_PROVIDER_POLICY})
|
||||
*/
|
||||
public RedisCacheRegionAdapter(
|
||||
RedisRuntimeOwner owner,
|
||||
CacheKeys keys,
|
||||
Function<K, String> keyDigest,
|
||||
Function<V, byte[]> encoder,
|
||||
Function<byte[], V> decoder,
|
||||
Clock clock,
|
||||
Duration softTtl,
|
||||
Duration hardTtl,
|
||||
Duration negativeTtl,
|
||||
Duration commandTimeout,
|
||||
int maximumValueBytes) {
|
||||
this.call =
|
||||
new SemanticRedisCall(
|
||||
owner,
|
||||
@@ -144,6 +191,11 @@ public final class RedisCacheRegionAdapter<K, V> implements CacheRegionPort<K, V
|
||||
this.softTtl = Objects.requireNonNull(softTtl, "soft TTL must be non-null");
|
||||
this.hardTtl = Objects.requireNonNull(hardTtl, "hard TTL must be non-null");
|
||||
this.negativeTtl = Objects.requireNonNull(negativeTtl, "negative TTL must be non-null");
|
||||
this.commandTimeout = commandTimeout;
|
||||
if (maximumValueBytes < 1) {
|
||||
throw new IllegalStateException("the maximum value size must be at least one byte");
|
||||
}
|
||||
this.maximumValueBytes = maximumValueBytes;
|
||||
if (softTtl.compareTo(hardTtl) > 0) {
|
||||
throw new IllegalArgumentException(
|
||||
"the soft TTL must not exceed the hard TTL: an entry cannot stop being fresh after it has"
|
||||
@@ -158,8 +210,9 @@ public final class RedisCacheRegionAdapter<K, V> implements CacheRegionPort<K, V
|
||||
// A read changed nothing, whether or not it reached the server: degraded, not broken. The
|
||||
// caller
|
||||
// loads from the source and carries on.
|
||||
return call.run(
|
||||
return call.runWithin(
|
||||
"lookup",
|
||||
commandTimeout,
|
||||
session -> {
|
||||
resolveGeneration(session, now);
|
||||
byte[] stored =
|
||||
@@ -233,7 +286,12 @@ public final class RedisCacheRegionAdapter<K, V> implements CacheRegionPort<K, V
|
||||
@Override
|
||||
public CacheRecordOutcome record(K key, V value, CacheRecordMetadata metadata) {
|
||||
Objects.requireNonNull(value, "value must be non-null");
|
||||
return write(key, encoder.apply(value), "", metadata, hardTtl);
|
||||
byte[] payload = encoder.apply(value);
|
||||
if (payload.length > maximumValueBytes) {
|
||||
// Refused before any round trip; the caller serves the value it loaded and carries on.
|
||||
return CacheRecordOutcome.NOT_RECORDED_PROVIDER_POLICY;
|
||||
}
|
||||
return write(key, payload, "", metadata, hardTtl);
|
||||
}
|
||||
|
||||
@Override
|
||||
@@ -268,8 +326,9 @@ public final class RedisCacheRegionAdapter<K, V> implements CacheRegionPort<K, V
|
||||
// The write is marked sent only when the write command itself is handed to the driver. A
|
||||
// failure before that point — lane refusal, connection, the generation read — wrote nothing.
|
||||
return (scripted ? scriptCall : call)
|
||||
.run(
|
||||
.runWithin(
|
||||
"record",
|
||||
commandTimeout,
|
||||
session -> {
|
||||
resolveGeneration(session, now);
|
||||
CacheEnvelope envelope =
|
||||
@@ -343,8 +402,9 @@ public final class RedisCacheRegionAdapter<K, V> implements CacheRegionPort<K, V
|
||||
@Override
|
||||
public CacheInvalidationOutcome invalidate(K key) {
|
||||
Objects.requireNonNull(key, "key must be non-null");
|
||||
return call.run(
|
||||
return call.runWithin(
|
||||
"invalidate",
|
||||
commandTimeout,
|
||||
session -> {
|
||||
byte[] physicalKey = keys.entryKey(keyDigest.apply(key));
|
||||
// A marker, not a delete. A delete leaves the key absent, and a source load that read the
|
||||
@@ -371,8 +431,9 @@ public final class RedisCacheRegionAdapter<K, V> implements CacheRegionPort<K, V
|
||||
// a server that is answering everything else at the same time, and the SDK blocks KEYS for
|
||||
// exactly that reason. Bumping the generation makes every older entry fail the generation
|
||||
// check on read and expire on its own schedule.
|
||||
return call.run(
|
||||
return call.runWithin(
|
||||
"invalidate-region",
|
||||
commandTimeout,
|
||||
session -> {
|
||||
session.markSent();
|
||||
Long generation = session.await(session.commands().incrementBy(keys.generationKey(), 1));
|
||||
|
||||
+2
-1
@@ -56,7 +56,8 @@ class RedisCacheConfig {
|
||||
cache.getPositiveSoftTtl(),
|
||||
cache.getPositiveHardTtl(),
|
||||
cache.getNegativeTtl(),
|
||||
cache.getCommandTimeout());
|
||||
cache.getCommandTimeout(),
|
||||
cache.getMaximumValueBytes());
|
||||
}
|
||||
|
||||
/**
|
||||
|
||||
+26
@@ -1,5 +1,6 @@
|
||||
package dev.caskeleton.adapter.outbound.cache.redis.capability;
|
||||
|
||||
import dev.caskeleton.adapter.outbound.cache.redis.cache.RedisCacheRegionAdapter;
|
||||
import dev.caskeleton.adapter.outbound.support.CapabilitySelectors;
|
||||
import dev.caskeleton.adapter.outbound.support.CapabilityStartupValidationException;
|
||||
import dev.caskeleton.adapter.outbound.support.StrictSettingsBinder;
|
||||
@@ -87,6 +88,12 @@ public class RedisCapabilitySettings {
|
||||
private Duration negativeTtl = Duration.ofSeconds(10);
|
||||
private Duration minimumHardTtl = Duration.ofSeconds(1);
|
||||
private int maximumScriptConnections = 4;
|
||||
private int maximumValueBytes = RedisCacheRegionAdapter.DEFAULT_MAXIMUM_VALUE_BYTES;
|
||||
|
||||
/**
|
||||
* The largest ceiling a deployment may set: sixteen mebibytes, as the env registry bounds it.
|
||||
*/
|
||||
static final int MAXIMUM_VALUE_BYTES_CEILING = 16_777_216;
|
||||
|
||||
/**
|
||||
* Reports whether the default binding selected Redis.
|
||||
@@ -124,6 +131,10 @@ public class RedisCapabilitySettings {
|
||||
throw new CapabilityStartupValidationException(
|
||||
"the cache script lane needs at least one connection");
|
||||
}
|
||||
if (maximumValueBytes < 1 || maximumValueBytes > MAXIMUM_VALUE_BYTES_CEILING) {
|
||||
throw new CapabilityStartupValidationException(
|
||||
"the cache value ceiling must be 1.." + MAXIMUM_VALUE_BYTES_CEILING + " bytes");
|
||||
}
|
||||
}
|
||||
|
||||
public Map<String, String> getBindings() {
|
||||
@@ -214,6 +225,21 @@ public class RedisCapabilitySettings {
|
||||
public void setMaximumScriptConnections(int maximumScriptConnections) {
|
||||
this.maximumScriptConnections = maximumScriptConnections;
|
||||
}
|
||||
|
||||
/**
|
||||
* The largest value the region records, in encoded bytes. A larger one is not written and the
|
||||
* read path falls back to the source, so one oversized entry cannot fill the instance or hold a
|
||||
* connection for the whole transfer.
|
||||
*
|
||||
* @return the ceiling, 1 to 16 MiB
|
||||
*/
|
||||
public int getMaximumValueBytes() {
|
||||
return maximumValueBytes;
|
||||
}
|
||||
|
||||
public void setMaximumValueBytes(int maximumValueBytes) {
|
||||
this.maximumValueBytes = maximumValueBytes;
|
||||
}
|
||||
}
|
||||
|
||||
/** Edge rate-limit provider composition. */
|
||||
|
||||
+3
@@ -43,6 +43,9 @@ app:
|
||||
# limiting and leases share (app.redis.capacity.maximum-script-connections). A cache under
|
||||
# load runs out of these and degrades; it cannot take the correctness features' connections.
|
||||
maximum-script-connections: ${APP_CACHE_REDIS_MAXIMUM_SCRIPT_CONNECTIONS:4}
|
||||
# The largest value the region records, in encoded bytes (1..16777216). A larger one is not
|
||||
# written; the read falls back to the source.
|
||||
maximum-value-bytes: ${APP_CACHE_REDIS_MAXIMUM_VALUE_BYTES:1048576}
|
||||
idempotency:
|
||||
# Redis store tuning only; the provider selector and lifecycle policy stay with the
|
||||
# application's idempotency executor.
|
||||
|
||||
+122
-5
@@ -1,12 +1,14 @@
|
||||
package dev.caskeleton.adapter.outbound.cache.redis.cache;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
import static org.assertj.core.api.Assertions.catchThrowable;
|
||||
|
||||
import dev.caskeleton.adapter.outbound.cache.redis.runtime.RedisDeploymentMode;
|
||||
import dev.caskeleton.adapter.outbound.cache.redis.runtime.command.InMemorySemanticRedisCommands;
|
||||
import dev.caskeleton.adapter.outbound.cache.redis.runtime.command.SemanticRedisCommands;
|
||||
import dev.caskeleton.adapter.outbound.cache.redis.runtime.connection.RedisConnectionKind;
|
||||
import dev.caskeleton.adapter.outbound.cache.redis.runtime.connection.RedisLane;
|
||||
import dev.caskeleton.adapter.outbound.cache.redis.runtime.connection.RedisLease;
|
||||
import dev.caskeleton.adapter.outbound.cache.redis.runtime.connection.RedisRuntimeClient;
|
||||
import dev.caskeleton.adapter.outbound.cache.redis.runtime.connection.RedisRuntimeOwner;
|
||||
import dev.caskeleton.adapter.outbound.cache.redis.runtime.connection.StubRedisRuntimeClient;
|
||||
@@ -23,7 +25,10 @@ import java.time.Clock;
|
||||
import java.time.Duration;
|
||||
import java.time.Instant;
|
||||
import java.time.ZoneOffset;
|
||||
import java.util.LinkedHashMap;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.Optional;
|
||||
import java.util.concurrent.CompletableFuture;
|
||||
import org.junit.jupiter.api.DisplayName;
|
||||
import org.junit.jupiter.api.Test;
|
||||
@@ -56,7 +61,7 @@ class RedisCacheRegionAdapterTest {
|
||||
|
||||
private RedisCacheRegionAdapter<String, String> cache(
|
||||
Instant now, RedisRuntimeClient client, RedisCacheRegionAdapter.CacheKeys cacheKeys) {
|
||||
Map<RedisLane, Integer> limits = new java.util.LinkedHashMap<>();
|
||||
Map<RedisLane, Integer> limits = new LinkedHashMap<>();
|
||||
for (RedisConnectionKind kind : RedisConnectionKind.values()) {
|
||||
limits.put(kind, 4);
|
||||
}
|
||||
@@ -82,6 +87,119 @@ class RedisCacheRegionAdapterTest {
|
||||
Duration.ofSeconds(2));
|
||||
}
|
||||
|
||||
@Test
|
||||
void aLookupOnAFullLaneGivesUpAtTheCommandTimeoutNotTheLaneWait() {
|
||||
// given
|
||||
// One connection, already borrowed; a borrow would wait five seconds for it. The cache's
|
||||
// command timeout is 100 ms and is the ceiling on the whole operation, lane wait included.
|
||||
Map<RedisLane, Integer> limits = new LinkedHashMap<>();
|
||||
for (RedisConnectionKind kind : RedisConnectionKind.values()) {
|
||||
limits.put(kind, 1);
|
||||
}
|
||||
limits.put(CacheScriptLane.CACHE_SCRIPT, 1);
|
||||
RedisRuntimeOwner owner =
|
||||
new RedisRuntimeOwner(
|
||||
StubRedisRuntimeClient.serving(gateway),
|
||||
limits,
|
||||
Duration.ofSeconds(5),
|
||||
Duration.ofSeconds(1));
|
||||
RedisCacheRegionAdapter<String, String> cache =
|
||||
new RedisCacheRegionAdapter<>(
|
||||
owner,
|
||||
keys,
|
||||
key -> "digest-" + key,
|
||||
value -> value.getBytes(StandardCharsets.UTF_8),
|
||||
bytes -> new String(bytes, StandardCharsets.UTF_8),
|
||||
Clock.fixed(T0, ZoneOffset.UTC),
|
||||
Duration.ofSeconds(30),
|
||||
Duration.ofSeconds(120),
|
||||
Duration.ofSeconds(10),
|
||||
Duration.ofMillis(100));
|
||||
|
||||
// when
|
||||
long started = System.nanoTime();
|
||||
CacheLookup<String> lookup;
|
||||
RedisLease held = owner.borrow(RedisConnectionKind.REGULAR);
|
||||
try {
|
||||
lookup = cache.lookup("order-1");
|
||||
} finally {
|
||||
held.close();
|
||||
}
|
||||
Duration waited = Duration.ofNanos(System.nanoTime() - started);
|
||||
|
||||
// then
|
||||
assertThat(
|
||||
List.of(
|
||||
lookup instanceof CacheLookup.Unavailable,
|
||||
waited.compareTo(Duration.ofSeconds(3)) < 0))
|
||||
.as("waited %s on a lane that would have held the caller five seconds", waited)
|
||||
.containsExactly(true, true);
|
||||
}
|
||||
|
||||
@Test
|
||||
void aValueLargerThanTheCeilingIsNotRecorded() {
|
||||
// given
|
||||
RedisCacheRegionAdapter<String, String> cache = withCeiling(8);
|
||||
|
||||
// when
|
||||
CacheRecordOutcome outcome = cache.record("order-1", "nine bytes", upsert());
|
||||
|
||||
// then
|
||||
assertThat(outcome).isEqualTo(CacheRecordOutcome.NOT_RECORDED_PROVIDER_POLICY);
|
||||
}
|
||||
|
||||
@Test
|
||||
void aValueExactlyAtTheCeilingIsRecorded() {
|
||||
// given
|
||||
RedisCacheRegionAdapter<String, String> cache = withCeiling(8);
|
||||
|
||||
// when
|
||||
CacheRecordOutcome outcome = cache.record("order-1", "eight by", upsert());
|
||||
|
||||
// then
|
||||
assertThat(outcome).isEqualTo(CacheRecordOutcome.RECORDED);
|
||||
}
|
||||
|
||||
private RedisCacheRegionAdapter<String, String> withCeiling(int maximumValueBytes) {
|
||||
return new RedisCacheRegionAdapter<>(
|
||||
new RedisRuntimeOwner(
|
||||
StubRedisRuntimeClient.serving(gateway),
|
||||
lanesOf(4),
|
||||
Duration.ofSeconds(1),
|
||||
Duration.ofSeconds(1)),
|
||||
keys,
|
||||
key -> "digest-" + key,
|
||||
value -> value.getBytes(StandardCharsets.UTF_8),
|
||||
bytes -> new String(bytes, StandardCharsets.UTF_8),
|
||||
Clock.fixed(T0, ZoneOffset.UTC),
|
||||
Duration.ofSeconds(30),
|
||||
Duration.ofSeconds(120),
|
||||
Duration.ofSeconds(10),
|
||||
Duration.ofSeconds(2),
|
||||
maximumValueBytes);
|
||||
}
|
||||
|
||||
@Test
|
||||
void theDefaultCeilingIsTheOneMebibyteTheRetiredSettingHad() {
|
||||
// given
|
||||
String justOver = "x".repeat(RedisCacheRegionAdapter.DEFAULT_MAXIMUM_VALUE_BYTES + 1);
|
||||
|
||||
// when
|
||||
CacheRecordOutcome outcome = cache(T0).record("order-1", justOver, upsert());
|
||||
|
||||
// then
|
||||
assertThat(outcome).isEqualTo(CacheRecordOutcome.NOT_RECORDED_PROVIDER_POLICY);
|
||||
}
|
||||
|
||||
private static Map<RedisLane, Integer> lanesOf(int limit) {
|
||||
Map<RedisLane, Integer> limits = new LinkedHashMap<>();
|
||||
for (RedisConnectionKind kind : RedisConnectionKind.values()) {
|
||||
limits.put(kind, limit);
|
||||
}
|
||||
limits.put(CacheScriptLane.CACHE_SCRIPT, limit);
|
||||
return limits;
|
||||
}
|
||||
|
||||
private static CacheRecordMetadata upsert() {
|
||||
return new CacheRecordMetadata("rev-1", CacheRecordIntent.UPSERT);
|
||||
}
|
||||
@@ -428,8 +546,7 @@ class RedisCacheRegionAdapterTest {
|
||||
|
||||
// when
|
||||
Throwable refused =
|
||||
org.assertj.core.api.Assertions.catchThrowable(
|
||||
() -> cache.record("order-1", "payload", newerThan("rev-2")));
|
||||
catchThrowable(() -> cache.record("order-1", "payload", newerThan("rev-2")));
|
||||
|
||||
// then
|
||||
assertThat(refused)
|
||||
@@ -441,7 +558,7 @@ class RedisCacheRegionAdapterTest {
|
||||
@DisplayName("멱등성·lease가 SCRIPT lane을 모두 잡고 있어도 캐시 조건부 기록은 캐시 전용 lane에서 기록된다")
|
||||
void aConditionalCacheWriteDoesNotWaitForTheCorrectnessScriptLane() {
|
||||
// given
|
||||
Map<RedisLane, Integer> limits = new java.util.LinkedHashMap<>();
|
||||
Map<RedisLane, Integer> limits = new LinkedHashMap<>();
|
||||
limits.put(RedisConnectionKind.REGULAR, 4);
|
||||
limits.put(RedisConnectionKind.SCRIPT, 1);
|
||||
limits.put(CacheScriptLane.CACHE_SCRIPT, 1);
|
||||
@@ -584,7 +701,7 @@ class RedisCacheRegionAdapterTest {
|
||||
}
|
||||
|
||||
@Override
|
||||
public RedisLaneConnection openLane(RedisLane kind, java.util.Optional<byte[]> routingKey) {
|
||||
public RedisLaneConnection openLane(RedisLane kind, Optional<byte[]> routingKey) {
|
||||
throw new IllegalStateException("the server is unreachable");
|
||||
}
|
||||
|
||||
|
||||
+40
@@ -0,0 +1,40 @@
|
||||
package dev.caskeleton.adapter.outbound.cache.redis.capability;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
import static org.assertj.core.api.Assertions.catchThrowable;
|
||||
|
||||
import dev.caskeleton.adapter.outbound.cache.redis.cache.RedisCacheRegionAdapter;
|
||||
import dev.caskeleton.adapter.outbound.support.CapabilityStartupValidationException;
|
||||
import org.junit.jupiter.api.DisplayName;
|
||||
import org.junit.jupiter.api.Test;
|
||||
|
||||
/**
|
||||
* The cache value ceiling is a deployment's setting, {@code app.capabilities.cache
|
||||
* .maximum-value-bytes} (round9 review 2026-11-12, RR9-01-18): the region refused larger values
|
||||
* with no way for the shipped composition to change the bound.
|
||||
*/
|
||||
class RedisCacheValueCeilingSettingTest {
|
||||
|
||||
@Test
|
||||
@DisplayName("설정하지 않으면 상한은 1 MiB다")
|
||||
void anUnsetCeilingIsOneMebibyte() {
|
||||
assertThat(new RedisCapabilitySettings().getCache().getMaximumValueBytes())
|
||||
.isEqualTo(RedisCacheRegionAdapter.DEFAULT_MAXIMUM_VALUE_BYTES);
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("16 MiB를 넘는 상한은 기동 때 거절한다")
|
||||
void aCeilingPastSixteenMebibytesFailsStartup() {
|
||||
// given
|
||||
RedisCapabilitySettings.Cache cache = new RedisCapabilitySettings().getCache();
|
||||
cache.setMaximumValueBytes(16_777_217);
|
||||
|
||||
// when
|
||||
Throwable refused = catchThrowable(cache::validate);
|
||||
|
||||
// then
|
||||
assertThat(refused)
|
||||
.isInstanceOf(CapabilityStartupValidationException.class)
|
||||
.hasMessageContaining("the cache value ceiling must be 1..16777216 bytes");
|
||||
}
|
||||
}
|
||||
+12
-9
@@ -111,15 +111,17 @@ public final class ApacheClientFactory {
|
||||
.evictIdleConnections(TimeValue.ofMilliseconds(profile.pool().maxIdleTime().toMillis()))
|
||||
.setDefaultRequestConfig(requestConfig(profile));
|
||||
|
||||
if (!profile.request().compression()) {
|
||||
clientBuilder.disableContentCompression();
|
||||
}
|
||||
// Apache never decodes the body. With its content compression on, the stream the response
|
||||
// reader counted as "wire" bytes was already decompressed, so a profile's wire limit was a
|
||||
// second decoded limit and a well-compressed response under it was refused for its decoded size
|
||||
// (round9 review 2026-11-12, RR9-01-30). The reader counts the bytes as they arrive and decodes
|
||||
// them itself. Whether to ask for compression is the profile's, sent by the RestClient for
|
||||
// every transport.
|
||||
clientBuilder.disableContentCompression();
|
||||
if (profile.proxy().enabled()) {
|
||||
clientBuilder.setProxy(
|
||||
new HttpHost(
|
||||
profile.proxy().type().name().toLowerCase(java.util.Locale.ROOT),
|
||||
profile.proxy().host(),
|
||||
profile.proxy().port()));
|
||||
// An HTTP forward proxy, the only kind ProxyType has: SOCKS used to bind and then become an
|
||||
// HTTP proxy at the SOCKS port.
|
||||
clientBuilder.setProxy(new HttpHost("http", profile.proxy().host(), profile.proxy().port()));
|
||||
}
|
||||
return new ApacheRuntime(clientBuilder.build(), pool);
|
||||
}
|
||||
@@ -131,7 +133,8 @@ public final class ApacheClientFactory {
|
||||
.setRedirectsEnabled(false)
|
||||
.setMaxRedirects(0)
|
||||
.setCircularRedirectsAllowed(false)
|
||||
.setContentCompressionEnabled(profile.request().compression())
|
||||
// Decoding is the response reader's, after it has counted the wire bytes; see create().
|
||||
.setContentCompressionEnabled(false)
|
||||
// Hard cancellation lets a deadline abort actually abort the socket instead of leaving a
|
||||
// request in flight after the caller has given up.
|
||||
.setHardCancellationEnabled(true)
|
||||
|
||||
+13
@@ -22,6 +22,7 @@ import org.eclipse.jetty.http3.client.HTTP3Client;
|
||||
import org.eclipse.jetty.http3.client.transport.HttpClientTransportOverHTTP3;
|
||||
import org.eclipse.jetty.io.Transport;
|
||||
import org.eclipse.jetty.quic.client.ClientQuicConfiguration;
|
||||
import org.eclipse.jetty.util.component.LifeCycle;
|
||||
import org.springframework.http.client.ClientHttpRequestFactory;
|
||||
import org.springframework.http.client.JettyClientHttpRequestFactory;
|
||||
|
||||
@@ -132,6 +133,18 @@ public final class JettyHttp3TransportProvider implements BlockingTransportProvi
|
||||
.getRequestListeners()
|
||||
.addQueuedListener(
|
||||
request -> AttemptDeadline.abortOnExpiry(() -> abortAtDeadline(request)));
|
||||
// Jetty registers its gzip decoder from the classpath when it starts, then asks for gzip on
|
||||
// every request, decodes the response itself and drops Content-Encoding — so the reader's wire
|
||||
// count saw the decoded size, whatever the profile said (round9 review 2026-11-12,
|
||||
// RR9-01-30). Cleared once started, because an empty list before start is what tells Jetty to
|
||||
// load them; the coded bytes then reach the reader, which counts and decodes them.
|
||||
client.addEventListener(
|
||||
new LifeCycle.Listener() {
|
||||
@Override
|
||||
public void lifeCycleStarted(LifeCycle started) {
|
||||
client.getContentDecoderFactories().clear();
|
||||
}
|
||||
});
|
||||
return client;
|
||||
}
|
||||
|
||||
|
||||
+9
-3
@@ -1,7 +1,13 @@
|
||||
package dev.caskeleton.adapter.outbound.httpclient.profile;
|
||||
|
||||
/** Forward proxy kind (design §24.3). */
|
||||
/**
|
||||
* Forward proxy kind (design §24.3).
|
||||
*
|
||||
* <p>HTTP only. Every shipped transport speaks an HTTP forward proxy and none speaks SOCKS; a
|
||||
* {@code SOCKS} value used to bind and then be wired as an HTTP proxy at the SOCKS port, and a
|
||||
* value whose only possible effect is a failure is not kept as an option ({@code
|
||||
* docs/manual/10-capabilities.md} standard 3). {@code proxy.type: SOCKS} now fails to bind.
|
||||
*/
|
||||
public enum ProxyType {
|
||||
HTTP,
|
||||
SOCKS
|
||||
HTTP
|
||||
}
|
||||
|
||||
+55
-7
@@ -5,6 +5,7 @@ import dev.caskeleton.adapter.outbound.httpclient.api.HttpStatus;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.error.HttpDeadlineExceededException;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.error.HttpFailureMetadata;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.error.HttpRemoteErrorException;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.error.HttpTargetRejectedException;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.operation.AttemptStage;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.operation.ExecutionEvidence;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.operation.HttpOperation;
|
||||
@@ -109,10 +110,38 @@ public final class BlockingStreamingGateway {
|
||||
// exceptions a buffered call throws, classified by the transport's own
|
||||
// classifier at the body stage — not as the engine's bare IOException, which
|
||||
// carried no category, no evidence and nothing to log (RR-04-03).
|
||||
// Decoded here as the buffered reader decodes: the transport hands over the coded
|
||||
// bytes, counted on the wire as they arrive, and what they decode to is counted
|
||||
// against the decoded budget as the caller reads it. A streamed download used to
|
||||
// reach the caller still compressed once the transport stopped decoding.
|
||||
String coding;
|
||||
try {
|
||||
coding = ContentDecoding.codingOf(rawResponse.getHeaders());
|
||||
} catch (ContentDecoding.RefusedCodingException refused) {
|
||||
// A coding this client did not ask for: nothing is handed over, and the
|
||||
// connection is not left open behind the refusal.
|
||||
rawResponse.close();
|
||||
throw new HttpTargetRejectedException(
|
||||
"response content coding is not one this client asked for",
|
||||
metadata
|
||||
.withStatus(new HttpStatus(status))
|
||||
.withEvidence(ExecutionEvidence.RESPONSE_RECEIVED),
|
||||
refused);
|
||||
}
|
||||
InputStream wire =
|
||||
new CountingBoundedInputStream(
|
||||
rawResponse.getBody(), limiter::recordWireBytes);
|
||||
// Counted against the decoded budget whatever the coding, identity included, as
|
||||
// the buffered reader counts it: an identity body is its own decoding.
|
||||
InputStream body =
|
||||
new CountingBoundedInputStream(
|
||||
ContentDecoding.decoding(coding, wire), limiter::recordDecodedBytes);
|
||||
if (!coding.isEmpty()) {
|
||||
ContentDecoding.dropCodingHeaders(responseHeaders);
|
||||
}
|
||||
InputStream bounded =
|
||||
new ClassifiedBodyInputStream(
|
||||
new CountingBoundedInputStream(
|
||||
rawResponse.getBody(), limiter::recordWireBytes),
|
||||
body,
|
||||
bodyFailure ->
|
||||
runtime
|
||||
.support()
|
||||
@@ -129,11 +158,7 @@ public final class BlockingStreamingGateway {
|
||||
false);
|
||||
} catch (RuntimeException failure) {
|
||||
closeQuietly(admission);
|
||||
if (setupDeadline.expired() && !(failure instanceof HttpRemoteErrorException)) {
|
||||
throw new HttpDeadlineExceededException(
|
||||
"effective deadline reached before the stream was set up", metadata, failure);
|
||||
}
|
||||
throw failure;
|
||||
throw setupFailure(failure, setupDeadline.expired(), metadata);
|
||||
}
|
||||
|
||||
BlockingStreamingResponse wrapped =
|
||||
@@ -155,6 +180,29 @@ public final class BlockingStreamingGateway {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* What a failure while setting up the stream reaches the caller as.
|
||||
*
|
||||
* <p>Once the setup budget is spent the failure is the deadline, whatever the transport said. A
|
||||
* response that arrived and was refused, or a remote error, stays what it is: its evidence is
|
||||
* that the server answered.
|
||||
*
|
||||
* @param failure what setting up the stream threw
|
||||
* @param budgetSpent whether the setup deadline had expired when it did
|
||||
* @param metadata the call's failure metadata
|
||||
* @return the exception to throw
|
||||
*/
|
||||
static RuntimeException setupFailure(
|
||||
RuntimeException failure, boolean budgetSpent, HttpFailureMetadata metadata) {
|
||||
if (budgetSpent
|
||||
&& !(failure instanceof HttpRemoteErrorException)
|
||||
&& !(failure instanceof HttpTargetRejectedException)) {
|
||||
return new HttpDeadlineExceededException(
|
||||
"effective deadline reached before the stream was set up", metadata, failure);
|
||||
}
|
||||
return failure;
|
||||
}
|
||||
|
||||
/** A permit release must not mask the failure that is already propagating. */
|
||||
private static void closeQuietly(AutoCloseable closeable) {
|
||||
try {
|
||||
|
||||
+162
@@ -0,0 +1,162 @@
|
||||
package dev.caskeleton.adapter.outbound.httpclient.restclient;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.io.InputStream;
|
||||
import java.io.PushbackInputStream;
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
import java.util.Locale;
|
||||
import java.util.Map;
|
||||
import java.util.Set;
|
||||
import java.util.zip.GZIPInputStream;
|
||||
import java.util.zip.InflaterInputStream;
|
||||
import org.springframework.http.HttpHeaders;
|
||||
|
||||
/**
|
||||
* Undoes a response's content coding, for the buffered reader and the streaming download alike.
|
||||
*
|
||||
* <p>The transport does not decode (round9 review 2026-11-12, RR9-01-30): the bytes that arrive are
|
||||
* counted against the wire budget, and what they decode to against the decoded budget. Both read
|
||||
* paths come through here, so a streamed download is decoded the way a buffered body is rather than
|
||||
* handed over compressed.
|
||||
*/
|
||||
final class ContentDecoding {
|
||||
|
||||
/**
|
||||
* The {@code Accept-Encoding} value a profile with compression on sends: the codings {@link
|
||||
* #decoding} undoes. Kept beside the decoder so the request never asks for a coding the reader
|
||||
* would hand over still compressed.
|
||||
*/
|
||||
static final String ACCEPTED = "gzip, deflate";
|
||||
|
||||
/** How much a decoder reads from the wire at a time. */
|
||||
static final int CHUNK = 8 * 1024;
|
||||
|
||||
private ContentDecoding() {}
|
||||
|
||||
/** The codings {@link #decoding} undoes; {@code x-gzip} is the old name of {@code gzip}. */
|
||||
private static final Set<String> DECODABLE = Set.of("gzip", "x-gzip", "deflate");
|
||||
|
||||
/**
|
||||
* A response coded in a way this client did not ask for, refused before its body is read.
|
||||
*
|
||||
* <p>Its own type so the gateways can report it as what it is — a response that arrived and was
|
||||
* refused — rather than as the bare {@link IOException} a transport classifier reads as a call
|
||||
* that got no response.
|
||||
*/
|
||||
static final class RefusedCodingException extends IOException {
|
||||
|
||||
private static final long serialVersionUID = 1L;
|
||||
|
||||
private RefusedCodingException(String message) {
|
||||
super(message);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* The response's content coding, lower case; empty for none or {@code identity}.
|
||||
*
|
||||
* @throws RefusedCodingException when the response names more than one coding — stacked codings
|
||||
* are not something this client asks for, and decoding only the first would hand the rest
|
||||
* over as the body — or a coding {@link #decoding} cannot undo
|
||||
*/
|
||||
static String codingOf(HttpHeaders headers) throws RefusedCodingException {
|
||||
List<String> codings = new ArrayList<>();
|
||||
for (String value : headers.getOrEmpty(HttpHeaders.CONTENT_ENCODING)) {
|
||||
// Limit -1: every element, an empty trailing one included, which the check below skips.
|
||||
for (String part : value.split(",", -1)) {
|
||||
String coding = part.trim().toLowerCase(Locale.ROOT);
|
||||
if (!coding.isEmpty() && !coding.equals("identity")) {
|
||||
codings.add(coding);
|
||||
}
|
||||
}
|
||||
}
|
||||
if (codings.size() > 1) {
|
||||
throw new RefusedCodingException(
|
||||
"the response stacked content codings this client did not ask for: " + codings);
|
||||
}
|
||||
String coding = codings.isEmpty() ? "" : codings.get(0);
|
||||
if (!coding.isEmpty() && !DECODABLE.contains(coding)) {
|
||||
throw new RefusedCodingException(
|
||||
"the response used a content coding this client did not ask for: " + coding);
|
||||
}
|
||||
return coding;
|
||||
}
|
||||
|
||||
/**
|
||||
* The decoded body, read lazily from {@code wire}.
|
||||
*
|
||||
* <p>Lazily, because a gzip stream reads its header when it is built: built eagerly, an empty
|
||||
* body under {@code Content-Encoding: gzip} — a {@code HEAD}, a 204 or a 304 — failed the whole
|
||||
* response, and a streamed download read from the network before it was handed over. Here nothing
|
||||
* is read until the caller reads, and a body that turns out empty decodes to nothing.
|
||||
*
|
||||
* @param coding a coding {@link #codingOf} returned
|
||||
* @param wire the counted wire stream
|
||||
*/
|
||||
static InputStream decoding(String coding, InputStream wire) {
|
||||
return coding.isEmpty() ? wire : new LazilyDecoded(coding, wire);
|
||||
}
|
||||
|
||||
/** {@code headers} without the two that describe the coded body rather than the decoded one. */
|
||||
static void dropCodingHeaders(Map<String, List<String>> headers) {
|
||||
headers
|
||||
.keySet()
|
||||
.removeIf(
|
||||
name ->
|
||||
name.equalsIgnoreCase(HttpHeaders.CONTENT_ENCODING)
|
||||
|| name.equalsIgnoreCase(HttpHeaders.CONTENT_LENGTH));
|
||||
}
|
||||
|
||||
private static final class LazilyDecoded extends InputStream {
|
||||
|
||||
private final String coding;
|
||||
private final InputStream wire;
|
||||
private InputStream decoded;
|
||||
|
||||
private LazilyDecoded(String coding, InputStream wire) {
|
||||
this.coding = coding;
|
||||
this.wire = wire;
|
||||
}
|
||||
|
||||
@Override
|
||||
public int read() throws IOException {
|
||||
return open().read();
|
||||
}
|
||||
|
||||
@Override
|
||||
public int read(byte[] buffer, int offset, int length) throws IOException {
|
||||
return open().read(buffer, offset, length);
|
||||
}
|
||||
|
||||
@Override
|
||||
public void close() throws IOException {
|
||||
try {
|
||||
if (decoded != null) {
|
||||
decoded.close();
|
||||
}
|
||||
} finally {
|
||||
wire.close();
|
||||
}
|
||||
}
|
||||
|
||||
private InputStream open() throws IOException {
|
||||
if (decoded == null) {
|
||||
PushbackInputStream peeked = new PushbackInputStream(wire, 1);
|
||||
int first = peeked.read();
|
||||
if (first == -1) {
|
||||
decoded = InputStream.nullInputStream();
|
||||
} else {
|
||||
peeked.unread(first);
|
||||
// deflate is the zlib-wrapped form RFC 9110 names; a server that sends raw deflate under
|
||||
// this name fails to decode, which is a failed response rather than a guessed one.
|
||||
decoded =
|
||||
coding.equals("deflate")
|
||||
? new InflaterInputStream(peeked)
|
||||
: new GZIPInputStream(peeked, CHUNK);
|
||||
}
|
||||
}
|
||||
return decoded;
|
||||
}
|
||||
}
|
||||
}
|
||||
+5
@@ -36,6 +36,11 @@ public final class ResponseSizeLimiter {
|
||||
this.metadata = metadata;
|
||||
}
|
||||
|
||||
/** What a failure of the response this limiter counts says about the call. */
|
||||
HttpFailureMetadata failureMetadata() {
|
||||
return metadata;
|
||||
}
|
||||
|
||||
public void recordWireBytes(long additional) {
|
||||
wireBytes += additional;
|
||||
if (wireBytes > maxWireBytes) {
|
||||
|
||||
+43
-9
@@ -1,8 +1,10 @@
|
||||
package dev.caskeleton.adapter.outbound.httpclient.restclient;
|
||||
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.HttpStatus;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.error.HttpFailureMetadata;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.error.HttpSerializationException;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.error.HttpTargetRejectedException;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.operation.ExecutionEvidence;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.result.ByteArrayResponseType;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.result.ClassResponseType;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.result.EmptyResponseType;
|
||||
@@ -10,6 +12,7 @@ import dev.caskeleton.adapter.outbound.httpclient.api.result.GenericResponseType
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.result.ResponseType;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.profile.ResponseLimits;
|
||||
import java.io.ByteArrayInputStream;
|
||||
import java.io.ByteArrayOutputStream;
|
||||
import java.io.IOException;
|
||||
import java.io.InputStream;
|
||||
import java.lang.reflect.Type;
|
||||
@@ -34,6 +37,11 @@ import org.springframework.http.converter.json.JacksonJsonHttpMessageConverter;
|
||||
* <p>The body is bounded while it is read and then decoded from memory by a Spring message
|
||||
* converter. Doing the bounding first is what makes a decompression bomb or an unexpectedly huge
|
||||
* payload a rejected request instead of an out-of-memory error.
|
||||
*
|
||||
* <p>Content coding is undone here, not by the transport: the bytes that arrive are counted against
|
||||
* the wire budget as they arrive, and the bytes they decode to are counted against the decoded
|
||||
* budget as they are produced — a chunk at a time, so a decompression bomb is refused before it is
|
||||
* held in memory (round9 review 2026-11-12, RR9-01-30).
|
||||
*/
|
||||
public final class RestClientResponseReader {
|
||||
|
||||
@@ -112,16 +120,42 @@ public final class RestClientResponseReader {
|
||||
java.util.function.LongConsumer responseBytes)
|
||||
throws IOException {
|
||||
Objects.requireNonNull(responseBytes, "response byte observer");
|
||||
String coding;
|
||||
try {
|
||||
coding = ContentDecoding.codingOf(headers);
|
||||
} catch (ContentDecoding.RefusedCodingException refused) {
|
||||
// A response arrived and was refused, which the transport classifier would otherwise read
|
||||
// from a bare IOException as a call that got no answer.
|
||||
throw new HttpTargetRejectedException(
|
||||
"response content coding is not one this client asked for",
|
||||
limiter
|
||||
.failureMetadata()
|
||||
.withStatus(new HttpStatus(status))
|
||||
.withEvidence(ExecutionEvidence.RESPONSE_RECEIVED),
|
||||
refused);
|
||||
}
|
||||
try (InputStream counted =
|
||||
new CountingBoundedInputStream(
|
||||
body,
|
||||
read -> {
|
||||
limiter.recordWireBytes(read);
|
||||
responseBytes.accept(read);
|
||||
})) {
|
||||
byte[] bytes = counted.readAllBytes();
|
||||
limiter.recordDecodedBytes(bytes.length);
|
||||
return new RawResponse(status, toMap(headers), bytes);
|
||||
new CountingBoundedInputStream(
|
||||
body,
|
||||
read -> {
|
||||
limiter.recordWireBytes(read);
|
||||
responseBytes.accept(read);
|
||||
});
|
||||
InputStream decoded = ContentDecoding.decoding(coding, counted)) {
|
||||
ByteArrayOutputStream bytes = new ByteArrayOutputStream();
|
||||
byte[] chunk = new byte[ContentDecoding.CHUNK];
|
||||
int read;
|
||||
while ((read = decoded.read(chunk)) != -1) {
|
||||
limiter.recordDecodedBytes(read);
|
||||
bytes.write(chunk, 0, read);
|
||||
}
|
||||
Map<String, List<String>> kept = toMap(headers);
|
||||
if (!coding.isEmpty()) {
|
||||
// The body handed on is the decoded one; keeping these would describe bytes it no longer
|
||||
// is.
|
||||
ContentDecoding.dropCodingHeaders(kept);
|
||||
}
|
||||
return new RawResponse(status, kept, bytes.toByteArray());
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
+10
-5
@@ -22,6 +22,7 @@ import java.util.Map;
|
||||
import java.util.Objects;
|
||||
import java.util.function.Supplier;
|
||||
import java.util.random.RandomGenerator;
|
||||
import org.springframework.http.HttpHeaders;
|
||||
import org.springframework.http.client.ClientHttpRequestFactory;
|
||||
import org.springframework.web.client.RestClient;
|
||||
|
||||
@@ -77,11 +78,15 @@ public final class RestClientRuntimeFactory implements ClientRuntimeFactory {
|
||||
|
||||
ClientHttpRequestFactory requestFactory =
|
||||
provider.create(profile, generation, lifecycleListener);
|
||||
RestClient restClient =
|
||||
RestClient.builder()
|
||||
.requestFactory(requestFactory)
|
||||
.baseUrl(profile.baseUrl().toString())
|
||||
.build();
|
||||
RestClient.Builder builder =
|
||||
RestClient.builder().requestFactory(requestFactory).baseUrl(profile.baseUrl().toString());
|
||||
if (profile.request().compression()) {
|
||||
// Asked for here, once, for every transport: none of them decodes any more, so the reader
|
||||
// counts the coded bytes on the wire and decodes them itself. The JDK transport used to read
|
||||
// the setting nowhere, and Apache and Jetty each asked on their own.
|
||||
builder.defaultHeader(HttpHeaders.ACCEPT_ENCODING, ContentDecoding.ACCEPTED);
|
||||
}
|
||||
RestClient restClient = builder.build();
|
||||
|
||||
AttemptResiliencePipeline pipeline = newResiliencePipeline(profile);
|
||||
|
||||
|
||||
+2
-1
@@ -8,7 +8,8 @@ import java.util.Set;
|
||||
* What a blocking transport can actually guarantee (design §13.3).
|
||||
*
|
||||
* <p>A capability that is weaker than the profile requires fails at startup rather than degrading
|
||||
* silently at runtime.
|
||||
* silently at runtime. {@code proxySupport} is an HTTP forward proxy, the only kind {@code
|
||||
* ProxyType} has.
|
||||
*/
|
||||
public record BlockingTransportCapabilities(
|
||||
Set<HttpProtocol> protocols,
|
||||
|
||||
+25
@@ -2,13 +2,18 @@ package dev.caskeleton.adapter.outbound.httpclient.autoconfigure;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
import static org.assertj.core.api.Assertions.assertThatThrownBy;
|
||||
import static org.assertj.core.api.Assertions.catchThrowable;
|
||||
|
||||
import dev.caskeleton.adapter.outbound.httpclient.profile.ClientMode;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.profile.HttpProtocol;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.profile.TransportType;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import org.junit.jupiter.api.DisplayName;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.springframework.boot.context.properties.bind.BindException;
|
||||
import org.springframework.boot.context.properties.bind.Binder;
|
||||
import org.springframework.boot.context.properties.source.MapConfigurationPropertySource;
|
||||
|
||||
/**
|
||||
* The aggregate rules a single client's own fields cannot express.
|
||||
@@ -96,6 +101,26 @@ class HttpClientPlatformSettingsTest {
|
||||
.hasMessageContaining("dynamic-targets");
|
||||
}
|
||||
|
||||
@Test
|
||||
void aSocksProxyTypeFailsToBind() {
|
||||
// given
|
||||
Binder binder =
|
||||
new Binder(
|
||||
new MapConfigurationPropertySource(
|
||||
Map.of("proxy.enabled", "true", "proxy.type", "SOCKS")));
|
||||
|
||||
// when
|
||||
Throwable refused =
|
||||
catchThrowable(() -> binder.bind("proxy", HttpClientPlatformSettings.Proxy.class));
|
||||
|
||||
// then
|
||||
assertThat(refused)
|
||||
.as("no shipped transport speaks SOCKS; the value used to bind and then misroute")
|
||||
.isInstanceOf(BindException.class)
|
||||
.rootCause()
|
||||
.hasMessageContaining("SOCKS");
|
||||
}
|
||||
|
||||
/**
|
||||
* A client whose only meaningful field here is its name.
|
||||
*
|
||||
|
||||
+23
-5
@@ -28,7 +28,18 @@ import org.springframework.http.client.JettyClientHttpRequestFactory;
|
||||
class JettyAttemptDeadlineTest {
|
||||
|
||||
private static final Duration DEADLINE = Duration.ofSeconds(1);
|
||||
private static final Duration TOLERANCE = Duration.ofMillis(500);
|
||||
private static final int DRIP_BYTES = 200;
|
||||
private static final Duration DRIP_GAP = Duration.ofMillis(50);
|
||||
|
||||
/** How long the drip alone takes to deliver the body: 200 bytes, 50 ms apart (10 s). */
|
||||
private static final Duration DRIP_TOTAL = DRIP_GAP.multipliedBy(DRIP_BYTES);
|
||||
|
||||
/**
|
||||
* Time a loaded machine may add between the deadline firing and the caller seeing the failure.
|
||||
* The bound only has to tell "cut at the deadline" from "waited for the drip", which the total
|
||||
* leaves room for (deadline + slack is under half of it).
|
||||
*/
|
||||
private static final Duration SCHEDULING_SLACK = Duration.ofSeconds(3);
|
||||
|
||||
@Test
|
||||
@DisplayName("Jetty 클라이언트는 바이트를 조금씩 흘리는 응답이라도 attempt deadline에 교환을 끊는다")
|
||||
@@ -39,13 +50,14 @@ class JettyAttemptDeadlineTest {
|
||||
client.start();
|
||||
try (MockHttpServer server = MockHttpServer.start()) {
|
||||
// given
|
||||
server.enqueueSlowDripBody(200, 100, Duration.ofMillis(50));
|
||||
server.enqueueSlowDripBody(200, DRIP_BYTES, DRIP_GAP);
|
||||
JettyClientHttpRequestFactory factory = new JettyClientHttpRequestFactory(client);
|
||||
long started = System.nanoTime();
|
||||
|
||||
// when
|
||||
Throwable failure;
|
||||
Duration elapsed;
|
||||
try (AttemptDeadline.Scope scope = AttemptDeadline.open(DEADLINE)) {
|
||||
long started = System.nanoTime();
|
||||
failure =
|
||||
catchThrowable(
|
||||
() -> {
|
||||
@@ -54,12 +66,18 @@ class JettyAttemptDeadlineTest {
|
||||
response.getBody().readAllBytes();
|
||||
}
|
||||
});
|
||||
// The exchange alone: client and server setup are not part of the deadline.
|
||||
elapsed = Duration.ofNanos(System.nanoTime() - started);
|
||||
}
|
||||
|
||||
// then
|
||||
Duration elapsed = Duration.ofNanos(System.nanoTime() - started);
|
||||
assertThat(failure).isInstanceOf(IOException.class);
|
||||
assertThat(elapsed).isLessThan(DEADLINE.plus(TOLERANCE));
|
||||
assertThat(DEADLINE.plus(SCHEDULING_SLACK))
|
||||
.as("the bound must stay well under the drip, or it proves nothing")
|
||||
.isLessThan(DRIP_TOTAL.dividedBy(2));
|
||||
assertThat(elapsed)
|
||||
.as("the deadline, not the drip, ended the exchange")
|
||||
.isLessThan(DEADLINE.plus(SCHEDULING_SLACK));
|
||||
} finally {
|
||||
client.stop();
|
||||
}
|
||||
|
||||
+38
@@ -0,0 +1,38 @@
|
||||
package dev.caskeleton.adapter.outbound.httpclient.http3;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
|
||||
import dev.caskeleton.adapter.outbound.httpclient.testkit.ClientProfiles;
|
||||
import org.eclipse.jetty.client.HttpClient;
|
||||
import org.junit.jupiter.api.Test;
|
||||
|
||||
/**
|
||||
* The experimental Jetty client hands the response reader the coded bytes (round9 review
|
||||
* 2026-11-12, RR9-01-30).
|
||||
*
|
||||
* <p>Jetty loads its gzip decoder from the classpath on start and then decodes every response
|
||||
* itself, which made the reader's wire count the decoded size. Over a plain TCP transport, as the
|
||||
* deadline test does: what is checked is the client's configuration, not the wire.
|
||||
*/
|
||||
class JettyContentDecodingTest {
|
||||
|
||||
@Test
|
||||
void aStartedClientDecodesNoContentCodingItself() throws Exception {
|
||||
// given
|
||||
HttpClient client =
|
||||
JettyHttp3TransportProvider.configure(
|
||||
new HttpClient(), ClientProfiles.http3Experimental("edge"));
|
||||
|
||||
// when
|
||||
client.start();
|
||||
|
||||
// then
|
||||
try {
|
||||
assertThat(client.getContentDecoderFactories().isEmpty())
|
||||
.as("a decoder left registered decodes the body before the reader can count it")
|
||||
.isTrue();
|
||||
} finally {
|
||||
client.stop();
|
||||
}
|
||||
}
|
||||
}
|
||||
+19
-3
@@ -28,7 +28,18 @@ import org.junit.jupiter.params.provider.EnumSource;
|
||||
class AttemptDeadlineEnforcementTest {
|
||||
|
||||
private static final Duration TOTAL_CALL = Duration.ofSeconds(1);
|
||||
private static final Duration TOLERANCE = Duration.ofMillis(500);
|
||||
private static final int DRIP_BYTES = 200;
|
||||
private static final Duration DRIP_GAP = Duration.ofMillis(50);
|
||||
|
||||
/** How long the drip alone takes to deliver the body: 200 bytes, 50 ms apart (10 s). */
|
||||
private static final Duration DRIP_TOTAL = DRIP_GAP.multipliedBy(DRIP_BYTES);
|
||||
|
||||
/**
|
||||
* Time a loaded machine may add between the deadline firing and the caller seeing the failure.
|
||||
* The bound only has to tell "cut at the deadline" from "waited for the server", which the
|
||||
* server's delay leaves room for (deadline + slack is under half of it).
|
||||
*/
|
||||
private static final Duration SCHEDULING_SLACK = Duration.ofSeconds(3);
|
||||
|
||||
@ParameterizedTest(name = "{0}")
|
||||
@EnumSource(
|
||||
@@ -45,7 +56,7 @@ class AttemptDeadlineEnforcementTest {
|
||||
.timeout(withTotalCall(TOTAL_CALL))
|
||||
.build())) {
|
||||
// given
|
||||
server.enqueueSlowDripBody(200, 100, Duration.ofMillis(50));
|
||||
server.enqueueSlowDripBody(200, DRIP_BYTES, DRIP_GAP);
|
||||
long started = System.nanoTime();
|
||||
|
||||
// when
|
||||
@@ -62,7 +73,12 @@ class AttemptDeadlineEnforcementTest {
|
||||
// then
|
||||
Duration elapsed = Duration.ofNanos(System.nanoTime() - started);
|
||||
assertThat(failure).isInstanceOf(HttpDeadlineExceededException.class);
|
||||
assertThat(elapsed).isLessThan(TOTAL_CALL.plus(TOLERANCE));
|
||||
assertThat(TOTAL_CALL.plus(SCHEDULING_SLACK))
|
||||
.as("the bound must stay well under the drip, or it proves nothing")
|
||||
.isLessThan(DRIP_TOTAL.dividedBy(2));
|
||||
assertThat(elapsed)
|
||||
.as("the deadline, not the drip, ended the call")
|
||||
.isLessThan(TOTAL_CALL.plus(SCHEDULING_SLACK));
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
+156
@@ -0,0 +1,156 @@
|
||||
package dev.caskeleton.adapter.outbound.httpclient.restclient;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
import static org.assertj.core.api.Assertions.catchThrowable;
|
||||
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.error.HttpResponseTooLargeException;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.error.HttpTargetRejectedException;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.operation.ExecutionEvidence;
|
||||
import java.io.ByteArrayInputStream;
|
||||
import java.io.ByteArrayOutputStream;
|
||||
import java.io.IOException;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.util.List;
|
||||
import java.util.zip.GZIPOutputStream;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.springframework.http.HttpHeaders;
|
||||
|
||||
/**
|
||||
* The response reader counts what arrives against the wire budget and what it decodes to against
|
||||
* the decoded budget, a chunk at a time.
|
||||
*
|
||||
* <p>The transport used to decode before the reader saw the body, so the "wire" count was the
|
||||
* decoded size and a well-compressed response was refused for a size it never had on the wire
|
||||
* (round9 review 2026-11-12, RR9-01-30).
|
||||
*/
|
||||
class RestClientResponseReaderEncodingTest {
|
||||
|
||||
private final RestClientResponseReader reader = new RestClientResponseReader();
|
||||
|
||||
@Test
|
||||
void aGzipBodyIsCountedCompressedOnTheWireAndDecodedForTheCaller() throws IOException {
|
||||
// given
|
||||
// Six thousand bytes that compress to a few dozen: inside a 1,000-byte wire budget only when
|
||||
// the wire count is the compressed size.
|
||||
byte[] decoded = "a".repeat(6_000).getBytes(StandardCharsets.US_ASCII);
|
||||
byte[] wire = gzip(decoded);
|
||||
ResponseSizeLimiter limiter = new ResponseSizeLimiter(1_000, 10_000);
|
||||
|
||||
// when
|
||||
RestClientResponseReader.RawResponse response =
|
||||
reader.readBounded(200, gzipHeaders(), new ByteArrayInputStream(wire), limiter);
|
||||
|
||||
// then
|
||||
assertThat(
|
||||
List.of(
|
||||
response.body().length,
|
||||
limiter.wireBytes(),
|
||||
limiter.decodedBytes(),
|
||||
response.headers().containsKey(HttpHeaders.CONTENT_ENCODING)))
|
||||
.containsExactly(6_000, (long) wire.length, 6_000L, false);
|
||||
}
|
||||
|
||||
@Test
|
||||
void aBodyThatDecodesPastTheBudgetIsRefusedWhileItIsDecoded() throws IOException {
|
||||
// given
|
||||
// A small wire payload that would expand to a megabyte: refused at the decoded budget, before
|
||||
// the whole expansion is held in memory.
|
||||
byte[] wire = gzip(new byte[1_048_576]);
|
||||
ResponseSizeLimiter limiter = new ResponseSizeLimiter(1_048_576, 64 * 1024);
|
||||
|
||||
// when
|
||||
Throwable refused =
|
||||
catchThrowable(
|
||||
() -> reader.readBounded(200, gzipHeaders(), new ByteArrayInputStream(wire), limiter));
|
||||
|
||||
// then
|
||||
assertThat(
|
||||
List.of(
|
||||
refused instanceof HttpResponseTooLargeException,
|
||||
limiter.decodedBytes() <= 64L * 1024 + 8 * 1024))
|
||||
.as("decoding stopped within one chunk past the budget: %s", refused)
|
||||
.containsExactly(true, true);
|
||||
}
|
||||
|
||||
@Test
|
||||
void anUnaskedForContentCodingIsRefused() {
|
||||
// given
|
||||
HttpHeaders brotli = new HttpHeaders();
|
||||
brotli.add(HttpHeaders.CONTENT_ENCODING, "br");
|
||||
ResponseSizeLimiter limiter = new ResponseSizeLimiter(1_000, 1_000);
|
||||
|
||||
// when
|
||||
Throwable refused =
|
||||
catchThrowable(
|
||||
() ->
|
||||
reader.readBounded(
|
||||
200, brotli, new ByteArrayInputStream(new byte[] {1, 2, 3}), limiter));
|
||||
|
||||
// then
|
||||
assertThat(refused)
|
||||
.as("a response arrived and was refused; it is not a call that got no answer")
|
||||
.isInstanceOfSatisfying(
|
||||
HttpTargetRejectedException.class,
|
||||
rejected ->
|
||||
assertThat(
|
||||
List.of(
|
||||
rejected.getMessage(),
|
||||
rejected.getCause().getMessage(),
|
||||
rejected.metadata().evidence()))
|
||||
.containsExactly(
|
||||
"response content coding is not one this client asked for",
|
||||
"the response used a content coding this client did not ask for: br",
|
||||
ExecutionEvidence.RESPONSE_RECEIVED));
|
||||
}
|
||||
|
||||
@Test
|
||||
void anEmptyBodyUnderGzipIsAnEmptyBody() throws IOException {
|
||||
// given
|
||||
// What a HEAD, a 204 or a 304 sends with the coding of the representation it describes.
|
||||
ResponseSizeLimiter limiter = new ResponseSizeLimiter(1_000, 1_000);
|
||||
|
||||
// when
|
||||
RestClientResponseReader.RawResponse response =
|
||||
reader.readBounded(204, gzipHeaders(), new ByteArrayInputStream(new byte[0]), limiter);
|
||||
|
||||
// then
|
||||
assertThat(response.body()).isEmpty();
|
||||
}
|
||||
|
||||
@Test
|
||||
void stackedContentCodingsAreRefused() {
|
||||
// given
|
||||
HttpHeaders stacked = new HttpHeaders();
|
||||
stacked.add(HttpHeaders.CONTENT_ENCODING, "gzip");
|
||||
stacked.add(HttpHeaders.CONTENT_ENCODING, "br");
|
||||
ResponseSizeLimiter limiter = new ResponseSizeLimiter(1_000, 1_000);
|
||||
|
||||
// when
|
||||
Throwable refused =
|
||||
catchThrowable(
|
||||
() ->
|
||||
reader.readBounded(
|
||||
200, stacked, new ByteArrayInputStream(new byte[] {1, 2, 3}), limiter));
|
||||
|
||||
// then
|
||||
assertThat(refused)
|
||||
.isInstanceOf(HttpTargetRejectedException.class)
|
||||
.hasMessage("response content coding is not one this client asked for")
|
||||
.cause()
|
||||
.hasMessageContaining("stacked content codings");
|
||||
}
|
||||
|
||||
private static HttpHeaders gzipHeaders() {
|
||||
HttpHeaders headers = new HttpHeaders();
|
||||
headers.add(HttpHeaders.CONTENT_ENCODING, "gzip");
|
||||
return headers;
|
||||
}
|
||||
|
||||
private static byte[] gzip(byte[] plain) throws IOException {
|
||||
ByteArrayOutputStream compressed = new ByteArrayOutputStream();
|
||||
try (GZIPOutputStream out = new GZIPOutputStream(compressed)) {
|
||||
out.write(plain);
|
||||
}
|
||||
return compressed.toByteArray();
|
||||
}
|
||||
}
|
||||
+149
@@ -0,0 +1,149 @@
|
||||
package dev.caskeleton.adapter.outbound.httpclient.restclient;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
import static org.assertj.core.api.Assertions.catchThrowable;
|
||||
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.OperationName;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.error.HttpResponseTooLargeException;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.error.HttpTargetRejectedException;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.operation.ExecutionEvidence;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.operation.HttpOperation;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.result.BlockingStreamingResponse;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.profile.ClientProfile;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.profile.RequestLimits;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.profile.ResponseLimits;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.profile.TransportType;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.testkit.ClientProfiles;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.testkit.MockHttpServer;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.testkit.TestGateways;
|
||||
import java.io.ByteArrayOutputStream;
|
||||
import java.io.IOException;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.Set;
|
||||
import java.util.zip.GZIPOutputStream;
|
||||
import org.junit.jupiter.params.ParameterizedTest;
|
||||
import org.junit.jupiter.params.provider.EnumSource;
|
||||
|
||||
/**
|
||||
* A streamed download is decoded the way a buffered body is, on every blocking transport.
|
||||
*
|
||||
* <p>Once the transports stopped decoding — so the wire budget counts what arrives — the streaming
|
||||
* path still handed the body straight on, and a compressed response reached the caller as gzip
|
||||
* bytes (round9 review 2026-11-12, RR9-01-30). The wire budget here is smaller than the decoded
|
||||
* body and larger than the compressed one: the download passes only when the wire is counted
|
||||
* compressed, which is also what fails if a transport goes back to decoding on its own.
|
||||
*/
|
||||
class StreamingContentCodingTest {
|
||||
|
||||
private static final byte[] DECODED = "a".repeat(6_000).getBytes(StandardCharsets.US_ASCII);
|
||||
|
||||
@ParameterizedTest(name = "{0}")
|
||||
@EnumSource(
|
||||
value = TransportType.class,
|
||||
names = {"APACHE", "JDK"})
|
||||
void aGzipDownloadReachesTheCallerDecodedAndCountedCompressedOnTheWire(TransportType transport)
|
||||
throws Exception {
|
||||
try (MockHttpServer server = MockHttpServer.start();
|
||||
TestGateways.Harness harness = TestGateways.forProfile(profile(server, transport))) {
|
||||
// given
|
||||
server.enqueueEncodedBody(200, "text/plain", "gzip", gzip(DECODED));
|
||||
BlockingStreamingGateway gateway = new BlockingStreamingGateway(harness.registry());
|
||||
|
||||
// when
|
||||
List<Object> received;
|
||||
try (BlockingStreamingResponse response = gateway.download(harness.profile().name(), get())) {
|
||||
received =
|
||||
List.of(
|
||||
response.body().readAllBytes().length,
|
||||
response.headers().keySet().stream()
|
||||
.anyMatch(name -> name.equalsIgnoreCase("Content-Encoding")));
|
||||
}
|
||||
|
||||
// then
|
||||
assertThat(received).containsExactly(DECODED.length, false);
|
||||
}
|
||||
}
|
||||
|
||||
@ParameterizedTest(name = "{0}")
|
||||
@EnumSource(
|
||||
value = TransportType.class,
|
||||
names = {"APACHE", "JDK"})
|
||||
void anIdentityDownloadPastTheDecodedBudgetFailsAsTheBufferedReadDoes(TransportType transport)
|
||||
throws Exception {
|
||||
try (MockHttpServer server = MockHttpServer.start();
|
||||
TestGateways.Harness harness =
|
||||
TestGateways.forProfile(
|
||||
profile(
|
||||
server, transport, new ResponseLimits(10_000, 1_000, Set.of("text/plain"))))) {
|
||||
// given
|
||||
// Within the wire budget and past the decoded one: an identity body is its own decoding.
|
||||
server.enqueueBody(200, "text/plain", DECODED);
|
||||
BlockingStreamingGateway gateway = new BlockingStreamingGateway(harness.registry());
|
||||
|
||||
// when
|
||||
Throwable failure;
|
||||
try (BlockingStreamingResponse response = gateway.download(harness.profile().name(), get())) {
|
||||
failure = catchThrowable(() -> response.body().readAllBytes());
|
||||
}
|
||||
|
||||
// then
|
||||
assertThat(failure)
|
||||
.isInstanceOf(HttpResponseTooLargeException.class)
|
||||
.hasMessageContaining("decoded bytes exceeded the profile limit of 1000");
|
||||
}
|
||||
}
|
||||
|
||||
@ParameterizedTest(name = "{0}")
|
||||
@EnumSource(
|
||||
value = TransportType.class,
|
||||
names = {"APACHE", "JDK"})
|
||||
void aStreamedResponseInACodingThisClientDidNotAskForIsRefusedAsReceived(TransportType transport)
|
||||
throws Exception {
|
||||
try (MockHttpServer server = MockHttpServer.start();
|
||||
TestGateways.Harness harness = TestGateways.forProfile(profile(server, transport))) {
|
||||
// given
|
||||
server.enqueueEncodedBody(200, "text/plain", "br", new byte[] {1, 2, 3});
|
||||
BlockingStreamingGateway gateway = new BlockingStreamingGateway(harness.registry());
|
||||
|
||||
// when
|
||||
Throwable refused = catchThrowable(() -> gateway.download(harness.profile().name(), get()));
|
||||
|
||||
// then
|
||||
assertThat(refused)
|
||||
.isInstanceOfSatisfying(
|
||||
HttpTargetRejectedException.class,
|
||||
rejected ->
|
||||
assertThat(rejected.metadata().evidence())
|
||||
.as("the server answered; the client refused what it sent")
|
||||
.isEqualTo(ExecutionEvidence.RESPONSE_RECEIVED));
|
||||
}
|
||||
}
|
||||
|
||||
private static HttpOperation get() {
|
||||
return HttpOperation.get(new OperationName("download"), "/download", Map.of());
|
||||
}
|
||||
|
||||
private static ClientProfile profile(MockHttpServer server, TransportType transport) {
|
||||
return profile(server, transport, new ResponseLimits(1_000, 10_000, Set.of("text/plain")));
|
||||
}
|
||||
|
||||
private static ClientProfile profile(
|
||||
MockHttpServer server, TransportType transport, ResponseLimits limits) {
|
||||
return ClientProfiles.builder("streaming-coding")
|
||||
.baseUrl(server.uri("/"))
|
||||
.transport(transport)
|
||||
.request(new RequestLimits(1024, true))
|
||||
.response(limits)
|
||||
.build();
|
||||
}
|
||||
|
||||
private static byte[] gzip(byte[] plain) throws IOException {
|
||||
ByteArrayOutputStream compressed = new ByteArrayOutputStream();
|
||||
try (GZIPOutputStream out = new GZIPOutputStream(compressed)) {
|
||||
out.write(plain);
|
||||
}
|
||||
return compressed.toByteArray();
|
||||
}
|
||||
}
|
||||
+15
-3
@@ -37,8 +37,15 @@ import org.junit.jupiter.params.provider.EnumSource;
|
||||
class StreamingSetupDeadlineTest {
|
||||
|
||||
private static final Duration TOTAL_CALL = Duration.ofSeconds(1);
|
||||
private static final Duration HEADERS_DELAY = Duration.ofMillis(1800);
|
||||
private static final Duration TOLERANCE = Duration.ofMillis(500);
|
||||
// Under the mock server's 5 s shutdown wait: it sleeps the delay out after the client gives up.
|
||||
private static final Duration HEADERS_DELAY = Duration.ofSeconds(5);
|
||||
|
||||
/**
|
||||
* Time a loaded machine may add between the deadline firing and the caller seeing the failure.
|
||||
* The bound only has to tell "cut at the deadline" from "waited for the late headers", which the
|
||||
* delay leaves room for (deadline + slack is under it).
|
||||
*/
|
||||
private static final Duration SCHEDULING_SLACK = Duration.ofSeconds(3);
|
||||
|
||||
@ParameterizedTest(name = "{0}")
|
||||
@EnumSource(
|
||||
@@ -70,7 +77,12 @@ class StreamingSetupDeadlineTest {
|
||||
// then
|
||||
Duration elapsed = Duration.ofNanos(System.nanoTime() - started);
|
||||
assertThat(failure).isInstanceOf(HttpDeadlineExceededException.class);
|
||||
assertThat(elapsed).isLessThan(TOTAL_CALL.plus(TOLERANCE));
|
||||
assertThat(TOTAL_CALL.plus(SCHEDULING_SLACK))
|
||||
.as("the bound must stay under the headers delay, or it proves nothing")
|
||||
.isLessThan(HEADERS_DELAY);
|
||||
assertThat(elapsed)
|
||||
.as("the deadline, not the late headers, ended the setup")
|
||||
.isLessThan(TOTAL_CALL.plus(SCHEDULING_SLACK));
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
+59
@@ -0,0 +1,59 @@
|
||||
package dev.caskeleton.adapter.outbound.httpclient.restclient;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.error.HttpDeadlineExceededException;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.error.HttpFailureMetadata;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.error.HttpRemoteErrorException;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.error.HttpTargetRejectedException;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.testkit.CoreFixtures;
|
||||
import java.util.stream.Stream;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.junit.jupiter.params.ParameterizedTest;
|
||||
import org.junit.jupiter.params.provider.Arguments;
|
||||
import org.junit.jupiter.params.provider.MethodSource;
|
||||
|
||||
/**
|
||||
* What a streaming setup failure reaches the caller as, decided apart from the race that produces
|
||||
* it: a test against a server cannot make the budget run out between the headers arriving and the
|
||||
* coding being refused, so the decision is held here directly.
|
||||
*/
|
||||
class StreamingSetupFailureTest {
|
||||
|
||||
private static final HttpFailureMetadata METADATA = CoreFixtures.ambiguousMetadata();
|
||||
|
||||
static Stream<Arguments> failuresThatKeepWhatTheyAre() {
|
||||
return Stream.of(
|
||||
Arguments.of(
|
||||
"예산을 다 쓴 뒤 거절한 응답", new HttpTargetRejectedException("refused coding", METADATA), true),
|
||||
Arguments.of(
|
||||
"예산을 다 쓴 뒤 원격 오류", new HttpRemoteErrorException("remote error", METADATA), true),
|
||||
Arguments.of("예산이 남은 transport 실패", new IllegalStateException("transport"), false));
|
||||
}
|
||||
|
||||
@ParameterizedTest(name = "{0}")
|
||||
@MethodSource("failuresThatKeepWhatTheyAre")
|
||||
void aFailureTheDeadlineDidNotCauseKeepsWhatItIs(
|
||||
String name, RuntimeException failure, boolean budgetSpent) {
|
||||
// when
|
||||
RuntimeException thrown = BlockingStreamingGateway.setupFailure(failure, budgetSpent, METADATA);
|
||||
|
||||
// then
|
||||
assertThat(thrown).isSameAs(failure);
|
||||
}
|
||||
|
||||
@Test
|
||||
void aTransportFailureAfterTheBudgetIsSpentIsTheDeadline() {
|
||||
// given
|
||||
IllegalStateException failure = new IllegalStateException("transport");
|
||||
|
||||
// when
|
||||
RuntimeException thrown = BlockingStreamingGateway.setupFailure(failure, true, METADATA);
|
||||
|
||||
// then
|
||||
assertThat(thrown)
|
||||
.isInstanceOf(HttpDeadlineExceededException.class)
|
||||
.hasMessage("effective deadline reached before the stream was set up")
|
||||
.hasCause(failure);
|
||||
}
|
||||
}
|
||||
+16
@@ -6,7 +6,10 @@ import static org.assertj.core.api.Assertions.assertThatThrownBy;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.api.error.HttpConfigurationException;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.profile.ClientMode;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.profile.ClientProfile;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.profile.ProxySettings;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.profile.ProxyType;
|
||||
import dev.caskeleton.adapter.outbound.httpclient.testkit.ClientProfiles;
|
||||
import java.util.Optional;
|
||||
import org.junit.jupiter.api.Test;
|
||||
|
||||
class TransportCapabilityValidatorTest {
|
||||
@@ -46,6 +49,19 @@ class TransportCapabilityValidatorTest {
|
||||
.hasMessageContaining("validated DNS pinning");
|
||||
}
|
||||
|
||||
@Test
|
||||
void acceptsAnHttpProxyOnATransportThatHasOne() {
|
||||
ClientProfile http =
|
||||
ClientProfiles.builder("egress")
|
||||
.proxy(
|
||||
new ProxySettings(
|
||||
true, "proxy.internal", 3128, ProxyType.HTTP, Optional.empty(), false))
|
||||
.build();
|
||||
|
||||
assertThatCode(() -> validator.validate(http, BlockingTransportCapabilities.apacheClassic()))
|
||||
.doesNotThrowAnyException();
|
||||
}
|
||||
|
||||
@Test
|
||||
void acceptsAMatchingCapabilitySet() {
|
||||
assertThatCode(
|
||||
|
||||
+16
@@ -121,6 +121,22 @@ public final class MockHttpServer implements AutoCloseable {
|
||||
.setBody(new okio.Buffer().write(body)));
|
||||
}
|
||||
|
||||
/**
|
||||
* Answers with a body already in {@code contentEncoding}, as a server that compresses would.
|
||||
*
|
||||
* @param body the coded bytes, sent as they are
|
||||
*/
|
||||
@SuppressWarnings("resource") // the okio Buffer, as in enqueueBody
|
||||
public void enqueueEncodedBody(
|
||||
int status, String contentType, String contentEncoding, byte[] body) {
|
||||
server.enqueue(
|
||||
new MockResponse()
|
||||
.setResponseCode(status)
|
||||
.setHeader("Content-Type", contentType)
|
||||
.setHeader("Content-Encoding", contentEncoding)
|
||||
.setBody(new okio.Buffer().write(body)));
|
||||
}
|
||||
|
||||
/** Accepts the request and never answers; drives response-header timeout tests. */
|
||||
public void enqueueNoResponse() {
|
||||
server.enqueue(new MockResponse().setSocketPolicy(SocketPolicy.NO_RESPONSE));
|
||||
|
||||
@@ -37,8 +37,8 @@
|
||||
- 주 classpath에 JSON Schema 엔진(NetworkNT)이 없다. 증거 manifest schema 검사와 capability card 검사
|
||||
(`release/`)는 이 leaf의 시험(`evidence/`)이다. snakeyaml은 시험에서만 쓴다(소유자 결정 2026-09-25).
|
||||
- 유스케이스 순서나 업무 라우팅 정책을 broker 어댑터 안에 숨기지 않는다.
|
||||
- `OutboxMessagePublishAdapter`는 위임만 하고 의존성 로그를 내지 않는다. 확정된 FAILED/DEAD 전이가 정해진
|
||||
ERROR 하나를 가진다.
|
||||
- `OutboxMessagePublishAdapter`는 발행을 위임하고, publisher가 알린 `longestAttempt`까지만 기다린다(넘으면 결과 불명).
|
||||
의존성 로그를 내지 않는다. 확정된 FAILED/DEAD 전이가 정해진 ERROR 하나를 가진다.
|
||||
|
||||
## 테스트
|
||||
|
||||
|
||||
@@ -35,6 +35,17 @@ publish pipeline을 탄다. producer id는 host/pod/process 이름에서 추측
|
||||
결과를 application 결과(CONFIRMED·AMBIGUOUS·REJECTED_BEFORE_SEND·REJECTED_AFTER_BROKER·
|
||||
REJECTED_RETRYABLE)로 번역해 relay에 돌려준다. 다시 직렬화하지 않는다.
|
||||
|
||||
그 결과를 기다리는 시간은 `longestAttempt()`(relay가 batch와 lease를 맞춰 본 값)까지다. 넘으면 기다리기를 멈추고
|
||||
`AMBIGUOUS`(`PUBLISH_WAIT_EXCEEDED`)로 돌려준다. broker에 닿았을 수 있어 확정도 거부도 아니다. 예전에는 상한 없이
|
||||
`join()`해서, 답하지 않는 publisher가 relay thread를 claim의 lease와 종료 대기 시간 너머까지 붙잡았고 그 사이 다른
|
||||
인스턴스가 같은 행을 다시 보냈다(round9 리뷰 2026-11-12, RR9-01-05). `longestAttempt()`를 모르는 publisher는 자기
|
||||
deadline이 유일한 상한이다. 0이나 음수를 알리는 publisher는 결함이다. 그 행은 보내지 않고 `REJECTED_RETRYABLE`
|
||||
(`PUBLISH_BOUND_NOT_POSITIVE`)로 돌려준다. relay는 시도로 세지 않고 backoff 뒤에 다시 보며, 실패 보고가 원인을 남긴다.
|
||||
|
||||
relay thread가 interrupt되면 둘로 갈린다. 기다리는 중이면 `AMBIGUOUS`(`PUBLISH_WAIT_INTERRUPTED`)이고 interrupt 표시를
|
||||
남긴다. 발행하기 전에 이미 interrupt됐으면 아무것도 보내지 않고 `REJECTED_RETRYABLE`(`PUBLISH_INTERRUPTED_BEFORE_SEND`)로
|
||||
돌려준다. relay는 그 행을 backoff 뒤로 미뤘다가 발행한다(실패 보고 한 줄).
|
||||
|
||||
platform `REJECTED` 중 `FailureDescriptor.retryable()`이 true인 것(종료 중 `SHUTTING_DOWN`, permit
|
||||
대기 초과, `PUBLISH_RUNTIME_UNAVAILABLE`, `KAFKA_TRANSPORT_CLOSED`, Kafka 인증·인가 실패)은
|
||||
`REJECTED_RETRYABLE`로 번역한다. relay는 이 값을 DEAD로 보내지 않고 재시도한다. 확정 거부 두 값으로
|
||||
|
||||
+67
-2
@@ -10,6 +10,11 @@ import dev.caskeleton.application.outbox.OutboxPublishRefusedException;
|
||||
import java.time.Duration;
|
||||
import java.util.Objects;
|
||||
import java.util.Optional;
|
||||
import java.util.concurrent.CompletableFuture;
|
||||
import java.util.concurrent.CompletionException;
|
||||
import java.util.concurrent.ExecutionException;
|
||||
import java.util.concurrent.TimeUnit;
|
||||
import java.util.concurrent.TimeoutException;
|
||||
|
||||
/**
|
||||
* The relay's publish port: a claimed row's validated event, delegated unchanged to the canonical
|
||||
@@ -19,9 +24,28 @@ import java.util.Optional;
|
||||
* here re-renders it. The canonical publisher's attempt — its outcome and its failure code — is
|
||||
* passed through unchanged, so the relay can tell a possibly-stored message (ambiguous) from a
|
||||
* refused one, and a refused credential from a closed transport.
|
||||
*
|
||||
* <p>The wait for that attempt is bounded by {@link #longestAttempt()}, the same value the relay
|
||||
* sized its batch against. It used to be an unbounded {@code join()}: a publisher whose stage never
|
||||
* completed held the relay thread past its claim's lease — another instance then sent the row again
|
||||
* — and past the shutdown wait the relay promises. A wait that runs out is {@link
|
||||
* OutboxPublishOutcome#AMBIGUOUS}: the message may have reached the broker, so it is neither
|
||||
* confirmed nor refused (round9 review 2026-11-12, RR9-01-05).
|
||||
*/
|
||||
public final class OutboxMessagePublishAdapter implements OutboxMessagePublishPort {
|
||||
|
||||
/** The code an attempt carries when this adapter stopped waiting for it. */
|
||||
static final String WAIT_EXCEEDED = "PUBLISH_WAIT_EXCEEDED";
|
||||
|
||||
/** The code an attempt carries when the relay thread was interrupted while waiting. */
|
||||
static final String WAIT_INTERRUPTED = "PUBLISH_WAIT_INTERRUPTED";
|
||||
|
||||
/** The code an attempt carries when the relay thread was interrupted before it published. */
|
||||
static final String INTERRUPTED_BEFORE_SEND = "PUBLISH_INTERRUPTED_BEFORE_SEND";
|
||||
|
||||
/** The code an attempt carries when the publisher reported a bound of nothing. */
|
||||
static final String BOUND_NOT_POSITIVE = "PUBLISH_BOUND_NOT_POSITIVE";
|
||||
|
||||
private final IntegrationEventPublishPort canonicalPublisher;
|
||||
|
||||
public OutboxMessagePublishAdapter(IntegrationEventPublishPort canonicalPublisher) {
|
||||
@@ -41,7 +65,10 @@ public final class OutboxMessagePublishAdapter implements OutboxMessagePublishPo
|
||||
throw new OutboxPublishRefusedException(outcome);
|
||||
}
|
||||
|
||||
/** The canonical publisher's, unchanged: this adapter adds no wait of its own. */
|
||||
/**
|
||||
* The canonical publisher's, unchanged: this adapter adds no wait of its own, and waits no longer
|
||||
* than this.
|
||||
*/
|
||||
@Override
|
||||
public Optional<Duration> longestAttempt() {
|
||||
return canonicalPublisher.longestAttempt();
|
||||
@@ -50,6 +77,44 @@ public final class OutboxMessagePublishAdapter implements OutboxMessagePublishPo
|
||||
@Override
|
||||
public OutboxPublishAttempt publishForOutcome(ClaimedOutboxEvent event) {
|
||||
Objects.requireNonNull(event, "event must not be null");
|
||||
return canonicalPublisher.publish(event.event()).toCompletableFuture().join();
|
||||
if (Thread.currentThread().isInterrupted()) {
|
||||
// Asked to stop before this row was sent: send nothing. The relay defers the row and
|
||||
// publishes it after its backoff; publishing and then reporting it ambiguous sent every
|
||||
// remaining row of the batch and left each to be sent again.
|
||||
return OutboxPublishAttempt.failed(
|
||||
OutboxPublishOutcome.REJECTED_RETRYABLE, INTERRUPTED_BEFORE_SEND);
|
||||
}
|
||||
Optional<Duration> bound = canonicalPublisher.longestAttempt();
|
||||
if (bound.isPresent() && (bound.get().isNegative() || bound.get().isZero())) {
|
||||
// A bound of nothing would make every publish ambiguous before the broker could answer. It
|
||||
// is the publisher's defect, not this row's: send nothing and let the relay retry the row
|
||||
// after its backoff, with the failure report naming the cause. Thrown, it counted against
|
||||
// the row's attempts and dead-lettered the whole outbox within minutes.
|
||||
return OutboxPublishAttempt.failed(
|
||||
OutboxPublishOutcome.REJECTED_RETRYABLE, BOUND_NOT_POSITIVE);
|
||||
}
|
||||
// The bound starts before publish() is called, not after it returns: a send can block the
|
||||
// caller for up to its own deadline before handing back a stage (a Kafka producer waiting on
|
||||
// metadata), and a wait measured from the return let one row take the bound twice over while
|
||||
// the relay sized its lease for once.
|
||||
long deadline = System.nanoTime() + bound.map(Duration::toNanos).orElse(0L);
|
||||
CompletableFuture<OutboxPublishAttempt> attempt =
|
||||
canonicalPublisher.publish(event.event()).toCompletableFuture();
|
||||
if (bound.isEmpty()) {
|
||||
// A publisher that cannot say how long it takes gave the relay nothing to size its batch
|
||||
// against either; its own deadline is the only bound there is.
|
||||
return attempt.join();
|
||||
}
|
||||
try {
|
||||
return attempt.get(Math.max(0L, deadline - System.nanoTime()), TimeUnit.NANOSECONDS);
|
||||
} catch (TimeoutException exceeded) {
|
||||
return OutboxPublishAttempt.failed(OutboxPublishOutcome.AMBIGUOUS, WAIT_EXCEEDED);
|
||||
} catch (InterruptedException interrupted) {
|
||||
Thread.currentThread().interrupt();
|
||||
return OutboxPublishAttempt.failed(OutboxPublishOutcome.AMBIGUOUS, WAIT_INTERRUPTED);
|
||||
} catch (ExecutionException failed) {
|
||||
// As join() reported it, so the relay's handling of a failed stage does not change.
|
||||
throw new CompletionException(failed.getCause());
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
+150
@@ -2,16 +2,24 @@ package dev.caskeleton.adapter.outbound.messaging.outbox;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
import static org.assertj.core.api.Assertions.catchThrowable;
|
||||
import static org.junit.jupiter.api.Assertions.assertTimeoutPreemptively;
|
||||
|
||||
import dev.caskeleton.application.messaging.event.testkit.BrokerAnswer;
|
||||
import dev.caskeleton.application.messaging.event.testkit.FakeIntegrationEventPublisher;
|
||||
import dev.caskeleton.application.outbox.ClaimedOutboxEvent;
|
||||
import dev.caskeleton.application.outbox.OutboxMessagePublishPort;
|
||||
import dev.caskeleton.application.outbox.OutboxPublishAmbiguousException;
|
||||
import dev.caskeleton.application.outbox.OutboxPublishAttempt;
|
||||
import dev.caskeleton.application.outbox.OutboxPublishOutcome;
|
||||
import dev.caskeleton.application.outbox.OutboxPublishRefusedException;
|
||||
import dev.caskeleton.application.outbox.testkit.OutboxMessagePublishContract;
|
||||
import java.time.Duration;
|
||||
import java.util.List;
|
||||
import java.util.concurrent.CompletionException;
|
||||
import java.util.concurrent.CountDownLatch;
|
||||
import java.util.concurrent.TimeUnit;
|
||||
import java.util.concurrent.atomic.AtomicBoolean;
|
||||
import java.util.concurrent.atomic.AtomicReference;
|
||||
import org.junit.jupiter.api.DisplayName;
|
||||
import org.junit.jupiter.api.Test;
|
||||
|
||||
@@ -49,6 +57,148 @@ class OutboxMessagePublishAdapterTest extends OutboxMessagePublishContract {
|
||||
assertThat(canonical.published()).containsExactly(claim.event());
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("broker가 longestAttempt 안에 답하지 않으면 더 기다리지 않고 결과 불명으로 끝낸다")
|
||||
void aPublishTheBrokerNeverAnswersEndsAmbiguousAtTheLongestAttempt() {
|
||||
// given
|
||||
OutboxMessagePublishPort adapter =
|
||||
new OutboxMessagePublishAdapter(canonical.silentPast(Duration.ofMillis(50)));
|
||||
|
||||
// when
|
||||
// A wait with no bound never returns here; ten seconds is far past the 50 ms the publisher
|
||||
// promised and far short of a hung relay.
|
||||
OutboxPublishAttempt attempt =
|
||||
assertTimeoutPreemptively(
|
||||
Duration.ofSeconds(10), () -> adapter.publishForOutcome(claimedRow()));
|
||||
|
||||
// then
|
||||
assertThat(attempt)
|
||||
.isEqualTo(
|
||||
OutboxPublishAttempt.failed(
|
||||
OutboxPublishOutcome.AMBIGUOUS, OutboxMessagePublishAdapter.WAIT_EXCEEDED));
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("publish()가 호출자를 붙잡은 시간도 longestAttempt 안에 센다")
|
||||
void theWaitIncludesTheTimePublishHeldTheCaller() {
|
||||
// given
|
||||
// Blocks 1 s before handing back a stage that never completes, with a 1.2 s promise: a wait
|
||||
// measured from the return would end at about 2.2 s, one measured from the call at 1.2 s. The
|
||||
// 2 s threshold sits between them, 0.8 s above the right answer, so a loaded runner has room.
|
||||
OutboxMessagePublishPort adapter =
|
||||
new OutboxMessagePublishAdapter(
|
||||
canonical.silentPast(Duration.ofMillis(1_200)).blockingFor(Duration.ofSeconds(1)));
|
||||
long started = System.nanoTime();
|
||||
|
||||
// when
|
||||
var unused =
|
||||
assertTimeoutPreemptively(
|
||||
Duration.ofSeconds(10), () -> adapter.publishForOutcome(claimedRow()));
|
||||
|
||||
// then
|
||||
assertThat(Duration.ofNanos(System.nanoTime() - started))
|
||||
.as("one row used the bound twice over while the relay sized its lease for once")
|
||||
.isLessThan(Duration.ofSeconds(2));
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("기다리던 relay 스레드가 interrupt되면 결과 불명으로 끝내고 interrupt 표시를 남긴다")
|
||||
void anInterruptedWaitEndsAmbiguousAndKeepsTheInterrupt() throws Exception {
|
||||
// given
|
||||
OutboxMessagePublishPort adapter =
|
||||
new OutboxMessagePublishAdapter(canonical.silentPast(Duration.ofSeconds(30)));
|
||||
AtomicReference<OutboxPublishAttempt> attempt = new AtomicReference<>();
|
||||
AtomicBoolean stillInterrupted = new AtomicBoolean();
|
||||
CountDownLatch waiting = new CountDownLatch(1);
|
||||
Thread relay =
|
||||
new Thread(
|
||||
() -> {
|
||||
waiting.countDown();
|
||||
attempt.set(adapter.publishForOutcome(claimedRow()));
|
||||
stillInterrupted.set(Thread.currentThread().isInterrupted());
|
||||
});
|
||||
relay.start();
|
||||
assertThat(waiting.await(5, TimeUnit.SECONDS)).as("the relay thread started").isTrue();
|
||||
// Until the publish has been handed out, an interrupt is the before-the-send case below.
|
||||
while (canonical.published().isEmpty()) {
|
||||
Thread.onSpinWait();
|
||||
}
|
||||
|
||||
// when
|
||||
relay.interrupt();
|
||||
boolean ended = relay.join(Duration.ofSeconds(10));
|
||||
|
||||
// then
|
||||
assertThat(List.of(ended, attempt.get(), stillInterrupted.get()))
|
||||
.containsExactly(
|
||||
true,
|
||||
OutboxPublishAttempt.failed(
|
||||
OutboxPublishOutcome.AMBIGUOUS, OutboxMessagePublishAdapter.WAIT_INTERRUPTED),
|
||||
true);
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("이미 interrupt된 relay 스레드는 발행하지 않고 다시 claim할 수 있는 거부로 답한다")
|
||||
void anAlreadyInterruptedRelayPublishesNothing() {
|
||||
// given
|
||||
OutboxMessagePublishPort adapter =
|
||||
new OutboxMessagePublishAdapter(canonical.silentPast(Duration.ofSeconds(30)));
|
||||
Thread.currentThread().interrupt();
|
||||
|
||||
// when
|
||||
OutboxPublishAttempt attempt;
|
||||
try {
|
||||
attempt = adapter.publishForOutcome(claimedRow());
|
||||
} finally {
|
||||
Thread.interrupted();
|
||||
}
|
||||
|
||||
// then
|
||||
assertThat(List.of(attempt, canonical.published()))
|
||||
.as("a relay asked to stop sent the rest of its batch and left each row to be sent again")
|
||||
.containsExactly(
|
||||
OutboxPublishAttempt.failed(
|
||||
OutboxPublishOutcome.REJECTED_RETRYABLE,
|
||||
OutboxMessagePublishAdapter.INTERRUPTED_BEFORE_SEND),
|
||||
List.of());
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("longestAttempt가 0인 publisher에는 보내지 않고, 시도로 세지 않는 거부로 답한다")
|
||||
void aPublisherReportingAZeroBoundIsSentNothing() {
|
||||
// given
|
||||
OutboxMessagePublishPort adapter =
|
||||
new OutboxMessagePublishAdapter(canonical.silentPast(Duration.ZERO));
|
||||
|
||||
// when
|
||||
OutboxPublishAttempt attempt = adapter.publishForOutcome(claimedRow());
|
||||
|
||||
// then
|
||||
assertThat(List.of(attempt, canonical.published()))
|
||||
.as("thrown, the defect counted against every row's attempts and dead-lettered the outbox")
|
||||
.containsExactly(
|
||||
OutboxPublishAttempt.failed(
|
||||
OutboxPublishOutcome.REJECTED_RETRYABLE,
|
||||
OutboxMessagePublishAdapter.BOUND_NOT_POSITIVE),
|
||||
List.of());
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("실패로 끝난 발행 stage는 join()과 같은 CompletionException으로 알린다")
|
||||
void aFailedStageIsReportedAsJoinReportedIt() {
|
||||
// given
|
||||
IllegalStateException broker = new IllegalStateException("broker refused the connection");
|
||||
OutboxMessagePublishPort adapter =
|
||||
new OutboxMessagePublishAdapter(
|
||||
canonical.silentPast(Duration.ofSeconds(5)).failingWith(broker));
|
||||
|
||||
// when
|
||||
Throwable failure = catchThrowable(() -> adapter.publishForOutcome(claimedRow()));
|
||||
|
||||
// then
|
||||
assertThat(failure).isInstanceOf(CompletionException.class).hasCause(broker);
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("확인을 받지 못한 발행은 예외로 알리는 메서드에서 AMBIGUOUS 신호를 던진다")
|
||||
void anAmbiguousPublishThrowsTheAmbiguousSignalFromTheThrowingMethod() {
|
||||
|
||||
+28
@@ -5,6 +5,7 @@ import dev.caskeleton.adapter.outbound.notification.platform.dispatch.ProviderRu
|
||||
import dev.caskeleton.application.notification.platform.api.ProviderProfileId;
|
||||
import dev.caskeleton.application.notification.platform.api.routing.Channel;
|
||||
import dev.caskeleton.application.notification.platform.provider.ReconciliationCapability;
|
||||
import java.time.Duration;
|
||||
import java.util.ArrayList;
|
||||
import java.util.EnumMap;
|
||||
import java.util.HashMap;
|
||||
@@ -112,6 +113,10 @@ public final class NotificationProviderAssembly {
|
||||
Map<ProviderProfileId, ReconciliationCapability> reconciliations = new LinkedHashMap<>();
|
||||
Map<Channel, List<String>> claimants = new EnumMap<>(Channel.class);
|
||||
|
||||
if (mode != NotificationPlatformMode.INGEST_ONLY) {
|
||||
requireFamiliesTogether(configured, settings.dispatch().leaseDuration());
|
||||
}
|
||||
|
||||
for (Map.Entry<String, NotificationPlatformSettings.Provider> entry : configured.entrySet()) {
|
||||
String profileId = entry.getKey();
|
||||
NotificationPlatformSettings.Provider profile = entry.getValue();
|
||||
@@ -198,6 +203,29 @@ public final class NotificationProviderAssembly {
|
||||
}
|
||||
}
|
||||
|
||||
/** Hands each family its enabled profiles at once, for the limits only the set can break. */
|
||||
private void requireFamiliesTogether(
|
||||
Map<String, NotificationPlatformSettings.Provider> configured, Duration deliveryLease) {
|
||||
Map<ProviderType, Map<String, NotificationPlatformSettings.Provider>> byFamily =
|
||||
new EnumMap<>(ProviderType.class);
|
||||
configured.forEach(
|
||||
(profileId, profile) -> {
|
||||
if (profile.enabled()) {
|
||||
byFamily
|
||||
.computeIfAbsent(
|
||||
ProviderType.parse(profileId, profile.type()), family -> new TreeMap<>())
|
||||
.put(profileId, profile);
|
||||
}
|
||||
});
|
||||
byFamily.forEach(
|
||||
(family, profiles) -> {
|
||||
ProviderRuntimeAssembler assembler = assemblers.get(family);
|
||||
if (assembler != null) {
|
||||
assembler.requireTogether(Map.copyOf(profiles), deliveryLease);
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
private static String missingAssemblerMessage(String profileId, ProviderType type) {
|
||||
if (type == ProviderType.SMTP) {
|
||||
return "notification provider profile '"
|
||||
|
||||
+17
-3
@@ -103,24 +103,38 @@ public class NotificationSmtpProviderConfig {
|
||||
return new SmtpFailureClassifier();
|
||||
}
|
||||
|
||||
/** How many sends may wait per send thread before the pool refuses one. */
|
||||
static final int QUEUED_SENDS_PER_THREAD = 4;
|
||||
|
||||
/**
|
||||
* Where an SMTP send runs.
|
||||
*
|
||||
* <p>Bounded, and it aborts rather than queues without limit: an SMTP relay that has stopped
|
||||
* answering turns an unbounded queue into retained heap and a dispatch worker that never notices.
|
||||
* {@code CallerRunsPolicy} would instead block the dispatch loop on the slow relay, which stalls
|
||||
* every other channel too.
|
||||
* every other channel too. A rejected send never reached the relay, and the adapter reports it so
|
||||
* ({@code ProviderCallNotStartedException}), not as an ambiguous submission.
|
||||
*
|
||||
* <p>{@code dispatch-concurrency} threads from the first send, idle ones timing out. The core
|
||||
* size used to be one: a {@code ThreadPoolExecutor} adds a thread past its core only when the
|
||||
* queue is full, and the profile's limiter admits fewer sends than the queue holds, so every SMTP
|
||||
* send ran on one thread whatever {@code dispatch-concurrency} said (round9 review 2026-11-12,
|
||||
* RR9-01-15).
|
||||
*
|
||||
* <p>A deployment's own executor must not run a send on the submitting thread, as {@code
|
||||
* CallerRunsPolicy} or a direct executor does: the send's two timeout deadlines complete its
|
||||
* future, and a dispatch thread blocked inside {@code execute} never gets to read it.
|
||||
*/
|
||||
@Bean(name = "notificationSmtpExecutor", destroyMethod = "shutdown")
|
||||
@ConditionalOnMissingBean(name = "notificationSmtpExecutor")
|
||||
public Executor notificationSmtpExecutor(NotificationSmtpSettings settings) {
|
||||
ThreadPoolExecutor executor =
|
||||
new ThreadPoolExecutor(
|
||||
1,
|
||||
settings.dispatchConcurrency(),
|
||||
settings.dispatchConcurrency(),
|
||||
60L,
|
||||
TimeUnit.SECONDS,
|
||||
new LinkedBlockingQueue<>(settings.dispatchConcurrency() * 4),
|
||||
new LinkedBlockingQueue<>(settings.dispatchConcurrency() * QUEUED_SENDS_PER_THREAD),
|
||||
runnable -> {
|
||||
Thread thread = new Thread(runnable, "notification-smtp");
|
||||
thread.setDaemon(true);
|
||||
|
||||
+18
@@ -1,5 +1,8 @@
|
||||
package dev.caskeleton.adapter.outbound.notification.platform.autoconfigure;
|
||||
|
||||
import java.time.Duration;
|
||||
import java.util.Map;
|
||||
|
||||
/**
|
||||
* Turns one configured profile into one working provider.
|
||||
*
|
||||
@@ -32,4 +35,19 @@ public interface ProviderRuntimeAssembler {
|
||||
* is missing
|
||||
*/
|
||||
AssembledProvider assemble(String profileId, NotificationPlatformSettings.Provider profile);
|
||||
|
||||
/**
|
||||
* Refuses this family's enabled profiles together, before any of them is assembled.
|
||||
*
|
||||
* <p>Some limits are shared by every profile of a family — one send pool — and can only be
|
||||
* checked against all of them at once; some depend on the platform's delivery lease, which no
|
||||
* single profile carries. Called once per boot with the whole set, so it holds no state between
|
||||
* calls. The default accepts.
|
||||
*
|
||||
* @param profiles the family's enabled profiles, by id
|
||||
* @param deliveryLease {@code app.notification.platform.dispatch.lease-duration}
|
||||
* @throws IllegalStateException naming the profile and the limit it breaks
|
||||
*/
|
||||
default void requireTogether(
|
||||
Map<String, NotificationPlatformSettings.Provider> profiles, Duration deliveryLease) {}
|
||||
}
|
||||
|
||||
+100
-1
@@ -15,9 +15,12 @@ import dev.caskeleton.application.notification.platform.provider.ProviderCapabil
|
||||
import dev.caskeleton.application.notification.platform.provider.ProviderProfileSnapshot;
|
||||
import dev.caskeleton.application.notification.platform.security.ContactPointProtector;
|
||||
import java.time.Clock;
|
||||
import java.time.Duration;
|
||||
import java.util.Map;
|
||||
import java.util.Objects;
|
||||
import java.util.TreeMap;
|
||||
import java.util.concurrent.Executor;
|
||||
import java.util.concurrent.ThreadPoolExecutor;
|
||||
|
||||
/**
|
||||
* The first production assembler: one configured SMTP profile becomes one working provider
|
||||
@@ -98,10 +101,18 @@ public final class SmtpProviderRuntimeAssembler implements ProviderRuntimeAssemb
|
||||
String profileId, NotificationPlatformSettings.Provider profile) {
|
||||
Objects.requireNonNull(profileId, "profileId");
|
||||
Objects.requireNonNull(profile, "profile");
|
||||
requireTimeoutOutlastsOneStalledStep(profileId, profile.timeout());
|
||||
|
||||
SmtpNotificationProviderAdapter adapter =
|
||||
new SmtpNotificationProviderAdapter(
|
||||
dispatch, mimeFactory, classifier, protector, transport, executor, attachmentGuard);
|
||||
dispatch,
|
||||
mimeFactory,
|
||||
classifier,
|
||||
protector,
|
||||
transport,
|
||||
executor,
|
||||
attachmentGuard,
|
||||
profile.timeout());
|
||||
|
||||
ProviderRuntime runtime =
|
||||
new ProviderRuntime(
|
||||
@@ -112,6 +123,94 @@ public final class SmtpProviderRuntimeAssembler implements ProviderRuntimeAssemb
|
||||
return AssembledProvider.dispatchOnly(runtime, Channel.EMAIL);
|
||||
}
|
||||
|
||||
/**
|
||||
* Refuses a profile timeout that a single stalled connect and command can outlast.
|
||||
*
|
||||
* <p>The profile timeout bounds the whole send and reports what outlasts it as ambiguous, since
|
||||
* the DATA may have gone through. A relay that accepts the connection and never answers the
|
||||
* greeting has been sent nothing; its socket deadline reports that, as a failure the classifier
|
||||
* knows is safe to retry. With the profile timeout shorter than the connect deadline plus one
|
||||
* command's, the profile timeout fired first and turned that safe retry into an ambiguous attempt
|
||||
* an operator reconciles by hand (round9 review 2026-11-12, RR9-01-15).
|
||||
*/
|
||||
private void requireTimeoutOutlastsOneStalledStep(String profileId, Duration timeout) {
|
||||
Duration oneCommand =
|
||||
transport.readTimeout().compareTo(transport.writeTimeout()) >= 0
|
||||
? transport.readTimeout()
|
||||
: transport.writeTimeout();
|
||||
Duration stalledStep = transport.connectTimeout().plus(oneCommand);
|
||||
if (timeout.compareTo(stalledStep) <= 0) {
|
||||
throw new IllegalStateException(
|
||||
"notification provider profile '"
|
||||
+ profileId
|
||||
+ "': timeout ("
|
||||
+ timeout
|
||||
+ ") must be longer than mail.smtp.connectiontimeout plus the longer of"
|
||||
+ " mail.smtp.timeout and mail.smtp.writetimeout ("
|
||||
+ stalledStep
|
||||
+ "), or a relay that never answers is reported as an ambiguous send");
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Refuses SMTP profiles whose in-flight limits together are more than the send pool can hold, and
|
||||
* a delivery lease one SMTP send can outlast.
|
||||
*
|
||||
* <p>The limiter lets {@code max-concurrency} sends through per profile; every SMTP profile
|
||||
* shares the one pool, which runs its threads, queues the rest up to its queue's capacity, and
|
||||
* rejects what is left. Profiles allowed more in flight together than threads plus queue had
|
||||
* their excess rejected on every burst. Only a {@link ThreadPoolExecutor} says how much it holds,
|
||||
* so a deployment's own executor is not checked (round9 review 2026-11-12, RR9-01-16).
|
||||
*
|
||||
* <p>One send may wait in the pool for up to the profile's timeout and then run for up to the
|
||||
* timeout again, so it holds its delivery's lease for up to twice the timeout. A lease shorter
|
||||
* than that shows a running send as a stuck lease; if the send is still running one lease later,
|
||||
* lease recovery takes the delivery for abandoned and, since an SMTP relay cannot be asked, parks
|
||||
* it for an operator. A send that finishes after that still records its result.
|
||||
*/
|
||||
@Override
|
||||
public void requireTogether(
|
||||
Map<String, NotificationPlatformSettings.Provider> profiles, Duration deliveryLease) {
|
||||
new TreeMap<>(profiles)
|
||||
.forEach(
|
||||
(profileId, profile) -> {
|
||||
Duration oneSend = profile.timeout().multipliedBy(2);
|
||||
if (deliveryLease.compareTo(oneSend) <= 0) {
|
||||
throw new IllegalStateException(
|
||||
"notification provider profile '"
|
||||
+ profileId
|
||||
+ "': app.notification.platform.dispatch.lease-duration ("
|
||||
+ deliveryLease
|
||||
+ ") must be longer than twice the SMTP timeout ("
|
||||
+ oneSend
|
||||
+ "): a send may wait in the pool for the timeout and then run for it");
|
||||
}
|
||||
});
|
||||
if (!(executor instanceof ThreadPoolExecutor pool)) {
|
||||
return;
|
||||
}
|
||||
int threads = pool.getMaximumPoolSize();
|
||||
int queued = pool.getQueue().remainingCapacity();
|
||||
long holds = (long) threads + queued;
|
||||
long together =
|
||||
profiles.values().stream()
|
||||
.mapToLong(NotificationPlatformSettings.Provider::maxConcurrency)
|
||||
.sum();
|
||||
if (together > holds) {
|
||||
throw new IllegalStateException(
|
||||
"notification provider profiles "
|
||||
+ new TreeMap<>(profiles).keySet()
|
||||
+ ": max-concurrency ("
|
||||
+ together
|
||||
+ " together) is more than the SMTP send pool holds ("
|
||||
+ threads
|
||||
+ " threads and "
|
||||
+ queued
|
||||
+ " queued) — raise app.notification.platform.smtp.dispatch-concurrency or lower"
|
||||
+ " max-concurrency");
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* The snapshot the dispatch loop plans against. Its capabilities are the adapter's own, as the
|
||||
* SES assembler does: a second copy here had drifted to a different payload limit.
|
||||
|
||||
+138
-2
@@ -1,13 +1,20 @@
|
||||
package dev.caskeleton.adapter.outbound.notification.platform.provider.smtp;
|
||||
|
||||
import static java.util.concurrent.TimeUnit.MILLISECONDS;
|
||||
|
||||
import dev.caskeleton.adapter.outbound.notification.platform.provider.EmailAttachments;
|
||||
import dev.caskeleton.adapter.outbound.notification.platform.provider.EmailRecipients;
|
||||
import dev.caskeleton.application.notification.platform.api.ProviderId;
|
||||
import dev.caskeleton.application.notification.platform.api.error.FailureCategory;
|
||||
import dev.caskeleton.application.notification.platform.api.error.NotificationFailureCode;
|
||||
import dev.caskeleton.application.notification.platform.api.routing.Channel;
|
||||
import dev.caskeleton.application.notification.platform.contact.EmailAddress;
|
||||
import dev.caskeleton.application.notification.platform.dispatch.AttachmentIntegrityGuard;
|
||||
import dev.caskeleton.application.notification.platform.provider.NotificationProviderAdapter;
|
||||
import dev.caskeleton.application.notification.platform.provider.ProviderCallNotStartedException;
|
||||
import dev.caskeleton.application.notification.platform.provider.ProviderCapabilities;
|
||||
import dev.caskeleton.application.notification.platform.provider.ProviderExecutionEvidence;
|
||||
import dev.caskeleton.application.notification.platform.provider.ProviderFailure;
|
||||
import dev.caskeleton.application.notification.platform.provider.ProviderSubmission;
|
||||
import dev.caskeleton.application.notification.platform.provider.ProviderSubmissionResult;
|
||||
import dev.caskeleton.application.notification.platform.provider.ResolvedAttachment;
|
||||
@@ -15,10 +22,13 @@ import dev.caskeleton.application.notification.platform.security.ContactPointPro
|
||||
import java.time.Duration;
|
||||
import java.util.List;
|
||||
import java.util.Objects;
|
||||
import java.util.Optional;
|
||||
import java.util.Set;
|
||||
import java.util.concurrent.CompletableFuture;
|
||||
import java.util.concurrent.CompletionStage;
|
||||
import java.util.concurrent.Executor;
|
||||
import java.util.concurrent.RejectedExecutionException;
|
||||
import java.util.concurrent.atomic.AtomicReference;
|
||||
|
||||
/**
|
||||
* SMTP email adapter.
|
||||
@@ -39,7 +49,14 @@ public final class SmtpNotificationProviderAdapter implements NotificationProvid
|
||||
private final SmtpProviderProperties properties;
|
||||
private final Executor executor;
|
||||
private final AttachmentIntegrityGuard attachmentGuard;
|
||||
private final Duration callTimeout;
|
||||
|
||||
/**
|
||||
* @param callTimeout the profile's {@code timeout}: the longest one send may wait in the pool,
|
||||
* and then run, before it is reported as not started or as ambiguous. Startup holds it above
|
||||
* one stalled connect and command
|
||||
* @throws IllegalStateException if the timeout is not positive, which is the assembly's fault
|
||||
*/
|
||||
public SmtpNotificationProviderAdapter(
|
||||
SmtpDispatch dispatch,
|
||||
SmtpMimeMessageFactory mimeFactory,
|
||||
@@ -47,7 +64,8 @@ public final class SmtpNotificationProviderAdapter implements NotificationProvid
|
||||
ContactPointProtector protector,
|
||||
SmtpProviderProperties properties,
|
||||
Executor executor,
|
||||
AttachmentIntegrityGuard attachmentGuard) {
|
||||
AttachmentIntegrityGuard attachmentGuard,
|
||||
Duration callTimeout) {
|
||||
this.dispatch = Objects.requireNonNull(dispatch, "dispatch");
|
||||
this.mimeFactory = Objects.requireNonNull(mimeFactory, "mimeFactory");
|
||||
this.classifier = Objects.requireNonNull(classifier, "classifier");
|
||||
@@ -55,6 +73,10 @@ public final class SmtpNotificationProviderAdapter implements NotificationProvid
|
||||
this.properties = Objects.requireNonNull(properties, "properties");
|
||||
this.executor = Objects.requireNonNull(executor, "executor");
|
||||
this.attachmentGuard = Objects.requireNonNull(attachmentGuard, "attachmentGuard");
|
||||
this.callTimeout = Objects.requireNonNull(callTimeout, "callTimeout");
|
||||
if (callTimeout.isNegative() || callTimeout.isZero()) {
|
||||
throw new IllegalStateException("callTimeout must be positive");
|
||||
}
|
||||
}
|
||||
|
||||
@Override
|
||||
@@ -77,7 +99,109 @@ public final class SmtpNotificationProviderAdapter implements NotificationProvid
|
||||
@Override
|
||||
public CompletionStage<ProviderSubmissionResult> submit(ProviderSubmission submission) {
|
||||
Objects.requireNonNull(submission, "submission");
|
||||
return CompletableFuture.supplyAsync(() -> send(submission), executor);
|
||||
PooledSend send = new PooledSend(submission);
|
||||
try {
|
||||
executor.execute(send);
|
||||
} catch (RejectedExecutionException full) {
|
||||
// The send pool refused the task: nothing was handed to the relay. Reported as not started,
|
||||
// so the delivery is retried, rather than falling through to the ambiguous residue an
|
||||
// operator has to reconcile by hand (round9 review 2026-11-12, RR9-01-16).
|
||||
throw new ProviderCallNotStartedException(
|
||||
NotificationFailureCode.PROVIDER_UNAVAILABLE,
|
||||
FailureCategory.CAPACITY_REJECTED,
|
||||
true,
|
||||
"the SMTP send pool is full; the message was not handed to the relay");
|
||||
}
|
||||
send.armQueueDeadline();
|
||||
return send.result();
|
||||
}
|
||||
|
||||
/**
|
||||
* One send through the pool, bounded by the profile's timeout twice over.
|
||||
*
|
||||
* <p>Waiting in the pool: a send that has not started when the timeout passes is abandoned and
|
||||
* never runs. Nothing reached the relay, so it is reported as not started and retried; the old
|
||||
* single timer, started at the submit, closed it as ambiguous and the send then went out anyway,
|
||||
* after its permit was released.
|
||||
*
|
||||
* <p>Running: from the moment a pool thread takes it, the send has the whole timeout. The socket
|
||||
* timeouts bound each SMTP command, not the conversation, so a relay answering slowly but within
|
||||
* each one held the send past the delivery lease: it showed as a stuck lease, and one lease later
|
||||
* lease recovery parked the delivery for an operator while the send still ran. Past the bound the
|
||||
* outcome is ambiguous — the DATA may have gone through — and is never retried automatically; the
|
||||
* send keeps its thread until the socket gives up. Startup holds the timeout above one stalled
|
||||
* connect and command, so a relay that never answers fails on its socket deadline first, as a
|
||||
* classified failure, rather than on this one.
|
||||
*/
|
||||
private final class PooledSend implements Runnable {
|
||||
|
||||
private final ProviderSubmission submission;
|
||||
private final AtomicReference<SendState> state = new AtomicReference<>(SendState.QUEUED);
|
||||
private final CompletableFuture<ProviderSubmissionResult> result = new CompletableFuture<>();
|
||||
|
||||
/**
|
||||
* Completes when a pool thread takes the send, or when the timeout passes first.
|
||||
*
|
||||
* <p>Both deadlines are {@code completeOnTimeout}: the JDK's delay thread completes the future
|
||||
* itself, so nothing waits behind the common pool, and completing the future early cancels the
|
||||
* timer, so a send that finishes in time leaves no timer holding it.
|
||||
*/
|
||||
private final CompletableFuture<Void> started = new CompletableFuture<>();
|
||||
|
||||
private PooledSend(ProviderSubmission submission) {
|
||||
this.submission = submission;
|
||||
}
|
||||
|
||||
/**
|
||||
* Starts the queue deadline, once the pool has taken the send: a send the pool refused needs
|
||||
* none, and one the pool already ran has completed {@link #started}, so no timer is made.
|
||||
*/
|
||||
private void armQueueDeadline() {
|
||||
// Nothing to observe on the stage it returns: the abandon reports through the result.
|
||||
var unused =
|
||||
started
|
||||
.completeOnTimeout(null, callTimeout.toMillis(), MILLISECONDS)
|
||||
.thenRun(this::abandonIfStillQueued);
|
||||
}
|
||||
|
||||
private CompletableFuture<ProviderSubmissionResult> result() {
|
||||
return result;
|
||||
}
|
||||
|
||||
private void abandonIfStillQueued() {
|
||||
if (state.compareAndSet(SendState.QUEUED, SendState.ABANDONED)) {
|
||||
result.completeExceptionally(
|
||||
new ProviderCallNotStartedException(
|
||||
NotificationFailureCode.PROVIDER_UNAVAILABLE,
|
||||
FailureCategory.CAPACITY_REJECTED,
|
||||
true,
|
||||
"the SMTP send waited in the pool past the profile timeout of "
|
||||
+ callTimeout
|
||||
+ "; the message was not handed to the relay"));
|
||||
}
|
||||
}
|
||||
|
||||
@Override
|
||||
public void run() {
|
||||
if (!state.compareAndSet(SendState.QUEUED, SendState.RUNNING)) {
|
||||
return;
|
||||
}
|
||||
// Cancels the queue deadline; its abandon then finds the send running and does nothing.
|
||||
started.complete(null);
|
||||
result.completeOnTimeout(callOutlastedItsTimeout(), callTimeout.toMillis(), MILLISECONDS);
|
||||
try {
|
||||
result.complete(send(submission));
|
||||
} catch (RuntimeException failure) {
|
||||
result.completeExceptionally(failure);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/** Where a pooled send is: only a queued one may still be abandoned. */
|
||||
private enum SendState {
|
||||
QUEUED,
|
||||
RUNNING,
|
||||
ABANDONED
|
||||
}
|
||||
|
||||
private ProviderSubmissionResult send(ProviderSubmission submission) {
|
||||
@@ -100,6 +224,18 @@ public final class SmtpNotificationProviderAdapter implements NotificationProvid
|
||||
}
|
||||
}
|
||||
|
||||
private ProviderSubmissionResult callOutlastedItsTimeout() {
|
||||
return ProviderSubmissionResult.ambiguous(
|
||||
new ProviderFailure(
|
||||
NotificationFailureCode.PROVIDER_RESPONSE_LOST,
|
||||
FailureCategory.AMBIGUOUS_SUBMISSION,
|
||||
false,
|
||||
Optional.empty(),
|
||||
Optional.of("the SMTP send outlasted the profile timeout of " + callTimeout)),
|
||||
ProviderExecutionEvidence.undetermined(),
|
||||
callTimeout);
|
||||
}
|
||||
|
||||
private static Duration elapsedSince(long startedNanos) {
|
||||
return Duration.ofNanos(System.nanoTime() - startedNanos);
|
||||
}
|
||||
|
||||
+118
@@ -2,6 +2,7 @@ package dev.caskeleton.adapter.outbound.notification.platform.autoconfigure;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
import static org.assertj.core.api.Assertions.assertThatThrownBy;
|
||||
import static org.assertj.core.api.Assertions.catchThrowable;
|
||||
|
||||
import dev.caskeleton.adapter.outbound.notification.platform.callback.CallbackVerificationResult;
|
||||
import dev.caskeleton.adapter.outbound.notification.platform.callback.ProviderCallbackAdapter;
|
||||
@@ -45,6 +46,9 @@ import java.util.Optional;
|
||||
import java.util.Set;
|
||||
import java.util.UUID;
|
||||
import java.util.concurrent.CompletionStage;
|
||||
import java.util.concurrent.LinkedBlockingQueue;
|
||||
import java.util.concurrent.ThreadPoolExecutor;
|
||||
import java.util.concurrent.TimeUnit;
|
||||
import org.junit.jupiter.api.DisplayName;
|
||||
import org.junit.jupiter.api.Test;
|
||||
|
||||
@@ -199,6 +203,120 @@ class NotificationProviderAssemblyTest {
|
||||
.hasMessageContaining("INGEST_ONLY");
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("SERVING 조립은 SMTP timeout 20초의 두 배(40초)에 못 미치는 lease 30초를 거절한다")
|
||||
void aServingAssemblyRefusesALeaseOneSmtpSendCanOutlast() {
|
||||
// given
|
||||
// The settings fallback, Dispatch.defaults(), leases for 30 s (the module defaults ship 2 m):
|
||||
// longer than the timeout, so the settings accept it, and shorter than twice the timeout, which
|
||||
// only the SMTP family check refuses.
|
||||
Map<String, NotificationPlatformSettings.Provider> providers =
|
||||
Map.of("primary-email", smtpProfile(true, Duration.ofSeconds(20), 4));
|
||||
|
||||
// when
|
||||
Throwable failure =
|
||||
catchThrowable(
|
||||
() -> assemble(providers, SmtpProviderRuntimeAssemblerTest.assembler(Runnable::run)));
|
||||
|
||||
// then
|
||||
assertThat(failure)
|
||||
.isInstanceOf(IllegalStateException.class)
|
||||
.hasMessageContaining(
|
||||
"lease-duration (PT30S) must be longer than twice the SMTP timeout (PT40S)");
|
||||
}
|
||||
|
||||
@Test
|
||||
void anIngestOnlyAssemblyDoesNotHoldSmtpToTheDispatchLease() {
|
||||
// given
|
||||
Map<String, NotificationPlatformSettings.Provider> providers =
|
||||
Map.of("primary-email", smtpProfile(true, Duration.ofSeconds(20), 4));
|
||||
|
||||
// when
|
||||
NotificationProviderAssembly.AssembledPlatform platform =
|
||||
assemble(
|
||||
providers,
|
||||
NotificationPlatformMode.INGEST_ONLY,
|
||||
SmtpProviderRuntimeAssemblerTest.assembler(Runnable::run));
|
||||
|
||||
// then
|
||||
assertThat(platform.routeFor(Channel.EMAIL))
|
||||
.as("an ingest-only deployment sends nothing, so no send can outlast its lease")
|
||||
.contains(new ProviderProfileId("primary-email"));
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName(
|
||||
"SERVING 조립은 켜진 SMTP profile의 max-concurrency 합(3+3)이 pool(thread 1, 대기열 4)을 넘으면 거절한다")
|
||||
void aServingAssemblyRefusesSmtpProfilesThatTogetherOverfillThePool() {
|
||||
// given
|
||||
Map<String, NotificationPlatformSettings.Provider> providers =
|
||||
Map.of(
|
||||
"primary-email", smtpProfile(true, Duration.ofSeconds(10), 3),
|
||||
"secondary-email", smtpSecondary(true, 3));
|
||||
|
||||
// when
|
||||
Throwable failure = catchThrowable(() -> assembleOnASmallPool(providers));
|
||||
|
||||
// then
|
||||
assertThat(failure)
|
||||
.isInstanceOf(IllegalStateException.class)
|
||||
.hasMessageContaining("max-concurrency (6 together) is more than the SMTP send pool holds");
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("꺼진 SMTP profile의 max-concurrency는 pool(thread 1, 대기열 4) 합에 들지 않는다")
|
||||
void aDisabledSmtpProfileIsLeftOutOfThePoolSum() {
|
||||
// given
|
||||
Map<String, NotificationPlatformSettings.Provider> providers =
|
||||
Map.of(
|
||||
"primary-email", smtpProfile(true, Duration.ofSeconds(10), 3),
|
||||
"secondary-email", smtpSecondary(false, 3));
|
||||
|
||||
// when
|
||||
NotificationProviderAssembly.AssembledPlatform platform = assembleOnASmallPool(providers);
|
||||
|
||||
// then
|
||||
assertThat(
|
||||
List.of(
|
||||
platform.runtimes().find(new ProviderProfileId("primary-email")).isPresent(),
|
||||
platform.runtimes().find(new ProviderProfileId("secondary-email")).isPresent()))
|
||||
.as("the enabled profile is assembled; the disabled one is neither assembled nor counted")
|
||||
.containsExactly(true, false);
|
||||
}
|
||||
|
||||
/** Assembles {@code providers} over one SMTP send thread with four queued. */
|
||||
private static NotificationProviderAssembly.AssembledPlatform assembleOnASmallPool(
|
||||
Map<String, NotificationPlatformSettings.Provider> providers) {
|
||||
ThreadPoolExecutor pool =
|
||||
new ThreadPoolExecutor(1, 1, 60, TimeUnit.SECONDS, new LinkedBlockingQueue<>(4));
|
||||
try {
|
||||
return assemble(providers, SmtpProviderRuntimeAssemblerTest.assembler(pool));
|
||||
} finally {
|
||||
pool.shutdownNow();
|
||||
}
|
||||
}
|
||||
|
||||
private static NotificationPlatformSettings.Provider smtpProfile(
|
||||
boolean enabled, Duration timeout, int maxConcurrency) {
|
||||
return new NotificationPlatformSettings.Provider(
|
||||
"SMTP", enabled, true, "local", "smtp-local", timeout, maxConcurrency, 8, null);
|
||||
}
|
||||
|
||||
/** A second SMTP profile on the same channel, not its primary. */
|
||||
private static NotificationPlatformSettings.Provider smtpSecondary(
|
||||
boolean enabled, int maxConcurrency) {
|
||||
return new NotificationPlatformSettings.Provider(
|
||||
"SMTP",
|
||||
enabled,
|
||||
false,
|
||||
"local",
|
||||
"smtp-local",
|
||||
Duration.ofSeconds(10),
|
||||
maxConcurrency,
|
||||
8,
|
||||
null);
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("INGEST_ONLY is a declaration, and it reports non-serving")
|
||||
void ingestOnlyIsADeclaration() {
|
||||
|
||||
+58
@@ -0,0 +1,58 @@
|
||||
package dev.caskeleton.adapter.outbound.notification.platform.autoconfigure;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
|
||||
import java.util.concurrent.CountDownLatch;
|
||||
import java.util.concurrent.Executor;
|
||||
import java.util.concurrent.ExecutorService;
|
||||
import java.util.concurrent.TimeUnit;
|
||||
import org.junit.jupiter.api.Test;
|
||||
|
||||
/**
|
||||
* The SMTP send pool runs {@code dispatch-concurrency} sends at once.
|
||||
*
|
||||
* <p>Its core size was one, and a {@code ThreadPoolExecutor} adds a thread past its core only when
|
||||
* its queue is full. The profile's limiter admits fewer sends than the queue holds, so the queue
|
||||
* never filled and every send waited for the one thread (round9 review 2026-11-12, RR9-01-15).
|
||||
*/
|
||||
class NotificationSmtpExecutorTest {
|
||||
|
||||
@Test
|
||||
void asManySendsAsTheDispatchConcurrencyRunAtOnce() throws InterruptedException {
|
||||
// given
|
||||
Executor executor =
|
||||
new NotificationSmtpProviderConfig()
|
||||
.notificationSmtpExecutor(new NotificationSmtpSettings("no-reply@example.test", 3));
|
||||
CountDownLatch allRunning = new CountDownLatch(3);
|
||||
CountDownLatch release = new CountDownLatch(1);
|
||||
|
||||
// when
|
||||
boolean concurrent;
|
||||
try {
|
||||
for (int send = 0; send < 3; send++) {
|
||||
executor.execute(
|
||||
() -> {
|
||||
allRunning.countDown();
|
||||
awaitQuietly(release);
|
||||
});
|
||||
}
|
||||
concurrent = allRunning.await(10, TimeUnit.SECONDS);
|
||||
} finally {
|
||||
release.countDown();
|
||||
((ExecutorService) executor).shutdownNow();
|
||||
}
|
||||
|
||||
// then
|
||||
assertThat(concurrent).as("three sends in flight on three threads").isTrue();
|
||||
}
|
||||
|
||||
private static void awaitQuietly(CountDownLatch latch) {
|
||||
try {
|
||||
if (!latch.await(30, TimeUnit.SECONDS)) {
|
||||
throw new IllegalStateException("the test never released the send");
|
||||
}
|
||||
} catch (InterruptedException interrupted) {
|
||||
Thread.currentThread().interrupt();
|
||||
}
|
||||
}
|
||||
}
|
||||
+112
-2
@@ -1,6 +1,7 @@
|
||||
package dev.caskeleton.adapter.outbound.notification.platform.autoconfigure;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
import static org.assertj.core.api.Assertions.catchThrowable;
|
||||
|
||||
import dev.caskeleton.adapter.outbound.notification.platform.provider.smtp.SmtpFailureClassifier;
|
||||
import dev.caskeleton.adapter.outbound.notification.platform.provider.smtp.SmtpMimeMessageFactory;
|
||||
@@ -10,7 +11,12 @@ import dev.caskeleton.application.notification.platform.dispatch.AttachmentInteg
|
||||
import jakarta.mail.Session;
|
||||
import java.time.Clock;
|
||||
import java.time.Duration;
|
||||
import java.util.Map;
|
||||
import java.util.Properties;
|
||||
import java.util.concurrent.Executor;
|
||||
import java.util.concurrent.LinkedBlockingQueue;
|
||||
import java.util.concurrent.ThreadPoolExecutor;
|
||||
import java.util.concurrent.TimeUnit;
|
||||
import org.junit.jupiter.api.DisplayName;
|
||||
import org.junit.jupiter.api.Test;
|
||||
|
||||
@@ -25,6 +31,8 @@ import org.junit.jupiter.api.Test;
|
||||
*/
|
||||
class SmtpProviderRuntimeAssemblerTest {
|
||||
|
||||
private static final Duration LEASE = Duration.ofMinutes(2);
|
||||
|
||||
@Test
|
||||
@DisplayName("the assembler serves the SMTP family on the email channel")
|
||||
void theAssemblerServesSmtpOnEmail() {
|
||||
@@ -81,7 +89,105 @@ class SmtpProviderRuntimeAssemblerTest {
|
||||
.isEqualTo(1L);
|
||||
}
|
||||
|
||||
@Test
|
||||
void aProfileAllowedMoreInFlightThanTheSendPoolHoldsFailsAssembly() {
|
||||
// given
|
||||
// One thread and four queued: five sends fit, the profile would let eight through.
|
||||
ThreadPoolExecutor pool =
|
||||
new ThreadPoolExecutor(1, 1, 60, TimeUnit.SECONDS, new LinkedBlockingQueue<>(4));
|
||||
|
||||
// when
|
||||
Throwable failure;
|
||||
try {
|
||||
failure =
|
||||
catchThrowable(
|
||||
() -> assembler(pool).requireTogether(Map.of("primary-email", profile(8)), LEASE));
|
||||
} finally {
|
||||
pool.shutdownNow();
|
||||
}
|
||||
|
||||
// then
|
||||
assertThat(failure)
|
||||
.isInstanceOf(IllegalStateException.class)
|
||||
.hasMessageContaining(
|
||||
"max-concurrency (8 together) is more than the SMTP send pool holds (1 threads and 4");
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("SMTP profile 둘이 합쳐 send pool이 담는 것보다 많이 내보내면 조립이 실패한다")
|
||||
void twoProfilesThatTogetherOverfillTheSendPoolFailAssembly() {
|
||||
// given
|
||||
// One thread and four queued: five sends fit; the two profiles would let six through.
|
||||
ThreadPoolExecutor pool =
|
||||
new ThreadPoolExecutor(1, 1, 60, TimeUnit.SECONDS, new LinkedBlockingQueue<>(4));
|
||||
SmtpProviderRuntimeAssembler assembler = assembler(pool);
|
||||
|
||||
// when
|
||||
Throwable failure;
|
||||
try {
|
||||
failure =
|
||||
catchThrowable(
|
||||
() ->
|
||||
assembler.requireTogether(
|
||||
Map.of("primary-email", profile(3), "secondary-email", profile(3)), LEASE));
|
||||
} finally {
|
||||
pool.shutdownNow();
|
||||
}
|
||||
|
||||
// then
|
||||
assertThat(failure)
|
||||
.isInstanceOf(IllegalStateException.class)
|
||||
.hasMessageContaining(
|
||||
"[primary-email, secondary-email]: max-concurrency (6 together) is more than the SMTP"
|
||||
+ " send pool holds");
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("SMTP timeout 10초의 두 배인 20초를 넘지 않는 delivery lease는 조립이 실패한다")
|
||||
void aDeliveryLeaseOneSendCanOutlastFailsAssembly() {
|
||||
// when
|
||||
Throwable failure =
|
||||
catchThrowable(
|
||||
() ->
|
||||
assembler()
|
||||
.requireTogether(Map.of("primary-email", profile(4)), Duration.ofSeconds(20)));
|
||||
|
||||
// then
|
||||
assertThat(failure)
|
||||
.as("a send may wait in the pool for the timeout and then run for it")
|
||||
.isInstanceOf(IllegalStateException.class)
|
||||
.hasMessageContaining("lease-duration (PT20S) must be longer than twice the SMTP timeout");
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("connect 2초와 명령 5초가 다 지나도록 기다릴 수 없는 7초 timeout은 조립이 실패한다")
|
||||
void aProfileTimeoutOneStalledConnectAndCommandOutlastFailsAssembly() {
|
||||
// given
|
||||
NotificationPlatformSettings.Provider sevenSeconds =
|
||||
new NotificationPlatformSettings.Provider(
|
||||
"SMTP", true, true, "local", "smtp-local", Duration.ofSeconds(7), 4, 8, null);
|
||||
|
||||
// when
|
||||
Throwable failure = catchThrowable(() -> assembler().assemble("primary-email", sevenSeconds));
|
||||
|
||||
// then
|
||||
assertThat(failure)
|
||||
.as(
|
||||
"a relay that never answers the greeting has been sent nothing; its socket deadline says so")
|
||||
.isInstanceOf(IllegalStateException.class)
|
||||
.hasMessageContaining("timeout (PT7S) must be longer than mail.smtp.connectiontimeout")
|
||||
.hasMessageContaining("(PT7S), or a relay that never answers");
|
||||
}
|
||||
|
||||
private static SmtpProviderRuntimeAssembler assembler() {
|
||||
return assembler(Runnable::run);
|
||||
}
|
||||
|
||||
/**
|
||||
* The real SMTP assembler over {@code executor}; also used by {@code
|
||||
* NotificationProviderAssemblyTest}.
|
||||
*/
|
||||
static SmtpProviderRuntimeAssembler assembler(Executor executor) {
|
||||
Session session = Session.getInstance(new Properties());
|
||||
return new SmtpProviderRuntimeAssembler(
|
||||
message -> {
|
||||
@@ -90,7 +196,7 @@ class SmtpProviderRuntimeAssemblerTest {
|
||||
new SmtpMimeMessageFactory(session),
|
||||
new SmtpFailureClassifier(),
|
||||
protector(),
|
||||
Runnable::run,
|
||||
executor,
|
||||
new AttachmentIntegrityGuard((reference, context) -> null),
|
||||
new SmtpProviderProperties(
|
||||
"localhost",
|
||||
@@ -108,8 +214,12 @@ class SmtpProviderRuntimeAssemblerTest {
|
||||
}
|
||||
|
||||
private static NotificationPlatformSettings.Provider profile() {
|
||||
return profile(4);
|
||||
}
|
||||
|
||||
private static NotificationPlatformSettings.Provider profile(int maxConcurrency) {
|
||||
return new NotificationPlatformSettings.Provider(
|
||||
"SMTP", true, true, "local", "smtp-local", Duration.ofSeconds(10), 4, 8, null);
|
||||
"SMTP", true, true, "local", "smtp-local", Duration.ofSeconds(10), maxConcurrency, 8, null);
|
||||
}
|
||||
|
||||
/** Not a lambda: the protector has three operations, and only one of them is exercised here. */
|
||||
|
||||
+128
-2
@@ -15,18 +15,28 @@ import dev.caskeleton.application.notification.platform.contact.EmailAddress;
|
||||
import dev.caskeleton.application.notification.platform.contact.PhoneNumber;
|
||||
import dev.caskeleton.application.notification.platform.provider.ProviderCallNotStartedException;
|
||||
import dev.caskeleton.application.notification.platform.provider.ProviderSubmission;
|
||||
import dev.caskeleton.application.notification.platform.provider.ProviderSubmissionResult;
|
||||
import dev.caskeleton.application.notification.platform.provider.testkit.ProviderFixtures;
|
||||
import dev.caskeleton.application.notification.platform.security.ContactPointProtector;
|
||||
import jakarta.mail.Session;
|
||||
import java.time.Duration;
|
||||
import java.util.List;
|
||||
import java.util.Optional;
|
||||
import java.util.Properties;
|
||||
import java.util.concurrent.CountDownLatch;
|
||||
import java.util.concurrent.Executor;
|
||||
import java.util.concurrent.ExecutorService;
|
||||
import java.util.concurrent.Executors;
|
||||
import java.util.concurrent.RejectedExecutionException;
|
||||
import java.util.concurrent.TimeUnit;
|
||||
import java.util.concurrent.atomic.AtomicInteger;
|
||||
import org.junit.jupiter.api.DisplayName;
|
||||
import org.junit.jupiter.api.Test;
|
||||
|
||||
class SmtpNotificationProviderAdapterTest {
|
||||
|
||||
private static final Duration CALL_TIMEOUT = Duration.ofSeconds(5);
|
||||
|
||||
private final ContactPointProtector protector =
|
||||
new AesGcmContactPointProtector(SecurityFixtures.keys());
|
||||
|
||||
@@ -138,7 +148,122 @@ class SmtpNotificationProviderAdapterTest {
|
||||
.hasMessageContaining("email contact point");
|
||||
}
|
||||
|
||||
@Test
|
||||
void aSendThePoolRejectsIsReportedAsNotStarted() {
|
||||
// given
|
||||
Executor full =
|
||||
command -> {
|
||||
throw new RejectedExecutionException("the send pool is full");
|
||||
};
|
||||
|
||||
// when
|
||||
Throwable failure = catchThrowable(() -> adapter(message -> {}, full).submit(submission()));
|
||||
|
||||
// then
|
||||
assertThat(failure)
|
||||
.isInstanceOf(ProviderCallNotStartedException.class)
|
||||
.extracting("code", "category", "retryable")
|
||||
.containsExactly("PROVIDER_UNAVAILABLE", FailureCategory.CAPACITY_REJECTED, true);
|
||||
}
|
||||
|
||||
private SmtpNotificationProviderAdapter adapter(SmtpDispatch dispatch) {
|
||||
return adapter(dispatch, Executors.newSingleThreadExecutor());
|
||||
}
|
||||
|
||||
@Test
|
||||
void aSendThatOutlastsTheProfileTimeoutIsAmbiguousNotRetried() throws Exception {
|
||||
// given
|
||||
CountDownLatch released = new CountDownLatch(1);
|
||||
SmtpDispatch slowRelay =
|
||||
message -> {
|
||||
try {
|
||||
released.await();
|
||||
} catch (InterruptedException interrupted) {
|
||||
Thread.currentThread().interrupt();
|
||||
}
|
||||
};
|
||||
ExecutorService pool = Executors.newSingleThreadExecutor();
|
||||
// A second, not milliseconds: the send has to leave the pool queue inside it on a loaded
|
||||
// runner, or it is abandoned before it starts and this is the other test.
|
||||
SmtpNotificationProviderAdapter adapter = adapter(slowRelay, pool, Duration.ofSeconds(1));
|
||||
|
||||
// when
|
||||
ProviderSubmissionResult result;
|
||||
try {
|
||||
result = adapter.submit(submission()).toCompletableFuture().get(5, TimeUnit.SECONDS);
|
||||
} finally {
|
||||
released.countDown();
|
||||
pool.shutdownNow();
|
||||
}
|
||||
|
||||
// then
|
||||
assertThat(result)
|
||||
.as("past the profile timeout the DATA may have gone through: never resend on our own")
|
||||
.extracting(
|
||||
ProviderSubmissionResult::confirmation,
|
||||
outcome -> outcome.failure().orElseThrow().code(),
|
||||
outcome -> outcome.failure().orElseThrow().retryable())
|
||||
.containsExactly(AttemptConfirmation.AMBIGUOUS, "PROVIDER_RESPONSE_LOST", false);
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("pool에서 profile timeout을 넘겨 기다린 발송은 relay에 보내지 않고 시작 전 실패로 답한다")
|
||||
void aSendThatWaitedInThePoolPastTheTimeoutIsNotStartedAndNeverSent() throws Exception {
|
||||
// given
|
||||
CountDownLatch released = new CountDownLatch(1);
|
||||
CountDownLatch entered = new CountDownLatch(1);
|
||||
AtomicInteger handedToTheRelay = new AtomicInteger();
|
||||
SmtpDispatch slowRelay =
|
||||
message -> {
|
||||
handedToTheRelay.incrementAndGet();
|
||||
entered.countDown();
|
||||
try {
|
||||
released.await();
|
||||
} catch (InterruptedException interrupted) {
|
||||
Thread.currentThread().interrupt();
|
||||
}
|
||||
};
|
||||
ExecutorService pool = Executors.newSingleThreadExecutor();
|
||||
SmtpNotificationProviderAdapter adapter = adapter(slowRelay, pool, Duration.ofSeconds(1));
|
||||
var occupyingThePool = adapter.submit(submission());
|
||||
// The pool's one thread is in the relay before the second send is queued behind it.
|
||||
assertThat(entered.await(5, TimeUnit.SECONDS)).as("the first send reached the relay").isTrue();
|
||||
|
||||
// when
|
||||
Throwable waited;
|
||||
boolean drained;
|
||||
try {
|
||||
waited =
|
||||
catchThrowable(
|
||||
() -> adapter.submit(submission()).toCompletableFuture().get(5, TimeUnit.SECONDS));
|
||||
} finally {
|
||||
released.countDown();
|
||||
pool.shutdown();
|
||||
// Every queued task has run once this returns true, the abandoned send's included, so the
|
||||
// relay count below is final rather than read before the late send could have gone out.
|
||||
drained = pool.awaitTermination(5, TimeUnit.SECONDS);
|
||||
}
|
||||
|
||||
// then
|
||||
assertThat(
|
||||
List.of(
|
||||
waited.getCause().getClass(),
|
||||
waited.getCause().getMessage().contains("waited in the pool past the profile"),
|
||||
drained,
|
||||
handedToTheRelay.get()))
|
||||
.as(
|
||||
"nothing reached the relay, so the delivery is retried rather than reconciled, and"
|
||||
+ " the abandoned send does not go out late beside %s",
|
||||
occupyingThePool)
|
||||
.containsExactly(ProviderCallNotStartedException.class, true, true, 1);
|
||||
}
|
||||
|
||||
private SmtpNotificationProviderAdapter adapter(SmtpDispatch dispatch, Executor executor) {
|
||||
return adapter(dispatch, executor, CALL_TIMEOUT);
|
||||
}
|
||||
|
||||
private SmtpNotificationProviderAdapter adapter(
|
||||
SmtpDispatch dispatch, Executor executor, Duration callTimeout) {
|
||||
var properties =
|
||||
new SmtpProviderProperties(
|
||||
"smtp.example.com",
|
||||
@@ -155,12 +280,13 @@ class SmtpNotificationProviderAdapterTest {
|
||||
new SmtpFailureClassifier(),
|
||||
protector,
|
||||
properties,
|
||||
Executors.newSingleThreadExecutor(),
|
||||
executor,
|
||||
// The unconfigured resolver refuses on use, which is the honest default: an attachment the
|
||||
// deployment has no way to fetch must fail before the send, not silently vanish from it.
|
||||
new dev.caskeleton.application.notification.platform.dispatch.AttachmentIntegrityGuard(
|
||||
new dev.caskeleton.adapter.outbound.notification.platform.provider
|
||||
.UnconfiguredAttachmentResolver()));
|
||||
.UnconfiguredAttachmentResolver()),
|
||||
callTimeout);
|
||||
}
|
||||
|
||||
private ProviderSubmission submission() {
|
||||
|
||||
+44
-8
@@ -29,6 +29,8 @@ import software.amazon.awssdk.services.s3.model.ServerSideEncryption;
|
||||
/** Exact-key S3 control store using native conditional PUT rather than check-then-act emulation. */
|
||||
public final class S3ConditionalObjectControlStore implements ObjectControlStore {
|
||||
|
||||
private static final String CONFLICT = "S3 control conditional mutation conflicted";
|
||||
|
||||
private static final int MAXIMUM_CONTROL_BYTES = 64 * 1024;
|
||||
|
||||
private final S3AsyncClient client;
|
||||
@@ -107,9 +109,11 @@ public final class S3ConditionalObjectControlStore implements ObjectControlStore
|
||||
true);
|
||||
return stored(response.eTag(), record);
|
||||
} catch (ObjectStorageProviderException exception) {
|
||||
if (exception.failure() == ObjectStorageProviderException.Failure.CONFLICT
|
||||
|| exception.failure() == ObjectStorageProviderException.Failure.PRECONDITION_FAILED) {
|
||||
throw new ObjectControlConflictException("S3 control conditional mutation conflicted");
|
||||
if (exception.failure() == ObjectStorageProviderException.Failure.PRECONDITION_FAILED) {
|
||||
return reconcilePreconditionFailure(key, record);
|
||||
}
|
||||
if (exception.failure() == ObjectStorageProviderException.Failure.CONFLICT) {
|
||||
throw new ObjectControlConflictException(CONFLICT);
|
||||
}
|
||||
if (exception.failure() == ObjectStorageProviderException.Failure.INDETERMINATE) {
|
||||
return reconcileLostResponse(key, record);
|
||||
@@ -118,20 +122,52 @@ public final class S3ConditionalObjectControlStore implements ObjectControlStore
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Decides a failed precondition by what the key holds now.
|
||||
*
|
||||
* <p>A 412 is not always somebody else's write. The SDK resends a conditional {@code PUT} whose
|
||||
* response it lost — the body is replayable — and when the first send landed, the resend fails
|
||||
* the very precondition the first one satisfied: {@code If-None-Match: *} now finds the object,
|
||||
* {@code If-Match} now finds a new ETag. Reported as a conflict, this store's own write was
|
||||
* called stale and the CAS above it gave up on a mutation that had succeeded (round9 review
|
||||
* 2026-11-12, RR9-01-19). The key holding exactly the record this call wrote is that case; any
|
||||
* other content, or none, is a real conflict.
|
||||
*
|
||||
* <p>"Exactly the record" is equality, not authorship: two writers that CAS the same version to
|
||||
* equal records — same revision, state and timestamp — are indistinguishable here, and both are
|
||||
* told they succeeded. The record's revision is what normally tells them apart; a caller that can
|
||||
* produce equal records from two places carries its own token in the record.
|
||||
*/
|
||||
private StoredControlRecord reconcilePreconditionFailure(
|
||||
String key, ObjectControlRecord expected) {
|
||||
return holdingExactly(key, expected)
|
||||
.orElseThrow(() -> new ObjectControlConflictException(CONFLICT));
|
||||
}
|
||||
|
||||
private StoredControlRecord reconcileLostResponse(String key, ObjectControlRecord expected) {
|
||||
Optional<StoredControlRecord> current = read(key);
|
||||
if (current.isPresent() && current.orElseThrow().record().equals(expected)) {
|
||||
return current.orElseThrow();
|
||||
}
|
||||
if (current.isPresent()) {
|
||||
throw new ObjectControlConflictException(
|
||||
"S3 control response loss resolved to different exact record evidence");
|
||||
return holding(current.orElseThrow(), expected)
|
||||
.orElseThrow(
|
||||
() ->
|
||||
new ObjectControlConflictException(
|
||||
"S3 control response loss resolved to different exact record evidence"));
|
||||
}
|
||||
throw new ObjectStorageProviderException(
|
||||
ObjectStorageProviderException.Failure.INDETERMINATE,
|
||||
"S3 control mutation response was lost and no authoritative applied evidence exists");
|
||||
}
|
||||
|
||||
/** What the key holds now, when it is exactly {@code expected}. */
|
||||
private Optional<StoredControlRecord> holdingExactly(String key, ObjectControlRecord expected) {
|
||||
return read(key).flatMap(current -> holding(current, expected));
|
||||
}
|
||||
|
||||
private static Optional<StoredControlRecord> holding(
|
||||
StoredControlRecord current, ObjectControlRecord expected) {
|
||||
return current.record().equals(expected) ? Optional.of(current) : Optional.empty();
|
||||
}
|
||||
|
||||
private static StoredControlRecord stored(String eTag, ObjectControlRecord record) {
|
||||
if (eTag == null || eTag.isBlank() || eTag.length() > 256) {
|
||||
throw new ObjectStorageProviderException(
|
||||
|
||||
+59
-2
@@ -2,6 +2,7 @@ package dev.caskeleton.adapter.outbound.objectstorage.s3;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
import static org.assertj.core.api.Assertions.assertThatThrownBy;
|
||||
import static org.assertj.core.api.Assertions.catchThrowable;
|
||||
import static org.mockito.ArgumentMatchers.any;
|
||||
import static org.mockito.Mockito.mock;
|
||||
import static org.mockito.Mockito.never;
|
||||
@@ -26,6 +27,7 @@ import java.util.List;
|
||||
import java.util.concurrent.CompletableFuture;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.mockito.ArgumentCaptor;
|
||||
import org.mockito.ArgumentMatchers;
|
||||
import software.amazon.awssdk.core.async.AsyncRequestBody;
|
||||
import software.amazon.awssdk.core.async.AsyncResponseTransformer;
|
||||
import software.amazon.awssdk.core.async.SdkPublisher;
|
||||
@@ -99,6 +101,9 @@ class S3ConditionalObjectControlStoreTest {
|
||||
S3AsyncClient client = mock(S3AsyncClient.class);
|
||||
when(client.putObject(any(PutObjectRequest.class), any(AsyncRequestBody.class)))
|
||||
.thenReturn(CompletableFuture.failedFuture(S3Exception.builder().statusCode(412).build()));
|
||||
// Another writer's record holds the key: the precondition failed on somebody else's write.
|
||||
respondToGet(
|
||||
client, CODEC.encode(record(3, PublishedReferenceState.PUBLISHED)), "\"other-writer\"");
|
||||
S3ConditionalObjectControlStore store =
|
||||
new S3ConditionalObjectControlStore(client, awsBinding(), CODEC);
|
||||
|
||||
@@ -118,12 +123,64 @@ class S3ConditionalObjectControlStoreTest {
|
||||
.class);
|
||||
}
|
||||
|
||||
@Test
|
||||
void aPreconditionFailedByTheResendOfOurOwnLandedWriteIsThatWrite() {
|
||||
// given
|
||||
// The SDK lost the first response and resent the conditional PUT; the first one had landed, so
|
||||
// the resend fails If-Match against the ETag it produced.
|
||||
S3AsyncClient client = mock(S3AsyncClient.class);
|
||||
ObjectReferenceRecord written = record(2, PublishedReferenceState.RETIRED);
|
||||
when(client.putObject(any(PutObjectRequest.class), any(AsyncRequestBody.class)))
|
||||
.thenReturn(CompletableFuture.failedFuture(S3Exception.builder().statusCode(412).build()));
|
||||
respondToGet(client, CODEC.encode(written), "\"after-our-write\"");
|
||||
S3ConditionalObjectControlStore store =
|
||||
new S3ConditionalObjectControlStore(client, awsBinding(), CODEC);
|
||||
|
||||
// when
|
||||
ObjectControlStore.StoredControlRecord stored =
|
||||
store.compareAndSet(
|
||||
KEY, new ObjectControlMutation(ObjectControlVersion.of("\"before\""), written));
|
||||
|
||||
// then
|
||||
assertThat(List.of(stored.record(), stored.version()))
|
||||
.containsExactly(written, ObjectControlVersion.of("\"after-our-write\""));
|
||||
}
|
||||
|
||||
@Test
|
||||
void aPreconditionFailureOnAnAbsentKeyIsAConflict() {
|
||||
// given
|
||||
S3AsyncClient client = mock(S3AsyncClient.class);
|
||||
when(client.putObject(any(PutObjectRequest.class), any(AsyncRequestBody.class)))
|
||||
.thenReturn(CompletableFuture.failedFuture(S3Exception.builder().statusCode(412).build()));
|
||||
when(client.getObject(
|
||||
any(GetObjectRequest.class),
|
||||
ArgumentMatchers.<AsyncResponseTransformer<GetObjectResponse, ?>>any()))
|
||||
.thenReturn(CompletableFuture.failedFuture(S3Exception.builder().statusCode(404).build()));
|
||||
S3ConditionalObjectControlStore store =
|
||||
new S3ConditionalObjectControlStore(client, awsBinding(), CODEC);
|
||||
|
||||
// when
|
||||
Throwable refused =
|
||||
catchThrowable(
|
||||
() ->
|
||||
store.compareAndSet(
|
||||
KEY,
|
||||
new ObjectControlMutation(
|
||||
ObjectControlVersion.of("\"stale\""),
|
||||
record(2, PublishedReferenceState.RETIRED))));
|
||||
|
||||
// then
|
||||
assertThat(refused)
|
||||
.isInstanceOf(ObjectControlConflictException.class)
|
||||
.hasMessageContaining("conflicted");
|
||||
}
|
||||
|
||||
@Test
|
||||
void exact404IsTheOnlyAbsentRead() {
|
||||
S3AsyncClient client = mock(S3AsyncClient.class);
|
||||
when(client.getObject(
|
||||
any(GetObjectRequest.class),
|
||||
org.mockito.ArgumentMatchers.<AsyncResponseTransformer<GetObjectResponse, ?>>any()))
|
||||
ArgumentMatchers.<AsyncResponseTransformer<GetObjectResponse, ?>>any()))
|
||||
.thenReturn(CompletableFuture.failedFuture(S3Exception.builder().statusCode(404).build()));
|
||||
S3ConditionalObjectControlStore store =
|
||||
new S3ConditionalObjectControlStore(client, awsBinding(), CODEC);
|
||||
@@ -154,7 +211,7 @@ class S3ConditionalObjectControlStoreTest {
|
||||
private static void respondToGet(S3AsyncClient client, byte[] encoded, String eTag) {
|
||||
when(client.getObject(
|
||||
any(GetObjectRequest.class),
|
||||
org.mockito.ArgumentMatchers.<AsyncResponseTransformer<GetObjectResponse, ?>>any()))
|
||||
ArgumentMatchers.<AsyncResponseTransformer<GetObjectResponse, ?>>any()))
|
||||
.thenAnswer(
|
||||
invocation -> {
|
||||
AsyncResponseTransformer<GetObjectResponse, Object> transformer =
|
||||
|
||||
+13
-6
@@ -17,6 +17,8 @@ import java.time.Duration;
|
||||
import java.time.Instant;
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.TreeMap;
|
||||
import java.util.concurrent.CountDownLatch;
|
||||
import java.util.concurrent.ExecutorService;
|
||||
import java.util.concurrent.Executors;
|
||||
@@ -87,23 +89,28 @@ class LiveEventLogContractTest {
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("two concurrent writers receive different positions and both commit")
|
||||
@DisplayName("두 writer가 동시에 쓰면 서로 다른 위치 1·2를 받고, replay는 받은 위치 순서대로 둘 다 돌려준다")
|
||||
void concurrentWritersAllocateDistinctPositions() throws Exception {
|
||||
// given
|
||||
CountDownLatch start = new CountDownLatch(1);
|
||||
// Which writer reaches the allocator first is up to the scheduler, so the expected replay
|
||||
// order is the order of the positions the writers got back, not the order they were started.
|
||||
Map<Long, String> payloadByPosition = new TreeMap<>();
|
||||
|
||||
// when
|
||||
try (ExecutorService executor = Executors.newFixedThreadPool(2)) {
|
||||
Future<Long> first =
|
||||
executor.submit(() -> allocateAndAppendAfter(start, "orders", "from-first"));
|
||||
Future<Long> second =
|
||||
executor.submit(() -> allocateAndAppendAfter(start, "orders", "from-second"));
|
||||
|
||||
start.countDown();
|
||||
|
||||
assertThat(List.of(first.get(10, TimeUnit.SECONDS), second.get(10, TimeUnit.SECONDS)))
|
||||
.containsExactlyInAnyOrder(1L, 2L);
|
||||
payloadByPosition.put(first.get(10, TimeUnit.SECONDS), "from-first");
|
||||
payloadByPosition.put(second.get(10, TimeUnit.SECONDS), "from-second");
|
||||
}
|
||||
|
||||
assertThat(replayAfter("orders", 0, NOW)).containsExactly("from-first", "from-second");
|
||||
// then
|
||||
assertThat(payloadByPosition).containsOnlyKeys(1L, 2L);
|
||||
assertThat(replayAfter("orders", 0, NOW)).containsExactlyElementsOf(payloadByPosition.values());
|
||||
}
|
||||
|
||||
@Test
|
||||
|
||||
+147
-33
@@ -7,9 +7,14 @@ import dev.caskeleton.adapter.outbound.mongo.api.error.MongoOperationRejectedExc
|
||||
import java.time.Clock;
|
||||
import java.time.Duration;
|
||||
import java.time.Instant;
|
||||
import java.util.HashMap;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.Objects;
|
||||
import java.util.Set;
|
||||
import java.util.concurrent.ConcurrentHashMap;
|
||||
import java.util.concurrent.atomic.AtomicLong;
|
||||
import java.util.function.Function;
|
||||
|
||||
/**
|
||||
* Bounds how many tenant clients exist at once (advanced plan Task 11).
|
||||
@@ -18,6 +23,13 @@ import java.util.Set;
|
||||
* connection pool and its own monitoring threads, so a thousand tenants is a thousand pools. The
|
||||
* registry caps the number and evicts idle ones, and refuses rather than exceeding the cap — an
|
||||
* unbounded registry fails later, as connection exhaustion on an unrelated request.
|
||||
*
|
||||
* <p>{@link #acquire} hands back a {@link Lease}; the client is in use until every lease on it is
|
||||
* closed, and only a client nobody holds is idle. Acquire used to return nothing, so idle meant
|
||||
* "acquired longest ago" and eviction could close a client a long-running caller was still using.
|
||||
* Closing an evicted client is this registry's own clean-up: if it fails, the tenant whose acquire
|
||||
* triggered the eviction is not told about another tenant's client — the failure is counted in
|
||||
* {@link #evictionCloseFailures()} (round9 review 2026-11-12, RR9-01-27).
|
||||
*/
|
||||
@MongoAdvancedEntryPoint(MongoCapability.DATABASE_PER_TENANT)
|
||||
public final class MongoTenantClientRegistry {
|
||||
@@ -26,11 +38,21 @@ public final class MongoTenantClientRegistry {
|
||||
|
||||
private final Duration idleEviction;
|
||||
|
||||
private final Map<String, Instant> lastUsed = new java.util.concurrent.ConcurrentHashMap<>();
|
||||
private final Map<String, Instant> lastUsed = new ConcurrentHashMap<>();
|
||||
|
||||
private final Map<String, AutoCloseable> clients = new java.util.concurrent.ConcurrentHashMap<>();
|
||||
private final Map<String, AutoCloseable> clients = new ConcurrentHashMap<>();
|
||||
|
||||
private final java.util.function.Function<String, AutoCloseable> clientFactory;
|
||||
private final Map<String, Integer> leases = new HashMap<>();
|
||||
|
||||
/**
|
||||
* Guards the cap, the lease counts and the eviction as one decision. A private monitor rather
|
||||
* than the registry itself, so code outside cannot take it and stall every acquire.
|
||||
*/
|
||||
private final Object lock = new Object();
|
||||
|
||||
private final AtomicLong evictionCloseFailures = new AtomicLong();
|
||||
|
||||
private final Function<String, AutoCloseable> clientFactory;
|
||||
|
||||
private final Clock clock;
|
||||
|
||||
@@ -47,15 +69,15 @@ public final class MongoTenantClientRegistry {
|
||||
MongoAdvancedCapabilityGuard guard,
|
||||
int maximumActiveClients,
|
||||
Duration idleEviction,
|
||||
java.util.function.Function<String, AutoCloseable> clientFactory) {
|
||||
Function<String, AutoCloseable> clientFactory) {
|
||||
this(guard, maximumActiveClients, idleEviction, clientFactory, Clock.systemUTC());
|
||||
}
|
||||
|
||||
/**
|
||||
* A registry that owns real clients.
|
||||
*
|
||||
* <p>Idle eviction reads {@code clock}; {@link #acquire(String)} used to read {@code
|
||||
* Instant.now()} beside it.
|
||||
* <p>Acquiring, closing a lease and idle eviction all read {@code clock}; {@link
|
||||
* #acquire(String)} used to read {@code Instant.now()} beside it.
|
||||
*
|
||||
* <p>The factory is called at most once per tenant, inside the map's own computation, so two
|
||||
* threads asking for the same tenant get one client rather than two — a per-tenant client carries
|
||||
@@ -65,7 +87,7 @@ public final class MongoTenantClientRegistry {
|
||||
MongoAdvancedCapabilityGuard guard,
|
||||
int maximumActiveClients,
|
||||
Duration idleEviction,
|
||||
java.util.function.Function<String, AutoCloseable> clientFactory,
|
||||
Function<String, AutoCloseable> clientFactory,
|
||||
Clock clock) {
|
||||
Objects.requireNonNull(guard, "guard").require(MongoTenantClientRegistry.class);
|
||||
this.clientFactory = Objects.requireNonNull(clientFactory, "clientFactory");
|
||||
@@ -80,41 +102,128 @@ public final class MongoTenantClientRegistry {
|
||||
/**
|
||||
* Acquires a client for a tenant.
|
||||
*
|
||||
* @return the lease to close when the work with the client is done
|
||||
* @throws MongoOperationRejectedException when the cap is reached and nothing can be evicted
|
||||
*/
|
||||
public void acquire(String tenantKey) {
|
||||
acquire(tenantKey, clock.instant());
|
||||
public Lease acquire(String tenantKey) {
|
||||
Objects.requireNonNull(tenantKey, "tenantKey");
|
||||
synchronized (lock) {
|
||||
return acquireHeld(tenantKey);
|
||||
}
|
||||
}
|
||||
|
||||
/** Acquires a client for a tenant at an explicit instant, so eviction is testable. */
|
||||
public synchronized void acquire(String tenantKey, Instant now) {
|
||||
Objects.requireNonNull(tenantKey, "tenantKey");
|
||||
Objects.requireNonNull(now, "now");
|
||||
private Lease acquireHeld(String tenantKey) {
|
||||
// One time source for the acquire, the close and the eviction: an overload that took the
|
||||
// instant as an argument let a caller acquire in one timeline and close in the clock's.
|
||||
Instant now = clock.instant();
|
||||
// Synchronized rather than a bare ConcurrentHashMap: the cap, the eviction and the create are
|
||||
// one decision. Checking the size on one thread while another inserts is how a cap of ten ends
|
||||
// up holding twelve connection pools.
|
||||
if (lastUsed.containsKey(tenantKey)) {
|
||||
lastUsed.put(tenantKey, now);
|
||||
return;
|
||||
if (!lastUsed.containsKey(tenantKey)) {
|
||||
evictIdle(now);
|
||||
if (lastUsed.size() >= maximumActiveClients) {
|
||||
throw MongoOperationRejectedException.of(
|
||||
"tenancy.client",
|
||||
"the tenant client cap of "
|
||||
+ maximumActiveClients
|
||||
+ " is reached and no client is idle; each tenant client carries its own"
|
||||
+ " connection pool and monitoring threads, so the cap is what stops one process"
|
||||
+ " from exhausting the cluster's connections");
|
||||
}
|
||||
clients.computeIfAbsent(tenantKey, clientFactory);
|
||||
}
|
||||
evictIdle(now);
|
||||
if (lastUsed.size() >= maximumActiveClients) {
|
||||
throw MongoOperationRejectedException.of(
|
||||
"tenancy.client",
|
||||
"the tenant client cap of "
|
||||
+ maximumActiveClients
|
||||
+ " is reached and no client is idle; each tenant client carries its own connection "
|
||||
+ "pool and monitoring threads, so the cap is what stops one process from exhausting "
|
||||
+ "the cluster's connections");
|
||||
}
|
||||
clients.computeIfAbsent(tenantKey, clientFactory);
|
||||
lastUsed.put(tenantKey, now);
|
||||
leases.merge(tenantKey, 1, Integer::sum);
|
||||
return new HeldLease(tenantKey, clients.get(tenantKey));
|
||||
}
|
||||
|
||||
/** Releases a tenant's client and closes it. */
|
||||
public synchronized void release(String tenantKey) {
|
||||
lastUsed.remove(Objects.requireNonNull(tenantKey, "tenantKey"));
|
||||
close(tenantKey);
|
||||
/**
|
||||
* Releases a tenant's client and closes it.
|
||||
*
|
||||
* @throws IllegalStateException when a lease on the client is still open, or the close fails
|
||||
*/
|
||||
public void release(String tenantKey) {
|
||||
Objects.requireNonNull(tenantKey, "tenantKey");
|
||||
synchronized (lock) {
|
||||
if (leases.containsKey(tenantKey)) {
|
||||
throw new IllegalStateException(
|
||||
"the client for tenant '" + tenantKey + "' is still in use and cannot be released");
|
||||
}
|
||||
lastUsed.remove(tenantKey);
|
||||
close(tenantKey);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* How many evicted clients failed to close.
|
||||
*
|
||||
* <p>The registry has dropped each of them and freed its slot; the count is how an operator
|
||||
* learns that clients — and their pools — may still be open behind it.
|
||||
*/
|
||||
public long evictionCloseFailures() {
|
||||
return evictionCloseFailures.get();
|
||||
}
|
||||
|
||||
/**
|
||||
* One use of a tenant client; close it when the work with that client is done.
|
||||
*
|
||||
* <p>An interface so the registry's own implementation stays private: only the registry makes a
|
||||
* lease, and only through {@link #acquire}.
|
||||
*/
|
||||
public interface Lease extends AutoCloseable {
|
||||
|
||||
/** The tenant's client, valid until this lease is closed. */
|
||||
AutoCloseable client();
|
||||
|
||||
/** Ends this use. Closing twice is the same as closing once. */
|
||||
@Override
|
||||
void close();
|
||||
}
|
||||
|
||||
private final class HeldLease implements Lease {
|
||||
|
||||
private final String tenantKey;
|
||||
|
||||
private final AutoCloseable client;
|
||||
|
||||
private boolean closed;
|
||||
|
||||
private HeldLease(String tenantKey, AutoCloseable client) {
|
||||
this.tenantKey = tenantKey;
|
||||
this.client = client;
|
||||
}
|
||||
|
||||
@Override
|
||||
public AutoCloseable client() {
|
||||
return client;
|
||||
}
|
||||
|
||||
@Override
|
||||
public void close() {
|
||||
synchronized (lock) {
|
||||
if (closed) {
|
||||
return;
|
||||
}
|
||||
closed = true;
|
||||
leases.computeIfPresent(tenantKey, (tenant, open) -> open == 1 ? null : open - 1);
|
||||
// Idle from the end of the last use, not its start: a client held for half an hour was in
|
||||
// use all that time, and measuring from the acquire evicted it the moment it was let go.
|
||||
Instant released = clock.instant();
|
||||
lastUsed.computeIfPresent(
|
||||
tenantKey, (tenant, used) -> released.isAfter(used) ? released : used);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* How many leases are open now, across all tenants.
|
||||
*
|
||||
* <p>A lease nobody closed keeps its client's slot for good; this is how that becomes visible.
|
||||
*/
|
||||
public int openLeases() {
|
||||
synchronized (lock) {
|
||||
return leases.values().stream().mapToInt(Integer::intValue).sum();
|
||||
}
|
||||
}
|
||||
|
||||
/** The tenants with an active client. */
|
||||
@@ -128,8 +237,9 @@ public final class MongoTenantClientRegistry {
|
||||
}
|
||||
|
||||
private void evictIdle(Instant now) {
|
||||
java.util.List<String> idle =
|
||||
List<String> idle =
|
||||
lastUsed.entrySet().stream()
|
||||
.filter(entry -> !leases.containsKey(entry.getKey()))
|
||||
.filter(entry -> entry.getValue().plus(idleEviction).isBefore(now))
|
||||
.map(Map.Entry::getKey)
|
||||
.toList();
|
||||
@@ -138,7 +248,11 @@ public final class MongoTenantClientRegistry {
|
||||
lastUsed.remove(tenant);
|
||||
// Evicting the bookkeeping entry without closing the client frees the slot and keeps the
|
||||
// connection pool, which is the exhaustion this cap exists to prevent, delayed.
|
||||
close(tenant);
|
||||
try {
|
||||
close(tenant);
|
||||
} catch (IllegalStateException closeFailed) {
|
||||
evictionCloseFailures.incrementAndGet();
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
|
||||
+52
-15
@@ -6,20 +6,30 @@ import dev.caskeleton.adapter.outbound.mongo.query.budget.MongoBudgetEnforcer;
|
||||
import dev.caskeleton.adapter.outbound.mongo.query.budget.MongoBudgetPolicyRegistry;
|
||||
import dev.caskeleton.adapter.outbound.mongo.query.budget.MongoOperationBudget;
|
||||
import java.time.Duration;
|
||||
import java.util.ArrayList;
|
||||
import java.util.Iterator;
|
||||
import java.util.List;
|
||||
import java.util.Objects;
|
||||
import java.util.stream.Stream;
|
||||
import org.bson.Document;
|
||||
import org.bson.RawBsonDocument;
|
||||
import org.bson.codecs.Codec;
|
||||
import org.springframework.data.mongodb.core.MongoOperations;
|
||||
import org.springframework.data.mongodb.core.aggregation.Aggregation;
|
||||
import org.springframework.data.mongodb.core.aggregation.AggregationOperation;
|
||||
import org.springframework.data.mongodb.core.aggregation.AggregationOptions;
|
||||
import org.springframework.data.mongodb.core.aggregation.AggregationResults;
|
||||
|
||||
/**
|
||||
* Runs an aggregation only after its stages, budget and lookups have been checked (design §17).
|
||||
*
|
||||
* <p>Every guard runs before the first stage reaches the server, because an aggregation that is
|
||||
* going to be refused should cost nothing. The one guard that runs afterwards is the result-count
|
||||
* check: {@code $limit} bounds what the pipeline emits, but a pipeline whose own shape produced
|
||||
* more than the budget allows has still told the caller something worth failing over.
|
||||
* going to be refused should cost nothing. Two run while the results arrive: the result count and
|
||||
* the result bytes. The pipeline gets a final {@code $limit} of one document past the count budget,
|
||||
* so the server never produces more than that, and the results are read from the cursor one
|
||||
* document at a time, the encoded size of each added up as it arrives. They used to be read whole —
|
||||
* every document in memory before anything was counted — and the byte budget was declared and never
|
||||
* checked (round9 review 2026-11-12, RR9-01-20; {@code docs/mongodb/query-aggregation-guide.md}
|
||||
* 「집계에 예산을 둔다」).
|
||||
*/
|
||||
public final class PolicyAwareMongoAggregationExecutor {
|
||||
|
||||
@@ -53,6 +63,14 @@ public final class PolicyAwareMongoAggregationExecutor {
|
||||
/**
|
||||
* Executes a typed aggregation.
|
||||
*
|
||||
* <p>Results are read as {@link Document}, sized, then mapped with the template's converter. Two
|
||||
* costs follow, both deliberate. Each document is encoded once more to be measured: the template
|
||||
* hands back decoded documents, so the bytes the server sent are not available, and the budget
|
||||
* has to be in bytes. And the mapping is the converter's alone: the template's typed aggregate
|
||||
* would also publish {@code AfterLoadEvent}/{@code AfterConvertEvent} and run entity callbacks,
|
||||
* and those do not run for an aggregation result read here. A listener that needs them maps from
|
||||
* the returned type itself.
|
||||
*
|
||||
* @throws MongoOperationRejectedException when a stage, lookup, budget or result size is not
|
||||
* permitted
|
||||
*/
|
||||
@@ -71,19 +89,38 @@ public final class PolicyAwareMongoAggregationExecutor {
|
||||
MongoOperationBudget effective =
|
||||
enforcer.narrow(budget, budget.narrowedTo(budgetFor(context, budget)));
|
||||
|
||||
List<AggregationOperation> bounded = new ArrayList<>(plan.operations());
|
||||
// One past the budget: enough to know the budget was exceeded, never more than that.
|
||||
bounded.add(Aggregation.limit(effective.maxResults() + 1L));
|
||||
Aggregation aggregation =
|
||||
Aggregation.newAggregation(plan.operations()).withOptions(optionsFor(profile, effective));
|
||||
AggregationResults<T> results = operations.aggregate(aggregation, collection, outputType);
|
||||
List<T> mapped = results.getMappedResults();
|
||||
if (mapped.size() > effective.maxResults()) {
|
||||
throw MongoOperationRejectedException.of(
|
||||
"aggregation.result",
|
||||
"the aggregation returned "
|
||||
+ mapped.size()
|
||||
+ " documents, above the budget of "
|
||||
+ effective.maxResults());
|
||||
Aggregation.newAggregation(bounded).withOptions(optionsFor(profile, effective));
|
||||
Codec<Document> sizing = operations.getConverter().getCodecRegistry().get(Document.class);
|
||||
List<T> mapped = new ArrayList<>();
|
||||
long bytes = 0;
|
||||
try (Stream<Document> documents =
|
||||
operations.aggregateStream(aggregation, collection, Document.class)) {
|
||||
Iterator<Document> cursor = documents.iterator();
|
||||
while (cursor.hasNext()) {
|
||||
Document document = cursor.next();
|
||||
if (mapped.size() == effective.maxResults()) {
|
||||
throw MongoOperationRejectedException.of(
|
||||
"aggregation.result",
|
||||
"the aggregation returned more than the budget of "
|
||||
+ effective.maxResults()
|
||||
+ " documents");
|
||||
}
|
||||
bytes += new RawBsonDocument(document, sizing).getByteBuffer().remaining();
|
||||
if (bytes > effective.maxResultBytes()) {
|
||||
throw MongoOperationRejectedException.of(
|
||||
"aggregation.result",
|
||||
"the aggregation returned more than the budget of "
|
||||
+ effective.maxResultBytes()
|
||||
+ " bytes");
|
||||
}
|
||||
mapped.add(operations.getConverter().read(outputType, document));
|
||||
}
|
||||
}
|
||||
return mapped;
|
||||
return List.copyOf(mapped);
|
||||
}
|
||||
|
||||
private static MongoOperationBudget budgetFor(
|
||||
|
||||
+60
-9
@@ -36,20 +36,46 @@ public final class MongoCollectionMigrationLedger implements MongoMigrationLedge
|
||||
|
||||
private static final String MIGRATION_ID = "migrationId";
|
||||
|
||||
private static final String FENCE = "fence";
|
||||
|
||||
private final MongoCollection<Document> ledger;
|
||||
|
||||
private final MongoCollection<Document> checkpoints;
|
||||
|
||||
private final MongoCollection<Document> locks;
|
||||
|
||||
/**
|
||||
* A ledger over the default collections of {@code database}, the lock's included.
|
||||
*
|
||||
* <p>The lock collection is {@link MongoCollectionMigrationLock#DEFAULT_COLLECTION}. A deployment
|
||||
* that keeps its runner lease in a collection of another name uses the three-collection
|
||||
* constructor with that collection; with this one the ledger would read a lock that is never
|
||||
* taken and refuse nothing on its account.
|
||||
*
|
||||
* @param database the database being migrated
|
||||
*/
|
||||
public MongoCollectionMigrationLedger(MongoDatabase database) {
|
||||
this(
|
||||
Objects.requireNonNull(database, "database").getCollection(DEFAULT_COLLECTION),
|
||||
database.getCollection(DEFAULT_CHECKPOINT_COLLECTION));
|
||||
database.getCollection(DEFAULT_CHECKPOINT_COLLECTION),
|
||||
database.getCollection(MongoCollectionMigrationLock.DEFAULT_COLLECTION));
|
||||
}
|
||||
|
||||
/**
|
||||
* A ledger over explicit collections.
|
||||
*
|
||||
* @param ledger the applied migrations
|
||||
* @param checkpoints the running migrations' positions
|
||||
* @param locks the collection {@link MongoCollectionMigrationLock} keeps the runner lease in,
|
||||
* read to refuse a replaced runner's ledger entry
|
||||
*/
|
||||
public MongoCollectionMigrationLedger(
|
||||
MongoCollection<Document> ledger, MongoCollection<Document> checkpoints) {
|
||||
MongoCollection<Document> ledger,
|
||||
MongoCollection<Document> checkpoints,
|
||||
MongoCollection<Document> locks) {
|
||||
this.ledger = Objects.requireNonNull(ledger, "ledger");
|
||||
this.checkpoints = Objects.requireNonNull(checkpoints, "checkpoints");
|
||||
this.locks = Objects.requireNonNull(locks, "locks");
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -90,13 +116,34 @@ public final class MongoCollectionMigrationLedger implements MongoMigrationLedge
|
||||
Objects.requireNonNull(checksum, "checksum");
|
||||
Objects.requireNonNull(operator, "operator");
|
||||
Objects.requireNonNull(appliedAt, "appliedAt");
|
||||
requireCurrentFence(fence, "ledger entry for " + migrationId.value());
|
||||
requireFenced(fence, "ledger entry for " + migrationId.value());
|
||||
// A superseded runner that wakes up after its replacement took the lease must not record the
|
||||
// migration as applied under it (round9 review 2026-11-12, RR9-01-28). Two pieces of evidence:
|
||||
// the lock's newest acquisition, which a replacement leaves the moment it takes the lease, and
|
||||
// a checkpoint a newer runner wrote.
|
||||
//
|
||||
// A narrowing, not a guarantee: the reads and the insert are separate operations, so a runner
|
||||
// replaced between them still records. What stops a superseded runner first is the lock's
|
||||
// refresh, which refuses it; this check is the second line for a runner that skipped a
|
||||
// refresh. Making it atomic needs the lock and the ledger in one transaction, which a
|
||||
// standalone deployment does not have.
|
||||
if (MongoCollectionMigrationLock.newestFence(locks) > fence
|
||||
|| checkpoints
|
||||
.find(
|
||||
Filters.and(
|
||||
Filters.eq(MIGRATION_ID, migrationId.value()), Filters.gt(FENCE, fence)))
|
||||
.first()
|
||||
!= null) {
|
||||
throw MongoOperationRejectedException.of(
|
||||
"migration.fence",
|
||||
"a newer migration runner owns the lease; this runner's ledger entry was refused");
|
||||
}
|
||||
ledger.insertOne(
|
||||
new Document(MIGRATION_ID, migrationId.value())
|
||||
.append("checksum", checksum.value())
|
||||
.append("operator", operator)
|
||||
.append("appliedAt", Date.from(appliedAt))
|
||||
.append("fence", fence));
|
||||
.append(FENCE, fence));
|
||||
}
|
||||
|
||||
@Override
|
||||
@@ -117,7 +164,7 @@ public final class MongoCollectionMigrationLedger implements MongoMigrationLedge
|
||||
@Override
|
||||
public void saveCheckpoint(MongoMigrationCheckpoint checkpoint, long fence) {
|
||||
Objects.requireNonNull(checkpoint, "checkpoint");
|
||||
requireCurrentFence(fence, "checkpoint for " + checkpoint.migrationId().value());
|
||||
requireFenced(fence, "checkpoint for " + checkpoint.migrationId().value());
|
||||
// Conditioned on the stored fence as well as matched by id: a superseded runner that wakes up
|
||||
// mid-batch must not move the checkpoint of the runner that replaced it, which would send the
|
||||
// live migration back over rows it has already done or forward past rows it has not.
|
||||
@@ -126,7 +173,7 @@ public final class MongoCollectionMigrationLedger implements MongoMigrationLedge
|
||||
.append("resumePosition", checkpoint.resumePosition())
|
||||
.append("processedCount", checkpoint.processedCount())
|
||||
.append("updatedAt", Date.from(checkpoint.updatedAt()))
|
||||
.append("fence", fence);
|
||||
.append(FENCE, fence);
|
||||
|
||||
// Replace-then-insert rather than one upsert, and the two cases this separates are the whole
|
||||
// behaviour of the fence.
|
||||
@@ -147,7 +194,7 @@ public final class MongoCollectionMigrationLedger implements MongoMigrationLedge
|
||||
.replaceOne(
|
||||
Filters.and(
|
||||
Filters.eq(MIGRATION_ID, checkpoint.migrationId().value()),
|
||||
Filters.or(Filters.exists("fence", false), Filters.lte("fence", fence))),
|
||||
Filters.or(Filters.exists(FENCE, false), Filters.lte(FENCE, fence))),
|
||||
document,
|
||||
new ReplaceOptions().upsert(false))
|
||||
.getMatchedCount();
|
||||
@@ -172,9 +219,13 @@ public final class MongoCollectionMigrationLedger implements MongoMigrationLedge
|
||||
* Refuses a write from an unfenced lease.
|
||||
*
|
||||
* <p>An unfenced lease cannot prove the writer is still the owner, and the whole point of the
|
||||
* ledger is that it says what actually happened.
|
||||
* ledger is that it says what actually happened. Whether a fenced writer is still the current one
|
||||
* is decided in two places, not here: the runner refreshes its lease, conditioned on the fence,
|
||||
* right before it writes, and each write is conditioned on the newest fence the ledger has seen
|
||||
* for the migration. The method was called {@code requireCurrentFence} and checked neither
|
||||
* (round9 review 2026-11-12, RR9-01-28).
|
||||
*/
|
||||
private static void requireCurrentFence(long fence, String what) {
|
||||
private static void requireFenced(long fence, String what) {
|
||||
if (fence == MongoMigrationLock.UNFENCED) {
|
||||
throw MongoOperationRejectedException.of(
|
||||
"migration.fence",
|
||||
|
||||
+19
-1
@@ -6,6 +6,7 @@ import com.mongodb.client.MongoCollection;
|
||||
import com.mongodb.client.MongoDatabase;
|
||||
import com.mongodb.client.model.Filters;
|
||||
import com.mongodb.client.model.Updates;
|
||||
import dev.caskeleton.adapter.outbound.mongo.api.error.MongoOperationRejectedException;
|
||||
import java.time.Clock;
|
||||
import java.time.Duration;
|
||||
import java.time.Instant;
|
||||
@@ -125,6 +126,19 @@ public final class MongoCollectionMigrationLock implements MongoMigrationLock {
|
||||
return fence;
|
||||
}
|
||||
|
||||
/**
|
||||
* The fence of the newest acquisition of the lock in {@code locks}, or {@link #UNFENCED} when the
|
||||
* lock was never taken there.
|
||||
*
|
||||
* <p>For the ledger: a write whose fence is older than this one comes from a runner that has been
|
||||
* replaced, whether or not its replacement has written anything yet.
|
||||
*/
|
||||
static long newestFence(MongoCollection<Document> locks) {
|
||||
Document current =
|
||||
Objects.requireNonNull(locks, "locks").find(Filters.eq("_id", LOCK_ID)).first();
|
||||
return current == null ? UNFENCED : fenceOf(current);
|
||||
}
|
||||
|
||||
private static long fenceOf(Document lock) {
|
||||
Object stored = lock.get("fence");
|
||||
return stored instanceof Number number ? number.longValue() : 1L;
|
||||
@@ -165,7 +179,11 @@ public final class MongoCollectionMigrationLock implements MongoMigrationLock {
|
||||
.getMatchedCount();
|
||||
if (stillOwned != 1) {
|
||||
held = false;
|
||||
throw new IllegalStateException(
|
||||
// The refusal the heartbeat promises (MongoMigrationHeartbeat#beat), so a migration that
|
||||
// stops on it stops for the same reason the heartbeat's own ownership check gives. It was an
|
||||
// IllegalStateException, which read as a server defect (round9 review 2026-11-12, RR9-01-28).
|
||||
throw MongoOperationRejectedException.of(
|
||||
"migration.lock",
|
||||
"the migration lease was lost before it could be refreshed; another runner may have "
|
||||
+ "taken it after this one appeared to stall");
|
||||
}
|
||||
|
||||
+89
@@ -224,6 +224,95 @@ class MongoMigrationLaneTest {
|
||||
assertThat(ledger.findCheckpoint(id).orElseThrow().processedCount()).isEqualTo(200);
|
||||
}
|
||||
|
||||
@Test
|
||||
void aSupersededRunnerCannotRecordTheMigrationAppliedUnderItsReplacement() {
|
||||
// given
|
||||
MongoDatabase database = freshDatabase();
|
||||
MongoCollectionMigrationLedger ledger = new MongoCollectionMigrationLedger(database);
|
||||
ledger.ensureIndexes();
|
||||
MongoMigrationId id = new MongoMigrationId("20260811-011");
|
||||
ledger.saveCheckpoint(
|
||||
MongoMigrationCheckpoint.start(id, FIXED.instant())
|
||||
.advancedTo("o-300", 300, FIXED.instant()),
|
||||
2L);
|
||||
|
||||
// when
|
||||
Throwable refused =
|
||||
catchThrowable(
|
||||
() ->
|
||||
ledger.recordApplied(
|
||||
id, new MongoMigrationChecksum("abc"), "operator", FIXED.instant(), 1L));
|
||||
|
||||
// then
|
||||
assertThat(refused)
|
||||
.isInstanceOf(MongoOperationRejectedException.class)
|
||||
.hasMessageContaining("a newer migration runner owns the lease");
|
||||
}
|
||||
|
||||
@Test
|
||||
void aReplacedRunnerCannotRecordTheMigrationBeforeItsReplacementWroteAnything() {
|
||||
// given
|
||||
MongoDatabase database = freshDatabase();
|
||||
MongoCollectionMigrationLedger ledger = new MongoCollectionMigrationLedger(database);
|
||||
ledger.ensureIndexes();
|
||||
MongoMigrationId id = new MongoMigrationId("20260811-012");
|
||||
Instant early = Instant.parse("2026-08-11T00:00:00Z");
|
||||
MongoMigrationLock stalled =
|
||||
MongoCollectionMigrationLock.tryAcquire(
|
||||
database, "stalled-runner", LEASE, Clock.fixed(early, ZoneOffset.UTC))
|
||||
.orElseThrow();
|
||||
var replacement =
|
||||
MongoCollectionMigrationLock.tryAcquire(
|
||||
database,
|
||||
"next-runner",
|
||||
LEASE,
|
||||
Clock.fixed(early.plus(LEASE).plusSeconds(1), ZoneOffset.UTC))
|
||||
.orElseThrow();
|
||||
|
||||
// when
|
||||
Throwable refused =
|
||||
catchThrowable(
|
||||
() ->
|
||||
ledger.recordApplied(
|
||||
id,
|
||||
new MongoMigrationChecksum("abc"),
|
||||
"operator",
|
||||
FIXED.instant(),
|
||||
stalled.fence()));
|
||||
|
||||
// then
|
||||
assertThat(refused)
|
||||
.as("%s holds the lease with a newer fence and has not checkpointed yet", replacement)
|
||||
.isInstanceOf(MongoOperationRejectedException.class)
|
||||
.hasMessageContaining("a newer migration runner owns the lease");
|
||||
}
|
||||
|
||||
@Test
|
||||
void aLeaseLostBeforeItsRefreshIsTheRefusalTheHeartbeatPromises() {
|
||||
// given
|
||||
MongoDatabase database = freshDatabase();
|
||||
Instant early = Instant.parse("2026-08-11T00:00:00Z");
|
||||
MongoMigrationLock stalled =
|
||||
MongoCollectionMigrationLock.tryAcquire(
|
||||
database, "stalled-runner", LEASE, Clock.fixed(early, ZoneOffset.UTC))
|
||||
.orElseThrow();
|
||||
var unused =
|
||||
MongoCollectionMigrationLock.tryAcquire(
|
||||
database,
|
||||
"next-runner",
|
||||
LEASE,
|
||||
Clock.fixed(early.plus(LEASE).plusSeconds(1), ZoneOffset.UTC))
|
||||
.orElseThrow();
|
||||
|
||||
// when
|
||||
Throwable refused = catchThrowable(() -> stalled.refresh(LEASE));
|
||||
|
||||
// then
|
||||
assertThat(refused)
|
||||
.isInstanceOf(MongoOperationRejectedException.class)
|
||||
.hasMessageContaining("the migration lease was lost");
|
||||
}
|
||||
|
||||
@Test
|
||||
void theLedgerRefusesADoubleApplyOfTheSameId() {
|
||||
MongoDatabase database = freshDatabase();
|
||||
|
||||
+174
-74
@@ -3,6 +3,7 @@ package dev.caskeleton.adapter.outbound.mongo.advanced.tenancy;
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
import static org.assertj.core.api.Assertions.assertThatCode;
|
||||
import static org.assertj.core.api.Assertions.assertThatThrownBy;
|
||||
import static org.assertj.core.api.Assertions.catchThrowable;
|
||||
|
||||
import dev.caskeleton.adapter.outbound.mongo.advanced.MongoAdvancedCapabilityFlags;
|
||||
import dev.caskeleton.adapter.outbound.mongo.advanced.MongoAdvancedCapabilityGuard;
|
||||
@@ -19,10 +20,25 @@ import dev.caskeleton.adapter.outbound.mongo.imperative.atomic.AtomicFilter;
|
||||
import dev.caskeleton.adapter.outbound.mongo.migration.MongoMigrationCheckpoint;
|
||||
import dev.caskeleton.adapter.outbound.mongo.schema.manifest.MongoCollectionManifest;
|
||||
import dev.caskeleton.adapter.outbound.mongo.schema.manifest.MongoIndexManifest;
|
||||
import java.io.IOException;
|
||||
import java.time.Clock;
|
||||
import java.time.Duration;
|
||||
import java.time.Instant;
|
||||
import java.time.ZoneId;
|
||||
import java.time.ZoneOffset;
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
import java.util.Optional;
|
||||
import java.util.Set;
|
||||
import java.util.concurrent.Callable;
|
||||
import java.util.concurrent.ExecutorService;
|
||||
import java.util.concurrent.Executors;
|
||||
import java.util.concurrent.Future;
|
||||
import java.util.concurrent.atomic.AtomicInteger;
|
||||
import java.util.concurrent.atomic.AtomicReference;
|
||||
import java.util.function.Function;
|
||||
import java.util.stream.IntStream;
|
||||
import org.junit.jupiter.api.DisplayName;
|
||||
import org.junit.jupiter.api.Tag;
|
||||
import org.junit.jupiter.api.Test;
|
||||
|
||||
@@ -37,6 +53,8 @@ class MongoTenancyContractTest {
|
||||
MongoAdvancedCapabilityFlags.allDisabled()
|
||||
.withEnabled(MongoCapability.DATABASE_PER_TENANT));
|
||||
|
||||
private static final Instant START = Instant.parse("2026-08-13T00:00:00Z");
|
||||
|
||||
@Test
|
||||
void missingTenantContextFailsClosed() {
|
||||
assertThatThrownBy(() -> injector.apply(Optional.empty(), AtomicFilter.id("o-1")))
|
||||
@@ -116,29 +134,99 @@ class MongoTenancyContractTest {
|
||||
|
||||
@Test
|
||||
void registryEnforcesMaximumActiveClients() {
|
||||
// given
|
||||
MongoTenantClientRegistry registry = new MongoTenantClientRegistry(TENANCY, 2);
|
||||
Instant now = Instant.parse("2026-08-13T00:00:00Z");
|
||||
registry.acquire("t1");
|
||||
registry.acquire("t2");
|
||||
|
||||
registry.acquire("t1", now);
|
||||
registry.acquire("t2", now);
|
||||
// when
|
||||
Throwable refused = catchThrowable(() -> registry.acquire("t3"));
|
||||
|
||||
assertThatThrownBy(() -> registry.acquire("t3", now))
|
||||
.isInstanceOf(MongoOperationRejectedException.class);
|
||||
assertThat(registry.activeCount()).isEqualTo(2);
|
||||
// then
|
||||
assertThat(List.of(refused instanceof MongoOperationRejectedException, registry.activeCount()))
|
||||
.containsExactly(true, 2);
|
||||
}
|
||||
|
||||
@Test
|
||||
void anIdleClientIsEvictedToMakeRoom() {
|
||||
MongoTenantClientRegistry registry =
|
||||
new MongoTenantClientRegistry(TENANCY, 1, Duration.ofMinutes(5));
|
||||
Instant start = Instant.parse("2026-08-13T00:00:00Z");
|
||||
// given
|
||||
MovingClock clock = new MovingClock(START);
|
||||
MongoTenantClientRegistry registry = registry(1, tenant -> () -> {}, clock);
|
||||
registry.acquire("t1").close();
|
||||
clock.advance(Duration.ofMinutes(10));
|
||||
|
||||
registry.acquire("t1", start);
|
||||
registry.acquire("t2", start.plus(Duration.ofMinutes(10)));
|
||||
// when
|
||||
registry.acquire("t2");
|
||||
|
||||
// then
|
||||
assertThat(registry.activeTenants()).containsExactly("t2");
|
||||
}
|
||||
|
||||
@Test
|
||||
void aClientIsIdleFromTheEndOfItsLastUseNotItsStart() {
|
||||
// given
|
||||
MovingClock clock = new MovingClock(START);
|
||||
MongoTenantClientRegistry registry = registry(1, tenant -> () -> {}, clock);
|
||||
MongoTenantClientRegistry.Lease held = registry.acquire("t1");
|
||||
clock.advance(Duration.ofMinutes(30));
|
||||
held.close();
|
||||
clock.advance(Duration.ofMinutes(1));
|
||||
|
||||
// when
|
||||
Throwable refused = catchThrowable(() -> registry.acquire("t2"));
|
||||
|
||||
// then
|
||||
assertThat(List.of(refused instanceof MongoOperationRejectedException, registry.openLeases()))
|
||||
.as("held for thirty minutes, it was in use all that time and is not idle yet")
|
||||
.containsExactly(true, 0);
|
||||
}
|
||||
|
||||
@Test
|
||||
void anUnclosedLeaseIsCounted() {
|
||||
// given
|
||||
MongoTenantClientRegistry registry = new MongoTenantClientRegistry(TENANCY, 2);
|
||||
|
||||
// when
|
||||
var unused = registry.acquire("t1");
|
||||
|
||||
// then
|
||||
assertThat(registry.openLeases()).isEqualTo(1);
|
||||
}
|
||||
|
||||
private static MongoTenantClientRegistry registry(
|
||||
int cap, Function<String, AutoCloseable> clients, Clock clock) {
|
||||
return new MongoTenantClientRegistry(TENANCY, cap, Duration.ofMinutes(5), clients, clock);
|
||||
}
|
||||
|
||||
/** A clock a test moves by hand: one time source for acquiring, closing and eviction. */
|
||||
private static final class MovingClock extends Clock {
|
||||
|
||||
private final AtomicReference<Instant> now;
|
||||
|
||||
private MovingClock(Instant start) {
|
||||
this.now = new AtomicReference<>(start);
|
||||
}
|
||||
|
||||
private void advance(Duration by) {
|
||||
now.updateAndGet(instant -> instant.plus(by));
|
||||
}
|
||||
|
||||
@Override
|
||||
public ZoneId getZone() {
|
||||
return ZoneOffset.UTC;
|
||||
}
|
||||
|
||||
@Override
|
||||
public Clock withZone(ZoneId zone) {
|
||||
return this;
|
||||
}
|
||||
|
||||
@Override
|
||||
public Instant instant() {
|
||||
return now.get();
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
void aTenantDatabaseCannotServeBeforeItsSchemaIsValidated() {
|
||||
MongoTenantLifecyclePolicy policy = MongoTenantLifecyclePolicy.standard();
|
||||
@@ -172,100 +260,112 @@ class MongoTenancyContractTest {
|
||||
}
|
||||
|
||||
@Test
|
||||
@org.junit.jupiter.api.DisplayName("the same tenant is created once under concurrency")
|
||||
void theSameTenantIsCreatedOnce() throws Exception {
|
||||
java.util.concurrent.atomic.AtomicInteger created =
|
||||
new java.util.concurrent.atomic.AtomicInteger();
|
||||
void theSameTenantIsCreatedOnceUnderConcurrency() throws Exception {
|
||||
// given
|
||||
AtomicInteger created = new AtomicInteger();
|
||||
MongoTenantClientRegistry registry =
|
||||
new MongoTenantClientRegistry(
|
||||
TENANCY,
|
||||
registry(
|
||||
8,
|
||||
Duration.ofMinutes(10),
|
||||
tenant -> {
|
||||
created.incrementAndGet();
|
||||
return () -> {};
|
||||
});
|
||||
java.util.concurrent.ExecutorService pool =
|
||||
java.util.concurrent.Executors.newFixedThreadPool(8);
|
||||
},
|
||||
new MovingClock(START));
|
||||
List<Callable<Void>> work =
|
||||
IntStream.range(0, 50)
|
||||
.<Callable<Void>>mapToObj(
|
||||
index ->
|
||||
() -> {
|
||||
registry.acquire("t1").close();
|
||||
return null;
|
||||
})
|
||||
.toList();
|
||||
ExecutorService pool = Executors.newFixedThreadPool(8);
|
||||
|
||||
// when
|
||||
try {
|
||||
java.util.List<java.util.concurrent.Callable<Void>> work =
|
||||
java.util.stream.IntStream.range(0, 50)
|
||||
.<java.util.concurrent.Callable<Void>>mapToObj(
|
||||
index ->
|
||||
() -> {
|
||||
registry.acquire("t1", Instant.parse("2026-08-13T00:00:00Z"));
|
||||
return null;
|
||||
})
|
||||
.toList();
|
||||
for (java.util.concurrent.Future<Void> future : pool.invokeAll(work)) {
|
||||
for (Future<Void> future : pool.invokeAll(work)) {
|
||||
future.get();
|
||||
}
|
||||
} finally {
|
||||
pool.shutdownNow();
|
||||
}
|
||||
|
||||
assertThat(created)
|
||||
// then
|
||||
assertThat(List.of(created.get(), registry.activeCount()))
|
||||
.as("a per-tenant client carries a pool and monitoring threads; a second one doubles both")
|
||||
.hasValue(1);
|
||||
assertThat(registry.activeCount()).isEqualTo(1);
|
||||
.containsExactly(1, 1);
|
||||
}
|
||||
|
||||
@Test
|
||||
@org.junit.jupiter.api.DisplayName("acquire(tenant)는 주입된 clock으로 idle 여부를 판단한다")
|
||||
void anAcquireWithoutAnInstantReadsTheInjectedClock() {
|
||||
void aClientStillHeldIsNotEvictedHoweverLongAgoItWasAcquired() {
|
||||
// given
|
||||
java.util.List<String> closed = new java.util.ArrayList<>();
|
||||
java.util.concurrent.atomic.AtomicReference<Instant> now =
|
||||
new java.util.concurrent.atomic.AtomicReference<>(Instant.parse("2026-08-13T00:00:00Z"));
|
||||
java.time.Clock clock =
|
||||
new java.time.Clock() {
|
||||
@Override
|
||||
public java.time.ZoneId getZone() {
|
||||
return java.time.ZoneOffset.UTC;
|
||||
}
|
||||
List<String> closed = new ArrayList<>();
|
||||
MovingClock clock = new MovingClock(START);
|
||||
MongoTenantClientRegistry registry = registry(1, tenant -> () -> closed.add(tenant), clock);
|
||||
MongoTenantClientRegistry.Lease longRunning = registry.acquire("t1");
|
||||
clock.advance(Duration.ofMinutes(10));
|
||||
|
||||
@Override
|
||||
public java.time.Clock withZone(java.time.ZoneId zone) {
|
||||
return this;
|
||||
}
|
||||
// when
|
||||
Throwable refused = catchThrowable(() -> registry.acquire("t2"));
|
||||
|
||||
@Override
|
||||
public Instant instant() {
|
||||
return now.get();
|
||||
}
|
||||
};
|
||||
// then
|
||||
assertThat(List.of(refused instanceof MongoOperationRejectedException, closed))
|
||||
.as(
|
||||
"t1 is in use by %s; closing it would pull the pool out from under that work",
|
||||
longRunning)
|
||||
.containsExactly(true, List.of());
|
||||
}
|
||||
|
||||
@Test
|
||||
void anotherTenantsClientThatFailsToCloseDoesNotFailThisAcquire() {
|
||||
// given
|
||||
MovingClock clock = new MovingClock(START);
|
||||
MongoTenantClientRegistry registry =
|
||||
new MongoTenantClientRegistry(
|
||||
TENANCY, 1, Duration.ofMinutes(5), tenant -> () -> closed.add(tenant), clock);
|
||||
registry.acquire("t1");
|
||||
now.set(now.get().plus(Duration.ofMinutes(10)));
|
||||
registry(
|
||||
1,
|
||||
tenant ->
|
||||
() -> {
|
||||
if (tenant.equals("t1")) {
|
||||
throw new IOException("t1's pool would not shut down");
|
||||
}
|
||||
},
|
||||
clock);
|
||||
registry.acquire("t1").close();
|
||||
clock.advance(Duration.ofMinutes(10));
|
||||
|
||||
// when
|
||||
MongoTenantClientRegistry.Lease admitted = registry.acquire("t2");
|
||||
|
||||
// then
|
||||
assertThat(
|
||||
List.of(
|
||||
admitted.client() != null,
|
||||
registry.activeTenants(),
|
||||
registry.evictionCloseFailures()))
|
||||
.containsExactly(true, Set.of("t2"), 1L);
|
||||
}
|
||||
|
||||
@Test
|
||||
void evictionClosesTheClientItEvicts() {
|
||||
// given
|
||||
List<String> closed = new ArrayList<>();
|
||||
MovingClock clock = new MovingClock(START);
|
||||
MongoTenantClientRegistry registry = registry(1, tenant -> () -> closed.add(tenant), clock);
|
||||
registry.acquire("t1").close();
|
||||
clock.advance(Duration.ofMinutes(10));
|
||||
|
||||
// when
|
||||
registry.acquire("t2");
|
||||
|
||||
// then
|
||||
assertThat(closed).containsExactly("t1");
|
||||
}
|
||||
|
||||
@Test
|
||||
@org.junit.jupiter.api.DisplayName("eviction closes the client it evicts")
|
||||
void evictionClosesTheClient() {
|
||||
java.util.List<String> closed = new java.util.ArrayList<>();
|
||||
MongoTenantClientRegistry registry =
|
||||
new MongoTenantClientRegistry(
|
||||
TENANCY, 1, Duration.ofMinutes(5), tenant -> () -> closed.add(tenant));
|
||||
Instant start = Instant.parse("2026-08-13T00:00:00Z");
|
||||
|
||||
registry.acquire("t1", start);
|
||||
registry.acquire("t2", start.plus(Duration.ofMinutes(10)));
|
||||
|
||||
assertThat(closed)
|
||||
.as("freeing the slot without closing the client keeps the connection pool it counted")
|
||||
.containsExactly("t1");
|
||||
}
|
||||
|
||||
@Test
|
||||
@org.junit.jupiter.api.DisplayName("a coordinator resumes from its durable store after a restart")
|
||||
@DisplayName("a coordinator resumes from its durable store after a restart")
|
||||
void aCoordinatorResumesFromItsDurableStore() {
|
||||
RecordingCheckpointStore store = new RecordingCheckpointStore();
|
||||
MongoTenantMigrationCoordinator first =
|
||||
|
||||
+157
@@ -3,11 +3,31 @@ package dev.caskeleton.adapter.outbound.mongo.aggregation;
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
import static org.assertj.core.api.Assertions.assertThatCode;
|
||||
import static org.assertj.core.api.Assertions.assertThatThrownBy;
|
||||
import static org.assertj.core.api.Assertions.catchThrowable;
|
||||
import static org.mockito.ArgumentMatchers.any;
|
||||
import static org.mockito.ArgumentMatchers.eq;
|
||||
import static org.mockito.Mockito.mock;
|
||||
import static org.mockito.Mockito.when;
|
||||
|
||||
import com.mongodb.MongoClientSettings;
|
||||
import dev.caskeleton.adapter.outbound.mongo.api.MongoOperationContext;
|
||||
import dev.caskeleton.adapter.outbound.mongo.api.consistency.MongoConsistencyProfile;
|
||||
import dev.caskeleton.adapter.outbound.mongo.api.error.MongoOperationRejectedException;
|
||||
import dev.caskeleton.adapter.outbound.mongo.query.budget.MongoBudgetEnforcer;
|
||||
import dev.caskeleton.adapter.outbound.mongo.query.budget.MongoBudgetPolicyRegistry;
|
||||
import dev.caskeleton.adapter.outbound.mongo.query.budget.MongoOperationBudget;
|
||||
import java.time.Duration;
|
||||
import java.util.List;
|
||||
import java.util.Set;
|
||||
import org.bson.Document;
|
||||
import org.bson.conversions.Bson;
|
||||
import org.junit.jupiter.api.Tag;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.mockito.ArgumentCaptor;
|
||||
import org.springframework.data.mongodb.core.MongoOperations;
|
||||
import org.springframework.data.mongodb.core.aggregation.Aggregation;
|
||||
import org.springframework.data.mongodb.core.convert.MongoConverter;
|
||||
import org.springframework.data.mongodb.core.query.Criteria;
|
||||
|
||||
/** Design §17 — write stages are D4, and stage cost is graded rather than assumed. */
|
||||
@Tag("mongodb-contract")
|
||||
@@ -105,4 +125,141 @@ class PolicyAwareMongoAggregationExecutorTest {
|
||||
assertThat(MongoAggregationProfile.stableRead().allowDiskUse()).isFalse();
|
||||
assertThat(MongoAggregationProfile.stableRead().strictMapping()).isTrue();
|
||||
}
|
||||
|
||||
private static final MongoOperationContext CONTEXT =
|
||||
MongoOperationContext.of(
|
||||
"orders.summary",
|
||||
"primary",
|
||||
"orders",
|
||||
MongoConsistencyProfile.PRIMARY_MAJORITY,
|
||||
Duration.ofSeconds(2));
|
||||
|
||||
private static final MongoAggregationPlan OPEN_ORDERS =
|
||||
MongoAggregationPlan.builder()
|
||||
.stage("$match", Aggregation.match(Criteria.where("status").is("OPEN")))
|
||||
.build();
|
||||
|
||||
@Test
|
||||
void thePipelineEndsWithALimitOnePastTheCountBudget() {
|
||||
// given
|
||||
ArgumentCaptor<Aggregation> sent = ArgumentCaptor.forClass(Aggregation.class);
|
||||
MongoOperations operations = returning(List.of(), sent);
|
||||
|
||||
// when
|
||||
var unused =
|
||||
executorOver(operations, budget(3, 1_048_576L))
|
||||
.execute(
|
||||
CONTEXT,
|
||||
MongoAggregationProfile.stableRead(),
|
||||
OPEN_ORDERS,
|
||||
"orders",
|
||||
Document.class);
|
||||
|
||||
// then
|
||||
List<Document> pipeline = sent.getValue().toPipeline(Aggregation.DEFAULT_CONTEXT);
|
||||
assertThat(pipeline.get(pipeline.size() - 1)).isEqualTo(new Document("$limit", 4L));
|
||||
}
|
||||
|
||||
@Test
|
||||
void moreDocumentsThanTheCountBudgetAreRefused() {
|
||||
// given
|
||||
MongoOperations operations =
|
||||
returning(
|
||||
List.of(order(1), order(2), order(3), order(4)),
|
||||
ArgumentCaptor.forClass(Aggregation.class));
|
||||
|
||||
// when
|
||||
Throwable refused =
|
||||
catchThrowable(
|
||||
() ->
|
||||
executorOver(operations, budget(3, 1_048_576L))
|
||||
.execute(
|
||||
CONTEXT,
|
||||
MongoAggregationProfile.stableRead(),
|
||||
OPEN_ORDERS,
|
||||
"orders",
|
||||
Document.class));
|
||||
|
||||
// then
|
||||
assertThat(refused)
|
||||
.isInstanceOf(MongoOperationRejectedException.class)
|
||||
.hasMessageContaining("more than the budget of 3 documents");
|
||||
}
|
||||
|
||||
@Test
|
||||
void documentsLargerTogetherThanTheByteBudgetAreRefused() {
|
||||
// given
|
||||
// Each order encodes to well over 40 bytes: two of them pass any 80-byte budget.
|
||||
MongoOperations operations =
|
||||
returning(List.of(order(1), order(2)), ArgumentCaptor.forClass(Aggregation.class));
|
||||
|
||||
// when
|
||||
Throwable refused =
|
||||
catchThrowable(
|
||||
() ->
|
||||
executorOver(operations, budget(10, 80L))
|
||||
.execute(
|
||||
CONTEXT,
|
||||
MongoAggregationProfile.stableRead(),
|
||||
OPEN_ORDERS,
|
||||
"orders",
|
||||
Document.class));
|
||||
|
||||
// then
|
||||
assertThat(refused)
|
||||
.isInstanceOf(MongoOperationRejectedException.class)
|
||||
.hasMessageContaining("more than the budget of 80 bytes");
|
||||
}
|
||||
|
||||
@Test
|
||||
void resultsInsideBothBudgetsAreReturnedInCursorOrder() {
|
||||
// given
|
||||
MongoOperations operations =
|
||||
returning(List.of(order(1), order(2)), ArgumentCaptor.forClass(Aggregation.class));
|
||||
|
||||
// when
|
||||
List<Document> results =
|
||||
executorOver(operations, budget(10, 1_048_576L))
|
||||
.execute(
|
||||
CONTEXT,
|
||||
MongoAggregationProfile.stableRead(),
|
||||
OPEN_ORDERS,
|
||||
"orders",
|
||||
Document.class);
|
||||
|
||||
// then
|
||||
assertThat(results).containsExactly(order(1), order(2));
|
||||
}
|
||||
|
||||
private static MongoOperationBudget budget(int maxResults, long maxResultBytes) {
|
||||
return new MongoOperationBudget(maxResults, maxResultBytes, 500L, 50);
|
||||
}
|
||||
|
||||
private static PolicyAwareMongoAggregationExecutor executorOver(
|
||||
MongoOperations operations, MongoOperationBudget budget) {
|
||||
return new PolicyAwareMongoAggregationExecutor(
|
||||
operations,
|
||||
MongoBudgetPolicyRegistry.builder().register("orders.summary", budget).build(),
|
||||
new MongoBudgetEnforcer());
|
||||
}
|
||||
|
||||
/** The driver boundary: a cursor over {@code documents}, mapped as the converter would. */
|
||||
private static MongoOperations returning(
|
||||
List<Document> documents, ArgumentCaptor<Aggregation> sent) {
|
||||
MongoOperations operations = mock(MongoOperations.class);
|
||||
MongoConverter converter = mock(MongoConverter.class);
|
||||
when(operations.getConverter()).thenReturn(converter);
|
||||
when(converter.getCodecRegistry()).thenReturn(MongoClientSettings.getDefaultCodecRegistry());
|
||||
when(converter.read(eq(Document.class), any(Bson.class)))
|
||||
.thenAnswer(invocation -> invocation.getArgument(1));
|
||||
when(operations.aggregateStream(sent.capture(), eq("orders"), eq(Document.class)))
|
||||
.thenAnswer(invocation -> documents.stream());
|
||||
return operations;
|
||||
}
|
||||
|
||||
private static Document order(int number) {
|
||||
return new Document("_id", "order-" + number)
|
||||
.append("status", "OPEN")
|
||||
.append("customer", "customer-" + number);
|
||||
}
|
||||
}
|
||||
|
||||
+7
-2
@@ -19,10 +19,15 @@ public interface IntegrationEventPublishPort {
|
||||
CompletionStage<OutboxPublishAttempt> publish(ValidatedIntegrationEvent event);
|
||||
|
||||
/**
|
||||
* The longest one publish can take before the returned stage completes, when the publisher knows
|
||||
* it. The outbox relay sizes its batch against it (see {@code
|
||||
* The longest one publish can take, from the call to {@code publish} until the returned stage
|
||||
* completes, when the publisher knows it. The outbox relay sizes its batch against it (see {@code
|
||||
* OutboxMessagePublishPort#longestAttempt}).
|
||||
*
|
||||
* <p>A bound, not a hint: a caller may stop waiting once it has passed and treat the attempt as
|
||||
* ambiguous, so a publisher that reports less than it can take turns slow answers into ambiguous
|
||||
* ones. Positive when present. Empty means unknown, and then the stage's own completion is the
|
||||
* only bound.
|
||||
*
|
||||
* @return the longest attempt, or empty when unknown
|
||||
*/
|
||||
default Optional<Duration> longestAttempt() {
|
||||
|
||||
+62
-3
@@ -4,11 +4,13 @@ import dev.caskeleton.application.messaging.event.IntegrationEventPublishPort;
|
||||
import dev.caskeleton.application.messaging.event.ValidatedIntegrationEvent;
|
||||
import dev.caskeleton.application.outbox.OutboxPublishAttempt;
|
||||
import dev.caskeleton.application.outbox.OutboxPublishOutcome;
|
||||
import java.util.ArrayList;
|
||||
import java.time.Duration;
|
||||
import java.util.List;
|
||||
import java.util.Objects;
|
||||
import java.util.Optional;
|
||||
import java.util.concurrent.CompletableFuture;
|
||||
import java.util.concurrent.CompletionStage;
|
||||
import java.util.concurrent.CopyOnWriteArrayList;
|
||||
|
||||
/**
|
||||
* The {@link IntegrationEventPublishPort} fake: records every event it is handed and answers with
|
||||
@@ -17,8 +19,55 @@ import java.util.concurrent.CompletionStage;
|
||||
*/
|
||||
public final class FakeIntegrationEventPublisher implements IntegrationEventPublishPort {
|
||||
|
||||
private final List<ValidatedIntegrationEvent> published = new ArrayList<>();
|
||||
// A relay thread publishes while the test thread reads.
|
||||
private final List<ValidatedIntegrationEvent> published = new CopyOnWriteArrayList<>();
|
||||
private OutboxPublishAttempt answer = OutboxPublishAttempt.confirmed();
|
||||
private boolean silent;
|
||||
private Duration blocksFor = Duration.ZERO;
|
||||
private RuntimeException fails;
|
||||
private Optional<Duration> longestAttempt = Optional.empty();
|
||||
|
||||
/**
|
||||
* Never completes the stage of any later publish, and reports {@code longestAttempt} as the
|
||||
* longest a publish takes: a broker that does not answer, behind a publisher that promised it
|
||||
* would within that bound.
|
||||
*
|
||||
* @param longestAttempt what {@link #longestAttempt()} reports
|
||||
* @return this fake
|
||||
*/
|
||||
public FakeIntegrationEventPublisher silentPast(Duration longestAttempt) {
|
||||
this.longestAttempt = Optional.of(Objects.requireNonNull(longestAttempt, "longestAttempt"));
|
||||
this.silent = true;
|
||||
return this;
|
||||
}
|
||||
|
||||
/**
|
||||
* Holds the caller for {@code blocking} inside every later {@code publish}, before the stage is
|
||||
* handed back: a send that blocks its caller, as a producer waiting on cluster metadata does.
|
||||
*
|
||||
* @param blocking how long {@code publish} blocks
|
||||
* @return this fake
|
||||
*/
|
||||
public FakeIntegrationEventPublisher blockingFor(Duration blocking) {
|
||||
this.blocksFor = Objects.requireNonNull(blocking, "blocking");
|
||||
return this;
|
||||
}
|
||||
|
||||
/**
|
||||
* Completes every later publish's stage exceptionally with {@code failure}.
|
||||
*
|
||||
* @param failure what the stage fails with
|
||||
* @return this fake
|
||||
*/
|
||||
public FakeIntegrationEventPublisher failingWith(RuntimeException failure) {
|
||||
this.fails = Objects.requireNonNull(failure, "failure");
|
||||
return this;
|
||||
}
|
||||
|
||||
@Override
|
||||
public Optional<Duration> longestAttempt() {
|
||||
return longestAttempt;
|
||||
}
|
||||
|
||||
/**
|
||||
* Answers every later publish with {@code outcome}, and for anything but a confirmation the
|
||||
@@ -45,7 +94,17 @@ public final class FakeIntegrationEventPublisher implements IntegrationEventPubl
|
||||
@Override
|
||||
public CompletionStage<OutboxPublishAttempt> publish(ValidatedIntegrationEvent event) {
|
||||
published.add(Objects.requireNonNull(event, "event must not be null"));
|
||||
return CompletableFuture.completedFuture(answer);
|
||||
if (!blocksFor.isZero()) {
|
||||
try {
|
||||
Thread.sleep(blocksFor);
|
||||
} catch (InterruptedException interrupted) {
|
||||
Thread.currentThread().interrupt();
|
||||
}
|
||||
}
|
||||
if (fails != null) {
|
||||
return CompletableFuture.failedFuture(fails);
|
||||
}
|
||||
return silent ? new CompletableFuture<>() : CompletableFuture.completedFuture(answer);
|
||||
}
|
||||
|
||||
/** Every event handed to this fake, in order. */
|
||||
|
||||
+1
-1
@@ -131,7 +131,7 @@ app:
|
||||
primary-for-channel: ${APP_NOTIFICATION_PLATFORM_SMTP_PRIMARY:true}
|
||||
environment: ${APP_NOTIFICATION_PLATFORM_SMTP_ENVIRONMENT:local}
|
||||
credential-profile: ${APP_NOTIFICATION_PLATFORM_SMTP_CREDENTIAL_PROFILE:default}
|
||||
timeout: ${APP_NOTIFICATION_PLATFORM_SMTP_TIMEOUT:10s}
|
||||
timeout: ${APP_NOTIFICATION_PLATFORM_SMTP_TIMEOUT:30s}
|
||||
max-concurrency: ${APP_NOTIFICATION_PLATFORM_SMTP_MAX_CONCURRENCY:4}
|
||||
rate-per-second: ${APP_NOTIFICATION_PLATFORM_SMTP_RATE_PER_SECOND:10}
|
||||
ses:
|
||||
|
||||
Reference in New Issue
Block a user