fix(outbound)!: 발행 대기·SMTP 발송·캐시 시간 상한을 지킨다

outbox 발행 대기가 publish()가 돌아온 뒤부터 재서, 호출을 붙잡은
publisher는 longestAttempt를 넘겨도 relay를 묶는다. SMTP 발송 pool은
대기열이 찰 때까지 thread 하나로 돌고, profile들의 max-concurrency가 pool이
담는 수를 넘어도 기동한다. 넘친 발송은 결과 불명으로 남는다. SMTP
profile timeout은 명령 하나만 묶어, 느린 relay에서 발송이 lease를 넘으면
lease 복구가 아직 도는 발송을 운영자 확인으로 넘긴다. Redis 캐시
command-timeout에 lane 대기 2초가 더해지고, 값 크기 상한은 설정할 길이
없다.

고른 방법: 발행 대기 마감은 publish() 전에 잡고 남은 시간만 기다린다.
기다리는 중 interrupt는 결과 불명, 이미 interrupt된 thread와 0 이하
longestAttempt는 보내지 않고 REJECTED_RETRYABLE로 돌려준다. SMTP pool은
dispatch-concurrency개 thread와 그 네 배 대기열로 고정한다. 한 family의
켜진 profile 전체를 보는 ProviderRuntimeAssembler.requireTogether를 두고,
SMTP는 max-concurrency 합이 pool보다 크거나 lease가 timeout 두 배 이하면
조립을 실패시킨다. profile timeout은 발송 한 번을 두 번 묶는다:
pool 대기가 넘으면 보내지 않고 재시도, 발송이 넘으면 PROVIDER_RESPONSE_LOST
로 남겨 다시 보내지 않는다. 두 시한은 completeOnTimeout이라 취소된다.
기본 timeout은 10초에서 30초로, socket timeout 합보다 길어야 한다.
Redis 연산 전체를 command-timeout 하나로 묶고
app.capabilities.cache.maximum-value-bytes를 다시 묶는다(1..16 MiB).
버린 대안: 발송 timeout에 재시도(같은 메일을 두 번 보낼 수 있다), 캐시
상한을 상수로 두기(큰 값을 쓰는 서비스가 바꿀 길이 없다), assembler가
assemble마다 합을 누적하기(조립 순서에 상태가 남는다).

위험·한계: 기동 조건이 늘었다(SMTP max-concurrency 합, lease와 SMTP
timeout, 캐시 값 상한 범위). 대기 시한이 돌려주는 stage는 일부러 버린다
(Error Prone 규칙대로 `unused`에 받고, SpotBugs DLS는 config/spotbugs/
exclude.xml에 그 메서드만 제외한다. Jetty abortAtDeadline 선례). send pool을 바꿔 끼운 배포는 부른 thread에서
돌리지 않는 executor여야 한다. outbox 새 거절 코드의 runbook 줄은 runbook
묶음에 넘긴다. claim·fence 커밋 위에 쌓았다.

검증: desktop RED(고치기 전 트리 + 새 시험, pr2-red2) 새 시험 실패.
GREEN은 PR 본문의 verify.sh pr 결과.

Refs: RR9-01-05, RR9-01-15, RR9-01-16, RR9-01-17, RR9-01-18
BREAKING CHANGE: SMTP profile들의 max-concurrency 합이 send pool(thread와 대기열)을 넘거나 dispatch.lease-duration이 SMTP timeout의 두 배 이하면 기동이 실패한다. SMTP 기본 timeout이 30초이고, 넘은 발송은 결과 불명이 된다. SmtpNotificationProviderAdapter 생성자가 timeout을 받는다. Redis 캐시 값이 maximum-value-bytes(기본 1 MiB)를 넘으면 기록하지 않는다. docs/upgrade/2026-11-round9-outbound.md
This commit is contained in:
DongHyeonka
2026-11-13 16:21:16 +09:00
parent 141c2a4f6d
commit 154cb4b795
30 changed files with 1384 additions and 43 deletions
+2 -1
View File
@@ -171,6 +171,7 @@ APP_CACHE_REDIS_COMMAND_TIMEOUT=200ms
APP_CACHE_REDIS_POSITIVE_SOFT_TTL=30s
APP_CACHE_REDIS_MINIMUM_HARD_TTL=1s
APP_CACHE_REDIS_SEMANTIC_REGION=default
APP_CACHE_REDIS_MAXIMUM_VALUE_BYTES=1048576
APP_MESSAGING_BROKER=kafka
APP_MESSAGING_PRODUCER_ID=
APP_MESSAGING_KAFKA_BROKERS=
@@ -293,7 +294,7 @@ APP_NOTIFICATION_PLATFORM_SMTP_ENABLED=false
APP_NOTIFICATION_PLATFORM_SMTP_PRIMARY=true
APP_NOTIFICATION_PLATFORM_SMTP_ENVIRONMENT=local
APP_NOTIFICATION_PLATFORM_SMTP_CREDENTIAL_PROFILE=default
APP_NOTIFICATION_PLATFORM_SMTP_TIMEOUT=10s
APP_NOTIFICATION_PLATFORM_SMTP_TIMEOUT=30s
APP_NOTIFICATION_PLATFORM_SMTP_MAX_CONCURRENCY=4
APP_NOTIFICATION_PLATFORM_SMTP_RATE_PER_SECOND=10
APP_NOTIFICATION_PLATFORM_SMTP_SENDER_IDENTITY=no-reply@example.invalid
+9
View File
@@ -359,6 +359,15 @@
<Method name="abortAtDeadline"/>
</Match>
<!-- DLS_DEAD_LOCAL_STORE, the same disagreement: the SMTP send's queue deadline discards the
stage `thenRun` returns into `unused`, because an abandoned send reports through its own
result, not through that stage. -->
<Match>
<Bug pattern="DLS_DEAD_LOCAL_STORE"/>
<Class name="dev.caskeleton.adapter.outbound.notification.platform.provider.smtp.SmtpNotificationProviderAdapter$PooledSend"/>
<Method name="armQueueDeadline"/>
</Match>
<!-- The servlet error-dispatch controller (re-review 2026-11-03 RR-02-02) must answer for any
HTTP method: the original request's method is whatever failed, and the container forwards it
unchanged to /error. CSRF protection guards a state change made with ambient credentials;
+31 -2
View File
@@ -26,7 +26,7 @@ support leaf의 `ModuleDefaultsEnvironmentPostProcessor`가 가장 낮은 우선
| 속성 | 환경 변수 | 기본값 | 범위 |
|---|---|---|---|
| `dispatch.claim-batch-size` | `APP_NOTIFICATION_PLATFORM_CLAIM_BATCH_SIZE` | `50` | 1..1000 |
| `dispatch.lease-duration` | `APP_NOTIFICATION_PLATFORM_LEASE_DURATION` | `2m` | 양수, 유한. `poll-interval`과 켜진 provider profile의 `timeout`보다 길어야 기동한다 |
| `dispatch.lease-duration` | `APP_NOTIFICATION_PLATFORM_LEASE_DURATION` | `2m` | 양수, 유한. `poll-interval`과 켜진 provider profile의 `timeout`보다 길어야 기동한다. SMTP profile은 `timeout`의 두 배보다 길어야 한다(「SMTP relay」) |
| `dispatch.poll-interval` | `APP_NOTIFICATION_PLATFORM_POLL_INTERVAL` | `1s` | 양수, 유한 |
| `dispatch.max-global-concurrency` | `APP_NOTIFICATION_PLATFORM_MAX_CONCURRENCY` | `64` | 양수 |
| `dispatch.max-additional-attempts` | `APP_NOTIFICATION_PLATFORM_MAX_ADDITIONAL_ATTEMPTS` | `4` | 0 이상 |
@@ -134,7 +134,7 @@ Spring이 이미 그것을 가지고 있고, 두 번째 표기는 맞춰 둘 것
| `providers.smtp.primary-for-channel` | `APP_NOTIFICATION_PLATFORM_SMTP_PRIMARY` | `true` | boolean. 채널마다 primary는 정확히 하나 |
| `providers.smtp.environment` | `APP_NOTIFICATION_PLATFORM_SMTP_ENVIRONMENT` | `local` | 켜면 필요하다 |
| `providers.smtp.credential-profile` | `APP_NOTIFICATION_PLATFORM_SMTP_CREDENTIAL_PROFILE` | `default` | `SecretMaterialProvider`로 푼다. 값을 직접 적지 않는다 |
| `providers.smtp.timeout` | `APP_NOTIFICATION_PLATFORM_SMTP_TIMEOUT` | `10s` | 양수, 유한 |
| `providers.smtp.timeout` | `APP_NOTIFICATION_PLATFORM_SMTP_TIMEOUT` | `30s` | 양수, 유한. `mail.smtp.connectiontimeout`에 `mail.smtp.timeout`·`mail.smtp.writetimeout` 중 긴 것을 더한 값보다 길어야 기동한다 |
| `providers.smtp.max-concurrency` | `APP_NOTIFICATION_PLATFORM_SMTP_MAX_CONCURRENCY` | `4` | 양수 |
| `providers.smtp.rate-per-second` | `APP_NOTIFICATION_PLATFORM_SMTP_RATE_PER_SECOND` | `10` | 양수 |
@@ -143,11 +143,40 @@ Spring이 이미 그것을 가지고 있고, 두 번째 표기는 맞춰 둘 것
| `smtp.sender-identity` | `APP_NOTIFICATION_PLATFORM_SMTP_SENDER_IDENTITY` | `no-reply@example.invalid` | 주소 |
| `smtp.dispatch-concurrency` | `APP_NOTIFICATION_PLATFORM_SMTP_DISPATCH_CONCURRENCY` | `4` | 양수 |
`smtp.dispatch-concurrency`는 SMTP 발송 pool의 thread 수다. 첫 발송부터 그만큼 함께 돌고, 대기열은 그 네 배다. 모든 SMTP
profile이 이 pool 하나를 같이 쓴다. 켜진 SMTP profile들의 `max-concurrency` 합이 thread와 대기열을 합친 것보다 크면 SMTP
조립이 기동 때 실패한다. pool이 가득 차서 거절한 발송은 relay에 닿지 않은 것으로 기록되어 다시 시도된다(결과 불명이
아니다).
SMTP 전송 설정의 소유자는 Spring Mail 하나다. 실제 `JavaMailSenderImpl`은 필수 STARTTLS
(`mail.smtp.starttls.enable=true`와 `mail.smtp.starttls.required=true`)나 암묵적 TLS(`mail.smtp.ssl.enable=true`)를
써야 한다. `mail.smtp.connectiontimeout`, `mail.smtp.timeout`, `mail.smtp.writetimeout`도 양수여야 한다. 이 속성이
없거나 틀리면 SMTP 조립이 트래픽을 받기 전에 실패한다.
이 socket timeout은 SMTP 명령 하나씩을 묶을 뿐 대화 전체를 묶지 않는다. 발송 한 번 전체는 provider profile의 `timeout`이
두 번 묶는다(2026-11-13).
- pool 대기: 발송이 send pool에서 `timeout`이 지나도록 시작하지 못하면 버려지고 relay에 가지 않는다. relay에 닿지 않은
것(`PROVIDER_UNAVAILABLE`, `CAPACITY_REJECTED`)으로 기록되어 다시 시도된다.
- 발송: pool thread가 잡은 때부터 `timeout`이 지나면 결과 불명(`PROVIDER_RESPONSE_LOST`)으로 기록되고 자동으로 다시 보내지
않는다. DATA가 이미 넘어갔을 수 있기 때문이다.
그래서 한 발송이 delivery lease를 쓰는 시간은 많아야 `timeout`의 두 배다. 기동 때 두 가지를 본다.
- `dispatch.lease-duration`은 SMTP `timeout`의 두 배보다 길어야 한다. 모든 provider에 거는 조건(lease가 `timeout`보다
길다)보다 엄격하다. 짧으면 생기는 일은 위 「발송」 절의 lease 문단과 같다. 기본값(lease 2분, timeout 30초)은 맞는다.
- profile `timeout`은 connect timeout에 명령 timeout 하나를 더한 값보다 길어야 한다. 그보다 짧으면 인사말을 주지 않는
relay처럼 아무것도 보내지 않은 실패가, socket timeout이 알려 주기 전에 결과 불명으로 기록된다. 기본값(30초, connect 5초 +
명령 10초)은 맞는다.
`dispatch.shutdown-grace`(기본 20초)가 `timeout`의 두 배보다 짧으면, 종료하는 동안 끝나지 않은 발송은 attempt가
`DISPATCHING`으로 남고, 위 「발송」 절의 lease 문단대로 운영자 확인으로 넘어간다(종료된 발송은 결과를 기록하지 못한다).
느린 relay에서 이것 없이 끝내려면 pod 종료 유예와 함께 grace를 늘린다.
send pool(`notificationSmtpExecutor` bean)을 바꿔 끼우면 발송을 부른 thread에서 돌리지 않는 executor여야 한다
(`CallerRunsPolicy`나 직접 실행 executor는 안 된다). 부른 thread에서 돌면 두 시한이 dispatch thread를 풀지 못한다.
`ThreadPoolExecutor`가 아니면 위의 pool 합 검사도 하지 않는다.
환경 변수 계약은 Spring 그대로다. `SPRING_MAIL_HOST`, `SPRING_MAIL_PORT`, 선택인 `SPRING_MAIL_USERNAME`과 secret
`SPRING_MAIL_PASSWORD`, 그리고 `.env.example`에 있는 `SPRING_MAIL_PROPERTIES_MAIL_SMTP_*` TLS·시간 상한 키다. Spring
Boot 자신이 `spring.mail.*`를 가지고 읽으므로 이름을 Spring 그대로 둔다. 알림 모듈은 같은 relay에 두 번째 별칭을
+5 -6
View File
@@ -2688,10 +2688,8 @@ env_keys:
required_test: cache-contract:redis-semantic-region-bound
- name: APP_CACHE_REDIS_MAXIMUM_VALUE_BYTES
# 폐기 2026-08-10: 이 속성을 바인딩하던 settings class를 옛 Redis 세대와 함께 지워서 이 키는 어디에도
# 닿지 않는다. 아직 이 키를 쓰는 배포에 무엇이 대신하는지 알려 주려고 행을 지우지 않고 둔다.
deprecated_orphaned: true
removal_deadline: 2026-11-30
# 2026-08-10에 옛 Redis 세대와 함께 폐기했다가, 2026-11-13 cache-redis 모듈 기본 설정이
# app.capabilities.cache.maximum-value-bytes에 다시 묶었다. 상한을 넘는 값은 기록하지 않는다(round9 리뷰 RR9-01-18).
type: int
default: 1048576
allowed_values: null
@@ -4780,9 +4778,10 @@ env_keys:
required_test: adapter-contract:notification-smtp-provider-assembled
- name: APP_NOTIFICATION_PLATFORM_SMTP_TIMEOUT
# source: NTF-INT-001 — 이 profile의 시도마다 provider timeout.
# source: NTF-INT-001 — 이 profile의 발송 한 번 상한: pool 대기와 발송이 각각 이 시간을 넘지 않는다.
# 2026-11-13 10s에서 30s로 올렸다. 기동 때 mail.smtp.connectiontimeout + 명령 timeout보다 길어야 한다.
type: duration
default: 10s
default: 30s
allowed_values: null
classification: public-config
required: false
+1 -1
View File
@@ -19,7 +19,7 @@ tenant pool 예산을 쓰는 서비스다.
| 3 | persistence-jpa의 `V23__durable_operation_lease_fence.sql`(`durable_operation.lease_fence BIGINT NOT NULL DEFAULT 0`)이 기본 stream에 더해진다. 같은 번호의 서비스 마이그레이션이 있으면 번호를 옮긴다. `durable_operation`을 건드린 트랜잭션이 열려 있으면 5초 뒤 실패하고 V22에 남는다(다시 배포) | Flyway가 번호 충돌로 기동하지 않는다 | persistence-jpa를 쓰는 모든 서비스 |
| 4 | 알림 stream에 `V14__notification_provider_event_open_index.sql`(부분 인덱스 `ix_notification_provider_event_open`)이 더해진다. `notification_provider_event`가 이미 큰 운영 DB는 배포 전에 같은 인덱스를 `CREATE INDEX CONCURRENTLY IF NOT EXISTS ix_notification_provider_event_open ON notification_provider_event (received_at) WHERE projection_status IN ('PENDING', 'FAILED') OR attempt_id IS NULL;`로 먼저 만든다. 미리 만든 인덱스가 실패해 남았으면(`SELECT indisvalid FROM pg_index WHERE indexrelid = 'ix_notification_provider_event_open'::regclass`가 false) `DROP INDEX CONCURRENTLY ix_notification_provider_event_open` 뒤 다시 만든다. `IF NOT EXISTS`는 쓸 수 없는 인덱스도 있는 것으로 보고 넘어간다 | 마이그레이션이 인덱스를 만드는 동안 provider callback 저장이 기다린다(provider가 callback을 다시 보낸다) | 알림 capability를 켠 서비스 |
| 5 | `RecipientLeaseStorePort.expiredDispatching`이 `List<RecipientLease>`(버려질 때의 owner·fence·만료 시각)를 돌려주고, `RecipientDeliveryStorePort`에 `requeueAbandoned(lease, nextDispatchAt)`가 생겼다. 직접 구현했으면 상태가 `DISPATCHING`이고 owner·fence가 같고 lease가 늘지 않았을 때만 `READY_TO_DISPATCH`로 바꾼다(`RecipientDeliveryStoreContract` 새 사례). owner가 없거나 fence가 0인 `DISPATCHING` 행(notification V4 전에 남은 행)은 복구가 더는 고르지 않는다. 그런 행은 `SELECT id FROM notification_recipient_delivery WHERE delivery_state = 'DISPATCHING' AND (lease_owner IS NULL OR lease_fence = 0)`로 찾는다. 배포 전에 `UPDATE notification_recipient_delivery SET lease_owner = COALESCE(lease_owner, 'pre-v4'), lease_fence = lease_fence + 1, lease_until = now() - interval '1 day', version = version + 1, updated_at = now() WHERE delivery_state = 'DISPATCHING' AND (lease_owner IS NULL OR lease_fence = 0)`로 owner를 채우고 fence를 올려 복구 패스에 넘긴다. 복구는 attempt를 보고 정한다: attempt가 없으면 다시 보낼 차례로 돌리고, 끝나지 않은 attempt가 있으면 맞추기로 넘긴다(그 뒤는 `docs/notification/configuration-reference.md` 「발송」의 lease 문단). 손으로 `READY_TO_DISPATCH`로 바꾸지 않는다. 발송 중에 멈춘 행이면 같은 알림이 한 번 더 나간다. owner가 없는 행은 stuck lease 지표에도 잡히지 않는다 | 컴파일이 실패한다. 조건 없이 바꾸면 끝났거나 다시 claim된 delivery가 다시 발송된다 | 알림 store를 직접 구현한 서비스 |
| 6 | 알림 provider profile의 `timeout`이 `app.notification.platform.dispatch.lease-duration`보다 짧은지 확인한다. 짧은 것은 필요조건일 뿐이다: lease는 claim 때 시작해서 attempt 기록·렌더링·provider 허가 대기도 lease를 쓴다. lease는 timeout보다 넉넉히 길게 둔다. 이 timeout이 provider 호출 전체를 묶는 것은 SES(API 호출 시간 상한)다. SMTP는 `spring.mail` socket timeout이 명령마다 묶을 뿐이라 이 검사만으로는 호출 전체가 lease 안에 끝난다고 말할 수 없다 | 짧지 않으면 기동이 실패한다(`notification provider profile '…': timeout (…) must be shorter than …lease-duration`). 전에는 provider 호출 하나가 lease보다 길어도 기동했다. 그때 생기는 일은 `docs/notification/configuration-reference.md` 「발송」의 lease 문단이다 | 알림 capability를 켠 서비스 |
| 6 | 알림 provider profile의 `timeout`이 `app.notification.platform.dispatch.lease-duration`보다 짧은지 확인한다. 짧은 것은 필요조건일 뿐이다: lease는 claim 때 시작해서 attempt 기록·렌더링·provider 허가 대기도 lease를 쓴다. lease는 timeout보다 넉넉히 길게 둔다. 이 timeout은 SES에서는 API 호출 하나의 상한이다. SMTP의 추가 조건은 `docs/upgrade/2026-11-round9-outbound.md` 9행이다 | 짧지 않으면 기동이 실패한다(`notification provider profile '…': timeout (…) must be shorter than …lease-duration`). 전에는 provider 호출 하나가 lease보다 길어도 기동했다. 그때 생기는 일은 `docs/notification/configuration-reference.md` 「발송」의 lease 문단이다 | 알림 capability를 켠 서비스 |
| 7 | `app.lock.wait-time`을 설정에서 지운다. 기다릴 시간은 `tryAcquire(key, waitTime, leaseTtl)`의 인자다 | `app.lock` 아래 모르는 키라 기동이 실패한다. 전에도 읽는 곳이 없었다 | `app.lock.wait-time`을 설정한 서비스 |
| 8 | `JpaLiveEventReplay.sweepExpired()`를 부르던 코드를 지운다. replay를 켜면 `LiveEventReaper`가 `app.realtime.replay.reaper-interval`(기본 `PT10M`)마다 `retained_until`이 지난 행을 1000행씩 지운다. `app.realtime.replay.*`는 이제 `LiveEventReplaySettings`가 엄격하게 묶는다: 그 아래 모르는 키, 0 이하의 `retention`·`reaper-interval`은 기동을 실패시킨다 | 컴파일이 실패한다. 잘못 쓴 키는 기동이 실패한다. 전에는 지우는 곳이 없어 `live_event_log`가 끝없이 자랐다 | `app.realtime.replay.enabled=true`인 서비스 |
| 9 | `IdempotencyReaper`를 직접 만들던 코드는 `new IdempotencyReaper(repository)`로 바꾼다(`Clock` 인자 삭제). 만료 비교는 DB 시계(`now()`)이고 1000행씩 지운다 | 컴파일이 실패한다 | reaper를 직접 조립한 서비스 |
+29
View File
@@ -0,0 +1,29 @@
# 업그레이드 안내 — 2026-11 outbound 발행·발송·캐시·S3·MongoDB·HTTP client
- 범위: outbox 발행 대기, SMTP 발송 pool과 시간 상한, Redis 캐시 시간·크기 상한을 고친 변경. 기동 조건이 바뀐 곳이
있다. 아래 표가 그 목록이다.
- 근거: 2026-11-12 전체 리뷰(`docs/reviews/2026-11-12-round9-review/`)의 발행·발송·캐시(B10a) 발견
## 1. 누가 읽나
이 템플릿을 `upstream` remote로 두고 merge해 받는 파생 서비스가 읽는다. 특히 SMTP 알림, Redis 캐시, outbox
publisher를 직접 구현한 서비스다.
## 2. 할 일
| # | 할 일 | 안 하면 | 해당 서비스 |
| --- | --- | --- | --- |
| 1 | SMTP provider profile들의 `max-concurrency` 합이 발송 pool이 담는 수(`app.notification.platform.smtp.dispatch-concurrency`개 thread와 그 네 배의 대기열) 이하인지 확인한다. 모든 SMTP profile이 pool 하나를 같이 쓴다. 기본값(profile 하나, 4와 4)은 맞는다 | 넘으면 넘게 만든 profile의 조립이 기동 때 실패한다. 전에는 넘친 발송이 대기열에서 거절되어 결과 불명으로 남았다 | SMTP 알림을 켠 서비스 |
| 2 | Redis 캐시의 `command-timeout`이 이제 한 캐시 연산 전체(연결 대기와 응답 전부)의 상한이다. 연결 대기가 길던 배포는 값을 다시 본다 | 상한 안에 못 끝난 연산이 캐시 없음(`DEGRADED_UNAVAILABLE`·`Unavailable`)으로 끝나는 일이 늘 수 있다. 전에는 lane 대기 2초가 더해졌다 | Redis 캐시를 켠 서비스 |
| 3 | encode한 값이 1 MiB(`RedisCacheRegionAdapter.DEFAULT_MAXIMUM_VALUE_BYTES`)를 넘는 캐시는 그 값을 기록하지 않는다(`NOT_RECORDED_PROVIDER_POLICY`). 더 큰 값이 필요하면 `app.capabilities.cache.maximum-value-bytes`(`APP_CACHE_REDIS_MAXIMUM_VALUE_BYTES`, 1..16777216)로 상한을 올린다. 2026-08-10에 폐기했던 그 env 키가 다시 묶였다 | 그 값은 매번 source에서 읽힌다 | 큰 값을 캐시하던 서비스 |
| 4 | SMTP provider profile의 `timeout`(기본이 `10s`에서 `30s`로 바뀌었다)이 발송 한 번을 두 번 묶는다. 발송이 send pool에서 그 시간이 지나도록 시작하지 못하면 relay에 보내지 않고 시작 전 실패(`PROVIDER_UNAVAILABLE`, 재시도)로 기록한다. 시작한 뒤 그 시간이 지나면 결과 불명(`PROVIDER_RESPONSE_LOST`)으로 기록하고 자동으로 다시 보내지 않는다. 값은 `mail.smtp.connectiontimeout`에 `mail.smtp.timeout`·`mail.smtp.writetimeout` 중 긴 것을 더한 값보다 길어야 하고, `dispatch.lease-duration`은 그 두 배보다 길어야 한다. `dispatch.shutdown-grace`가 그 두 배보다 짧으면 종료 중에 끝나지 않은 발송이 결과 불명으로 남는다(`docs/notification/configuration-reference.md` 「SMTP relay」). `SmtpNotificationProviderAdapter`를 직접 만들던 코드는 마지막 인자로 이 timeout을 넘긴다. 직접 만든 `ProviderRuntimeAssembler`는 새 default 메서드 `requireTogether(profiles, deliveryLease)`로 한 family의 켜진 profile 전체에 걸린 검사를 할 수 있다(`NotificationProviderAssembly`가 SERVING일 때 family마다 한 번 부른다). `assemble`을 직접 부르는 코드는 그 전에 `requireTogether`를 부른다 | timeout이 socket timeout 합 이하이거나 lease가 timeout 두 배 이하면 SMTP 조립이 기동 때 실패한다. 상한 안에 끝나지 않는 발송이 결과 불명으로 남는다. 전에는 socket timeout이 명령마다만 묶어, 느린 relay에서 발송이 lease를 넘으면 lease 복구가 아직 도는 발송을 운영자 확인으로 넘겼다. | SMTP 알림을 켠 서비스 |
## 3. 바뀐 동작 (따라 할 일 없음)
- outbox relay의 발행 대기는 `publish()`를 부른 때부터 publisher의 `longestAttempt()`까지다. 넘으면 결과 불명(`PUBLISH_WAIT_EXCEEDED`)으로 기록하고
다음 claim에서 다시 본다. 직접 만든 `IntegrationEventPublishPort`는 `longestAttempt()`를 실제 최장 시간으로 답한다. 0이나
음수를 답하면 그 행은 보내지 않고 `REJECTED_RETRYABLE`(`PUBLISH_BOUND_NOT_POSITIVE`)로 backoff 뒤에 다시 본다.
- 이미 interrupt된 relay thread는 그 행을 발행하지 않고 `REJECTED_RETRYABLE`(`PUBLISH_INTERRUPTED_BEFORE_SEND`)로 돌려준다.
relay가 그 행을 backoff 뒤로 미뤘다가 발행한다. 기다리는 중의 interrupt는 전처럼 결과 불명(`PUBLISH_WAIT_INTERRUPTED`)이다.
- SMTP 발송 pool은 첫 발송부터 `dispatch-concurrency`개 thread로 돈다(전에는 대기열이 찰 때까지 thread 하나였다).
pool이 거절한 발송은 relay에 닿지 않은 것(`PROVIDER_UNAVAILABLE`, `CAPACITY_REJECTED`)으로 기록되어 재시도된다.
+6 -1
View File
@@ -132,7 +132,12 @@ credential이 깨지면 regular lane PING이 UP을 알리게 두지 않고 pod
- 부정 항목은 더 짧은 부정 TTL을 쓴다;
- Redis 물리 TTL은 hard 만료다;
- cache 장애는 애플리케이션의 타입 있는 unavailable·degraded 결과로 떨어진다;
- 식별자는 keyspace에 들어가기 전에 키 있는 digest를 거친다.
- 식별자는 keyspace에 들어가기 전에 키 있는 digest를 거친다;
- `app.capabilities.cache.command-timeout`은 한 cache 연산 전체(lane 연결 대기와 응답 전부)의 상한이다. rate limit과
같은 방식이다. 2026-11-13 전에는 lane 대기 2초가 그 위에 더해졌다. 상한 안에 못 끝난 연산은 `DEGRADED_UNAVAILABLE`이나
`Unavailable`이다;
- encode한 값이 `app.capabilities.cache.maximum-value-bytes`(`APP_CACHE_REDIS_MAXIMUM_VALUE_BYTES`, 기본 1 MiB,
1..16777216)를 넘으면 기록하지 않는다(`NOT_RECORDED_PROVIDER_POLICY`, 보낸 것 없음). 그 값은 매번 source에서 읽힌다.
### 조건부 바꾸기
@@ -116,6 +116,30 @@ public final class SemanticRedisCall {
return call(operation, null, body, notSent, unknown, false);
}
/**
* Runs a call that decides for itself when its effect was handed to the driver, and must finish
* within {@code budget}.
*
* <p>The budget bounds the wait for a lane connection and every awaited reply together, as {@link
* #effectWithin} does. Without it each await gets the frame's timeout of its own and the lane
* wait comes on top, so a caller that called its timeout the ceiling of one operation waited the
* lane's acquire timeout plus one timeout per reply (round9 review 2026-11-12, RR9-01-17).
*
* @param operation the semantic operation, for the failure log
* @param budget the ceiling on the whole call
* @param body what to send, and how to read the reply; calls {@link Session#markSent()} before
* the command whose loss would leave the outcome unknown
* @param notSent the answer when nothing can have happened
* @param unknown the answer when the command may have run
* @param <R> the answer type
* @return the body's answer, or one of the two failure answers
*/
public <R> R runWithin(
String operation, Duration budget, Body<R> body, Supplier<R> notSent, Supplier<R> unknown) {
Objects.requireNonNull(budget, "budget must be non-null");
return call(operation, budget, body, notSent, unknown, false);
}
private <R> R call(
String operation,
Duration budget,
@@ -91,6 +91,20 @@ public final class RedisCacheRegionAdapter<K, V> implements CacheRegionPort<K, V
private final Duration negativeTtl;
private final Duration commandTimeout;
private final int maximumValueBytes;
/**
* The largest encoded value one entry carries by default: 1 MiB, the default the old {@code
* APP_CACHE_REDIS_MAXIMUM_VALUE_BYTES} had before that key stopped reaching anything.
*
* <p>Without a ceiling a single oversized value — a list that grew, a payload nobody sized — was
* written whole: one {@code SET} that holds the connection and the server's single thread while
* it is copied, then a key every reader fetches whole (round9 review 2026-11-12, RR9-01-18).
*/
public static final int DEFAULT_MAXIMUM_VALUE_BYTES = 1_048_576;
private final CacheScripts scripts = new CacheScripts();
private static final String ABSENT = "absent";
@@ -115,7 +129,8 @@ public final class RedisCacheRegionAdapter<K, V> implements CacheRegionPort<K, V
* @param softTtl how long an entry stays fresh
* @param hardTtl how long an entry stays usable
* @param negativeTtl how long an authoritative absence is cached
* @param commandTimeout the ceiling on one cache operation
* @param commandTimeout the ceiling on one cache operation, the lane wait and every reply
* together
*/
public RedisCacheRegionAdapter(
RedisRuntimeOwner owner,
@@ -128,6 +143,38 @@ public final class RedisCacheRegionAdapter<K, V> implements CacheRegionPort<K, V
Duration hardTtl,
Duration negativeTtl,
Duration commandTimeout) {
this(
owner,
keys,
keyDigest,
encoder,
decoder,
clock,
softTtl,
hardTtl,
negativeTtl,
commandTimeout,
DEFAULT_MAXIMUM_VALUE_BYTES);
}
/**
* Creates the adapter with its own ceiling on an encoded value.
*
* @param maximumValueBytes the largest encoded value one entry may carry; a larger one is not
* recorded ({@link CacheRecordOutcome#NOT_RECORDED_PROVIDER_POLICY})
*/
public RedisCacheRegionAdapter(
RedisRuntimeOwner owner,
CacheKeys keys,
Function<K, String> keyDigest,
Function<V, byte[]> encoder,
Function<byte[], V> decoder,
Clock clock,
Duration softTtl,
Duration hardTtl,
Duration negativeTtl,
Duration commandTimeout,
int maximumValueBytes) {
this.call =
new SemanticRedisCall(
owner,
@@ -144,6 +191,11 @@ public final class RedisCacheRegionAdapter<K, V> implements CacheRegionPort<K, V
this.softTtl = Objects.requireNonNull(softTtl, "soft TTL must be non-null");
this.hardTtl = Objects.requireNonNull(hardTtl, "hard TTL must be non-null");
this.negativeTtl = Objects.requireNonNull(negativeTtl, "negative TTL must be non-null");
this.commandTimeout = commandTimeout;
if (maximumValueBytes < 1) {
throw new IllegalStateException("the maximum value size must be at least one byte");
}
this.maximumValueBytes = maximumValueBytes;
if (softTtl.compareTo(hardTtl) > 0) {
throw new IllegalArgumentException(
"the soft TTL must not exceed the hard TTL: an entry cannot stop being fresh after it has"
@@ -158,8 +210,9 @@ public final class RedisCacheRegionAdapter<K, V> implements CacheRegionPort<K, V
// A read changed nothing, whether or not it reached the server: degraded, not broken. The
// caller
// loads from the source and carries on.
return call.run(
return call.runWithin(
"lookup",
commandTimeout,
session -> {
resolveGeneration(session, now);
byte[] stored =
@@ -233,7 +286,12 @@ public final class RedisCacheRegionAdapter<K, V> implements CacheRegionPort<K, V
@Override
public CacheRecordOutcome record(K key, V value, CacheRecordMetadata metadata) {
Objects.requireNonNull(value, "value must be non-null");
return write(key, encoder.apply(value), "", metadata, hardTtl);
byte[] payload = encoder.apply(value);
if (payload.length > maximumValueBytes) {
// Refused before any round trip; the caller serves the value it loaded and carries on.
return CacheRecordOutcome.NOT_RECORDED_PROVIDER_POLICY;
}
return write(key, payload, "", metadata, hardTtl);
}
@Override
@@ -268,8 +326,9 @@ public final class RedisCacheRegionAdapter<K, V> implements CacheRegionPort<K, V
// The write is marked sent only when the write command itself is handed to the driver. A
// failure before that point — lane refusal, connection, the generation read — wrote nothing.
return (scripted ? scriptCall : call)
.run(
.runWithin(
"record",
commandTimeout,
session -> {
resolveGeneration(session, now);
CacheEnvelope envelope =
@@ -343,8 +402,9 @@ public final class RedisCacheRegionAdapter<K, V> implements CacheRegionPort<K, V
@Override
public CacheInvalidationOutcome invalidate(K key) {
Objects.requireNonNull(key, "key must be non-null");
return call.run(
return call.runWithin(
"invalidate",
commandTimeout,
session -> {
byte[] physicalKey = keys.entryKey(keyDigest.apply(key));
// A marker, not a delete. A delete leaves the key absent, and a source load that read the
@@ -371,8 +431,9 @@ public final class RedisCacheRegionAdapter<K, V> implements CacheRegionPort<K, V
// a server that is answering everything else at the same time, and the SDK blocks KEYS for
// exactly that reason. Bumping the generation makes every older entry fail the generation
// check on read and expire on its own schedule.
return call.run(
return call.runWithin(
"invalidate-region",
commandTimeout,
session -> {
session.markSent();
Long generation = session.await(session.commands().incrementBy(keys.generationKey(), 1));
@@ -56,7 +56,8 @@ class RedisCacheConfig {
cache.getPositiveSoftTtl(),
cache.getPositiveHardTtl(),
cache.getNegativeTtl(),
cache.getCommandTimeout());
cache.getCommandTimeout(),
cache.getMaximumValueBytes());
}
/**
@@ -1,5 +1,6 @@
package dev.caskeleton.adapter.outbound.cache.redis.capability;
import dev.caskeleton.adapter.outbound.cache.redis.cache.RedisCacheRegionAdapter;
import dev.caskeleton.adapter.outbound.support.CapabilitySelectors;
import dev.caskeleton.adapter.outbound.support.CapabilityStartupValidationException;
import dev.caskeleton.adapter.outbound.support.StrictSettingsBinder;
@@ -87,6 +88,12 @@ public class RedisCapabilitySettings {
private Duration negativeTtl = Duration.ofSeconds(10);
private Duration minimumHardTtl = Duration.ofSeconds(1);
private int maximumScriptConnections = 4;
private int maximumValueBytes = RedisCacheRegionAdapter.DEFAULT_MAXIMUM_VALUE_BYTES;
/**
* The largest ceiling a deployment may set: sixteen mebibytes, as the env registry bounds it.
*/
static final int MAXIMUM_VALUE_BYTES_CEILING = 16_777_216;
/**
* Reports whether the default binding selected Redis.
@@ -124,6 +131,10 @@ public class RedisCapabilitySettings {
throw new CapabilityStartupValidationException(
"the cache script lane needs at least one connection");
}
if (maximumValueBytes < 1 || maximumValueBytes > MAXIMUM_VALUE_BYTES_CEILING) {
throw new CapabilityStartupValidationException(
"the cache value ceiling must be 1.." + MAXIMUM_VALUE_BYTES_CEILING + " bytes");
}
}
public Map<String, String> getBindings() {
@@ -214,6 +225,21 @@ public class RedisCapabilitySettings {
public void setMaximumScriptConnections(int maximumScriptConnections) {
this.maximumScriptConnections = maximumScriptConnections;
}
/**
* The largest value the region records, in encoded bytes. A larger one is not written and the
* read path falls back to the source, so one oversized entry cannot fill the instance or hold a
* connection for the whole transfer.
*
* @return the ceiling, 1 to 16 MiB
*/
public int getMaximumValueBytes() {
return maximumValueBytes;
}
public void setMaximumValueBytes(int maximumValueBytes) {
this.maximumValueBytes = maximumValueBytes;
}
}
/** Edge rate-limit provider composition. */
@@ -43,6 +43,9 @@ app:
# limiting and leases share (app.redis.capacity.maximum-script-connections). A cache under
# load runs out of these and degrades; it cannot take the correctness features' connections.
maximum-script-connections: ${APP_CACHE_REDIS_MAXIMUM_SCRIPT_CONNECTIONS:4}
# The largest value the region records, in encoded bytes (1..16777216). A larger one is not
# written; the read falls back to the source.
maximum-value-bytes: ${APP_CACHE_REDIS_MAXIMUM_VALUE_BYTES:1048576}
idempotency:
# Redis store tuning only; the provider selector and lifecycle policy stay with the
# application's idempotency executor.
@@ -1,12 +1,14 @@
package dev.caskeleton.adapter.outbound.cache.redis.cache;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.catchThrowable;
import dev.caskeleton.adapter.outbound.cache.redis.runtime.RedisDeploymentMode;
import dev.caskeleton.adapter.outbound.cache.redis.runtime.command.InMemorySemanticRedisCommands;
import dev.caskeleton.adapter.outbound.cache.redis.runtime.command.SemanticRedisCommands;
import dev.caskeleton.adapter.outbound.cache.redis.runtime.connection.RedisConnectionKind;
import dev.caskeleton.adapter.outbound.cache.redis.runtime.connection.RedisLane;
import dev.caskeleton.adapter.outbound.cache.redis.runtime.connection.RedisLease;
import dev.caskeleton.adapter.outbound.cache.redis.runtime.connection.RedisRuntimeClient;
import dev.caskeleton.adapter.outbound.cache.redis.runtime.connection.RedisRuntimeOwner;
import dev.caskeleton.adapter.outbound.cache.redis.runtime.connection.StubRedisRuntimeClient;
@@ -23,7 +25,10 @@ import java.time.Clock;
import java.time.Duration;
import java.time.Instant;
import java.time.ZoneOffset;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.Optional;
import java.util.concurrent.CompletableFuture;
import org.junit.jupiter.api.DisplayName;
import org.junit.jupiter.api.Test;
@@ -56,7 +61,7 @@ class RedisCacheRegionAdapterTest {
private RedisCacheRegionAdapter<String, String> cache(
Instant now, RedisRuntimeClient client, RedisCacheRegionAdapter.CacheKeys cacheKeys) {
Map<RedisLane, Integer> limits = new java.util.LinkedHashMap<>();
Map<RedisLane, Integer> limits = new LinkedHashMap<>();
for (RedisConnectionKind kind : RedisConnectionKind.values()) {
limits.put(kind, 4);
}
@@ -82,6 +87,119 @@ class RedisCacheRegionAdapterTest {
Duration.ofSeconds(2));
}
@Test
void aLookupOnAFullLaneGivesUpAtTheCommandTimeoutNotTheLaneWait() {
// given
// One connection, already borrowed; a borrow would wait five seconds for it. The cache's
// command timeout is 100 ms and is the ceiling on the whole operation, lane wait included.
Map<RedisLane, Integer> limits = new LinkedHashMap<>();
for (RedisConnectionKind kind : RedisConnectionKind.values()) {
limits.put(kind, 1);
}
limits.put(CacheScriptLane.CACHE_SCRIPT, 1);
RedisRuntimeOwner owner =
new RedisRuntimeOwner(
StubRedisRuntimeClient.serving(gateway),
limits,
Duration.ofSeconds(5),
Duration.ofSeconds(1));
RedisCacheRegionAdapter<String, String> cache =
new RedisCacheRegionAdapter<>(
owner,
keys,
key -> "digest-" + key,
value -> value.getBytes(StandardCharsets.UTF_8),
bytes -> new String(bytes, StandardCharsets.UTF_8),
Clock.fixed(T0, ZoneOffset.UTC),
Duration.ofSeconds(30),
Duration.ofSeconds(120),
Duration.ofSeconds(10),
Duration.ofMillis(100));
// when
long started = System.nanoTime();
CacheLookup<String> lookup;
RedisLease held = owner.borrow(RedisConnectionKind.REGULAR);
try {
lookup = cache.lookup("order-1");
} finally {
held.close();
}
Duration waited = Duration.ofNanos(System.nanoTime() - started);
// then
assertThat(
List.of(
lookup instanceof CacheLookup.Unavailable,
waited.compareTo(Duration.ofSeconds(3)) < 0))
.as("waited %s on a lane that would have held the caller five seconds", waited)
.containsExactly(true, true);
}
@Test
void aValueLargerThanTheCeilingIsNotRecorded() {
// given
RedisCacheRegionAdapter<String, String> cache = withCeiling(8);
// when
CacheRecordOutcome outcome = cache.record("order-1", "nine bytes", upsert());
// then
assertThat(outcome).isEqualTo(CacheRecordOutcome.NOT_RECORDED_PROVIDER_POLICY);
}
@Test
void aValueExactlyAtTheCeilingIsRecorded() {
// given
RedisCacheRegionAdapter<String, String> cache = withCeiling(8);
// when
CacheRecordOutcome outcome = cache.record("order-1", "eight by", upsert());
// then
assertThat(outcome).isEqualTo(CacheRecordOutcome.RECORDED);
}
private RedisCacheRegionAdapter<String, String> withCeiling(int maximumValueBytes) {
return new RedisCacheRegionAdapter<>(
new RedisRuntimeOwner(
StubRedisRuntimeClient.serving(gateway),
lanesOf(4),
Duration.ofSeconds(1),
Duration.ofSeconds(1)),
keys,
key -> "digest-" + key,
value -> value.getBytes(StandardCharsets.UTF_8),
bytes -> new String(bytes, StandardCharsets.UTF_8),
Clock.fixed(T0, ZoneOffset.UTC),
Duration.ofSeconds(30),
Duration.ofSeconds(120),
Duration.ofSeconds(10),
Duration.ofSeconds(2),
maximumValueBytes);
}
@Test
void theDefaultCeilingIsTheOneMebibyteTheRetiredSettingHad() {
// given
String justOver = "x".repeat(RedisCacheRegionAdapter.DEFAULT_MAXIMUM_VALUE_BYTES + 1);
// when
CacheRecordOutcome outcome = cache(T0).record("order-1", justOver, upsert());
// then
assertThat(outcome).isEqualTo(CacheRecordOutcome.NOT_RECORDED_PROVIDER_POLICY);
}
private static Map<RedisLane, Integer> lanesOf(int limit) {
Map<RedisLane, Integer> limits = new LinkedHashMap<>();
for (RedisConnectionKind kind : RedisConnectionKind.values()) {
limits.put(kind, limit);
}
limits.put(CacheScriptLane.CACHE_SCRIPT, limit);
return limits;
}
private static CacheRecordMetadata upsert() {
return new CacheRecordMetadata("rev-1", CacheRecordIntent.UPSERT);
}
@@ -428,8 +546,7 @@ class RedisCacheRegionAdapterTest {
// when
Throwable refused =
org.assertj.core.api.Assertions.catchThrowable(
() -> cache.record("order-1", "payload", newerThan("rev-2")));
catchThrowable(() -> cache.record("order-1", "payload", newerThan("rev-2")));
// then
assertThat(refused)
@@ -441,7 +558,7 @@ class RedisCacheRegionAdapterTest {
@DisplayName("멱등성·lease가 SCRIPT lane을 모두 잡고 있어도 캐시 조건부 기록은 캐시 전용 lane에서 기록된다")
void aConditionalCacheWriteDoesNotWaitForTheCorrectnessScriptLane() {
// given
Map<RedisLane, Integer> limits = new java.util.LinkedHashMap<>();
Map<RedisLane, Integer> limits = new LinkedHashMap<>();
limits.put(RedisConnectionKind.REGULAR, 4);
limits.put(RedisConnectionKind.SCRIPT, 1);
limits.put(CacheScriptLane.CACHE_SCRIPT, 1);
@@ -584,7 +701,7 @@ class RedisCacheRegionAdapterTest {
}
@Override
public RedisLaneConnection openLane(RedisLane kind, java.util.Optional<byte[]> routingKey) {
public RedisLaneConnection openLane(RedisLane kind, Optional<byte[]> routingKey) {
throw new IllegalStateException("the server is unreachable");
}
@@ -0,0 +1,40 @@
package dev.caskeleton.adapter.outbound.cache.redis.capability;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.catchThrowable;
import dev.caskeleton.adapter.outbound.cache.redis.cache.RedisCacheRegionAdapter;
import dev.caskeleton.adapter.outbound.support.CapabilityStartupValidationException;
import org.junit.jupiter.api.DisplayName;
import org.junit.jupiter.api.Test;
/**
* The cache value ceiling is a deployment's setting, {@code app.capabilities.cache
* .maximum-value-bytes} (round9 review 2026-11-12, RR9-01-18): the region refused larger values
* with no way for the shipped composition to change the bound.
*/
class RedisCacheValueCeilingSettingTest {
@Test
@DisplayName("설정하지 않으면 상한은 1 MiB다")
void anUnsetCeilingIsOneMebibyte() {
assertThat(new RedisCapabilitySettings().getCache().getMaximumValueBytes())
.isEqualTo(RedisCacheRegionAdapter.DEFAULT_MAXIMUM_VALUE_BYTES);
}
@Test
@DisplayName("16 MiB를 넘는 상한은 기동 때 거절한다")
void aCeilingPastSixteenMebibytesFailsStartup() {
// given
RedisCapabilitySettings.Cache cache = new RedisCapabilitySettings().getCache();
cache.setMaximumValueBytes(16_777_217);
// when
Throwable refused = catchThrowable(cache::validate);
// then
assertThat(refused)
.isInstanceOf(CapabilityStartupValidationException.class)
.hasMessageContaining("the cache value ceiling must be 1..16777216 bytes");
}
}
+2 -2
View File
@@ -37,8 +37,8 @@
- 주 classpath에 JSON Schema 엔진(NetworkNT)이 없다. 증거 manifest schema 검사와 capability card 검사
(`release/`)는 이 leaf의 시험(`evidence/`)이다. snakeyaml은 시험에서만 쓴다(소유자 결정 2026-09-25).
- 유스케이스 순서나 업무 라우팅 정책을 broker 어댑터 안에 숨기지 않는다.
- `OutboxMessagePublishAdapter`는 위임만 하고 의존성 로그를 내지 않는다. 확정된 FAILED/DEAD 전이가 정해진
ERROR 하나를 가진다.
- `OutboxMessagePublishAdapter`는 발행을 위임하고, publisher가 알린 `longestAttempt`까지만 기다린다(넘으면 결과 불명).
의존성 로그를 내지 않는다. 확정된 FAILED/DEAD 전이가 정해진 ERROR 하나를 가진다.
## 테스트
+11
View File
@@ -35,6 +35,17 @@ publish pipeline을 탄다. producer id는 host/pod/process 이름에서 추측
결과를 application 결과(CONFIRMED·AMBIGUOUS·REJECTED_BEFORE_SEND·REJECTED_AFTER_BROKER·
REJECTED_RETRYABLE)로 번역해 relay에 돌려준다. 다시 직렬화하지 않는다.
그 결과를 기다리는 시간은 `longestAttempt()`(relay가 batch와 lease를 맞춰 본 값)까지다. 넘으면 기다리기를 멈추고
`AMBIGUOUS`(`PUBLISH_WAIT_EXCEEDED`)로 돌려준다. broker에 닿았을 수 있어 확정도 거부도 아니다. 예전에는 상한 없이
`join()`해서, 답하지 않는 publisher가 relay thread를 claim의 lease와 종료 대기 시간 너머까지 붙잡았고 그 사이 다른
인스턴스가 같은 행을 다시 보냈다(round9 리뷰 2026-11-12, RR9-01-05). `longestAttempt()`를 모르는 publisher는 자기
deadline이 유일한 상한이다. 0이나 음수를 알리는 publisher는 결함이다. 그 행은 보내지 않고 `REJECTED_RETRYABLE`
(`PUBLISH_BOUND_NOT_POSITIVE`)로 돌려준다. relay는 시도로 세지 않고 backoff 뒤에 다시 보며, 실패 보고가 원인을 남긴다.
relay thread가 interrupt되면 둘로 갈린다. 기다리는 중이면 `AMBIGUOUS`(`PUBLISH_WAIT_INTERRUPTED`)이고 interrupt 표시를
남긴다. 발행하기 전에 이미 interrupt됐으면 아무것도 보내지 않고 `REJECTED_RETRYABLE`(`PUBLISH_INTERRUPTED_BEFORE_SEND`)로
돌려준다. relay는 그 행을 backoff 뒤로 미뤘다가 발행한다(실패 보고 한 줄).
platform `REJECTED` 중 `FailureDescriptor.retryable()`이 true인 것(종료 중 `SHUTTING_DOWN`, permit
대기 초과, `PUBLISH_RUNTIME_UNAVAILABLE`, `KAFKA_TRANSPORT_CLOSED`, Kafka 인증·인가 실패)은
`REJECTED_RETRYABLE`로 번역한다. relay는 이 값을 DEAD로 보내지 않고 재시도한다. 확정 거부 두 값으로
@@ -10,6 +10,11 @@ import dev.caskeleton.application.outbox.OutboxPublishRefusedException;
import java.time.Duration;
import java.util.Objects;
import java.util.Optional;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.CompletionException;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;
/**
* The relay's publish port: a claimed row's validated event, delegated unchanged to the canonical
@@ -19,9 +24,28 @@ import java.util.Optional;
* here re-renders it. The canonical publisher's attempt — its outcome and its failure code — is
* passed through unchanged, so the relay can tell a possibly-stored message (ambiguous) from a
* refused one, and a refused credential from a closed transport.
*
* <p>The wait for that attempt is bounded by {@link #longestAttempt()}, the same value the relay
* sized its batch against. It used to be an unbounded {@code join()}: a publisher whose stage never
* completed held the relay thread past its claim's lease — another instance then sent the row again
* — and past the shutdown wait the relay promises. A wait that runs out is {@link
* OutboxPublishOutcome#AMBIGUOUS}: the message may have reached the broker, so it is neither
* confirmed nor refused (round9 review 2026-11-12, RR9-01-05).
*/
public final class OutboxMessagePublishAdapter implements OutboxMessagePublishPort {
/** The code an attempt carries when this adapter stopped waiting for it. */
static final String WAIT_EXCEEDED = "PUBLISH_WAIT_EXCEEDED";
/** The code an attempt carries when the relay thread was interrupted while waiting. */
static final String WAIT_INTERRUPTED = "PUBLISH_WAIT_INTERRUPTED";
/** The code an attempt carries when the relay thread was interrupted before it published. */
static final String INTERRUPTED_BEFORE_SEND = "PUBLISH_INTERRUPTED_BEFORE_SEND";
/** The code an attempt carries when the publisher reported a bound of nothing. */
static final String BOUND_NOT_POSITIVE = "PUBLISH_BOUND_NOT_POSITIVE";
private final IntegrationEventPublishPort canonicalPublisher;
public OutboxMessagePublishAdapter(IntegrationEventPublishPort canonicalPublisher) {
@@ -41,7 +65,10 @@ public final class OutboxMessagePublishAdapter implements OutboxMessagePublishPo
throw new OutboxPublishRefusedException(outcome);
}
/** The canonical publisher's, unchanged: this adapter adds no wait of its own. */
/**
* The canonical publisher's, unchanged: this adapter adds no wait of its own, and waits no longer
* than this.
*/
@Override
public Optional<Duration> longestAttempt() {
return canonicalPublisher.longestAttempt();
@@ -50,6 +77,44 @@ public final class OutboxMessagePublishAdapter implements OutboxMessagePublishPo
@Override
public OutboxPublishAttempt publishForOutcome(ClaimedOutboxEvent event) {
Objects.requireNonNull(event, "event must not be null");
return canonicalPublisher.publish(event.event()).toCompletableFuture().join();
if (Thread.currentThread().isInterrupted()) {
// Asked to stop before this row was sent: send nothing. The relay defers the row and
// publishes it after its backoff; publishing and then reporting it ambiguous sent every
// remaining row of the batch and left each to be sent again.
return OutboxPublishAttempt.failed(
OutboxPublishOutcome.REJECTED_RETRYABLE, INTERRUPTED_BEFORE_SEND);
}
Optional<Duration> bound = canonicalPublisher.longestAttempt();
if (bound.isPresent() && (bound.get().isNegative() || bound.get().isZero())) {
// A bound of nothing would make every publish ambiguous before the broker could answer. It
// is the publisher's defect, not this row's: send nothing and let the relay retry the row
// after its backoff, with the failure report naming the cause. Thrown, it counted against
// the row's attempts and dead-lettered the whole outbox within minutes.
return OutboxPublishAttempt.failed(
OutboxPublishOutcome.REJECTED_RETRYABLE, BOUND_NOT_POSITIVE);
}
// The bound starts before publish() is called, not after it returns: a send can block the
// caller for up to its own deadline before handing back a stage (a Kafka producer waiting on
// metadata), and a wait measured from the return let one row take the bound twice over while
// the relay sized its lease for once.
long deadline = System.nanoTime() + bound.map(Duration::toNanos).orElse(0L);
CompletableFuture<OutboxPublishAttempt> attempt =
canonicalPublisher.publish(event.event()).toCompletableFuture();
if (bound.isEmpty()) {
// A publisher that cannot say how long it takes gave the relay nothing to size its batch
// against either; its own deadline is the only bound there is.
return attempt.join();
}
try {
return attempt.get(Math.max(0L, deadline - System.nanoTime()), TimeUnit.NANOSECONDS);
} catch (TimeoutException exceeded) {
return OutboxPublishAttempt.failed(OutboxPublishOutcome.AMBIGUOUS, WAIT_EXCEEDED);
} catch (InterruptedException interrupted) {
Thread.currentThread().interrupt();
return OutboxPublishAttempt.failed(OutboxPublishOutcome.AMBIGUOUS, WAIT_INTERRUPTED);
} catch (ExecutionException failed) {
// As join() reported it, so the relay's handling of a failed stage does not change.
throw new CompletionException(failed.getCause());
}
}
}
@@ -2,16 +2,24 @@ package dev.caskeleton.adapter.outbound.messaging.outbox;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.catchThrowable;
import static org.junit.jupiter.api.Assertions.assertTimeoutPreemptively;
import dev.caskeleton.application.messaging.event.testkit.BrokerAnswer;
import dev.caskeleton.application.messaging.event.testkit.FakeIntegrationEventPublisher;
import dev.caskeleton.application.outbox.ClaimedOutboxEvent;
import dev.caskeleton.application.outbox.OutboxMessagePublishPort;
import dev.caskeleton.application.outbox.OutboxPublishAmbiguousException;
import dev.caskeleton.application.outbox.OutboxPublishAttempt;
import dev.caskeleton.application.outbox.OutboxPublishOutcome;
import dev.caskeleton.application.outbox.OutboxPublishRefusedException;
import dev.caskeleton.application.outbox.testkit.OutboxMessagePublishContract;
import java.time.Duration;
import java.util.List;
import java.util.concurrent.CompletionException;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicBoolean;
import java.util.concurrent.atomic.AtomicReference;
import org.junit.jupiter.api.DisplayName;
import org.junit.jupiter.api.Test;
@@ -49,6 +57,148 @@ class OutboxMessagePublishAdapterTest extends OutboxMessagePublishContract {
assertThat(canonical.published()).containsExactly(claim.event());
}
@Test
@DisplayName("broker가 longestAttempt 안에 답하지 않으면 더 기다리지 않고 결과 불명으로 끝낸다")
void aPublishTheBrokerNeverAnswersEndsAmbiguousAtTheLongestAttempt() {
// given
OutboxMessagePublishPort adapter =
new OutboxMessagePublishAdapter(canonical.silentPast(Duration.ofMillis(50)));
// when
// A wait with no bound never returns here; ten seconds is far past the 50 ms the publisher
// promised and far short of a hung relay.
OutboxPublishAttempt attempt =
assertTimeoutPreemptively(
Duration.ofSeconds(10), () -> adapter.publishForOutcome(claimedRow()));
// then
assertThat(attempt)
.isEqualTo(
OutboxPublishAttempt.failed(
OutboxPublishOutcome.AMBIGUOUS, OutboxMessagePublishAdapter.WAIT_EXCEEDED));
}
@Test
@DisplayName("publish()가 호출자를 붙잡은 시간도 longestAttempt 안에 센다")
void theWaitIncludesTheTimePublishHeldTheCaller() {
// given
// Blocks 1 s before handing back a stage that never completes, with a 1.2 s promise: a wait
// measured from the return would end at about 2.2 s, one measured from the call at 1.2 s. The
// 2 s threshold sits between them, 0.8 s above the right answer, so a loaded runner has room.
OutboxMessagePublishPort adapter =
new OutboxMessagePublishAdapter(
canonical.silentPast(Duration.ofMillis(1_200)).blockingFor(Duration.ofSeconds(1)));
long started = System.nanoTime();
// when
var unused =
assertTimeoutPreemptively(
Duration.ofSeconds(10), () -> adapter.publishForOutcome(claimedRow()));
// then
assertThat(Duration.ofNanos(System.nanoTime() - started))
.as("one row used the bound twice over while the relay sized its lease for once")
.isLessThan(Duration.ofSeconds(2));
}
@Test
@DisplayName("기다리던 relay 스레드가 interrupt되면 결과 불명으로 끝내고 interrupt 표시를 남긴다")
void anInterruptedWaitEndsAmbiguousAndKeepsTheInterrupt() throws Exception {
// given
OutboxMessagePublishPort adapter =
new OutboxMessagePublishAdapter(canonical.silentPast(Duration.ofSeconds(30)));
AtomicReference<OutboxPublishAttempt> attempt = new AtomicReference<>();
AtomicBoolean stillInterrupted = new AtomicBoolean();
CountDownLatch waiting = new CountDownLatch(1);
Thread relay =
new Thread(
() -> {
waiting.countDown();
attempt.set(adapter.publishForOutcome(claimedRow()));
stillInterrupted.set(Thread.currentThread().isInterrupted());
});
relay.start();
assertThat(waiting.await(5, TimeUnit.SECONDS)).as("the relay thread started").isTrue();
// Until the publish has been handed out, an interrupt is the before-the-send case below.
while (canonical.published().isEmpty()) {
Thread.onSpinWait();
}
// when
relay.interrupt();
boolean ended = relay.join(Duration.ofSeconds(10));
// then
assertThat(List.of(ended, attempt.get(), stillInterrupted.get()))
.containsExactly(
true,
OutboxPublishAttempt.failed(
OutboxPublishOutcome.AMBIGUOUS, OutboxMessagePublishAdapter.WAIT_INTERRUPTED),
true);
}
@Test
@DisplayName("이미 interrupt된 relay 스레드는 발행하지 않고 다시 claim할 수 있는 거부로 답한다")
void anAlreadyInterruptedRelayPublishesNothing() {
// given
OutboxMessagePublishPort adapter =
new OutboxMessagePublishAdapter(canonical.silentPast(Duration.ofSeconds(30)));
Thread.currentThread().interrupt();
// when
OutboxPublishAttempt attempt;
try {
attempt = adapter.publishForOutcome(claimedRow());
} finally {
Thread.interrupted();
}
// then
assertThat(List.of(attempt, canonical.published()))
.as("a relay asked to stop sent the rest of its batch and left each row to be sent again")
.containsExactly(
OutboxPublishAttempt.failed(
OutboxPublishOutcome.REJECTED_RETRYABLE,
OutboxMessagePublishAdapter.INTERRUPTED_BEFORE_SEND),
List.of());
}
@Test
@DisplayName("longestAttempt가 0인 publisher에는 보내지 않고, 시도로 세지 않는 거부로 답한다")
void aPublisherReportingAZeroBoundIsSentNothing() {
// given
OutboxMessagePublishPort adapter =
new OutboxMessagePublishAdapter(canonical.silentPast(Duration.ZERO));
// when
OutboxPublishAttempt attempt = adapter.publishForOutcome(claimedRow());
// then
assertThat(List.of(attempt, canonical.published()))
.as("thrown, the defect counted against every row's attempts and dead-lettered the outbox")
.containsExactly(
OutboxPublishAttempt.failed(
OutboxPublishOutcome.REJECTED_RETRYABLE,
OutboxMessagePublishAdapter.BOUND_NOT_POSITIVE),
List.of());
}
@Test
@DisplayName("실패로 끝난 발행 stage는 join()과 같은 CompletionException으로 알린다")
void aFailedStageIsReportedAsJoinReportedIt() {
// given
IllegalStateException broker = new IllegalStateException("broker refused the connection");
OutboxMessagePublishPort adapter =
new OutboxMessagePublishAdapter(
canonical.silentPast(Duration.ofSeconds(5)).failingWith(broker));
// when
Throwable failure = catchThrowable(() -> adapter.publishForOutcome(claimedRow()));
// then
assertThat(failure).isInstanceOf(CompletionException.class).hasCause(broker);
}
@Test
@DisplayName("확인을 받지 못한 발행은 예외로 알리는 메서드에서 AMBIGUOUS 신호를 던진다")
void anAmbiguousPublishThrowsTheAmbiguousSignalFromTheThrowingMethod() {
@@ -5,6 +5,7 @@ import dev.caskeleton.adapter.outbound.notification.platform.dispatch.ProviderRu
import dev.caskeleton.application.notification.platform.api.ProviderProfileId;
import dev.caskeleton.application.notification.platform.api.routing.Channel;
import dev.caskeleton.application.notification.platform.provider.ReconciliationCapability;
import java.time.Duration;
import java.util.ArrayList;
import java.util.EnumMap;
import java.util.HashMap;
@@ -112,6 +113,10 @@ public final class NotificationProviderAssembly {
Map<ProviderProfileId, ReconciliationCapability> reconciliations = new LinkedHashMap<>();
Map<Channel, List<String>> claimants = new EnumMap<>(Channel.class);
if (mode != NotificationPlatformMode.INGEST_ONLY) {
requireFamiliesTogether(configured, settings.dispatch().leaseDuration());
}
for (Map.Entry<String, NotificationPlatformSettings.Provider> entry : configured.entrySet()) {
String profileId = entry.getKey();
NotificationPlatformSettings.Provider profile = entry.getValue();
@@ -198,6 +203,29 @@ public final class NotificationProviderAssembly {
}
}
/** Hands each family its enabled profiles at once, for the limits only the set can break. */
private void requireFamiliesTogether(
Map<String, NotificationPlatformSettings.Provider> configured, Duration deliveryLease) {
Map<ProviderType, Map<String, NotificationPlatformSettings.Provider>> byFamily =
new EnumMap<>(ProviderType.class);
configured.forEach(
(profileId, profile) -> {
if (profile.enabled()) {
byFamily
.computeIfAbsent(
ProviderType.parse(profileId, profile.type()), family -> new TreeMap<>())
.put(profileId, profile);
}
});
byFamily.forEach(
(family, profiles) -> {
ProviderRuntimeAssembler assembler = assemblers.get(family);
if (assembler != null) {
assembler.requireTogether(Map.copyOf(profiles), deliveryLease);
}
});
}
private static String missingAssemblerMessage(String profileId, ProviderType type) {
if (type == ProviderType.SMTP) {
return "notification provider profile '"
@@ -103,24 +103,38 @@ public class NotificationSmtpProviderConfig {
return new SmtpFailureClassifier();
}
/** How many sends may wait per send thread before the pool refuses one. */
static final int QUEUED_SENDS_PER_THREAD = 4;
/**
* Where an SMTP send runs.
*
* <p>Bounded, and it aborts rather than queues without limit: an SMTP relay that has stopped
* answering turns an unbounded queue into retained heap and a dispatch worker that never notices.
* {@code CallerRunsPolicy} would instead block the dispatch loop on the slow relay, which stalls
* every other channel too.
* every other channel too. A rejected send never reached the relay, and the adapter reports it so
* ({@code ProviderCallNotStartedException}), not as an ambiguous submission.
*
* <p>{@code dispatch-concurrency} threads from the first send, idle ones timing out. The core
* size used to be one: a {@code ThreadPoolExecutor} adds a thread past its core only when the
* queue is full, and the profile's limiter admits fewer sends than the queue holds, so every SMTP
* send ran on one thread whatever {@code dispatch-concurrency} said (round9 review 2026-11-12,
* RR9-01-15).
*
* <p>A deployment's own executor must not run a send on the submitting thread, as {@code
* CallerRunsPolicy} or a direct executor does: the send's two timeout deadlines complete its
* future, and a dispatch thread blocked inside {@code execute} never gets to read it.
*/
@Bean(name = "notificationSmtpExecutor", destroyMethod = "shutdown")
@ConditionalOnMissingBean(name = "notificationSmtpExecutor")
public Executor notificationSmtpExecutor(NotificationSmtpSettings settings) {
ThreadPoolExecutor executor =
new ThreadPoolExecutor(
1,
settings.dispatchConcurrency(),
settings.dispatchConcurrency(),
60L,
TimeUnit.SECONDS,
new LinkedBlockingQueue<>(settings.dispatchConcurrency() * 4),
new LinkedBlockingQueue<>(settings.dispatchConcurrency() * QUEUED_SENDS_PER_THREAD),
runnable -> {
Thread thread = new Thread(runnable, "notification-smtp");
thread.setDaemon(true);
@@ -1,5 +1,8 @@
package dev.caskeleton.adapter.outbound.notification.platform.autoconfigure;
import java.time.Duration;
import java.util.Map;
/**
* Turns one configured profile into one working provider.
*
@@ -32,4 +35,19 @@ public interface ProviderRuntimeAssembler {
* is missing
*/
AssembledProvider assemble(String profileId, NotificationPlatformSettings.Provider profile);
/**
* Refuses this family's enabled profiles together, before any of them is assembled.
*
* <p>Some limits are shared by every profile of a family — one send pool — and can only be
* checked against all of them at once; some depend on the platform's delivery lease, which no
* single profile carries. Called once per boot with the whole set, so it holds no state between
* calls. The default accepts.
*
* @param profiles the family's enabled profiles, by id
* @param deliveryLease {@code app.notification.platform.dispatch.lease-duration}
* @throws IllegalStateException naming the profile and the limit it breaks
*/
default void requireTogether(
Map<String, NotificationPlatformSettings.Provider> profiles, Duration deliveryLease) {}
}
@@ -15,9 +15,12 @@ import dev.caskeleton.application.notification.platform.provider.ProviderCapabil
import dev.caskeleton.application.notification.platform.provider.ProviderProfileSnapshot;
import dev.caskeleton.application.notification.platform.security.ContactPointProtector;
import java.time.Clock;
import java.time.Duration;
import java.util.Map;
import java.util.Objects;
import java.util.TreeMap;
import java.util.concurrent.Executor;
import java.util.concurrent.ThreadPoolExecutor;
/**
* The first production assembler: one configured SMTP profile becomes one working provider
@@ -98,10 +101,18 @@ public final class SmtpProviderRuntimeAssembler implements ProviderRuntimeAssemb
String profileId, NotificationPlatformSettings.Provider profile) {
Objects.requireNonNull(profileId, "profileId");
Objects.requireNonNull(profile, "profile");
requireTimeoutOutlastsOneStalledStep(profileId, profile.timeout());
SmtpNotificationProviderAdapter adapter =
new SmtpNotificationProviderAdapter(
dispatch, mimeFactory, classifier, protector, transport, executor, attachmentGuard);
dispatch,
mimeFactory,
classifier,
protector,
transport,
executor,
attachmentGuard,
profile.timeout());
ProviderRuntime runtime =
new ProviderRuntime(
@@ -112,6 +123,94 @@ public final class SmtpProviderRuntimeAssembler implements ProviderRuntimeAssemb
return AssembledProvider.dispatchOnly(runtime, Channel.EMAIL);
}
/**
* Refuses a profile timeout that a single stalled connect and command can outlast.
*
* <p>The profile timeout bounds the whole send and reports what outlasts it as ambiguous, since
* the DATA may have gone through. A relay that accepts the connection and never answers the
* greeting has been sent nothing; its socket deadline reports that, as a failure the classifier
* knows is safe to retry. With the profile timeout shorter than the connect deadline plus one
* command's, the profile timeout fired first and turned that safe retry into an ambiguous attempt
* an operator reconciles by hand (round9 review 2026-11-12, RR9-01-15).
*/
private void requireTimeoutOutlastsOneStalledStep(String profileId, Duration timeout) {
Duration oneCommand =
transport.readTimeout().compareTo(transport.writeTimeout()) >= 0
? transport.readTimeout()
: transport.writeTimeout();
Duration stalledStep = transport.connectTimeout().plus(oneCommand);
if (timeout.compareTo(stalledStep) <= 0) {
throw new IllegalStateException(
"notification provider profile '"
+ profileId
+ "': timeout ("
+ timeout
+ ") must be longer than mail.smtp.connectiontimeout plus the longer of"
+ " mail.smtp.timeout and mail.smtp.writetimeout ("
+ stalledStep
+ "), or a relay that never answers is reported as an ambiguous send");
}
}
/**
* Refuses SMTP profiles whose in-flight limits together are more than the send pool can hold, and
* a delivery lease one SMTP send can outlast.
*
* <p>The limiter lets {@code max-concurrency} sends through per profile; every SMTP profile
* shares the one pool, which runs its threads, queues the rest up to its queue's capacity, and
* rejects what is left. Profiles allowed more in flight together than threads plus queue had
* their excess rejected on every burst. Only a {@link ThreadPoolExecutor} says how much it holds,
* so a deployment's own executor is not checked (round9 review 2026-11-12, RR9-01-16).
*
* <p>One send may wait in the pool for up to the profile's timeout and then run for up to the
* timeout again, so it holds its delivery's lease for up to twice the timeout. A lease shorter
* than that shows a running send as a stuck lease; if the send is still running one lease later,
* lease recovery takes the delivery for abandoned and, since an SMTP relay cannot be asked, parks
* it for an operator. A send that finishes after that still records its result.
*/
@Override
public void requireTogether(
Map<String, NotificationPlatformSettings.Provider> profiles, Duration deliveryLease) {
new TreeMap<>(profiles)
.forEach(
(profileId, profile) -> {
Duration oneSend = profile.timeout().multipliedBy(2);
if (deliveryLease.compareTo(oneSend) <= 0) {
throw new IllegalStateException(
"notification provider profile '"
+ profileId
+ "': app.notification.platform.dispatch.lease-duration ("
+ deliveryLease
+ ") must be longer than twice the SMTP timeout ("
+ oneSend
+ "): a send may wait in the pool for the timeout and then run for it");
}
});
if (!(executor instanceof ThreadPoolExecutor pool)) {
return;
}
int threads = pool.getMaximumPoolSize();
int queued = pool.getQueue().remainingCapacity();
long holds = (long) threads + queued;
long together =
profiles.values().stream()
.mapToLong(NotificationPlatformSettings.Provider::maxConcurrency)
.sum();
if (together > holds) {
throw new IllegalStateException(
"notification provider profiles "
+ new TreeMap<>(profiles).keySet()
+ ": max-concurrency ("
+ together
+ " together) is more than the SMTP send pool holds ("
+ threads
+ " threads and "
+ queued
+ " queued) — raise app.notification.platform.smtp.dispatch-concurrency or lower"
+ " max-concurrency");
}
}
/**
* The snapshot the dispatch loop plans against. Its capabilities are the adapter's own, as the
* SES assembler does: a second copy here had drifted to a different payload limit.
@@ -1,13 +1,20 @@
package dev.caskeleton.adapter.outbound.notification.platform.provider.smtp;
import static java.util.concurrent.TimeUnit.MILLISECONDS;
import dev.caskeleton.adapter.outbound.notification.platform.provider.EmailAttachments;
import dev.caskeleton.adapter.outbound.notification.platform.provider.EmailRecipients;
import dev.caskeleton.application.notification.platform.api.ProviderId;
import dev.caskeleton.application.notification.platform.api.error.FailureCategory;
import dev.caskeleton.application.notification.platform.api.error.NotificationFailureCode;
import dev.caskeleton.application.notification.platform.api.routing.Channel;
import dev.caskeleton.application.notification.platform.contact.EmailAddress;
import dev.caskeleton.application.notification.platform.dispatch.AttachmentIntegrityGuard;
import dev.caskeleton.application.notification.platform.provider.NotificationProviderAdapter;
import dev.caskeleton.application.notification.platform.provider.ProviderCallNotStartedException;
import dev.caskeleton.application.notification.platform.provider.ProviderCapabilities;
import dev.caskeleton.application.notification.platform.provider.ProviderExecutionEvidence;
import dev.caskeleton.application.notification.platform.provider.ProviderFailure;
import dev.caskeleton.application.notification.platform.provider.ProviderSubmission;
import dev.caskeleton.application.notification.platform.provider.ProviderSubmissionResult;
import dev.caskeleton.application.notification.platform.provider.ResolvedAttachment;
@@ -15,10 +22,13 @@ import dev.caskeleton.application.notification.platform.security.ContactPointPro
import java.time.Duration;
import java.util.List;
import java.util.Objects;
import java.util.Optional;
import java.util.Set;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.CompletionStage;
import java.util.concurrent.Executor;
import java.util.concurrent.RejectedExecutionException;
import java.util.concurrent.atomic.AtomicReference;
/**
* SMTP email adapter.
@@ -39,7 +49,14 @@ public final class SmtpNotificationProviderAdapter implements NotificationProvid
private final SmtpProviderProperties properties;
private final Executor executor;
private final AttachmentIntegrityGuard attachmentGuard;
private final Duration callTimeout;
/**
* @param callTimeout the profile's {@code timeout}: the longest one send may wait in the pool,
* and then run, before it is reported as not started or as ambiguous. Startup holds it above
* one stalled connect and command
* @throws IllegalStateException if the timeout is not positive, which is the assembly's fault
*/
public SmtpNotificationProviderAdapter(
SmtpDispatch dispatch,
SmtpMimeMessageFactory mimeFactory,
@@ -47,7 +64,8 @@ public final class SmtpNotificationProviderAdapter implements NotificationProvid
ContactPointProtector protector,
SmtpProviderProperties properties,
Executor executor,
AttachmentIntegrityGuard attachmentGuard) {
AttachmentIntegrityGuard attachmentGuard,
Duration callTimeout) {
this.dispatch = Objects.requireNonNull(dispatch, "dispatch");
this.mimeFactory = Objects.requireNonNull(mimeFactory, "mimeFactory");
this.classifier = Objects.requireNonNull(classifier, "classifier");
@@ -55,6 +73,10 @@ public final class SmtpNotificationProviderAdapter implements NotificationProvid
this.properties = Objects.requireNonNull(properties, "properties");
this.executor = Objects.requireNonNull(executor, "executor");
this.attachmentGuard = Objects.requireNonNull(attachmentGuard, "attachmentGuard");
this.callTimeout = Objects.requireNonNull(callTimeout, "callTimeout");
if (callTimeout.isNegative() || callTimeout.isZero()) {
throw new IllegalStateException("callTimeout must be positive");
}
}
@Override
@@ -77,7 +99,109 @@ public final class SmtpNotificationProviderAdapter implements NotificationProvid
@Override
public CompletionStage<ProviderSubmissionResult> submit(ProviderSubmission submission) {
Objects.requireNonNull(submission, "submission");
return CompletableFuture.supplyAsync(() -> send(submission), executor);
PooledSend send = new PooledSend(submission);
try {
executor.execute(send);
} catch (RejectedExecutionException full) {
// The send pool refused the task: nothing was handed to the relay. Reported as not started,
// so the delivery is retried, rather than falling through to the ambiguous residue an
// operator has to reconcile by hand (round9 review 2026-11-12, RR9-01-16).
throw new ProviderCallNotStartedException(
NotificationFailureCode.PROVIDER_UNAVAILABLE,
FailureCategory.CAPACITY_REJECTED,
true,
"the SMTP send pool is full; the message was not handed to the relay");
}
send.armQueueDeadline();
return send.result();
}
/**
* One send through the pool, bounded by the profile's timeout twice over.
*
* <p>Waiting in the pool: a send that has not started when the timeout passes is abandoned and
* never runs. Nothing reached the relay, so it is reported as not started and retried; the old
* single timer, started at the submit, closed it as ambiguous and the send then went out anyway,
* after its permit was released.
*
* <p>Running: from the moment a pool thread takes it, the send has the whole timeout. The socket
* timeouts bound each SMTP command, not the conversation, so a relay answering slowly but within
* each one held the send past the delivery lease: it showed as a stuck lease, and one lease later
* lease recovery parked the delivery for an operator while the send still ran. Past the bound the
* outcome is ambiguous — the DATA may have gone through — and is never retried automatically; the
* send keeps its thread until the socket gives up. Startup holds the timeout above one stalled
* connect and command, so a relay that never answers fails on its socket deadline first, as a
* classified failure, rather than on this one.
*/
private final class PooledSend implements Runnable {
private final ProviderSubmission submission;
private final AtomicReference<SendState> state = new AtomicReference<>(SendState.QUEUED);
private final CompletableFuture<ProviderSubmissionResult> result = new CompletableFuture<>();
/**
* Completes when a pool thread takes the send, or when the timeout passes first.
*
* <p>Both deadlines are {@code completeOnTimeout}: the JDK's delay thread completes the future
* itself, so nothing waits behind the common pool, and completing the future early cancels the
* timer, so a send that finishes in time leaves no timer holding it.
*/
private final CompletableFuture<Void> started = new CompletableFuture<>();
private PooledSend(ProviderSubmission submission) {
this.submission = submission;
}
/**
* Starts the queue deadline, once the pool has taken the send: a send the pool refused needs
* none, and one the pool already ran has completed {@link #started}, so no timer is made.
*/
private void armQueueDeadline() {
// Nothing to observe on the stage it returns: the abandon reports through the result.
var unused =
started
.completeOnTimeout(null, callTimeout.toMillis(), MILLISECONDS)
.thenRun(this::abandonIfStillQueued);
}
private CompletableFuture<ProviderSubmissionResult> result() {
return result;
}
private void abandonIfStillQueued() {
if (state.compareAndSet(SendState.QUEUED, SendState.ABANDONED)) {
result.completeExceptionally(
new ProviderCallNotStartedException(
NotificationFailureCode.PROVIDER_UNAVAILABLE,
FailureCategory.CAPACITY_REJECTED,
true,
"the SMTP send waited in the pool past the profile timeout of "
+ callTimeout
+ "; the message was not handed to the relay"));
}
}
@Override
public void run() {
if (!state.compareAndSet(SendState.QUEUED, SendState.RUNNING)) {
return;
}
// Cancels the queue deadline; its abandon then finds the send running and does nothing.
started.complete(null);
result.completeOnTimeout(callOutlastedItsTimeout(), callTimeout.toMillis(), MILLISECONDS);
try {
result.complete(send(submission));
} catch (RuntimeException failure) {
result.completeExceptionally(failure);
}
}
}
/** Where a pooled send is: only a queued one may still be abandoned. */
private enum SendState {
QUEUED,
RUNNING,
ABANDONED
}
private ProviderSubmissionResult send(ProviderSubmission submission) {
@@ -100,6 +224,18 @@ public final class SmtpNotificationProviderAdapter implements NotificationProvid
}
}
private ProviderSubmissionResult callOutlastedItsTimeout() {
return ProviderSubmissionResult.ambiguous(
new ProviderFailure(
NotificationFailureCode.PROVIDER_RESPONSE_LOST,
FailureCategory.AMBIGUOUS_SUBMISSION,
false,
Optional.empty(),
Optional.of("the SMTP send outlasted the profile timeout of " + callTimeout)),
ProviderExecutionEvidence.undetermined(),
callTimeout);
}
private static Duration elapsedSince(long startedNanos) {
return Duration.ofNanos(System.nanoTime() - startedNanos);
}
@@ -2,6 +2,7 @@ package dev.caskeleton.adapter.outbound.notification.platform.autoconfigure;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatThrownBy;
import static org.assertj.core.api.Assertions.catchThrowable;
import dev.caskeleton.adapter.outbound.notification.platform.callback.CallbackVerificationResult;
import dev.caskeleton.adapter.outbound.notification.platform.callback.ProviderCallbackAdapter;
@@ -45,6 +46,9 @@ import java.util.Optional;
import java.util.Set;
import java.util.UUID;
import java.util.concurrent.CompletionStage;
import java.util.concurrent.LinkedBlockingQueue;
import java.util.concurrent.ThreadPoolExecutor;
import java.util.concurrent.TimeUnit;
import org.junit.jupiter.api.DisplayName;
import org.junit.jupiter.api.Test;
@@ -199,6 +203,120 @@ class NotificationProviderAssemblyTest {
.hasMessageContaining("INGEST_ONLY");
}
@Test
@DisplayName("SERVING 조립은 SMTP timeout 20초의 두 배(40초)에 못 미치는 lease 30초를 거절한다")
void aServingAssemblyRefusesALeaseOneSmtpSendCanOutlast() {
// given
// The settings fallback, Dispatch.defaults(), leases for 30 s (the module defaults ship 2 m):
// longer than the timeout, so the settings accept it, and shorter than twice the timeout, which
// only the SMTP family check refuses.
Map<String, NotificationPlatformSettings.Provider> providers =
Map.of("primary-email", smtpProfile(true, Duration.ofSeconds(20), 4));
// when
Throwable failure =
catchThrowable(
() -> assemble(providers, SmtpProviderRuntimeAssemblerTest.assembler(Runnable::run)));
// then
assertThat(failure)
.isInstanceOf(IllegalStateException.class)
.hasMessageContaining(
"lease-duration (PT30S) must be longer than twice the SMTP timeout (PT40S)");
}
@Test
void anIngestOnlyAssemblyDoesNotHoldSmtpToTheDispatchLease() {
// given
Map<String, NotificationPlatformSettings.Provider> providers =
Map.of("primary-email", smtpProfile(true, Duration.ofSeconds(20), 4));
// when
NotificationProviderAssembly.AssembledPlatform platform =
assemble(
providers,
NotificationPlatformMode.INGEST_ONLY,
SmtpProviderRuntimeAssemblerTest.assembler(Runnable::run));
// then
assertThat(platform.routeFor(Channel.EMAIL))
.as("an ingest-only deployment sends nothing, so no send can outlast its lease")
.contains(new ProviderProfileId("primary-email"));
}
@Test
@DisplayName(
"SERVING 조립은 켜진 SMTP profile의 max-concurrency 합(3+3)이 pool(thread 1, 대기열 4)을 넘으면 거절한다")
void aServingAssemblyRefusesSmtpProfilesThatTogetherOverfillThePool() {
// given
Map<String, NotificationPlatformSettings.Provider> providers =
Map.of(
"primary-email", smtpProfile(true, Duration.ofSeconds(10), 3),
"secondary-email", smtpSecondary(true, 3));
// when
Throwable failure = catchThrowable(() -> assembleOnASmallPool(providers));
// then
assertThat(failure)
.isInstanceOf(IllegalStateException.class)
.hasMessageContaining("max-concurrency (6 together) is more than the SMTP send pool holds");
}
@Test
@DisplayName("꺼진 SMTP profile의 max-concurrency는 pool(thread 1, 대기열 4) 합에 들지 않는다")
void aDisabledSmtpProfileIsLeftOutOfThePoolSum() {
// given
Map<String, NotificationPlatformSettings.Provider> providers =
Map.of(
"primary-email", smtpProfile(true, Duration.ofSeconds(10), 3),
"secondary-email", smtpSecondary(false, 3));
// when
NotificationProviderAssembly.AssembledPlatform platform = assembleOnASmallPool(providers);
// then
assertThat(
List.of(
platform.runtimes().find(new ProviderProfileId("primary-email")).isPresent(),
platform.runtimes().find(new ProviderProfileId("secondary-email")).isPresent()))
.as("the enabled profile is assembled; the disabled one is neither assembled nor counted")
.containsExactly(true, false);
}
/** Assembles {@code providers} over one SMTP send thread with four queued. */
private static NotificationProviderAssembly.AssembledPlatform assembleOnASmallPool(
Map<String, NotificationPlatformSettings.Provider> providers) {
ThreadPoolExecutor pool =
new ThreadPoolExecutor(1, 1, 60, TimeUnit.SECONDS, new LinkedBlockingQueue<>(4));
try {
return assemble(providers, SmtpProviderRuntimeAssemblerTest.assembler(pool));
} finally {
pool.shutdownNow();
}
}
private static NotificationPlatformSettings.Provider smtpProfile(
boolean enabled, Duration timeout, int maxConcurrency) {
return new NotificationPlatformSettings.Provider(
"SMTP", enabled, true, "local", "smtp-local", timeout, maxConcurrency, 8, null);
}
/** A second SMTP profile on the same channel, not its primary. */
private static NotificationPlatformSettings.Provider smtpSecondary(
boolean enabled, int maxConcurrency) {
return new NotificationPlatformSettings.Provider(
"SMTP",
enabled,
false,
"local",
"smtp-local",
Duration.ofSeconds(10),
maxConcurrency,
8,
null);
}
@Test
@DisplayName("INGEST_ONLY is a declaration, and it reports non-serving")
void ingestOnlyIsADeclaration() {
@@ -0,0 +1,58 @@
package dev.caskeleton.adapter.outbound.notification.platform.autoconfigure;
import static org.assertj.core.api.Assertions.assertThat;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.Executor;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.TimeUnit;
import org.junit.jupiter.api.Test;
/**
* The SMTP send pool runs {@code dispatch-concurrency} sends at once.
*
* <p>Its core size was one, and a {@code ThreadPoolExecutor} adds a thread past its core only when
* its queue is full. The profile's limiter admits fewer sends than the queue holds, so the queue
* never filled and every send waited for the one thread (round9 review 2026-11-12, RR9-01-15).
*/
class NotificationSmtpExecutorTest {
@Test
void asManySendsAsTheDispatchConcurrencyRunAtOnce() throws InterruptedException {
// given
Executor executor =
new NotificationSmtpProviderConfig()
.notificationSmtpExecutor(new NotificationSmtpSettings("no-reply@example.test", 3));
CountDownLatch allRunning = new CountDownLatch(3);
CountDownLatch release = new CountDownLatch(1);
// when
boolean concurrent;
try {
for (int send = 0; send < 3; send++) {
executor.execute(
() -> {
allRunning.countDown();
awaitQuietly(release);
});
}
concurrent = allRunning.await(10, TimeUnit.SECONDS);
} finally {
release.countDown();
((ExecutorService) executor).shutdownNow();
}
// then
assertThat(concurrent).as("three sends in flight on three threads").isTrue();
}
private static void awaitQuietly(CountDownLatch latch) {
try {
if (!latch.await(30, TimeUnit.SECONDS)) {
throw new IllegalStateException("the test never released the send");
}
} catch (InterruptedException interrupted) {
Thread.currentThread().interrupt();
}
}
}
@@ -1,6 +1,7 @@
package dev.caskeleton.adapter.outbound.notification.platform.autoconfigure;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.catchThrowable;
import dev.caskeleton.adapter.outbound.notification.platform.provider.smtp.SmtpFailureClassifier;
import dev.caskeleton.adapter.outbound.notification.platform.provider.smtp.SmtpMimeMessageFactory;
@@ -10,7 +11,12 @@ import dev.caskeleton.application.notification.platform.dispatch.AttachmentInteg
import jakarta.mail.Session;
import java.time.Clock;
import java.time.Duration;
import java.util.Map;
import java.util.Properties;
import java.util.concurrent.Executor;
import java.util.concurrent.LinkedBlockingQueue;
import java.util.concurrent.ThreadPoolExecutor;
import java.util.concurrent.TimeUnit;
import org.junit.jupiter.api.DisplayName;
import org.junit.jupiter.api.Test;
@@ -25,6 +31,8 @@ import org.junit.jupiter.api.Test;
*/
class SmtpProviderRuntimeAssemblerTest {
private static final Duration LEASE = Duration.ofMinutes(2);
@Test
@DisplayName("the assembler serves the SMTP family on the email channel")
void theAssemblerServesSmtpOnEmail() {
@@ -81,7 +89,105 @@ class SmtpProviderRuntimeAssemblerTest {
.isEqualTo(1L);
}
@Test
void aProfileAllowedMoreInFlightThanTheSendPoolHoldsFailsAssembly() {
// given
// One thread and four queued: five sends fit, the profile would let eight through.
ThreadPoolExecutor pool =
new ThreadPoolExecutor(1, 1, 60, TimeUnit.SECONDS, new LinkedBlockingQueue<>(4));
// when
Throwable failure;
try {
failure =
catchThrowable(
() -> assembler(pool).requireTogether(Map.of("primary-email", profile(8)), LEASE));
} finally {
pool.shutdownNow();
}
// then
assertThat(failure)
.isInstanceOf(IllegalStateException.class)
.hasMessageContaining(
"max-concurrency (8 together) is more than the SMTP send pool holds (1 threads and 4");
}
@Test
@DisplayName("SMTP profile 둘이 합쳐 send pool이 담는 것보다 많이 내보내면 조립이 실패한다")
void twoProfilesThatTogetherOverfillTheSendPoolFailAssembly() {
// given
// One thread and four queued: five sends fit; the two profiles would let six through.
ThreadPoolExecutor pool =
new ThreadPoolExecutor(1, 1, 60, TimeUnit.SECONDS, new LinkedBlockingQueue<>(4));
SmtpProviderRuntimeAssembler assembler = assembler(pool);
// when
Throwable failure;
try {
failure =
catchThrowable(
() ->
assembler.requireTogether(
Map.of("primary-email", profile(3), "secondary-email", profile(3)), LEASE));
} finally {
pool.shutdownNow();
}
// then
assertThat(failure)
.isInstanceOf(IllegalStateException.class)
.hasMessageContaining(
"[primary-email, secondary-email]: max-concurrency (6 together) is more than the SMTP"
+ " send pool holds");
}
@Test
@DisplayName("SMTP timeout 10초의 두 배인 20초를 넘지 않는 delivery lease는 조립이 실패한다")
void aDeliveryLeaseOneSendCanOutlastFailsAssembly() {
// when
Throwable failure =
catchThrowable(
() ->
assembler()
.requireTogether(Map.of("primary-email", profile(4)), Duration.ofSeconds(20)));
// then
assertThat(failure)
.as("a send may wait in the pool for the timeout and then run for it")
.isInstanceOf(IllegalStateException.class)
.hasMessageContaining("lease-duration (PT20S) must be longer than twice the SMTP timeout");
}
@Test
@DisplayName("connect 2초와 명령 5초가 다 지나도록 기다릴 수 없는 7초 timeout은 조립이 실패한다")
void aProfileTimeoutOneStalledConnectAndCommandOutlastFailsAssembly() {
// given
NotificationPlatformSettings.Provider sevenSeconds =
new NotificationPlatformSettings.Provider(
"SMTP", true, true, "local", "smtp-local", Duration.ofSeconds(7), 4, 8, null);
// when
Throwable failure = catchThrowable(() -> assembler().assemble("primary-email", sevenSeconds));
// then
assertThat(failure)
.as(
"a relay that never answers the greeting has been sent nothing; its socket deadline says so")
.isInstanceOf(IllegalStateException.class)
.hasMessageContaining("timeout (PT7S) must be longer than mail.smtp.connectiontimeout")
.hasMessageContaining("(PT7S), or a relay that never answers");
}
private static SmtpProviderRuntimeAssembler assembler() {
return assembler(Runnable::run);
}
/**
* The real SMTP assembler over {@code executor}; also used by {@code
* NotificationProviderAssemblyTest}.
*/
static SmtpProviderRuntimeAssembler assembler(Executor executor) {
Session session = Session.getInstance(new Properties());
return new SmtpProviderRuntimeAssembler(
message -> {
@@ -90,7 +196,7 @@ class SmtpProviderRuntimeAssemblerTest {
new SmtpMimeMessageFactory(session),
new SmtpFailureClassifier(),
protector(),
Runnable::run,
executor,
new AttachmentIntegrityGuard((reference, context) -> null),
new SmtpProviderProperties(
"localhost",
@@ -108,8 +214,12 @@ class SmtpProviderRuntimeAssemblerTest {
}
private static NotificationPlatformSettings.Provider profile() {
return profile(4);
}
private static NotificationPlatformSettings.Provider profile(int maxConcurrency) {
return new NotificationPlatformSettings.Provider(
"SMTP", true, true, "local", "smtp-local", Duration.ofSeconds(10), 4, 8, null);
"SMTP", true, true, "local", "smtp-local", Duration.ofSeconds(10), maxConcurrency, 8, null);
}
/** Not a lambda: the protector has three operations, and only one of them is exercised here. */
@@ -15,18 +15,28 @@ import dev.caskeleton.application.notification.platform.contact.EmailAddress;
import dev.caskeleton.application.notification.platform.contact.PhoneNumber;
import dev.caskeleton.application.notification.platform.provider.ProviderCallNotStartedException;
import dev.caskeleton.application.notification.platform.provider.ProviderSubmission;
import dev.caskeleton.application.notification.platform.provider.ProviderSubmissionResult;
import dev.caskeleton.application.notification.platform.provider.testkit.ProviderFixtures;
import dev.caskeleton.application.notification.platform.security.ContactPointProtector;
import jakarta.mail.Session;
import java.time.Duration;
import java.util.List;
import java.util.Optional;
import java.util.Properties;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.Executor;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.RejectedExecutionException;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicInteger;
import org.junit.jupiter.api.DisplayName;
import org.junit.jupiter.api.Test;
class SmtpNotificationProviderAdapterTest {
private static final Duration CALL_TIMEOUT = Duration.ofSeconds(5);
private final ContactPointProtector protector =
new AesGcmContactPointProtector(SecurityFixtures.keys());
@@ -138,7 +148,122 @@ class SmtpNotificationProviderAdapterTest {
.hasMessageContaining("email contact point");
}
@Test
void aSendThePoolRejectsIsReportedAsNotStarted() {
// given
Executor full =
command -> {
throw new RejectedExecutionException("the send pool is full");
};
// when
Throwable failure = catchThrowable(() -> adapter(message -> {}, full).submit(submission()));
// then
assertThat(failure)
.isInstanceOf(ProviderCallNotStartedException.class)
.extracting("code", "category", "retryable")
.containsExactly("PROVIDER_UNAVAILABLE", FailureCategory.CAPACITY_REJECTED, true);
}
private SmtpNotificationProviderAdapter adapter(SmtpDispatch dispatch) {
return adapter(dispatch, Executors.newSingleThreadExecutor());
}
@Test
void aSendThatOutlastsTheProfileTimeoutIsAmbiguousNotRetried() throws Exception {
// given
CountDownLatch released = new CountDownLatch(1);
SmtpDispatch slowRelay =
message -> {
try {
released.await();
} catch (InterruptedException interrupted) {
Thread.currentThread().interrupt();
}
};
ExecutorService pool = Executors.newSingleThreadExecutor();
// A second, not milliseconds: the send has to leave the pool queue inside it on a loaded
// runner, or it is abandoned before it starts and this is the other test.
SmtpNotificationProviderAdapter adapter = adapter(slowRelay, pool, Duration.ofSeconds(1));
// when
ProviderSubmissionResult result;
try {
result = adapter.submit(submission()).toCompletableFuture().get(5, TimeUnit.SECONDS);
} finally {
released.countDown();
pool.shutdownNow();
}
// then
assertThat(result)
.as("past the profile timeout the DATA may have gone through: never resend on our own")
.extracting(
ProviderSubmissionResult::confirmation,
outcome -> outcome.failure().orElseThrow().code(),
outcome -> outcome.failure().orElseThrow().retryable())
.containsExactly(AttemptConfirmation.AMBIGUOUS, "PROVIDER_RESPONSE_LOST", false);
}
@Test
@DisplayName("pool에서 profile timeout을 넘겨 기다린 발송은 relay에 보내지 않고 시작 전 실패로 답한다")
void aSendThatWaitedInThePoolPastTheTimeoutIsNotStartedAndNeverSent() throws Exception {
// given
CountDownLatch released = new CountDownLatch(1);
CountDownLatch entered = new CountDownLatch(1);
AtomicInteger handedToTheRelay = new AtomicInteger();
SmtpDispatch slowRelay =
message -> {
handedToTheRelay.incrementAndGet();
entered.countDown();
try {
released.await();
} catch (InterruptedException interrupted) {
Thread.currentThread().interrupt();
}
};
ExecutorService pool = Executors.newSingleThreadExecutor();
SmtpNotificationProviderAdapter adapter = adapter(slowRelay, pool, Duration.ofSeconds(1));
var occupyingThePool = adapter.submit(submission());
// The pool's one thread is in the relay before the second send is queued behind it.
assertThat(entered.await(5, TimeUnit.SECONDS)).as("the first send reached the relay").isTrue();
// when
Throwable waited;
boolean drained;
try {
waited =
catchThrowable(
() -> adapter.submit(submission()).toCompletableFuture().get(5, TimeUnit.SECONDS));
} finally {
released.countDown();
pool.shutdown();
// Every queued task has run once this returns true, the abandoned send's included, so the
// relay count below is final rather than read before the late send could have gone out.
drained = pool.awaitTermination(5, TimeUnit.SECONDS);
}
// then
assertThat(
List.of(
waited.getCause().getClass(),
waited.getCause().getMessage().contains("waited in the pool past the profile"),
drained,
handedToTheRelay.get()))
.as(
"nothing reached the relay, so the delivery is retried rather than reconciled, and"
+ " the abandoned send does not go out late beside %s",
occupyingThePool)
.containsExactly(ProviderCallNotStartedException.class, true, true, 1);
}
private SmtpNotificationProviderAdapter adapter(SmtpDispatch dispatch, Executor executor) {
return adapter(dispatch, executor, CALL_TIMEOUT);
}
private SmtpNotificationProviderAdapter adapter(
SmtpDispatch dispatch, Executor executor, Duration callTimeout) {
var properties =
new SmtpProviderProperties(
"smtp.example.com",
@@ -155,12 +280,13 @@ class SmtpNotificationProviderAdapterTest {
new SmtpFailureClassifier(),
protector,
properties,
Executors.newSingleThreadExecutor(),
executor,
// The unconfigured resolver refuses on use, which is the honest default: an attachment the
// deployment has no way to fetch must fail before the send, not silently vanish from it.
new dev.caskeleton.application.notification.platform.dispatch.AttachmentIntegrityGuard(
new dev.caskeleton.adapter.outbound.notification.platform.provider
.UnconfiguredAttachmentResolver()));
.UnconfiguredAttachmentResolver()),
callTimeout);
}
private ProviderSubmission submission() {
@@ -19,10 +19,15 @@ public interface IntegrationEventPublishPort {
CompletionStage<OutboxPublishAttempt> publish(ValidatedIntegrationEvent event);
/**
* The longest one publish can take before the returned stage completes, when the publisher knows
* it. The outbox relay sizes its batch against it (see {@code
* The longest one publish can take, from the call to {@code publish} until the returned stage
* completes, when the publisher knows it. The outbox relay sizes its batch against it (see {@code
* OutboxMessagePublishPort#longestAttempt}).
*
* <p>A bound, not a hint: a caller may stop waiting once it has passed and treat the attempt as
* ambiguous, so a publisher that reports less than it can take turns slow answers into ambiguous
* ones. Positive when present. Empty means unknown, and then the stage's own completion is the
* only bound.
*
* @return the longest attempt, or empty when unknown
*/
default Optional<Duration> longestAttempt() {
@@ -4,11 +4,13 @@ import dev.caskeleton.application.messaging.event.IntegrationEventPublishPort;
import dev.caskeleton.application.messaging.event.ValidatedIntegrationEvent;
import dev.caskeleton.application.outbox.OutboxPublishAttempt;
import dev.caskeleton.application.outbox.OutboxPublishOutcome;
import java.util.ArrayList;
import java.time.Duration;
import java.util.List;
import java.util.Objects;
import java.util.Optional;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.CompletionStage;
import java.util.concurrent.CopyOnWriteArrayList;
/**
* The {@link IntegrationEventPublishPort} fake: records every event it is handed and answers with
@@ -17,8 +19,55 @@ import java.util.concurrent.CompletionStage;
*/
public final class FakeIntegrationEventPublisher implements IntegrationEventPublishPort {
private final List<ValidatedIntegrationEvent> published = new ArrayList<>();
// A relay thread publishes while the test thread reads.
private final List<ValidatedIntegrationEvent> published = new CopyOnWriteArrayList<>();
private OutboxPublishAttempt answer = OutboxPublishAttempt.confirmed();
private boolean silent;
private Duration blocksFor = Duration.ZERO;
private RuntimeException fails;
private Optional<Duration> longestAttempt = Optional.empty();
/**
* Never completes the stage of any later publish, and reports {@code longestAttempt} as the
* longest a publish takes: a broker that does not answer, behind a publisher that promised it
* would within that bound.
*
* @param longestAttempt what {@link #longestAttempt()} reports
* @return this fake
*/
public FakeIntegrationEventPublisher silentPast(Duration longestAttempt) {
this.longestAttempt = Optional.of(Objects.requireNonNull(longestAttempt, "longestAttempt"));
this.silent = true;
return this;
}
/**
* Holds the caller for {@code blocking} inside every later {@code publish}, before the stage is
* handed back: a send that blocks its caller, as a producer waiting on cluster metadata does.
*
* @param blocking how long {@code publish} blocks
* @return this fake
*/
public FakeIntegrationEventPublisher blockingFor(Duration blocking) {
this.blocksFor = Objects.requireNonNull(blocking, "blocking");
return this;
}
/**
* Completes every later publish's stage exceptionally with {@code failure}.
*
* @param failure what the stage fails with
* @return this fake
*/
public FakeIntegrationEventPublisher failingWith(RuntimeException failure) {
this.fails = Objects.requireNonNull(failure, "failure");
return this;
}
@Override
public Optional<Duration> longestAttempt() {
return longestAttempt;
}
/**
* Answers every later publish with {@code outcome}, and for anything but a confirmation the
@@ -45,7 +94,17 @@ public final class FakeIntegrationEventPublisher implements IntegrationEventPubl
@Override
public CompletionStage<OutboxPublishAttempt> publish(ValidatedIntegrationEvent event) {
published.add(Objects.requireNonNull(event, "event must not be null"));
return CompletableFuture.completedFuture(answer);
if (!blocksFor.isZero()) {
try {
Thread.sleep(blocksFor);
} catch (InterruptedException interrupted) {
Thread.currentThread().interrupt();
}
}
if (fails != null) {
return CompletableFuture.failedFuture(fails);
}
return silent ? new CompletableFuture<>() : CompletableFuture.completedFuture(answer);
}
/** Every event handed to this fake, in order. */
@@ -118,7 +118,7 @@ app:
primary-for-channel: ${APP_NOTIFICATION_PLATFORM_SMTP_PRIMARY:true}
environment: ${APP_NOTIFICATION_PLATFORM_SMTP_ENVIRONMENT:local}
credential-profile: ${APP_NOTIFICATION_PLATFORM_SMTP_CREDENTIAL_PROFILE:default}
timeout: ${APP_NOTIFICATION_PLATFORM_SMTP_TIMEOUT:10s}
timeout: ${APP_NOTIFICATION_PLATFORM_SMTP_TIMEOUT:30s}
max-concurrency: ${APP_NOTIFICATION_PLATFORM_SMTP_MAX_CONCURRENCY:4}
rate-per-second: ${APP_NOTIFICATION_PLATFORM_SMTP_RATE_PER_SECOND:10}
ses: