fix(persistence)!: 트랜잭션 시한과 정리 경로를 닫는다
pr / done (pull_request) Successful in 12m42s

남은 트랜잭션·정리 결함 일곱. live event log는 지우는 곳이 없어 끝없이
자란다. app.lock.wait-time은 읽는 곳이 없다. 알림 scheduler는 poll마다
상태 쿼리 넷을 돌리고, projection 지연은 닫힌 event까지 읽는다.
stateless runner 트랜잭션에 시한이 없다. tenant pool 예산은 새 pool의
크기를 모른 채 열어 연결 합계가 상한을 넘는다. runbook SQL 설명이
lane 넷 중 셋만 말한다.

고른 방법: LiveEventReaper(1000행씩, 트랜잭션 30초)와 공용
BatchedDeletes를 더한다. app.realtime.replay.*는 LiveEventReplaySettings가
엄격하게 묶고, 조립 설정이 SchedulingConfigurer로 reaper를
reaper-interval(기본 10분)마다 등록한다. LockSettings는 lease-ttl 하나만
받는다(모르는 키로 기동 실패). scheduler는 backlogDepth() 한 문장만 읽고,
그 WHERE는 claim과 같은 SQL 조각(RecipientClaimSql.CLAIMABLE_AT_NOW)이다.
projection 지연은 V14 부분 인덱스로 읽는다(lock_timeout, V21 선례대로
CONCURRENTLY 아님). runner 기본 30초(SpringTransaction 쓰기 시한과 같음을
시험이 본다), tenant 예산은 계획된 pool 크기로 열기 전에 본다. 버린
대안: V14를 CONCURRENTLY로(Flyway 트랜잭션 안에서 못 돈다, 큰 DB는
안내대로 미리 만든다), reaper @Scheduled에 SpEL로 settings bean 이름
읽기(문자열 bean 조회는 검사가 못 본다).

위험·한계: LockSettings·TenantDataSourceRegistry·TenantPoolBudget의
공개 생성자가 바뀐다. V14는 큰 notification_provider_event에서 쓰기를
잠깐 막는다(안내 4행에 미리 만드는 법과 INVALID 인덱스 확인). backlog
수치는 창이 닫힌 delivery를 더는 세지 않는다.

검증: desktop RED(pr1-red3) 새 시험 실패 포함. GREEN은 PR 본문.

Refs: RR9-01-03, RR9-01-07, RR9-01-09, RR9-01-10, RR9-01-14, RR9-01-25, RR9-01-26
BREAKING CHANGE: JpaLiveEventReplay.sweepExpired 삭제, app.realtime.replay 아래 모르는 키와 app.lock.wait-time은 기동 실패, HibernateStatelessSessionRunner 트랜잭션 30초 상한, TenantPoolBudget·TenantDataSourceRegistry 시그니처 변경. docs/upgrade/2026-11-round9-claim-fence.md
This commit is contained in:
DongHyeonka
2026-11-13 13:17:33 +09:00
parent afdedabf2a
commit 4da5143906
32 changed files with 1038 additions and 140 deletions
+3 -1
View File
@@ -5,7 +5,7 @@
# root yet.
# Update only after review with:
# ./gradlew :adapter:outbound:persistence-jpa:updateJpaApiSurface -PapproveJpaApiSurfaceChange
# types: 166
# types: 168
dev.caskeleton.adapter.outbound.persistence.api.PersistenceOperationName
dev.caskeleton.adapter.outbound.persistence.api.capability.CapabilitySupport
dev.caskeleton.adapter.outbound.persistence.api.capability.JpaCapability
@@ -104,6 +104,8 @@ dev.caskeleton.adapter.outbound.persistence.idempotency.postgresql.IdempotencySc
dev.caskeleton.adapter.outbound.persistence.idempotency.postgresql.PostgreSqlIdempotencyStore
dev.caskeleton.adapter.outbound.persistence.liveevent.JpaLiveEventReplay
dev.caskeleton.adapter.outbound.persistence.liveevent.LiveEventJpaRepository
dev.caskeleton.adapter.outbound.persistence.liveevent.LiveEventReaper
dev.caskeleton.adapter.outbound.persistence.liveevent.LiveEventReplaySettings
dev.caskeleton.adapter.outbound.persistence.liveevent.entity.LiveEventEntity
dev.caskeleton.adapter.outbound.persistence.lock.DistributedLockPersistenceConfig
dev.caskeleton.adapter.outbound.persistence.lock.LockRegistryDistributedLock
+21 -10
View File
@@ -1,13 +1,14 @@
# 업그레이드 안내 — 2026-11 claim·fence·트랜잭션 시한·정리
- 범위: durable operation·알림 수신자·fileserver cleanup·멱등성 기록의 lease, fence, 조건부 쓰기, 정리를 고친
변경. 포트 시그니처와 기동 조건이 바뀐 곳이 있다. 아래 표가 그 목록이다.
- 근거: 2026-11-12 전체 리뷰(`docs/reviews/2026-11-12-round9-review/`)의 claim·fence(B8) 발견
- 범위: durable operation·알림 수신자·fileserver cleanup·멱등성 기록·live event·분산 락·stateless runner의 lease,
fence, 조건부 쓰기, 정리, 시간 상한을 고친 변경. 포트 시그니처와 기동 조건이 바뀐 곳이 있다. 아래 표가 그 목록이다.
- 근거: 2026-11-12 전체 리뷰(`docs/reviews/2026-11-12-round9-review/`)의 claim·fence(B8)와 트랜잭션 시한·정리(B9) 발견
## 1. 누가 읽나
이 템플릿을 `upstream` remote로 두고 merge해 받는 파생 서비스가 읽는다. 특히 durable operation worker를 직접 쓴 서비스,
알림 recipient store를 직접 구현한 서비스, 알림 provider timeout을 설정한 서비스다.
알림 recipient store를 직접 구현한 서비스, `app.lock.wait-time`이나 알림 provider timeout을 설정한 서비스, experimental
tenant pool 예산을 쓰는 서비스다.
## 2. 할 일
@@ -16,17 +17,22 @@
| 1 | durable operation worker는 `claimNext`가 돌려주는 `DurableOperationClaim`을 들고 있다가 `heartbeat(claim, …)`·`reportProgress(claim, …)`·`succeed(claim, …)`·`fail(claim, …)`에 넘긴다. 예전에는 `(operationId, workerId, …)`였다 | 컴파일이 실패한다 | durable operation worker를 쓴 서비스 |
| 2 | `DurableOperationStorePort`를 직접 구현했으면 claim마다 오르는 fence를 저장하고, 네 쓰기가 owner와 fence를 함께 확인하게 한다. `DurableOperationStoreContract`의 새 사례(같은 worker id가 다시 claim한 뒤 앞 실행의 쓰기는 거절)로 확인한다 | 재시작한 worker나 한 프로세스의 두 실행이 같은 id를 쓰면 앞 실행의 결과가 넘겨받은 실행 위에 기록된다 | 자기 durable operation store를 둔 서비스 |
| 3 | persistence-jpa의 `V23__durable_operation_lease_fence.sql`(`durable_operation.lease_fence BIGINT NOT NULL DEFAULT 0`)이 기본 stream에 더해진다. 같은 번호의 서비스 마이그레이션이 있으면 번호를 옮긴다. `durable_operation`을 건드린 트랜잭션이 열려 있으면 5초 뒤 실패하고 V22에 남는다(다시 배포) | Flyway가 번호 충돌로 기동하지 않는다 | persistence-jpa를 쓰는 모든 서비스 |
| 4 | `RecipientLeaseStorePort.expiredDispatching`이 `List<RecipientLease>`(버려질 때의 owner·fence·만료 시각)를 돌려주고, `RecipientDeliveryStorePort`에 `requeueAbandoned(lease, nextDispatchAt)`가 생겼다. 직접 구현했으면 상태가 `DISPATCHING`이고 owner·fence가 같고 lease가 늘지 않았을 때만 `READY_TO_DISPATCH`로 바꾼다(`RecipientDeliveryStoreContract` 새 사례). owner가 없거나 fence가 0인 `DISPATCHING` 행(notification V4 전에 남은 행)은 복구가 더는 고르지 않는다. 그런 행은 `SELECT id FROM notification_recipient_delivery WHERE delivery_state = 'DISPATCHING' AND (lease_owner IS NULL OR lease_fence = 0)`로 찾는다. 배포 전에 `UPDATE notification_recipient_delivery SET lease_owner = COALESCE(lease_owner, 'pre-v4'), lease_fence = lease_fence + 1, lease_until = now() - interval '1 day', version = version + 1, updated_at = now() WHERE delivery_state = 'DISPATCHING' AND (lease_owner IS NULL OR lease_fence = 0)`로 owner를 채우고 fence를 올려 복구 패스에 넘긴다. 복구는 attempt를 보고 정한다: attempt가 없으면 다시 보낼 차례로 돌리고, 끝나지 않은 attempt가 있으면 맞추기로 넘긴다(그 뒤는 `docs/notification/configuration-reference.md` 「발송」의 lease 문단). 손으로 `READY_TO_DISPATCH`로 바꾸지 않는다. 발송 중에 멈춘 행이면 같은 알림이 한 번 더 나간다. owner가 없는 행은 stuck lease 지표에도 잡히지 않는다 | 컴파일이 실패한다. 조건 없이 바꾸면 끝났거나 다시 claim된 delivery가 다시 발송된다 | 알림 store를 직접 구현한 서비스 |
| 5 | 알림 provider profile의 `timeout`이 `app.notification.platform.dispatch.lease-duration`보다 짧은지 확인한다. 짧은 것은 필요조건일 뿐이다: lease는 claim 때 시작해서 attempt 기록·렌더링·provider 허가 대기도 lease를 쓴다. lease는 timeout보다 넉넉히 길게 둔다. 이 timeout이 provider 호출 전체를 묶는 것은 SES(API 호출 시간 상한)다. SMTP는 `spring.mail` socket timeout이 명령마다 묶을 뿐이라 이 검사만으로는 호출 전체가 lease 안에 끝난다고 말할 수 없다 | 짧지 않으면 기동이 실패한다(`notification provider profile '…': timeout (…) must be shorter than …lease-duration`). 전에는 provider 호출 하나가 lease보다 길어도 기동했다. 그때 생기는 일은 `docs/notification/configuration-reference.md` 「발송」의 lease 문단이다 | 알림 capability를 켠 서비스 |
| 6 | `IdempotencyReaper`를 직접 만들던 코드는 `new IdempotencyReaper(repository)`로 바꾼다(`Clock` 인자 삭제). 만료 비교는 DB 시계(`now()`)이고 1000행씩 지운다 | 컴파일이 실패한다 | reaper를 직접 조립한 서비스 |
| 7 | fileserver cleanup 묶음의 `max-items`를 정할 때, 한 묶음이 claim lease(10분)의 첫 8분 안에 시작할 수 있는 만큼만 처리된다는 것을 안다. 나머지는 시도하지 않은 채 queue에 돌아가 다음 묶음이 다시 claim한다. 시도 횟수는 그대로다(바이트 예산이 바닥나 돌려준 항목도 이제 같다. 전에는 시도 한 번으로 세어 8번째에 보류됐다). 직접 구현한 `CleanupQueuePort`는 새 연산 `release`를 구현한다(`CleanupQueuePortContract` 새 사례) | 느린 저장소에서 큰 `max-items`는 그 사유로 도는 항목이 늘어난다. 전에는 lease가 끝난 뒤에도 남은 항목을 처리해 다른 worker와 겹쳤다 | fileserver capability를 켠 서비스 |
| 4 | 알림 stream에 `V14__notification_provider_event_open_index.sql`(부분 인덱스 `ix_notification_provider_event_open`)이 더해진다. `notification_provider_event`가 이미 큰 운영 DB는 배포 전에 같은 인덱스를 `CREATE INDEX CONCURRENTLY IF NOT EXISTS ix_notification_provider_event_open ON notification_provider_event (received_at) WHERE projection_status IN ('PENDING', 'FAILED') OR attempt_id IS NULL;`로 먼저 만든다. 미리 만든 인덱스가 실패해 남았으면(`SELECT indisvalid FROM pg_index WHERE indexrelid = 'ix_notification_provider_event_open'::regclass`가 false) `DROP INDEX CONCURRENTLY ix_notification_provider_event_open` 뒤 다시 만든다. `IF NOT EXISTS`는 쓸 수 없는 인덱스도 있는 것으로 보고 넘어간다 | 마이그레이션이 인덱스를 만드는 동안 provider callback 저장이 기다린다(provider가 callback을 다시 보낸다) | 알림 capability를 켠 서비스 |
| 5 | `RecipientLeaseStorePort.expiredDispatching`이 `List<RecipientLease>`(버려질 때의 owner·fence·만료 시각)를 돌려주고, `RecipientDeliveryStorePort`에 `requeueAbandoned(lease, nextDispatchAt)`가 생겼다. 직접 구현했으면 상태가 `DISPATCHING`이고 owner·fence가 같고 lease가 늘지 않았을 때만 `READY_TO_DISPATCH`로 바꾼다(`RecipientDeliveryStoreContract` 새 사례). owner가 없거나 fence가 0인 `DISPATCHING` 행(notification V4 전에 남은 행)은 복구가 더는 고르지 않는다. 그런 행은 `SELECT id FROM notification_recipient_delivery WHERE delivery_state = 'DISPATCHING' AND (lease_owner IS NULL OR lease_fence = 0)`로 찾는다. 배포 전에 `UPDATE notification_recipient_delivery SET lease_owner = COALESCE(lease_owner, 'pre-v4'), lease_fence = lease_fence + 1, lease_until = now() - interval '1 day', version = version + 1, updated_at = now() WHERE delivery_state = 'DISPATCHING' AND (lease_owner IS NULL OR lease_fence = 0)`로 owner를 채우고 fence를 올려 복구 패스에 넘긴다. 복구는 attempt를 보고 정한다: attempt가 없으면 다시 보낼 차례로 돌리고, 끝나지 않은 attempt가 있으면 맞추기로 넘긴다(그 뒤는 `docs/notification/configuration-reference.md` 「발송」의 lease 문단). 손으로 `READY_TO_DISPATCH`로 바꾸지 않는다. 발송 중에 멈춘 행이면 같은 알림이 한 번 더 나간다. owner가 없는 행은 stuck lease 지표에도 잡히지 않는다 | 컴파일이 실패한다. 조건 없이 바꾸면 끝났거나 다시 claim된 delivery가 다시 발송된다 | 알림 store를 직접 구현한 서비스 |
| 6 | 알림 provider profile의 `timeout`이 `app.notification.platform.dispatch.lease-duration`보다 짧은지 확인한다. 짧은 것은 필요조건일 뿐이다: lease는 claim 때 시작해서 attempt 기록·렌더링·provider 허가 대기도 lease를 쓴다. lease는 timeout보다 넉넉히 길게 둔다. 이 timeout이 provider 호출 전체를 묶는 것은 SES(API 호출 시간 상한)다. SMTP는 `spring.mail` socket timeout이 명령마다 묶을 뿐이라 이 검사만으로는 호출 전체가 lease 안에 끝난다고 말할 수 없다 | 짧지 않으면 기동이 실패한다(`notification provider profile '…': timeout (…) must be shorter than …lease-duration`). 전에는 provider 호출 하나가 lease보다 길어도 기동했다. 그때 생기는 일은 `docs/notification/configuration-reference.md` 「발송」의 lease 문단이다 | 알림 capability를 켠 서비스 |
| 7 | `app.lock.wait-time`을 설정에서 지운다. 기다릴 시간은 `tryAcquire(key, waitTime, leaseTtl)`의 인자다 | `app.lock` 아래 모르는 키라 기동이 실패한다. 전에도 읽는 곳이 없었다 | `app.lock.wait-time`을 설정한 서비스 |
| 8 | `JpaLiveEventReplay.sweepExpired()`를 부르던 코드를 지운다. replay를 켜면 `LiveEventReaper`가 `app.realtime.replay.reaper-interval`(기본 `PT10M`)마다 `retained_until`이 지난 행을 1000행씩 지운다. `app.realtime.replay.*`는 이제 `LiveEventReplaySettings`가 엄격하게 묶는다: 그 아래 모르는 키, 0 이하의 `retention`·`reaper-interval`은 기동을 실패시킨다 | 컴파일이 실패한다. 잘못 쓴 키는 기동이 실패한다. 전에는 지우는 곳이 없어 `live_event_log`가 끝없이 자랐다 | `app.realtime.replay.enabled=true`인 서비스 |
| 9 | `IdempotencyReaper`를 직접 만들던 코드는 `new IdempotencyReaper(repository)`로 바꾼다(`Clock` 인자 삭제). 만료 비교는 DB 시계(`now()`)이고 1000행씩 지운다 | 컴파일이 실패한다 | reaper를 직접 조립한 서비스 |
| 10 | `HibernateStatelessSessionRunner`의 트랜잭션은 이제 30초 상한이 있다(`DEFAULT_TIMEOUT`). 더 긴 일이면 셋째 인자로 상한을 넘기거나 일을 나눈다 | 30초를 넘는 문장이 취소되고 일 전체가 롤백된다. 전에는 상한이 없었다 | stateless runner를 쓰는 서비스 |
| 11 | experimental `TenantPoolBudget.requireCapacity`·`hasCapacity`가 새 pool의 연결 수를 셋째 인자로 받는다. pool 수만 볼 때는 `requirePoolSlot(openPools)`를 쓴다. `TenantDataSourceRegistry`의 둘째 인자는 이제 열린 pool의 크기를 재는 함수가 아니라 tenant의 계획된 pool 크기(`ToIntFunction<TenantId>`)다. 예산은 pool을 열기 전에 그 크기로 본다 | 컴파일이 실패한다. 전에는 새 pool 크기를 모른 채 열어 연결 합계가 상한을 넘었다 | experimental tenant pool을 쓰는 서비스 |
| 12 | fileserver cleanup 묶음의 `max-items`를 정할 때, 한 묶음이 claim lease(10분)의 첫 8분 안에 시작할 수 있는 만큼만 처리된다는 것을 안다. 나머지는 시도하지 않은 채 queue에 돌아가 다음 묶음이 다시 claim한다. 시도 횟수는 그대로다(바이트 예산이 바닥나 돌려준 항목도 이제 같다. 전에는 시도 한 번으로 세어 8번째에 보류됐다). 직접 구현한 `CleanupQueuePort`는 새 연산 `release`를 구현한다(`CleanupQueuePortContract` 새 사례) | 느린 저장소에서 큰 `max-items`는 그 사유로 도는 항목이 늘어난다. 전에는 lease가 끝난 뒤에도 남은 항목을 처리해 다른 worker와 겹쳤다 | fileserver capability를 켠 서비스 |
## 3. 바뀐 동작 (따라 할 일 없음)
- reaper 둘(outbox·멱등성)의 트랜잭션은 `SpringTransaction.DEFAULT_TIMEOUT_SECONDS`(30초)로 묶인다.
- reaper 셋(outbox·멱등성·live event)의 트랜잭션은 `SpringTransaction.DEFAULT_TIMEOUT_SECONDS`(30초)로 묶인다.
`app.persistence-jpa.transaction.*`를 바꿔도 reaper 시한은 그대로다. 한 트랜잭션이 1000행 한 묶음만 지운다.
- 저장소가 행에서 만드는 `DurableOperationClaim`·`OperationLease`·`RecipientLease`·`OutboxLease`·`ClaimedOutboxEvent`·
`ReconciliationJob`·`AdminOperationClaim`이 값을 거절할 때 `IllegalStateException`을 던진다
`ReconciliationJob`·`AdminOperationClaim`과 stateless runner의 timeout이 값을 거절할 때 `IllegalStateException`을 던진다
(전에는 `IllegalArgumentException`). 요청에서 오는 값이 아니라 서버 결함이다. 이 IAE를 잡던 코드는 ISE를 잡는다.
- 알림 lease 회복은 버려질 때의 lease 그대로일 때만 delivery를 다시 넣는다. 다른 인스턴스가 이미 다시 넣었거나 holder가
끝냈으면 아무것도 쓰지 않고 처리 수에 세지 않는다.
@@ -36,3 +42,8 @@
- 멱등성 reaper는 다른 트랜잭션이 쥔 기록을 건너뛰고, 바깥 `DELETE`에서도 만료를 다시 본다. 같은 키 요청이 만료된 기록을
되살리는 동안 돈 reap이 그 claim을 지우지 않는다.
- fileserver recovery 대기열 enqueue는 `INSERT … ON CONFLICT` 한 문장이다. 같은 파일을 동시에 넣어도 25P02로 실패하지 않는다.
- 알림 scheduler는 poll마다 backlog 한 문장만 읽는다(전에는 넷). health snapshot의 projection 지연은 열린 provider event만
읽는다.
- 알림 backlog(`backlogDepth`, scheduler의 queue-depth gauge, health 임계값)는 claim이 지금 가져갈 delivery만 센다. claim과
같은 SQL 조각이다. 전에는 창이 닫힌(`expires_at`이 지난) delivery도 셌고, lease가 지금 막 끝나는 행(`lease_until = now`)도
셌다. 만료가 쌓인 플랫폼은 배포 직후 backlog 수치가 내려간다. 보낼 일이 줄어든 것이 아니라 보낼 수 없는 행을 빼고 센 것이다.
@@ -81,10 +81,10 @@ public final class NotificationSchedulerWorker implements AutoCloseable {
// by the claim batch size — so a queue of ten and a queue of ten million published the same
// number, and the one metric named "queue depth" was the one that could not show a queue
// growing.
// The depth alone: the full serving snapshot is several statements and is the health
// endpoint's, not something to run on every poll.
metrics.gauge(
NotificationMetricName.QUEUE_DEPTH,
Map.of(),
(double) servingState.currentState().backlogDepth());
NotificationMetricName.QUEUE_DEPTH, Map.of(), (double) servingState.backlogDepth());
for (RecipientLease lease : claimed) {
globalConcurrency.acquireUninterruptibly();
@@ -389,6 +389,22 @@ PUBLISHED backlog 를 한 transaction 으로 지우면 그 사이 모든 row loc
cadence 측정 필요). retention 한 값은 reaper-local 이라 `@Value` 로 받지만, canonical 6-property
문서는 app-bootstrap `OutboxSettings` / `application.yml` 에 있다.
## liveevent — `JpaLiveEventReplay`와 `LiveEventReaper`
`app.realtime.replay.enabled=true`이고 vendor가 PostgreSQL이면 replay store와 reaper가 함께 조립된다. 설정은
`LiveEventReplaySettings`가 `app.realtime.replay.*`를 엄격하게 묶는다. 모르는 키는 기동을 실패시킨다.
- `retention`: 한 event를 다시 받을 수 있는 시간. 기본값이 없다. 모듈 기본값이 `APP_REALTIME_REPLAY_RETENTION`(1시간)으로
채운다. 0이나 음수는 기동 실패다.
- `reaper-interval`: `LiveEventReaper`가 retention이 지난 행을 지우는 간격(기본 10분, 모듈 기본값이 같은 값을 적는다). 0이나
음수는 기동 실패다. reaper 자체에는 일정이 없다. 조립 설정이 이 값으로 `reap()`을 등록한다(첫 실행도 한 간격 뒤).
reaper를 직접 만드는 서비스는 일정도 직접 건다.
reaper는 `BatchedDeletes`로 1,000행씩, 배치마다 따로 commit하고, 배치가 덜 차면 멈춘다. 배치 문장은 `SKIP LOCKED`라 여러
replica가 같이 돌아도 서로 기다리지 않는다. 트랜잭션 시한은 `SpringTransaction.DEFAULT_TIMEOUT_SECONDS`(30초)다. replay 읽기는
`retained_until`로 지난 행을 이미 거르므로, reaper가 늦어도 지난 event를 내주지는 않는다. reaper가 없으면 테이블이 끝없이
자란다(2026-11-13 전에는 지우는 곳이 없었다).
## operation — `JpaDurableOperationStore`
### claim과 fence (`V23`)
@@ -8,6 +8,8 @@ import dev.caskeleton.adapter.outbound.persistence.idempotency.jpa.IdempotencyRe
import dev.caskeleton.adapter.outbound.persistence.idempotency.jpa.IdempotencyRecordJpaRepository;
import dev.caskeleton.adapter.outbound.persistence.liveevent.JpaLiveEventReplay;
import dev.caskeleton.adapter.outbound.persistence.liveevent.LiveEventJpaRepository;
import dev.caskeleton.adapter.outbound.persistence.liveevent.LiveEventReaper;
import dev.caskeleton.adapter.outbound.persistence.liveevent.LiveEventReplaySettings;
import dev.caskeleton.adapter.outbound.persistence.lock.DistributedLockPersistenceConfig;
import dev.caskeleton.adapter.outbound.persistence.operation.DurableOperationJpaRepository;
import dev.caskeleton.adapter.outbound.persistence.operation.JpaDurableOperationStore;
@@ -26,7 +28,11 @@ import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.Import;
import org.springframework.core.env.Environment;
import org.springframework.jdbc.core.JdbcOperations;
import org.springframework.scheduling.annotation.EnableScheduling;
import org.springframework.scheduling.annotation.SchedulingConfigurer;
import org.springframework.scheduling.config.FixedDelayTask;
import org.springframework.transaction.PlatformTransactionManager;
/**
@@ -120,11 +126,15 @@ class JpaAdapterBeansConfig {
}
/**
* Retained live-event history: on when replay is switched on, and only on PostgreSQL (the
* position allocator is an {@code INSERT … ON CONFLICT … RETURNING}). The retention has no
* default on purpose: a replay window nobody chose is one nobody sized.
* Retained live-event history and the reaper that removes it past retention: on when replay is
* switched on, and only on PostgreSQL (the position allocator is an {@code INSERT … ON CONFLICT …
* RETURNING}). The settings are {@link LiveEventReplaySettings}, bound strictly: the retention is
* one hour unless the deployment sets {@code APP_REALTIME_REPLAY_RETENTION} (the module
* defaults), the reaper runs every {@code reaper-interval}. Scheduling is enabled here because
* the reaper is part of the store, as for the idempotency reaper.
*/
@Configuration(proxyBeanMethods = false)
@EnableScheduling
@ConditionalOnProperty(prefix = "app.realtime.replay", name = "enabled", havingValue = "true")
@ConditionalOnProperty(
prefix = "app.persistence",
@@ -133,13 +143,36 @@ class JpaAdapterBeansConfig {
matchIfMissing = true)
static class LiveEventReplay {
@Bean
LiveEventReplaySettings liveEventReplaySettings(Environment environment) {
return LiveEventReplaySettings.bind(environment);
}
@Bean
JpaLiveEventReplay jpaLiveEventReplay(
LiveEventJpaRepository repository,
JdbcOperations jdbc,
Clock clock,
@Value("${app.realtime.replay.retention}") Duration retention) {
return new JpaLiveEventReplay(repository, jdbc, clock, retention);
LiveEventReplaySettings settings) {
return new JpaLiveEventReplay(repository, jdbc, clock, settings.retention());
}
@Bean
LiveEventReaper liveEventReaper(LiveEventJpaRepository repository, Clock clock) {
return new LiveEventReaper(repository, clock);
}
/**
* Runs the reaper every bound {@code reaper-interval}, first one interval after startup. The
* interval is handed over as the settings' value, so it has one default and a zero has already
* failed startup in the settings rather than scheduling a loop.
*/
@Bean
SchedulingConfigurer liveEventReaperSchedule(
LiveEventReaper reaper, LiveEventReplaySettings settings) {
Duration interval = settings.reaperInterval();
return registrar ->
registrar.addFixedDelayTask(new FixedDelayTask(reaper::reap, interval, interval));
}
}
@@ -23,26 +23,32 @@ import javax.sql.DataSource;
public final class TenantDataSourceRegistry implements AutoCloseable {
private final Map<TenantId, DataSource> pools = new LinkedHashMap<>();
private final Map<TenantId, Integer> poolSizes = new LinkedHashMap<>();
private final Function<TenantId, DataSource> poolFactory;
private final ToIntFunction<DataSource> poolSize;
private final ToIntFunction<TenantId> plannedPoolSize;
private final TenantPoolBudget budget;
/**
* @param poolFactory opens a pool for a tenant from its secret-backed connection profile
* @param poolSize reports a pool's maximum connection count, for the global budget
* @param plannedPoolSize the maximum connection count a tenant's pool will have, read from its
* connection profile before the pool is opened, for the global budget
*/
public TenantDataSourceRegistry(
Function<TenantId, DataSource> poolFactory,
ToIntFunction<DataSource> poolSize,
ToIntFunction<TenantId> plannedPoolSize,
TenantPoolBudget budget) {
this.poolFactory = Objects.requireNonNull(poolFactory, "poolFactory");
this.poolSize = Objects.requireNonNull(poolSize, "poolSize");
this.plannedPoolSize = Objects.requireNonNull(plannedPoolSize, "plannedPoolSize");
this.budget = Objects.requireNonNull(budget, "budget");
}
/**
* The tenant's pool, opening it if the budget allows.
*
* <p>The budget is checked against the pool's planned size before the pool is opened: a pool
* opens its first connections as it is built, so one checked afterwards had already taken the
* server past the ceiling, if only until it was closed again.
*
* @throws IllegalStateException when the global pool budget is exhausted
*/
public synchronized DataSource require(TenantId tenant) {
@@ -51,9 +57,12 @@ public final class TenantDataSourceRegistry implements AutoCloseable {
if (existing != null) {
return existing;
}
budget.requireCapacity(pools.size(), allocatedConnections());
budget.requirePoolSlot(pools.size());
int planned = plannedPoolSize.applyAsInt(tenant);
budget.requireCapacity(pools.size(), allocatedConnections(), planned);
DataSource opened = poolFactory.apply(tenant);
pools.put(tenant, opened);
poolSizes.put(tenant, planned);
return opened;
}
@@ -72,6 +81,7 @@ public final class TenantDataSourceRegistry implements AutoCloseable {
*/
public synchronized void evict(TenantId tenant) {
DataSource removed = pools.remove(tenant);
poolSizes.remove(tenant);
if (removed instanceof AutoCloseable closeable) {
TenantDataSourceLifecycle.drainAndClose(closeable);
}
@@ -84,7 +94,7 @@ public final class TenantDataSourceRegistry implements AutoCloseable {
/** How many connections all open pools may hold between them. */
public synchronized int allocatedConnections() {
return pools.values().stream().mapToInt(poolSize).sum();
return poolSizes.values().stream().mapToInt(Integer::intValue).sum();
}
/** The tenants with an open pool. */
@@ -99,5 +109,6 @@ public final class TenantDataSourceRegistry implements AutoCloseable {
.map(AutoCloseable.class::cast)
.forEach(TenantDataSourceLifecycle::drainAndClose);
pools.clear();
poolSizes.clear();
}
}
@@ -23,19 +23,44 @@ public record TenantPoolBudget(int maxOpenPools, int maxConnectionsAcrossPools)
}
/**
* Fails when another tenant pool would exceed either ceiling.
* Fails when another tenant pool would exceed the pool count, before that pool is opened.
*
* @throws IllegalStateException when the budget is exhausted
* @param openPools how many pools are open now
* @throws IllegalStateException when no other pool fits
*/
public void requireCapacity(int openPools, int allocatedConnections) {
if (openPools >= maxOpenPools || allocatedConnections >= maxConnectionsAcrossPools) {
public void requirePoolSlot(int openPools) {
if (openPools >= maxOpenPools) {
throw new IllegalStateException("tenant pool budget exhausted");
}
}
/** Whether another tenant pool fits inside both ceilings. */
public boolean hasCapacity(int openPools, int allocatedConnections) {
return openPools < maxOpenPools && allocatedConnections < maxConnectionsAcrossPools;
/**
* Fails when a new tenant pool of {@code newPoolConnections} would exceed either ceiling.
*
* <p>The new pool's own size is part of the question. Checking only that the pools already open
* stay under the ceiling let a ten-connection pool open with one connection to spare, and the
* total then stood nine above the ceiling the budget exists to keep (round9 review 2026-11-12,
* RR9-01-26).
*
* @param openPools how many pools are open now, the new one not included
* @param allocatedConnections how many connections the open pools may hold between them
* @param newPoolConnections how many the new pool may hold
* @throws IllegalStateException when the budget cannot take the new pool
*/
public void requireCapacity(int openPools, int allocatedConnections, int newPoolConnections) {
if (!hasCapacity(openPools, allocatedConnections, newPoolConnections)) {
throw new IllegalStateException("tenant pool budget exhausted");
}
}
/** Whether a new tenant pool of {@code newPoolConnections} fits inside both ceilings. */
public boolean hasCapacity(int openPools, int allocatedConnections, int newPoolConnections) {
if (newPoolConnections < 1) {
// The size is the server's own configuration, not the caller's input: a state error.
throw new IllegalStateException("a tenant pool holds at least one connection");
}
return openPools < maxOpenPools
&& (long) allocatedConnections + newPoolConnections <= maxConnectionsAcrossPools;
}
/** The maximum number of tenants that can be open at once. */
@@ -1,5 +1,6 @@
package dev.caskeleton.adapter.outbound.persistence.hibernate.stateless;
import java.time.Duration;
import java.util.Objects;
import java.util.Set;
import java.util.function.Function;
@@ -19,23 +20,63 @@ import org.hibernate.Transaction;
*
* <p>Only registered work names may run, and each carries a declared row cap. An admin bulk path
* without a cap is how a maintenance job becomes an outage.
*
* <p>The transaction has the same time bound as the application's write transactions (30 seconds
* unless the constructor is given another, {@code docs/work/reference/decisions.md} 2.5). The row
* cap can only be checked once the work reports what it touched, so without a bound a runaway
* statement held its locks for as long as it ran before the cap could roll it back. Hibernate hands
* every statement of the work the time left, and the database cancels the one that runs past it
* (round9 review 2026-11-12, RR9-01-25).
*/
public final class HibernateStatelessSessionRunner {
/**
* The default write-transaction bound, which this runner's transactions share: {@code
* SpringTransaction.DEFAULT_TIMEOUT_SECONDS}. Restated rather than read from it, because the
* {@code hibernate} package does not depend on {@code transaction} (JpaModuleBoundaryTest);
* {@code StatelessRunnerTimeoutTest} holds the two equal. A deployment that moves its write bound
* passes the same value to the three-argument constructor.
*/
public static final Duration DEFAULT_TIMEOUT = Duration.ofSeconds(30);
private final SessionFactory sessionFactory;
private final Set<StatelessWorkName> registeredWork;
private final int timeoutSeconds;
/** A runner whose transactions are bounded by {@link #DEFAULT_TIMEOUT}. */
public HibernateStatelessSessionRunner(
SessionFactory sessionFactory, Set<StatelessWorkName> registeredWork) {
this(sessionFactory, registeredWork, DEFAULT_TIMEOUT);
}
/**
* A runner whose transactions are bounded by {@code timeout}.
*
* @param timeout a whole, positive number of seconds; Hibernate's transaction timeout has no
* finer unit, so a fraction would be dropped without a word
*/
public HibernateStatelessSessionRunner(
SessionFactory sessionFactory, Set<StatelessWorkName> registeredWork, Duration timeout) {
this.sessionFactory = Objects.requireNonNull(sessionFactory, "sessionFactory");
this.registeredWork = Set.copyOf(Objects.requireNonNull(registeredWork, "registeredWork"));
Objects.requireNonNull(timeout, "timeout");
if (timeout.isNegative()
|| timeout.isZero()
|| timeout.toNanosPart() != 0
|| timeout.toSeconds() > Integer.MAX_VALUE) {
throw new IllegalStateException(
"a stateless transaction timeout must be a positive whole number of seconds, was "
+ timeout);
}
this.timeoutSeconds = Math.toIntExact(timeout.toSeconds());
}
/**
* Runs {@code work} in its own stateless session and transaction.
*
* <p>The work reports how many rows it touched, and exceeding {@code maxRows} rolls the
* transaction back.
* transaction back. A statement still running when the transaction's time is up is cancelled, and
* the failure rolls the transaction back.
*
* @param maxRows the row cap for this unit of work
* @throws StatelessRowCapExceededException when the work touched more, after rolling back
@@ -54,7 +95,9 @@ public final class HibernateStatelessSessionRunner {
}
try (StatelessSession session = sessionFactory.openStatelessSession()) {
Transaction transaction = session.beginTransaction();
Transaction transaction = session.getTransaction();
transaction.setTimeout(timeoutSeconds);
transaction.begin();
try {
StatelessWorkResult<T> result = work.apply(session);
if (result.affectedRows() > maxRows) {
@@ -134,16 +134,6 @@ public final class JpaLiveEventReplay implements LiveEventReplayPort {
.toList();
}
/**
* Removes events past their retention.
*
* @return how many went
*/
public int sweepExpired() {
requireCallerTransaction("sweepExpired");
return repository.sweepExpired(clock.instant());
}
private static void requireCallerTransaction(String operation) {
CallerTransaction.require("live-event " + operation);
}
@@ -1,6 +1,7 @@
package dev.caskeleton.adapter.outbound.persistence.liveevent;
import dev.caskeleton.adapter.outbound.persistence.liveevent.entity.LiveEventEntity;
import dev.caskeleton.adapter.outbound.persistence.transaction.SpringTransaction;
import java.time.Instant;
import java.util.List;
import org.springframework.data.domain.Limit;
@@ -8,6 +9,7 @@ import org.springframework.data.jpa.repository.JpaRepository;
import org.springframework.data.jpa.repository.Modifying;
import org.springframework.data.jpa.repository.Query;
import org.springframework.data.repository.query.Param;
import org.springframework.transaction.annotation.Transactional;
/**
* Spring Data access for {@code live_event_log}.
@@ -66,12 +68,27 @@ public interface LiveEventJpaRepository
Long latestRetained(@Param("streamId") String streamId, @Param("now") Instant now);
/**
* Removes expired rows.
* Deletes at most {@code limit} rows retained no later than {@code now}, in a transaction of its
* own unless the caller already has one; returns the count.
*
* <p>Bounded, and skipping rows another replica's reaper is deleting, for the reason the outbox
* purge is bounded: a backlog of expired history is one long transaction holding every row lock
* otherwise. {@code retained_until} is written from the application clock, so it is compared with
* the application clock.
*
* @param now rows retained no later than this are removed
* @param limit the most rows one statement removes
* @return how many went
*/
// Bounded by the default transaction bound: a statement stuck behind a lock gives the connection
// back instead of holding it until the pool is empty.
@Transactional(timeout = SpringTransaction.DEFAULT_TIMEOUT_SECONDS)
@Modifying
@Query("delete from LiveEventEntity e where e.retainedUntil <= :now")
int sweepExpired(@Param("now") Instant now);
@Query(
value =
"DELETE FROM live_event_log WHERE (stream_id, position) IN ("
+ "SELECT stream_id, position FROM live_event_log WHERE retained_until <= :now"
+ " FOR UPDATE SKIP LOCKED LIMIT :limit)",
nativeQuery = true)
int deleteExpiredBatch(@Param("now") Instant now, @Param("limit") int limit);
}
@@ -0,0 +1,53 @@
package dev.caskeleton.adapter.outbound.persistence.liveevent;
import dev.caskeleton.adapter.outbound.persistence.transaction.BatchedDeletes;
import java.time.Clock;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
/**
* Periodically deletes {@code live_event_log} rows past their retention.
*
* <p>A replay read already filters on {@code retained_until}, so an expired row is never served;
* without this nothing ever removed one, and the table grew by every event ever published for as
* long as replay was on (round9 review 2026-11-12, RR9-01-03). It is assembled with the replay
* store, as the idempotency and outbox reapers are with theirs, rather than left to each service to
* schedule: a duty that every deployment must remember is one some deployment forgets.
*
* <p>Deletes in batches of {@link #BATCH_SIZE}, each committed on its own, until a batch comes back
* short. Every replica runs it; the batch skips rows another replica's batch holds.
*
* <p>It carries no schedule of its own. The replay configuration registers {@link #reap()} at the
* bound {@link LiveEventReplaySettings#reaperInterval()}, so the interval reaches the schedule as a
* value rather than through a bean name an expression looks up. A deployment that builds a reaper
* itself schedules it too.
*/
public class LiveEventReaper {
/** Rows per delete statement and transaction. */
static final int BATCH_SIZE = BatchedDeletes.BATCH_SIZE;
private static final Logger log = LoggerFactory.getLogger(LiveEventReaper.class);
private final LiveEventJpaRepository repository;
private final Clock clock;
public LiveEventReaper(LiveEventJpaRepository repository, Clock clock) {
this.repository = repository;
this.clock = clock;
}
/**
* Deletes every row retained no later than now; returns the count.
*
* <p>Not transactional itself: each batch is its own transaction.
*/
public int reap() {
int purged =
BatchedDeletes.untilShort(limit -> repository.deleteExpiredBatch(clock.instant(), limit));
if (purged > 0) {
log.debug("live event reaper purged {} expired row(s)", purged);
}
return purged;
}
}
@@ -0,0 +1,55 @@
package dev.caskeleton.adapter.outbound.persistence.liveevent;
import dev.caskeleton.adapter.outbound.support.StrictSettingsBinder;
import java.time.Duration;
import org.springframework.core.env.Environment;
/**
* The replay store's settings, bound strictly from {@code app.realtime.replay.*}: a misspelt key
* fails startup rather than leaving a default in force.
*
* <p>The reaper's interval was read only by the {@code @Scheduled} placeholder, with its default
* written twice and nothing to refuse a zero (round9 review 2026-11-12). It is bound here with the
* rest of the store's settings and read from this record by the reaper's schedule.
*
* @param enabled whether the replay store is assembled; the configuration's condition reads it
* @param retention how long an event stays replayable. No default: how long a reconnecting client
* may resume is the deployment's decision, made against what it is willing to store
* @param reaperInterval how often {@link LiveEventReaper} deletes rows past retention; 10 minutes
* unless set, which the module defaults mirror
*/
public record LiveEventReplaySettings(
boolean enabled, Duration retention, Duration reaperInterval) {
static final String PREFIX = "app.realtime.replay";
private static final Duration DEFAULT_REAPER_INTERVAL = Duration.ofMinutes(10);
public LiveEventReplaySettings {
if (retention == null) {
throw new IllegalArgumentException(
"app.realtime.replay.retention must be set when replay is enabled: how long a client may"
+ " resume is the deployment's decision");
}
if (retention.isNegative() || retention.isZero()) {
throw new IllegalArgumentException(
"app.realtime.replay.retention must be positive, was " + retention);
}
if (reaperInterval == null) {
reaperInterval = DEFAULT_REAPER_INTERVAL;
} else if (reaperInterval.isNegative() || reaperInterval.isZero()) {
throw new IllegalArgumentException(
"app.realtime.replay.reaper-interval must be positive, was " + reaperInterval);
}
}
/**
* Binds {@link #PREFIX} strictly.
*
* @param environment the environment to bind from
* @return the bound settings
*/
public static LiveEventReplaySettings bind(Environment environment) {
return StrictSettingsBinder.bind(environment, PREFIX, LiveEventReplaySettings.class);
}
}
@@ -5,41 +5,28 @@ import java.time.Duration;
import org.springframework.core.env.Environment;
/**
* Distributed-lock tuning knobs bound from {@code app.lock.*} (yaml-only defaults, not {@code
* APP_*} env keys). {@code waitTime} = max wait before timeout (default 3s); {@code leaseTtl} = max
* lock hold before auto-expiry (default 30s, must be ≥ {@code waitTime}). See README "lock".
* Distributed-lock tuning bound from {@code app.lock.*} (yaml-only defaults, not {@code APP_*} env
* keys). {@code leaseTtl} = the longest a lock may be held before the provider expires it (default
* 30s); a caller's {@code tryAcquire} may ask for that long or less. See README "lock".
*
* <p>There is no wait setting. How long to wait is every caller's own argument to {@code
* tryAcquire}; an {@code app.lock.wait-time} used to be bound here, checked against the lease, and
* read by nothing (round9 review 2026-11-12, RR9-01-07). It now fails startup as an unknown key, so
* a deployment that set it learns that it never did anything.
*/
public record LockSettings(Duration waitTime, Duration leaseTtl) {
public record LockSettings(Duration leaseTtl) {
static final String PREFIX = "app.lock";
private static final Duration DEFAULT_WAIT_TIME = Duration.ofSeconds(3);
private static final Duration DEFAULT_LEASE_TTL = Duration.ofSeconds(30);
public LockSettings {
if (waitTime == null) {
waitTime = DEFAULT_WAIT_TIME;
} else if (waitTime.isNegative() || waitTime.isZero()) {
throw new IllegalArgumentException(
"app.lock.wait-time must be positive (> 0), was " + waitTime);
}
if (leaseTtl == null) {
leaseTtl = DEFAULT_LEASE_TTL;
} else if (leaseTtl.isNegative() || leaseTtl.isZero()) {
throw new IllegalArgumentException(
"app.lock.lease-ttl must be positive (> 0), was " + leaseTtl);
}
// Cross-field invariant: a lease shorter than the max wait is nonsensical.
// If the TTL fires before the waitTime elapses, a second holder could acquire
// the lock before the first holder's protected work is complete.
if (leaseTtl.compareTo(waitTime) < 0) {
throw new IllegalArgumentException(
"app.lock.lease-ttl ("
+ leaseTtl
+ ") must be >= app.lock.wait-time ("
+ waitTime
+ ") — a lease shorter than the max wait is nonsensical");
}
}
/**
@@ -199,10 +199,10 @@ app:
# true | false (whether to run Flyway migrations automatically at startup)
migration-on-startup: ${APP_MIGRATION_ON_STARTUP:true}
# feature-distributed-lock-contract D5 — distributed lock acquisition contract (try-lock +
# finite wait + lease TTL). Bound to LockSettings. Plain values, not APP_* env keys; the code
# defaults in LockSettings mirror these.
# finite wait + lease TTL). Bound to LockSettings. A plain value, not an APP_* env key; the code
# default in LockSettings mirrors it. The wait is each caller's tryAcquire argument, so there is
# no wait setting (app.lock.wait-time was read by nothing and is now an unknown key).
lock:
wait-time: 3s
lease-ttl: 30s
realtime:
replay:
@@ -210,6 +210,10 @@ app:
# events into the store and exposes replay/resume semantics.
enabled: ${APP_REALTIME_REPLAY_ENABLED:false}
retention: ${APP_REALTIME_REPLAY_RETENTION:1h}
# ISO-8601 duration — how often LiveEventReaper deletes rows past retention, 1 000 a batch.
# A plain value like the outbox reaper's interval. Bound strictly by LiveEventReplaySettings,
# whose default mirrors it; zero or negative fails startup.
reaper-interval: PT10M
outbox:
# feature-domain-event-outbox-contract I11 — the tuning values are literal defaults;
# NO env placeholders (spec: 신규 env key 없음). Declared, validated and strictly bound by
@@ -1,5 +1,7 @@
package dev.caskeleton.adapter.outbound.persistence.liveevent;
import static org.assertj.core.api.Assertions.assertThat;
import dev.caskeleton.adapter.outbound.persistence.liveevent.entity.LiveEventEntity;
import dev.caskeleton.adapter.outbound.persistence.platform.JpaPlatformContractSupport;
import dev.caskeleton.adapter.outbound.persistence.transaction.SpringTransaction;
@@ -13,12 +15,17 @@ import java.nio.charset.StandardCharsets;
import java.sql.Connection;
import java.sql.SQLException;
import java.sql.Statement;
import java.time.Clock;
import java.time.Duration;
import java.time.Instant;
import java.time.ZoneOffset;
import java.util.Map;
import java.util.UUID;
import javax.sql.DataSource;
import org.junit.jupiter.api.AfterAll;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.Tag;
import org.junit.jupiter.api.Test;
import org.springframework.context.annotation.AnnotationConfigApplicationContext;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
@@ -98,6 +105,40 @@ class JpaLiveEventReplayContractTest extends LiveEventReplayContract {
return RETENTION;
}
@Test
void theReaperRemovesRowsPastTheirRetentionAndKeepsTheRest() {
// given
String stream = "reaper-" + UUID.randomUUID();
Instant start = Instant.parse("2026-11-03T00:00:00Z");
appendAt(stream, "expired", start);
appendAt(stream, "retained", start.plus(RETENTION).minusSeconds(60));
LiveEventReaper reaper =
new LiveEventReaper(
context.getBean(LiveEventJpaRepository.class),
Clock.fixed(start.plus(RETENTION).plusSeconds(1), ZoneOffset.UTC));
// when
var unused = reaper.reap();
// then
assertThat(
new JdbcTemplate(support.dataSource())
.queryForList(
"SELECT payload FROM live_event_log WHERE stream_id = ?", String.class, stream))
.containsExactly("retained");
}
private void appendAt(String streamId, String payload, Instant at) {
JpaLiveEventReplay writer =
new JpaLiveEventReplay(
context.getBean(LiveEventJpaRepository.class),
new JdbcTemplate(support.dataSource()),
Clock.fixed(at, ZoneOffset.UTC),
RETENTION);
var unused =
context.getBean(TransactionPort.class).inWrite(() -> writer.append(streamId, payload));
}
private static void execute(String sql) throws SQLException {
try (Connection connection = support.connection();
Statement statement = connection.createStatement()) {
@@ -49,7 +49,8 @@ final class NotificationFixtures {
migration("jpa/notification-platform", "V10__variables_payload_envelope_guard.sql"),
migration("jpa/notification-platform", "V11__notification_reconciliation_claim_lease.sql"),
migration("jpa/notification-platform", "V12__notification_suppression_target_unique.sql"),
migration("jpa/notification-platform", "V13__notification_request_acceptance.sql"));
migration("jpa/notification-platform", "V13__notification_request_acceptance.sql"),
migration("jpa/notification-platform", "V14__notification_provider_event_open_index.sql"));
}
/**
@@ -8,6 +8,7 @@ import dev.caskeleton.application.notification.platform.observation.Notification
import java.io.IOException;
import java.sql.Connection;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;
import java.sql.Timestamp;
@@ -15,6 +16,9 @@ import java.time.Clock;
import java.time.Duration;
import java.time.Instant;
import java.time.ZoneOffset;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
import java.util.UUID;
import javax.sql.DataSource;
import org.junit.jupiter.api.AfterAll;
@@ -23,7 +27,12 @@ import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.DisplayName;
import org.junit.jupiter.api.Tag;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.CsvSource;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.jdbc.core.PreparedStatementCreator;
import org.springframework.jdbc.core.RowMapper;
import org.springframework.jdbc.core.SqlProvider;
import org.springframework.jdbc.datasource.SingleConnectionDataSource;
/**
@@ -100,6 +109,39 @@ class ServingStateContractTest {
assertThat(measure().backlogDepth()).isZero();
}
@Test
void aDeliveryWhoseWindowClosedIsNotBacklog() throws SQLException {
// given
// Due, but past its expiry: the claim refuses it and the expiry sweep retires it unsent.
insertDelivery(
"READY_TO_DISPATCH", NOW.minus(Duration.ofMinutes(5)), null, null, NOW.minusSeconds(1));
// when
long depth = measure().backlogDepth();
// then
assertThat(depth).isZero();
}
@ParameterizedTest(name = "[{index}] lease_until = NOW에서 {0}초 → backlog {1}")
@CsvSource({"-1, 1", "0, 0"})
@DisplayName("lease는 NOW 전에 끝났을 때만 끝난 것이다: claim과 같은 lease_until < now")
void aLeaseHasRunOutOnlyOnceItEndedBeforeNow(long offsetSeconds, long expectedDepth)
throws SQLException {
// given
insertDelivery(
"READY_TO_DISPATCH",
NOW.minus(Duration.ofMinutes(1)),
"worker-1",
NOW.plusSeconds(offsetSeconds));
// when
long depth = measure().backlogDepth();
// then
assertThat(depth).isEqualTo(expectedDepth);
}
@Test
@DisplayName("a delivery held under a live lease is not backlog")
void aDeliveryUnderALiveLeaseIsNotBacklog() throws SQLException {
@@ -113,22 +155,24 @@ class ServingStateContractTest {
}
@Test
@DisplayName("a lease whose holder never came back is counted as stuck")
void anExpiredLeaseIsStuck() throws SQLException {
// given
insertDelivery(
"DISPATCHING",
NOW.minus(Duration.ofMinutes(20)),
"worker-1",
NOW.minus(Duration.ofMinutes(10)));
// when
NotificationServingState state = measure();
assertThat(state.stuckLeases())
.as("a worker died mid-dispatch, so a provider call may have happened with no record")
.isEqualTo(1);
assertThat(state.backlogDepth())
.as("an expired lease is claimable again, but this row is DISPATCHING, not due")
.isZero();
// then
assertThat(state)
.as(
"a worker died mid-dispatch, so a provider call may have happened with no record; a"
+ " dispatching row is never claimed, so it is not backlog")
.extracting(NotificationServingState::stuckLeases, NotificationServingState::backlogDepth)
.containsExactly(1L, 0L);
}
@Test
@@ -162,6 +206,108 @@ class ServingStateContractTest {
assertThat(state.oldestReconciliationAge()).isEqualTo(Duration.ofHours(2));
}
@Test
void theSchedulersBacklogReadIsOneStatement() throws SQLException {
// given
insertDelivery("READY_TO_DISPATCH", NOW.minus(Duration.ofMinutes(30)), null, null);
// when
List<Object> outcome;
try (Connection connection = support.connection()) {
RecordingJdbc jdbc = new RecordingJdbc(new SingleConnectionDataSource(connection, true));
long depth = new JdbcNotificationServingState(jdbc, CLOCK).backlogDepth();
outcome = List.of(depth, jdbc.statements.size());
}
// then
assertThat(outcome)
.as("the scheduler asks this on every poll; the full snapshot is four statements")
.containsExactly(1L, 1);
}
@Test
void theProjectionLagReadsOpenEventsThroughTheirIndex() throws SQLException {
// given
insertProviderEvent("PENDING", NOW.minus(Duration.ofMinutes(45)), false);
// when
String plan;
try (Connection connection = support.connection()) {
RecordingJdbc jdbc = new RecordingJdbc(new SingleConnectionDataSource(connection, true));
var unused = new JdbcNotificationServingState(jdbc, CLOCK).currentState();
RecordedStatement lag =
jdbc.statements.stream()
.filter(statement -> statement.sql().contains("notification_provider_event"))
.findFirst()
.orElseThrow();
plan = explainWithoutSequentialScans(connection, lag);
}
// then
assertThat(plan)
.as("the event table is never purged; only the index over the open events bounds this read")
.contains("ix_notification_provider_event_open");
}
/** The plan PostgreSQL picks for {@code statement} when it may not scan a table sequentially. */
private static String explainWithoutSequentialScans(
Connection connection, RecordedStatement statement) throws SQLException {
boolean autoCommit = connection.getAutoCommit();
connection.setAutoCommit(false);
try {
try (Statement off = connection.createStatement()) {
off.execute("SET LOCAL enable_seqscan = off");
}
StringBuilder plan = new StringBuilder();
try (PreparedStatement explain = connection.prepareStatement("EXPLAIN " + statement.sql())) {
for (int index = 0; index < statement.args().size(); index++) {
explain.setObject(index + 1, statement.args().get(index));
}
try (ResultSet rows = explain.executeQuery()) {
while (rows.next()) {
plan.append(rows.getString(1)).append('\n');
}
}
}
return plan.toString();
} finally {
connection.rollback();
connection.setAutoCommit(autoCommit);
}
}
/** A statement the serving state ran, as it ran it. */
private record RecordedStatement(String sql, List<Object> args) {}
/** Plain JDBC that keeps the statements it was given, so a case reads the shipped SQL. */
private static final class RecordingJdbc extends JdbcTemplate {
private final List<RecordedStatement> statements = new ArrayList<>();
private RecordingJdbc(DataSource dataSource) {
super(dataSource);
}
@Override
public <T> T queryForObject(String sql, RowMapper<T> rowMapper, Object... args) {
statements.add(new RecordedStatement(sql, Arrays.asList(args)));
return super.queryForObject(sql, rowMapper, args);
}
@Override
public <T> T queryForObject(String sql, Class<T> requiredType, Object... args) {
statements.add(new RecordedStatement(sql, Arrays.asList(args)));
return super.queryForObject(sql, requiredType, args);
}
/** The named-parameter backlog read arrives here; its parameters stay inside the creator. */
@Override
public <T> List<T> query(PreparedStatementCreator creator, RowMapper<T> rowMapper) {
statements.add(new RecordedStatement(((SqlProvider) creator).getSql(), List.of()));
return super.query(creator, rowMapper);
}
}
private NotificationServingState measure() throws SQLException {
// try-with-resources: the pool holds four connections, and a measurement that keeps one is a
// measurement that starves the next test.
@@ -174,6 +320,16 @@ class ServingStateContractTest {
private void insertDelivery(
String state, Instant nextDispatchAt, String leaseOwner, Instant leaseUntil)
throws SQLException {
insertDelivery(state, nextDispatchAt, leaseOwner, leaseUntil, null);
}
private void insertDelivery(
String state,
Instant nextDispatchAt,
String leaseOwner,
Instant leaseUntil,
Instant expiresAt)
throws SQLException {
UUID requestId = UUID.randomUUID();
try (Connection connection = support.connection()) {
try (PreparedStatement statement =
@@ -197,17 +353,18 @@ class ServingStateContractTest {
+ " id, notification_id, tenant_id, recipient_ref, routing_plan_json,"
+ " route_cursor, delivery_state, submission_outcome, delivery_outcome,"
+ " evidence_level, attempt_count, next_dispatch_at, lease_owner, lease_until,"
+ " created_at, updated_at)"
+ " expires_at, created_at, updated_at)"
+ " VALUES (?, ?, 'tenant-a', 'user-1', '[]'::jsonb, 0, ?, 'NOT_SUBMITTED',"
+ " 'UNKNOWN', 'PLATFORM_QUEUED', 0, ?, ?, ?, ?, ?)")) {
+ " 'UNKNOWN', 'PLATFORM_QUEUED', 0, ?, ?, ?, ?, ?, ?)")) {
statement.setObject(1, UUID.randomUUID());
statement.setObject(2, requestId);
statement.setString(3, state);
statement.setTimestamp(4, Timestamp.from(nextDispatchAt));
statement.setString(5, leaseOwner);
statement.setTimestamp(6, leaseUntil == null ? null : Timestamp.from(leaseUntil));
statement.setTimestamp(7, Timestamp.from(NOW));
statement.setTimestamp(7, expiresAt == null ? null : Timestamp.from(expiresAt));
statement.setTimestamp(8, Timestamp.from(NOW));
statement.setTimestamp(9, Timestamp.from(NOW));
statement.executeUpdate();
}
}
@@ -2,12 +2,16 @@ package dev.caskeleton.adapter.outbound.persistence.platform;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatThrownBy;
import static org.assertj.core.api.Assertions.catchThrowable;
import dev.caskeleton.adapter.outbound.persistence.hibernate.stateless.HibernateStatelessSessionRunner;
import dev.caskeleton.adapter.outbound.persistence.hibernate.stateless.StatelessRowCapExceededException;
import dev.caskeleton.adapter.outbound.persistence.hibernate.stateless.StatelessWorkName;
import dev.caskeleton.adapter.outbound.persistence.hibernate.stateless.StatelessWorkResult;
import dev.caskeleton.adapter.outbound.persistence.testkit.id.SequenceEntity;
import java.time.Duration;
import java.util.List;
import java.util.Locale;
import java.util.Set;
import org.junit.jupiter.api.AfterAll;
import org.junit.jupiter.api.BeforeAll;
@@ -104,6 +108,53 @@ class HibernateStatelessSessionRunnerIntegrationTest {
.hasMessageContaining("positive row cap");
}
@Test
void workStillRunningWhenTheTransactionTimeIsUpIsCancelledAndRolledBack() {
// given
var runner =
new HibernateStatelessSessionRunner(
jpa.sessionFactory(), Set.of(WORK), Duration.ofSeconds(1));
long before = countRows();
// when
Throwable failure =
catchThrowable(
() ->
runner.execute(
WORK,
10,
session -> {
session.insert(new SequenceEntity("timed-out"));
var unused =
session
.createNativeQuery("SELECT 1 FROM pg_sleep(5)", Integer.class)
.getSingleResult();
return StatelessWorkResult.of(1, 1);
}));
// then
assertThat(List.of(failure != null && causedByTimeout(failure), countRows()))
.as(
"the transaction's bound cancels the statement; nothing the work wrote is kept: %s",
failure)
.containsExactly(true, before);
}
/** Whether the failure is the database or Hibernate stopping work that ran out of time. */
private static boolean causedByTimeout(Throwable failure) {
for (Throwable current = failure; current != null; current = current.getCause()) {
String message = String.valueOf(current.getMessage()).toLowerCase(Locale.ROOT);
if (current instanceof org.hibernate.QueryTimeoutException
|| current instanceof jakarta.persistence.QueryTimeoutException
|| message.contains("canceling statement due to statement timeout")
|| message.contains("canceling statement due to user request")
|| message.contains("transaction timeout expired")) {
return true;
}
}
return false;
}
private static long countRows() {
return jpa.inTransaction(
entityManager ->
@@ -2,6 +2,7 @@ package dev.caskeleton.adapter.outbound.persistence.platform.experimental;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatThrownBy;
import static org.assertj.core.api.Assertions.catchThrowable;
import com.zaxxer.hikari.HikariConfig;
import com.zaxxer.hikari.HikariDataSource;
@@ -12,6 +13,7 @@ import dev.caskeleton.application.tenant.TenantId;
import java.sql.Connection;
import java.sql.SQLException;
import java.sql.Statement;
import java.util.ArrayList;
import java.util.List;
import org.junit.jupiter.api.AfterAll;
import org.junit.jupiter.api.BeforeAll;
@@ -51,9 +53,9 @@ class TenantPoolCapacityContractTest {
void budgetRefusesTheTenantThatWouldExceedTheCeiling() {
var budget = new TenantPoolBudget(4, 8);
budget.requireCapacity(3, 6);
assertThat(budget.hasCapacity(4, 8)).isFalse();
assertThatThrownBy(() -> budget.requireCapacity(4, 8))
budget.requireCapacity(3, 6, 2);
assertThat(budget.hasCapacity(3, 7, 2)).isFalse();
assertThatThrownBy(() -> budget.requireCapacity(3, 7, 2))
.isInstanceOf(IllegalStateException.class);
}
@@ -77,30 +79,50 @@ class TenantPoolCapacityContractTest {
void openingPastTheBudgetIsRefusedBeforeAPoolIsCreated() {
var budget = new TenantPoolBudget(2, 4);
try (var registry =
new TenantDataSourceRegistry(
TenantPoolCapacityContractTest::poolFor,
TenantPoolCapacityContractTest::poolSizeOf,
budget)) {
new TenantDataSourceRegistry(this::poolFor, tenant -> POOL_SIZE_PER_TENANT, budget)) {
registry.openTenants(2);
assertThat(registry.openPools()).isEqualTo(2);
assertThat(registry.allocatedConnections()).isEqualTo(4);
assertThatThrownBy(() -> registry.openTenants(3)).isInstanceOf(IllegalStateException.class);
assertThat(registry.openPools())
.as("a refused tenant must not leave a half-built pool behind")
.isEqualTo(2);
assertThat(List.of(registry.openPools(), opened.size()))
.as("a refused tenant must not leave a half-built pool behind, or have one built at all")
.containsExactly(2, 2);
}
}
@Test
void aTenantWhosePoolWouldPassTheConnectionCeilingIsRefusedBeforeItsPoolOpens() {
// given
// Two pools of two use four of five connections; the third pool would make six.
var budget = new TenantPoolBudget(4, 5);
List<Object> outcome;
try (var registry =
new TenantDataSourceRegistry(this::poolFor, tenant -> POOL_SIZE_PER_TENANT, budget)) {
registry.openTenants(2);
// when
Throwable refused = catchThrowable(() -> registry.require(new TenantId("tenant-2")));
outcome =
List.of(
refused instanceof IllegalStateException,
registry.openPools(),
registry.allocatedConnections(),
opened.size());
}
// then
assertThat(outcome)
.as("the third pool is refused on its planned size, so it never opens a connection")
.containsExactly(true, 2, 4, 2);
}
@Test
@DisplayName("every opened tenant pool reaches the real server")
void everyOpenedTenantPoolReachesTheServer() throws SQLException {
var budget = new TenantPoolBudget(3, 6);
try (var registry =
new TenantDataSourceRegistry(
TenantPoolCapacityContractTest::poolFor,
TenantPoolCapacityContractTest::poolSizeOf,
budget)) {
new TenantDataSourceRegistry(this::poolFor, tenant -> POOL_SIZE_PER_TENANT, budget)) {
registry.openTenants(3);
for (TenantId tenant : List.copyOf(registry.openTenantIds())) {
@@ -118,10 +140,7 @@ class TenantPoolCapacityContractTest {
void evictingATenantReturnsItsConnectionsToTheBudget() {
var budget = new TenantPoolBudget(2, 4);
try (var registry =
new TenantDataSourceRegistry(
TenantPoolCapacityContractTest::poolFor,
TenantPoolCapacityContractTest::poolSizeOf,
budget)) {
new TenantDataSourceRegistry(this::poolFor, tenant -> POOL_SIZE_PER_TENANT, budget)) {
registry.openTenants(2);
TenantId evicted = List.copyOf(registry.openTenantIds()).get(0);
@@ -133,18 +152,19 @@ class TenantPoolCapacityContractTest {
}
}
private static javax.sql.DataSource poolFor(TenantId tenant) {
/** Every pool this test's factory built, so a test can see one was never opened. */
private final List<HikariDataSource> opened = new ArrayList<>();
private javax.sql.DataSource poolFor(TenantId tenant) {
var config = new HikariConfig();
config.setJdbcUrl(support.jdbcUrl());
config.setUsername(support.username());
config.setPassword(support.password());
config.setMaximumPoolSize(POOL_SIZE_PER_TENANT);
config.setPoolName("tenant-" + tenant.value());
return new HikariDataSource(config);
}
private static int poolSizeOf(javax.sql.DataSource pool) {
return ((HikariDataSource) pool).getMaximumPoolSize();
HikariDataSource pool = new HikariDataSource(config);
opened.add(pool);
return pool;
}
private static int serverMaxConnections() throws SQLException {
@@ -104,12 +104,14 @@ final class PostgreSqlOptionalStreamLifecycle {
// CONFLICT instead of a find-then-insert two webhooks could both win.
// V13 records whether a stored request was accepted or dropped by a deduplication window,
// so a retry with the same idempotency key gets the same receipt (review 05-23c).
// V14 indexes the provider events still waiting for something, so the projection lag reads
// them and not the whole history (round9 review 2026-11-12, RR9-01-14).
//
// This list is the stream's contract, not a note about its length: the lifecycle below
// disables, re-enables and interrupts the stream and asserts the applied set is unchanged
// each time. So a migration added to the stream belongs here, and the version that landed
// without being added is why the lane failed the first time anybody ran it.
List.of("0", "1", "2", "3", "4", "5", "6", "7", "8", "9", "10", "11", "12", "13"),
List.of("0", "1", "2", "3", "4", "5", "6", "7", "8", "9", "10", "11", "12", "13", "14"),
List.of());
}
@@ -8,9 +8,13 @@ import java.util.Map;
/**
* The SQL a runbook gives an operator, read out of the runbook itself. Every lane that runs a
* runbook's SQL reads it through this one class: so far the outbox runbooks and the migration
* runbook's recovery of a failed concurrent index migration (re-review 2026-11-03 round 8,
* RR8-03-03: the class kept an outbox name after the migration runbook began to use it).
* runbook's SQL reads it through this one class: so far the outbox runbooks ({@code
* PostgreSqlOutboxDispositionRunbookIntegrationTest}, {@code
* PostgreSqlOutboxOrderKeyRunbookIntegrationTest}), the migration runbook's recovery of a failed
* concurrent index migration ({@code PostgreSqlConcurrentIndexMigrationIntegrationTest}) and the
* lock-timeout runbook's holder lookup ({@code PostgreSqlDistributedLockIntegrationTest}).
* Re-review 2026-11-03 round 8, RR8-03-03: the class kept an outbox name after the migration
* runbook began to use it; round9 review 2026-11-12, RR9-01-10: this list missed the lock lane.
*
* <p>A runbook block is marked {@code <!-- runbook-sql: id -->} and followed by a {@code sql}
* fence. The lanes that run these blocks against real PostgreSQL read them from the document rather
@@ -1,18 +1,34 @@
package dev.caskeleton.adapter.outbound.persistence.autoconfigure;
import static org.assertj.core.api.Assertions.assertThat;
import static org.mockito.ArgumentMatchers.any;
import static org.mockito.ArgumentMatchers.anyInt;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.verify;
import dev.caskeleton.adapter.outbound.persistence.liveevent.JpaLiveEventReplay;
import dev.caskeleton.adapter.outbound.persistence.liveevent.LiveEventJpaRepository;
import dev.caskeleton.adapter.outbound.persistence.liveevent.LiveEventReaper;
import java.time.Clock;
import java.time.Duration;
import java.time.Instant;
import java.util.ArrayList;
import java.util.List;
import org.junit.jupiter.api.DisplayName;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.CsvSource;
import org.springframework.boot.context.properties.bind.UnboundConfigurationPropertiesException;
import org.springframework.boot.convert.ApplicationConversionService;
import org.springframework.boot.test.context.runner.ApplicationContextRunner;
import org.springframework.context.ApplicationContext;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.Import;
import org.springframework.jdbc.core.JdbcOperations;
import org.springframework.scheduling.config.FixedDelayTask;
import org.springframework.scheduling.config.ScheduledTask;
import org.springframework.scheduling.config.ScheduledTaskHolder;
class JpaLiveEventReplayActivationTest {
@@ -34,6 +50,110 @@ class JpaLiveEventReplayActivationTest {
});
}
@Test
void theReaperIsAssembledWithTheReplayStore() {
runner
.withPropertyValues(
"app.realtime.replay.enabled=true", "app.realtime.replay.retention=PT1H")
.run(
context ->
assertThat(context)
.as("without the reaper nothing removes a row past its retention")
.hasSingleBean(LiveEventReaper.class));
}
@Test
void noReaperIsAssembledWithoutTheReplayStore() {
runner.run(context -> assertThat(context).doesNotHaveBean(LiveEventReaper.class));
}
@Test
void aZeroReaperIntervalFailsStartup() {
runner
.withPropertyValues(
"app.realtime.replay.enabled=true",
"app.realtime.replay.retention=PT1H",
"app.realtime.replay.reaper-interval=PT0S")
.run(
context ->
assertThat(context.getStartupFailure())
.rootCause()
.hasMessageContaining("app.realtime.replay.reaper-interval must be positive"));
}
@Test
void aMisspeltReplayKeyFailsStartup() {
runner
.withPropertyValues(
"app.realtime.replay.enabled=true",
"app.realtime.replay.retention=PT1H",
"app.realtime.replay.reaper-intervall=PT1M")
.run(
context ->
assertThat(context.getStartupFailure())
.rootCause()
.isInstanceOf(UnboundConfigurationPropertiesException.class)
.hasMessageContaining("reaper-intervall"));
}
@ParameterizedTest(name = "[{index}] reaper-interval {0} → {1}")
@CsvSource(
value = {"UNSET, PT10M", "PT3M, PT3M"},
nullValues = "UNSET")
@DisplayName("reaper는 묶인 reaper-interval(기본 PT10M)마다 돌고, 첫 회도 그 간격 뒤다")
void theReaperRunsAtTheBoundInterval(String configured, Duration expected) {
// given
ApplicationContextRunner replay =
runner.withPropertyValues(
"app.realtime.replay.enabled=true", "app.realtime.replay.retention=PT1H");
if (configured != null) {
replay = replay.withPropertyValues("app.realtime.replay.reaper-interval=" + configured);
}
// when
List<List<Duration>> schedules = new ArrayList<>();
replay.run(context -> schedules.addAll(fixedDelaySchedules(context)));
// then
assertThat(schedules)
.as("the schedule is the bound settings' interval, its default included")
.containsExactly(List.of(expected, expected));
}
@Test
void theScheduledTaskIsTheReaper() {
// given
ApplicationContextRunner replay =
runner.withPropertyValues(
"app.realtime.replay.enabled=true", "app.realtime.replay.retention=PT1H");
// when
List<LiveEventJpaRepository> reapedThrough = new ArrayList<>();
replay.run(
context -> {
fixedDelayTasks(context).forEach(task -> task.getRunnable().run());
reapedThrough.add(context.getBean(LiveEventJpaRepository.class));
});
// then
verify(reapedThrough.getFirst()).deleteExpiredBatch(any(Instant.class), anyInt());
}
/** Interval and initial delay of every fixed-delay task the context scheduled. */
private static List<List<Duration>> fixedDelaySchedules(ApplicationContext context) {
return fixedDelayTasks(context).stream()
.map(task -> List.of(task.getIntervalDuration(), task.getInitialDelayDuration()))
.toList();
}
private static List<FixedDelayTask> fixedDelayTasks(ApplicationContext context) {
return context.getBean(ScheduledTaskHolder.class).getScheduledTasks().stream()
.map(ScheduledTask::getTask)
.filter(FixedDelayTask.class::isInstance)
.map(FixedDelayTask.class::cast)
.toList();
}
@Test
void enabledReplayCapabilityRequiresExplicitRetention() {
runner
@@ -42,7 +162,8 @@ class JpaLiveEventReplayActivationTest {
context -> {
assertThat(context).hasFailed();
assertThat(context.getStartupFailure())
.hasMessageContaining("app.realtime.replay.retention");
.rootCause()
.hasMessageContaining("app.realtime.replay.retention must be set");
});
}
@@ -2,6 +2,7 @@ package dev.caskeleton.adapter.outbound.persistence.experimental;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatThrownBy;
import static org.assertj.core.api.Assertions.catchThrowable;
import dev.caskeleton.adapter.outbound.persistence.experimental.database.TenantPoolBudget;
import dev.caskeleton.adapter.outbound.persistence.experimental.replica.ConsistencyAwareDataSourceRouter;
@@ -103,14 +104,28 @@ class ExperimentalFailClosedTest {
void tenantPoolBudgetIsBounded() {
var budget = new TenantPoolBudget(2, 20);
assertThat(budget.hasCapacity(1, 10)).isTrue();
assertThatThrownBy(() -> budget.requireCapacity(2, 10))
assertThat(budget.hasCapacity(1, 10, 10)).isTrue();
assertThatThrownBy(() -> budget.requireCapacity(2, 10, 1))
.isInstanceOf(IllegalStateException.class)
.hasMessageContaining("tenant pool budget");
assertThatThrownBy(() -> budget.requireCapacity(1, 20))
assertThatThrownBy(() -> budget.requireCapacity(1, 20, 1))
.isInstanceOf(IllegalStateException.class);
}
@Test
void aNewPoolLargerThanTheConnectionsLeftIsRefused() {
// given
var budget = new TenantPoolBudget(4, 20);
// when
Throwable refused = catchThrowable(() -> budget.requireCapacity(1, 15, 10));
// then
assertThat(refused)
.isInstanceOf(IllegalStateException.class)
.hasMessageContaining("tenant pool budget exhausted");
}
/** A lag monitor with a fixed answer, including "I do not know". */
private record FixedLagMonitor(Optional<Instant> replayedThrough, Optional<Duration> lag)
implements ReplicaLagMonitor {}
@@ -0,0 +1,45 @@
package dev.caskeleton.adapter.outbound.persistence.hibernate.stateless;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.catchThrowable;
import static org.mockito.Mockito.mock;
import dev.caskeleton.adapter.outbound.persistence.transaction.SpringTransaction;
import java.time.Duration;
import java.util.Set;
import org.hibernate.SessionFactory;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.ValueSource;
/**
* The stateless runner's transaction bound is the write bound every other transaction here has.
*
* <p>The runner restates the value instead of reading it, because the {@code hibernate} package may
* not depend on {@code transaction} (JpaModuleBoundaryTest). A test may depend on both, so it is
* the place the two copies are held to each other (round9 review 2026-11-12, RR9-01-25).
*/
class StatelessRunnerTimeoutTest {
@Test
void theRunnersDefaultIsSpringTransactionsWriteDefault() {
assertThat(HibernateStatelessSessionRunner.DEFAULT_TIMEOUT)
.isEqualTo(SpringTransaction.Timeouts.DEFAULT.write());
}
@ParameterizedTest
@ValueSource(strings = {"PT0S", "PT-1S", "PT1.5S", "PT2147483648S"})
void aTimeoutThatIsNotAWholePositiveNumberOfSecondsIsTheAssemblysFault(String timeout) {
// when
Throwable refused =
catchThrowable(
() ->
new HibernateStatelessSessionRunner(
mock(SessionFactory.class), Set.of(), Duration.parse(timeout)));
// then
assertThat(refused)
.isInstanceOf(IllegalStateException.class)
.hasMessageContaining("must be a positive whole number of seconds");
}
}
@@ -0,0 +1,53 @@
package dev.caskeleton.adapter.outbound.persistence.liveevent;
import static org.assertj.core.api.Assertions.assertThat;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.times;
import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.when;
import java.time.Clock;
import java.time.Instant;
import java.time.ZoneOffset;
import org.junit.jupiter.api.Test;
/**
* Unit tests for {@link LiveEventReaper}: rows retained no later than the clock's now go, in
* bounded batches until one comes back short.
*
* <p>The repository is a Spring Data boundary, so Mockito stands in for it. The statement runs
* against PostgreSQL in {@code JpaLiveEventReplayContractTest}.
*/
class LiveEventReaperTest {
private static final Instant NOW = Instant.parse("2026-11-13T00:00:00Z");
private static final int BATCH = LiveEventReaper.BATCH_SIZE;
private final LiveEventJpaRepository repository = mock(LiveEventJpaRepository.class);
private final LiveEventReaper reaper =
new LiveEventReaper(repository, Clock.fixed(NOW, ZoneOffset.UTC));
@Test
void reapKeepsDeletingBatchesUntilOneComesBackShortAndReturnsTheTotal() {
// given
when(repository.deleteExpiredBatch(NOW, BATCH)).thenReturn(BATCH, 4);
// when
int purged = reaper.reap();
// then
assertThat(purged).isEqualTo(BATCH + 4);
}
@Test
void eachBatchIsItsOwnDeleteStatement() {
// given
when(repository.deleteExpiredBatch(NOW, BATCH)).thenReturn(BATCH, 4);
// when
var unused = reaper.reap();
// then
verify(repository, times(2)).deleteExpiredBatch(NOW, BATCH);
}
}
@@ -49,6 +49,20 @@ class DistributedLockPersistenceConfigTest {
});
}
@Test
@DisplayName("읽는 곳이 없던 app.lock.wait-time은 모르는 키로 기동을 실패시킨다")
void theRemovedWaitTimeKeyFailsStartup() {
// when
contextRunner
.withPropertyValues("app.lock.wait-time=3s")
.run(
context -> {
// then
assertThat(context).hasFailed();
assertThat(context.getStartupFailure()).hasStackTraceContaining("wait-time");
});
}
@Test
void multiInstanceBuildsTheJdbcProviderAndNotTheInProcessProvider() {
contextRunner
@@ -14,6 +14,7 @@
be51fabd07220f9ef885534ba36ada5f1491baf24cdfb54d25d93a19ddc29926 jpa/notification-platform/V11__notification_reconciliation_claim_lease.sql
43f9429f91c07528154ec01b721649b2c25fa98857ac154c7c08f4655634657f jpa/notification-platform/V12__notification_suppression_target_unique.sql
d01aaf59f66834358abf18246faa9a9eb8dfa9f42097ca9d968c44edff564630 jpa/notification-platform/V13__notification_request_acceptance.sql
8a7dc3604d44f1438b39dfec3aa688f792e7363cd3c1b74bf0c799a6b53f9f0d jpa/notification-platform/V14__notification_provider_event_open_index.sql
1ac7cc1c2509868eb0ce202e1917bb36846f98f17ad4922abe3fac0ef128199b jpa/notification-platform/V1__notification_platform_core.sql
0add67aa23580e51712f84fabe9b09e08401ef708cfa1b3292b120fdb5c12fa2 jpa/notification-platform/V2__notification_platform_contact_template_policy.sql
f5b98598a59cc4376f5d233b410f8a5ae9f3cd94156291ebdf2d0da5a66f34ef jpa/notification-platform/V3__notification_platform_inbox_admin.sql
@@ -1,13 +1,17 @@
package dev.caskeleton.adapter.outbound.persistence.notification.platform;
import dev.caskeleton.adapter.outbound.persistence.notification.platform.repository.RecipientClaimSql;
import dev.caskeleton.application.notification.platform.observation.NotificationServingState;
import dev.caskeleton.application.notification.platform.observation.NotificationServingStatePort;
import java.sql.Timestamp;
import java.time.Clock;
import java.time.Duration;
import java.time.Instant;
import java.util.Map;
import java.util.Objects;
import org.springframework.jdbc.core.JdbcOperations;
import org.springframework.jdbc.core.namedparam.NamedParameterJdbcOperations;
import org.springframework.jdbc.core.namedparam.NamedParameterJdbcTemplate;
/**
* Measures the platform's serving state from the durable rows.
@@ -17,34 +21,44 @@ import org.springframework.jdbc.core.JdbcOperations;
* identical. These are the questions an operator actually asks during an incident: how far behind
* are we, how long has the oldest thing waited, and is anything stuck.
*
* <p>One statement per question rather than one joined statement. They are read independently, a
* slow one does not delay the others, and each index the queries rely on is the same index the
* corresponding worker already uses to claim.
* <p>One statement per question rather than one joined statement. They are read independently and a
* slow one does not delay the others. Every statement is bounded by an index: the backlog and
* stuck-lease counts by the predicates the claim uses, the reconciliation count by the job's due
* time, and the projection lag by {@code ix_notification_provider_event_open} (V14), which holds
* only the events still waiting for something — the provider-event table itself is never purged, so
* the lag query used to read all of it (round9 review 2026-11-12, RR9-01-14).
*
* <p>The scheduler reads only {@link #backlogDepth()}, on every poll; the full snapshot is for the
* health endpoint.
*/
public final class JdbcNotificationServingState implements NotificationServingStatePort {
/**
* Deliveries that are due and unclaimed.
* Deliveries the claim would take now.
*
* <p>The lease predicate matches the claim's: a row whose lease has expired is available again,
* so it is backlog. Counting it as in-flight is how a stuck worker hides a growing queue.
* <p>The {@code WHERE} is the claim's own fragment ({@link RecipientClaimSql#CLAIMABLE_AT_NOW}):
* the same states, due, not expired, and a lease that is absent or has run out. A row whose lease
* has expired is available again, so it is backlog; counting it as in-flight is how a stuck
* worker hides a growing queue. An expired delivery is not backlog: the expiry sweep retires it
* unsent. The fragment's parameter is named, so this statement is too.
*/
private static final String BACKLOG =
"""
SELECT count(*) AS depth,
COALESCE(EXTRACT(EPOCH FROM (? - min(next_dispatch_at))), 0) AS oldest_seconds
COALESCE(EXTRACT(EPOCH FROM (:now - min(next_dispatch_at))), 0) AS oldest_seconds
FROM notification_recipient_delivery
WHERE delivery_state IN ('PENDING', 'READY_TO_DISPATCH', 'RETRY_WAITING')
AND next_dispatch_at IS NOT NULL
AND next_dispatch_at <= ?
AND (lease_until IS NULL OR lease_until <= ?)
""";
WHERE\s"""
+ RecipientClaimSql.CLAIMABLE_AT_NOW;
/**
* Leases whose holder never came back.
*
* <p>Distinct from backlog: these rows are claimable again, but the fact that they had to expire
* means a worker died mid-dispatch, and a provider call may have happened without a record.
* <p>Distinct from backlog: these rows are still dispatching, which the claim never takes; lease
* recovery requeues or reconciles them one lease after they ran out. A row left from before
* notification V4 (an owner but fence 0) is counted here and not recovered; {@code
* docs/upgrade/2026-11-round9-claim-fence.md}, row 5, says how to hand it to recovery. That a
* lease had to run out means a worker died or stalled mid-dispatch, and a provider call may have
* happened without a record.
*/
private static final String STUCK_LEASES =
"""
@@ -56,7 +70,21 @@ public final class JdbcNotificationServingState implements NotificationServingSt
AND delivery_state = 'DISPATCHING'
""";
/** Provider events stored and not yet applied to a delivery. */
/**
* The provider events still waiting for something: the lag query's {@code WHERE}, and the
* predicate of the partial index {@code V14} builds for it. One string, held to the migration by
* {@code JdbcNotificationServingStateOpenEventsTest}: a query the index does not cover reads the
* whole table again, which is what the index was made to stop.
*/
static final String OPEN_EVENTS =
"projection_status IN ('PENDING', 'FAILED') OR attempt_id IS NULL";
/**
* Provider events stored and not yet applied to a delivery.
*
* <p>The {@code WHERE} is exactly the predicate of {@code ix_notification_provider_event_open}:
* every row any of the four counts can see, and none of the applied, matched history.
*/
private static final String PROJECTION_LAG =
"""
SELECT
@@ -66,7 +94,9 @@ public final class JdbcNotificationServingState implements NotificationServingSt
COALESCE(EXTRACT(EPOCH FROM (? - min(received_at)
FILTER (WHERE projection_status = 'PENDING'))), 0) AS oldest_seconds
FROM notification_provider_event
""";
WHERE
"""
+ OPEN_EVENTS;
/** Questions to providers that are due and unanswered. */
private static final String RECONCILIATION_LAG =
@@ -78,26 +108,36 @@ public final class JdbcNotificationServingState implements NotificationServingSt
""";
private final JdbcOperations jdbc;
private final NamedParameterJdbcOperations named;
private final Clock clock;
public JdbcNotificationServingState(JdbcOperations jdbc, Clock clock) {
this.jdbc = Objects.requireNonNull(jdbc, "jdbc");
this.named = new NamedParameterJdbcTemplate(jdbc);
this.clock = Objects.requireNonNull(clock, "clock");
}
@Override
public long backlogDepth() {
Measurement backlog = measureBacklog(Timestamp.from(clock.instant()));
return backlog == null ? 0 : backlog.count();
}
/** The backlog query alone: the one statement the scheduler's queue-depth gauge needs. */
private Measurement measureBacklog(Timestamp at) {
return named.queryForObject(
BACKLOG,
Map.of("now", at),
(rows, rowNumber) ->
new Measurement(rows.getLong("depth"), rows.getDouble("oldest_seconds")));
}
@Override
public NotificationServingState currentState() {
Instant now = clock.instant();
Timestamp at = Timestamp.from(now);
Measurement backlog =
jdbc.queryForObject(
BACKLOG,
(rows, rowNumber) ->
new Measurement(rows.getLong("depth"), rows.getDouble("oldest_seconds")),
at,
at,
at);
Measurement backlog = measureBacklog(at);
Long stuck = jdbc.queryForObject(STUCK_LEASES, Long.class, at);
ProjectionMeasurement projection =
jdbc.queryForObject(
@@ -0,0 +1,26 @@
-- An index over the provider events still waiting for something.
--
-- The serving-state projection lag counts pending, failed and unmatched events. It had no WHERE and
-- read the whole notification_provider_event table, which nothing ever purges, and the scheduler
-- asked for it on every poll (round9 review 2026-11-12, RR9-01-14). The query now names exactly
-- this predicate, so it reads only these rows; applied, matched events — the history that grows —
-- are outside the index.
--
-- Built inside this migration's transaction, not CONCURRENTLY, as V21 of the core stream builds its
-- index: a transactional file fails whole and leaves nothing to repair, where a failed concurrent
-- build leaves an INVALID index and a failed history row an operator must clear by hand
-- (docs/runbooks/migration-failed.md). The cost is that the build holds SHARE on
-- notification_provider_event: provider callbacks wait for it, and providers retry a callback that
-- does not get an answer. On a table already large in production, create the same index
-- CONCURRENTLY by hand before deploying; IF NOT EXISTS then makes this migration a no-op.
-- Bound the wait for the lock (SHARE on notification_provider_event, held until commit). Unbounded,
-- it queues behind any open transaction that touched the table, and every callback insert queues
-- behind the migration. Five seconds, then the migration fails, rolls back whole and the stream
-- stays at V13: deploy again once that transaction has ended (docs/jpa/migration-guide.md "Bound
-- the lock wait").
SET LOCAL lock_timeout = '5s';
CREATE INDEX IF NOT EXISTS ix_notification_provider_event_open
ON notification_provider_event (received_at)
WHERE projection_status IN ('PENDING', 'FAILED') OR attempt_id IS NULL;
@@ -0,0 +1,42 @@
package dev.caskeleton.adapter.outbound.persistence.notification.platform;
import static java.nio.charset.StandardCharsets.UTF_8;
import static org.assertj.core.api.Assertions.assertThat;
import java.io.IOException;
import java.io.InputStream;
import java.util.Objects;
import org.junit.jupiter.api.Test;
/**
* The projection-lag query and the partial index built for it name the same events.
*
* <p>The index is what keeps the lag query, which the scheduler runs on every poll, from reading
* the whole provider-event history. A predicate that drifted on either side would leave the query
* outside the index and scanning the table again, with nothing failing.
*/
class JdbcNotificationServingStateOpenEventsTest {
private static final String V14 =
"db/migration/jpa/notification-platform/V14__notification_provider_event_open_index.sql";
@Test
void theLagQueryAndItsPartialIndexNameTheSameEvents() throws IOException {
// given
String migration;
try (InputStream sql = getClass().getClassLoader().getResourceAsStream(V14)) {
migration = new String(Objects.requireNonNull(sql, V14).readAllBytes(), UTF_8);
}
int index = migration.indexOf("CREATE INDEX");
// when
String indexPredicate =
migration
.substring(
migration.indexOf("WHERE ", index) + "WHERE ".length(), migration.lastIndexOf(';'))
.trim();
// then
assertThat(indexPredicate).isEqualTo(JdbcNotificationServingState.OPEN_EVENTS);
}
}
@@ -12,7 +12,7 @@ import java.util.Objects;
* being applied reported exactly the same numbers as an idle healthy one. Readiness that cannot
* distinguish those two states is not readiness.
*
* @param backlogDepth deliveries due now and not yet claimed
* @param backlogDepth deliveries a claim would take now: due, not expired, not under a live lease
* @param oldestDueAge how long the oldest of them has been due
* @param stuckLeases leases whose holder has not finished and whose expiry has passed
* @param pendingProjections provider events stored and not yet applied
@@ -6,7 +6,25 @@ public interface NotificationServingStatePort {
/**
* Measure the backlog, the stuck leases and the projection and reconciliation lag.
*
* <p>Several questions, for the health snapshot. A caller that needs only the queue depth asks
* {@link #backlogDepth()} instead: the scheduler reads the depth on every poll, and asking all of
* them that often ran four statements a poll, one of them over the whole provider-event table
* (round9 review 2026-11-12, RR9-01-14).
*
* @return the current state
*/
NotificationServingState currentState();
/**
* How many deliveries a claim would take now — due, not expired, and not under a live lease: the
* {@link NotificationServingState#backlogDepth()} of {@link #currentState()}, measured alone.
*
* <p>The default takes it from the full snapshot, so an implementation that can only measure
* everything at once still answers; a store that can count the backlog by itself overrides it.
*
* @return the backlog depth
*/
default long backlogDepth() {
return currentState().backlogDepth();
}
}