From 4da5143906bdea2c08bb39ce44a8738b5a87f3d0 Mon Sep 17 00:00:00 2001 From: DongHyeonka Date: Fri, 13 Nov 2026 13:17:33 +0900 Subject: [PATCH] =?UTF-8?q?fix(persistence)!:=20=ED=8A=B8=EB=9E=9C?= =?UTF-8?q?=EC=9E=AD=EC=85=98=20=EC=8B=9C=ED=95=9C=EA=B3=BC=20=EC=A0=95?= =?UTF-8?q?=EB=A6=AC=20=EA=B2=BD=EB=A1=9C=EB=A5=BC=20=EB=8B=AB=EB=8A=94?= =?UTF-8?q?=EB=8B=A4?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 남은 트랜잭션·정리 결함 일곱. live event log는 지우는 곳이 없어 끝없이 자란다. app.lock.wait-time은 읽는 곳이 없다. 알림 scheduler는 poll마다 상태 쿼리 넷을 돌리고, projection 지연은 닫힌 event까지 읽는다. stateless runner 트랜잭션에 시한이 없다. tenant pool 예산은 새 pool의 크기를 모른 채 열어 연결 합계가 상한을 넘는다. runbook SQL 설명이 lane 넷 중 셋만 말한다. 고른 방법: LiveEventReaper(1000행씩, 트랜잭션 30초)와 공용 BatchedDeletes를 더한다. app.realtime.replay.*는 LiveEventReplaySettings가 엄격하게 묶고, 조립 설정이 SchedulingConfigurer로 reaper를 reaper-interval(기본 10분)마다 등록한다. LockSettings는 lease-ttl 하나만 받는다(모르는 키로 기동 실패). scheduler는 backlogDepth() 한 문장만 읽고, 그 WHERE는 claim과 같은 SQL 조각(RecipientClaimSql.CLAIMABLE_AT_NOW)이다. projection 지연은 V14 부분 인덱스로 읽는다(lock_timeout, V21 선례대로 CONCURRENTLY 아님). runner 기본 30초(SpringTransaction 쓰기 시한과 같음을 시험이 본다), tenant 예산은 계획된 pool 크기로 열기 전에 본다. 버린 대안: V14를 CONCURRENTLY로(Flyway 트랜잭션 안에서 못 돈다, 큰 DB는 안내대로 미리 만든다), reaper @Scheduled에 SpEL로 settings bean 이름 읽기(문자열 bean 조회는 검사가 못 본다). 위험·한계: LockSettings·TenantDataSourceRegistry·TenantPoolBudget의 공개 생성자가 바뀐다. V14는 큰 notification_provider_event에서 쓰기를 잠깐 막는다(안내 4행에 미리 만드는 법과 INVALID 인덱스 확인). backlog 수치는 창이 닫힌 delivery를 더는 세지 않는다. 검증: desktop RED(pr1-red3) 새 시험 실패 포함. GREEN은 PR 본문. Refs: RR9-01-03, RR9-01-07, RR9-01-09, RR9-01-10, RR9-01-14, RR9-01-25, RR9-01-26 BREAKING CHANGE: JpaLiveEventReplay.sweepExpired 삭제, app.realtime.replay 아래 모르는 키와 app.lock.wait-time은 기동 실패, HibernateStatelessSessionRunner 트랜잭션 30초 상한, TenantPoolBudget·TenantDataSourceRegistry 시그니처 변경. docs/upgrade/2026-11-round9-claim-fence.md --- docs/architecture/jpa-api-surface.txt | 4 +- docs/upgrade/2026-11-round9-claim-fence.md | 31 ++- .../dispatch/NotificationSchedulerWorker.java | 6 +- .../outbound/persistence-jpa/README.md | 16 ++ .../autoconfigure/JpaAdapterBeansConfig.java | 43 ++++- .../database/TenantDataSourceRegistry.java | 23 ++- .../database/TenantPoolBudget.java | 39 +++- .../HibernateStatelessSessionRunner.java | 47 ++++- .../liveevent/JpaLiveEventReplay.java | 10 - .../liveevent/LiveEventJpaRepository.java | 23 ++- .../liveevent/LiveEventReaper.java | 53 ++++++ .../liveevent/LiveEventReplaySettings.java | 55 ++++++ .../persistence/lock/LockSettings.java | 31 +-- .../module-defaults/persistence-jpa.yml | 10 +- .../JpaLiveEventReplayContractTest.java | 41 ++++ .../notification/NotificationFixtures.java | 3 +- .../ServingStateContractTest.java | 177 +++++++++++++++++- ...StatelessSessionRunnerIntegrationTest.java | 51 +++++ .../TenantPoolCapacityContractTest.java | 68 ++++--- .../PostgreSqlOptionalStreamLifecycle.java | 4 +- .../persistence/readiness/RunbookSql.java | 10 +- .../JpaLiveEventReplayActivationTest.java | 123 +++++++++++- .../ExperimentalFailClosedTest.java | 21 ++- .../stateless/StatelessRunnerTimeoutTest.java | 45 +++++ .../liveevent/LiveEventReaperTest.java | 53 ++++++ .../DistributedLockPersistenceConfigTest.java | 14 ++ .../resources/db/applied-migrations.sha256 | 1 + .../JdbcNotificationServingState.java | 88 ++++++--- ...notification_provider_event_open_index.sql | 26 +++ ...otificationServingStateOpenEventsTest.java | 42 +++++ .../observation/NotificationServingState.java | 2 +- .../NotificationServingStatePort.java | 18 ++ 32 files changed, 1038 insertions(+), 140 deletions(-) create mode 100644 src/adapter/outbound/persistence-jpa/src/main/java/dev/caskeleton/adapter/outbound/persistence/liveevent/LiveEventReaper.java create mode 100644 src/adapter/outbound/persistence-jpa/src/main/java/dev/caskeleton/adapter/outbound/persistence/liveevent/LiveEventReplaySettings.java create mode 100644 src/adapter/outbound/persistence-jpa/src/test/java/dev/caskeleton/adapter/outbound/persistence/hibernate/stateless/StatelessRunnerTimeoutTest.java create mode 100644 src/adapter/outbound/persistence-jpa/src/test/java/dev/caskeleton/adapter/outbound/persistence/liveevent/LiveEventReaperTest.java create mode 100644 src/adapter/outbound/persistence-notification/src/main/resources/db/migration/jpa/notification-platform/V14__notification_provider_event_open_index.sql create mode 100644 src/adapter/outbound/persistence-notification/src/test/java/dev/caskeleton/adapter/outbound/persistence/notification/platform/JdbcNotificationServingStateOpenEventsTest.java diff --git a/docs/architecture/jpa-api-surface.txt b/docs/architecture/jpa-api-surface.txt index 0063f347..06fc95b3 100644 --- a/docs/architecture/jpa-api-surface.txt +++ b/docs/architecture/jpa-api-surface.txt @@ -5,7 +5,7 @@ # root yet. # Update only after review with: # ./gradlew :adapter:outbound:persistence-jpa:updateJpaApiSurface -PapproveJpaApiSurfaceChange -# types: 166 +# types: 168 dev.caskeleton.adapter.outbound.persistence.api.PersistenceOperationName dev.caskeleton.adapter.outbound.persistence.api.capability.CapabilitySupport dev.caskeleton.adapter.outbound.persistence.api.capability.JpaCapability @@ -104,6 +104,8 @@ dev.caskeleton.adapter.outbound.persistence.idempotency.postgresql.IdempotencySc dev.caskeleton.adapter.outbound.persistence.idempotency.postgresql.PostgreSqlIdempotencyStore dev.caskeleton.adapter.outbound.persistence.liveevent.JpaLiveEventReplay dev.caskeleton.adapter.outbound.persistence.liveevent.LiveEventJpaRepository +dev.caskeleton.adapter.outbound.persistence.liveevent.LiveEventReaper +dev.caskeleton.adapter.outbound.persistence.liveevent.LiveEventReplaySettings dev.caskeleton.adapter.outbound.persistence.liveevent.entity.LiveEventEntity dev.caskeleton.adapter.outbound.persistence.lock.DistributedLockPersistenceConfig dev.caskeleton.adapter.outbound.persistence.lock.LockRegistryDistributedLock diff --git a/docs/upgrade/2026-11-round9-claim-fence.md b/docs/upgrade/2026-11-round9-claim-fence.md index 99ae53f9..940011b1 100644 --- a/docs/upgrade/2026-11-round9-claim-fence.md +++ b/docs/upgrade/2026-11-round9-claim-fence.md @@ -1,13 +1,14 @@ # 업그레이드 안내 — 2026-11 claim·fence·트랜잭션 시한·정리 -- 범위: durable operation·알림 수신자·fileserver cleanup·멱등성 기록의 lease, fence, 조건부 쓰기, 정리를 고친 - 변경. 포트 시그니처와 기동 조건이 바뀐 곳이 있다. 아래 표가 그 목록이다. -- 근거: 2026-11-12 전체 리뷰(`docs/reviews/2026-11-12-round9-review/`)의 claim·fence(B8) 발견 +- 범위: durable operation·알림 수신자·fileserver cleanup·멱등성 기록·live event·분산 락·stateless runner의 lease, + fence, 조건부 쓰기, 정리, 시간 상한을 고친 변경. 포트 시그니처와 기동 조건이 바뀐 곳이 있다. 아래 표가 그 목록이다. +- 근거: 2026-11-12 전체 리뷰(`docs/reviews/2026-11-12-round9-review/`)의 claim·fence(B8)와 트랜잭션 시한·정리(B9) 발견 ## 1. 누가 읽나 이 템플릿을 `upstream` remote로 두고 merge해 받는 파생 서비스가 읽는다. 특히 durable operation worker를 직접 쓴 서비스, -알림 recipient store를 직접 구현한 서비스, 알림 provider timeout을 설정한 서비스다. +알림 recipient store를 직접 구현한 서비스, `app.lock.wait-time`이나 알림 provider timeout을 설정한 서비스, experimental +tenant pool 예산을 쓰는 서비스다. ## 2. 할 일 @@ -16,17 +17,22 @@ | 1 | durable operation worker는 `claimNext`가 돌려주는 `DurableOperationClaim`을 들고 있다가 `heartbeat(claim, …)`·`reportProgress(claim, …)`·`succeed(claim, …)`·`fail(claim, …)`에 넘긴다. 예전에는 `(operationId, workerId, …)`였다 | 컴파일이 실패한다 | durable operation worker를 쓴 서비스 | | 2 | `DurableOperationStorePort`를 직접 구현했으면 claim마다 오르는 fence를 저장하고, 네 쓰기가 owner와 fence를 함께 확인하게 한다. `DurableOperationStoreContract`의 새 사례(같은 worker id가 다시 claim한 뒤 앞 실행의 쓰기는 거절)로 확인한다 | 재시작한 worker나 한 프로세스의 두 실행이 같은 id를 쓰면 앞 실행의 결과가 넘겨받은 실행 위에 기록된다 | 자기 durable operation store를 둔 서비스 | | 3 | persistence-jpa의 `V23__durable_operation_lease_fence.sql`(`durable_operation.lease_fence BIGINT NOT NULL DEFAULT 0`)이 기본 stream에 더해진다. 같은 번호의 서비스 마이그레이션이 있으면 번호를 옮긴다. `durable_operation`을 건드린 트랜잭션이 열려 있으면 5초 뒤 실패하고 V22에 남는다(다시 배포) | Flyway가 번호 충돌로 기동하지 않는다 | persistence-jpa를 쓰는 모든 서비스 | -| 4 | `RecipientLeaseStorePort.expiredDispatching`이 `List`(버려질 때의 owner·fence·만료 시각)를 돌려주고, `RecipientDeliveryStorePort`에 `requeueAbandoned(lease, nextDispatchAt)`가 생겼다. 직접 구현했으면 상태가 `DISPATCHING`이고 owner·fence가 같고 lease가 늘지 않았을 때만 `READY_TO_DISPATCH`로 바꾼다(`RecipientDeliveryStoreContract` 새 사례). owner가 없거나 fence가 0인 `DISPATCHING` 행(notification V4 전에 남은 행)은 복구가 더는 고르지 않는다. 그런 행은 `SELECT id FROM notification_recipient_delivery WHERE delivery_state = 'DISPATCHING' AND (lease_owner IS NULL OR lease_fence = 0)`로 찾는다. 배포 전에 `UPDATE notification_recipient_delivery SET lease_owner = COALESCE(lease_owner, 'pre-v4'), lease_fence = lease_fence + 1, lease_until = now() - interval '1 day', version = version + 1, updated_at = now() WHERE delivery_state = 'DISPATCHING' AND (lease_owner IS NULL OR lease_fence = 0)`로 owner를 채우고 fence를 올려 복구 패스에 넘긴다. 복구는 attempt를 보고 정한다: attempt가 없으면 다시 보낼 차례로 돌리고, 끝나지 않은 attempt가 있으면 맞추기로 넘긴다(그 뒤는 `docs/notification/configuration-reference.md` 「발송」의 lease 문단). 손으로 `READY_TO_DISPATCH`로 바꾸지 않는다. 발송 중에 멈춘 행이면 같은 알림이 한 번 더 나간다. owner가 없는 행은 stuck lease 지표에도 잡히지 않는다 | 컴파일이 실패한다. 조건 없이 바꾸면 끝났거나 다시 claim된 delivery가 다시 발송된다 | 알림 store를 직접 구현한 서비스 | -| 5 | 알림 provider profile의 `timeout`이 `app.notification.platform.dispatch.lease-duration`보다 짧은지 확인한다. 짧은 것은 필요조건일 뿐이다: lease는 claim 때 시작해서 attempt 기록·렌더링·provider 허가 대기도 lease를 쓴다. lease는 timeout보다 넉넉히 길게 둔다. 이 timeout이 provider 호출 전체를 묶는 것은 SES(API 호출 시간 상한)다. SMTP는 `spring.mail` socket timeout이 명령마다 묶을 뿐이라 이 검사만으로는 호출 전체가 lease 안에 끝난다고 말할 수 없다 | 짧지 않으면 기동이 실패한다(`notification provider profile '…': timeout (…) must be shorter than …lease-duration`). 전에는 provider 호출 하나가 lease보다 길어도 기동했다. 그때 생기는 일은 `docs/notification/configuration-reference.md` 「발송」의 lease 문단이다 | 알림 capability를 켠 서비스 | -| 6 | `IdempotencyReaper`를 직접 만들던 코드는 `new IdempotencyReaper(repository)`로 바꾼다(`Clock` 인자 삭제). 만료 비교는 DB 시계(`now()`)이고 1000행씩 지운다 | 컴파일이 실패한다 | reaper를 직접 조립한 서비스 | -| 7 | fileserver cleanup 묶음의 `max-items`를 정할 때, 한 묶음이 claim lease(10분)의 첫 8분 안에 시작할 수 있는 만큼만 처리된다는 것을 안다. 나머지는 시도하지 않은 채 queue에 돌아가 다음 묶음이 다시 claim한다. 시도 횟수는 그대로다(바이트 예산이 바닥나 돌려준 항목도 이제 같다. 전에는 시도 한 번으로 세어 8번째에 보류됐다). 직접 구현한 `CleanupQueuePort`는 새 연산 `release`를 구현한다(`CleanupQueuePortContract` 새 사례) | 느린 저장소에서 큰 `max-items`는 그 사유로 도는 항목이 늘어난다. 전에는 lease가 끝난 뒤에도 남은 항목을 처리해 다른 worker와 겹쳤다 | fileserver capability를 켠 서비스 | +| 4 | 알림 stream에 `V14__notification_provider_event_open_index.sql`(부분 인덱스 `ix_notification_provider_event_open`)이 더해진다. `notification_provider_event`가 이미 큰 운영 DB는 배포 전에 같은 인덱스를 `CREATE INDEX CONCURRENTLY IF NOT EXISTS ix_notification_provider_event_open ON notification_provider_event (received_at) WHERE projection_status IN ('PENDING', 'FAILED') OR attempt_id IS NULL;`로 먼저 만든다. 미리 만든 인덱스가 실패해 남았으면(`SELECT indisvalid FROM pg_index WHERE indexrelid = 'ix_notification_provider_event_open'::regclass`가 false) `DROP INDEX CONCURRENTLY ix_notification_provider_event_open` 뒤 다시 만든다. `IF NOT EXISTS`는 쓸 수 없는 인덱스도 있는 것으로 보고 넘어간다 | 마이그레이션이 인덱스를 만드는 동안 provider callback 저장이 기다린다(provider가 callback을 다시 보낸다) | 알림 capability를 켠 서비스 | +| 5 | `RecipientLeaseStorePort.expiredDispatching`이 `List`(버려질 때의 owner·fence·만료 시각)를 돌려주고, `RecipientDeliveryStorePort`에 `requeueAbandoned(lease, nextDispatchAt)`가 생겼다. 직접 구현했으면 상태가 `DISPATCHING`이고 owner·fence가 같고 lease가 늘지 않았을 때만 `READY_TO_DISPATCH`로 바꾼다(`RecipientDeliveryStoreContract` 새 사례). owner가 없거나 fence가 0인 `DISPATCHING` 행(notification V4 전에 남은 행)은 복구가 더는 고르지 않는다. 그런 행은 `SELECT id FROM notification_recipient_delivery WHERE delivery_state = 'DISPATCHING' AND (lease_owner IS NULL OR lease_fence = 0)`로 찾는다. 배포 전에 `UPDATE notification_recipient_delivery SET lease_owner = COALESCE(lease_owner, 'pre-v4'), lease_fence = lease_fence + 1, lease_until = now() - interval '1 day', version = version + 1, updated_at = now() WHERE delivery_state = 'DISPATCHING' AND (lease_owner IS NULL OR lease_fence = 0)`로 owner를 채우고 fence를 올려 복구 패스에 넘긴다. 복구는 attempt를 보고 정한다: attempt가 없으면 다시 보낼 차례로 돌리고, 끝나지 않은 attempt가 있으면 맞추기로 넘긴다(그 뒤는 `docs/notification/configuration-reference.md` 「발송」의 lease 문단). 손으로 `READY_TO_DISPATCH`로 바꾸지 않는다. 발송 중에 멈춘 행이면 같은 알림이 한 번 더 나간다. owner가 없는 행은 stuck lease 지표에도 잡히지 않는다 | 컴파일이 실패한다. 조건 없이 바꾸면 끝났거나 다시 claim된 delivery가 다시 발송된다 | 알림 store를 직접 구현한 서비스 | +| 6 | 알림 provider profile의 `timeout`이 `app.notification.platform.dispatch.lease-duration`보다 짧은지 확인한다. 짧은 것은 필요조건일 뿐이다: lease는 claim 때 시작해서 attempt 기록·렌더링·provider 허가 대기도 lease를 쓴다. lease는 timeout보다 넉넉히 길게 둔다. 이 timeout이 provider 호출 전체를 묶는 것은 SES(API 호출 시간 상한)다. SMTP는 `spring.mail` socket timeout이 명령마다 묶을 뿐이라 이 검사만으로는 호출 전체가 lease 안에 끝난다고 말할 수 없다 | 짧지 않으면 기동이 실패한다(`notification provider profile '…': timeout (…) must be shorter than …lease-duration`). 전에는 provider 호출 하나가 lease보다 길어도 기동했다. 그때 생기는 일은 `docs/notification/configuration-reference.md` 「발송」의 lease 문단이다 | 알림 capability를 켠 서비스 | +| 7 | `app.lock.wait-time`을 설정에서 지운다. 기다릴 시간은 `tryAcquire(key, waitTime, leaseTtl)`의 인자다 | `app.lock` 아래 모르는 키라 기동이 실패한다. 전에도 읽는 곳이 없었다 | `app.lock.wait-time`을 설정한 서비스 | +| 8 | `JpaLiveEventReplay.sweepExpired()`를 부르던 코드를 지운다. replay를 켜면 `LiveEventReaper`가 `app.realtime.replay.reaper-interval`(기본 `PT10M`)마다 `retained_until`이 지난 행을 1000행씩 지운다. `app.realtime.replay.*`는 이제 `LiveEventReplaySettings`가 엄격하게 묶는다: 그 아래 모르는 키, 0 이하의 `retention`·`reaper-interval`은 기동을 실패시킨다 | 컴파일이 실패한다. 잘못 쓴 키는 기동이 실패한다. 전에는 지우는 곳이 없어 `live_event_log`가 끝없이 자랐다 | `app.realtime.replay.enabled=true`인 서비스 | +| 9 | `IdempotencyReaper`를 직접 만들던 코드는 `new IdempotencyReaper(repository)`로 바꾼다(`Clock` 인자 삭제). 만료 비교는 DB 시계(`now()`)이고 1000행씩 지운다 | 컴파일이 실패한다 | reaper를 직접 조립한 서비스 | +| 10 | `HibernateStatelessSessionRunner`의 트랜잭션은 이제 30초 상한이 있다(`DEFAULT_TIMEOUT`). 더 긴 일이면 셋째 인자로 상한을 넘기거나 일을 나눈다 | 30초를 넘는 문장이 취소되고 일 전체가 롤백된다. 전에는 상한이 없었다 | stateless runner를 쓰는 서비스 | +| 11 | experimental `TenantPoolBudget.requireCapacity`·`hasCapacity`가 새 pool의 연결 수를 셋째 인자로 받는다. pool 수만 볼 때는 `requirePoolSlot(openPools)`를 쓴다. `TenantDataSourceRegistry`의 둘째 인자는 이제 열린 pool의 크기를 재는 함수가 아니라 tenant의 계획된 pool 크기(`ToIntFunction`)다. 예산은 pool을 열기 전에 그 크기로 본다 | 컴파일이 실패한다. 전에는 새 pool 크기를 모른 채 열어 연결 합계가 상한을 넘었다 | experimental tenant pool을 쓰는 서비스 | +| 12 | fileserver cleanup 묶음의 `max-items`를 정할 때, 한 묶음이 claim lease(10분)의 첫 8분 안에 시작할 수 있는 만큼만 처리된다는 것을 안다. 나머지는 시도하지 않은 채 queue에 돌아가 다음 묶음이 다시 claim한다. 시도 횟수는 그대로다(바이트 예산이 바닥나 돌려준 항목도 이제 같다. 전에는 시도 한 번으로 세어 8번째에 보류됐다). 직접 구현한 `CleanupQueuePort`는 새 연산 `release`를 구현한다(`CleanupQueuePortContract` 새 사례) | 느린 저장소에서 큰 `max-items`는 그 사유로 도는 항목이 늘어난다. 전에는 lease가 끝난 뒤에도 남은 항목을 처리해 다른 worker와 겹쳤다 | fileserver capability를 켠 서비스 | ## 3. 바뀐 동작 (따라 할 일 없음) -- reaper 둘(outbox·멱등성)의 트랜잭션은 `SpringTransaction.DEFAULT_TIMEOUT_SECONDS`(30초)로 묶인다. +- reaper 셋(outbox·멱등성·live event)의 트랜잭션은 `SpringTransaction.DEFAULT_TIMEOUT_SECONDS`(30초)로 묶인다. `app.persistence-jpa.transaction.*`를 바꿔도 reaper 시한은 그대로다. 한 트랜잭션이 1000행 한 묶음만 지운다. - 저장소가 행에서 만드는 `DurableOperationClaim`·`OperationLease`·`RecipientLease`·`OutboxLease`·`ClaimedOutboxEvent`· - `ReconciliationJob`·`AdminOperationClaim`이 값을 거절할 때 `IllegalStateException`을 던진다 + `ReconciliationJob`·`AdminOperationClaim`과 stateless runner의 timeout이 값을 거절할 때 `IllegalStateException`을 던진다 (전에는 `IllegalArgumentException`). 요청에서 오는 값이 아니라 서버 결함이다. 이 IAE를 잡던 코드는 ISE를 잡는다. - 알림 lease 회복은 버려질 때의 lease 그대로일 때만 delivery를 다시 넣는다. 다른 인스턴스가 이미 다시 넣었거나 holder가 끝냈으면 아무것도 쓰지 않고 처리 수에 세지 않는다. @@ -36,3 +42,8 @@ - 멱등성 reaper는 다른 트랜잭션이 쥔 기록을 건너뛰고, 바깥 `DELETE`에서도 만료를 다시 본다. 같은 키 요청이 만료된 기록을 되살리는 동안 돈 reap이 그 claim을 지우지 않는다. - fileserver recovery 대기열 enqueue는 `INSERT … ON CONFLICT` 한 문장이다. 같은 파일을 동시에 넣어도 25P02로 실패하지 않는다. +- 알림 scheduler는 poll마다 backlog 한 문장만 읽는다(전에는 넷). health snapshot의 projection 지연은 열린 provider event만 + 읽는다. +- 알림 backlog(`backlogDepth`, scheduler의 queue-depth gauge, health 임계값)는 claim이 지금 가져갈 delivery만 센다. claim과 + 같은 SQL 조각이다. 전에는 창이 닫힌(`expires_at`이 지난) delivery도 셌고, lease가 지금 막 끝나는 행(`lease_until = now`)도 + 셌다. 만료가 쌓인 플랫폼은 배포 직후 backlog 수치가 내려간다. 보낼 일이 줄어든 것이 아니라 보낼 수 없는 행을 빼고 센 것이다. diff --git a/src/adapter/outbound/notification/src/main/java/dev/caskeleton/adapter/outbound/notification/platform/dispatch/NotificationSchedulerWorker.java b/src/adapter/outbound/notification/src/main/java/dev/caskeleton/adapter/outbound/notification/platform/dispatch/NotificationSchedulerWorker.java index d761c923..f54fa001 100644 --- a/src/adapter/outbound/notification/src/main/java/dev/caskeleton/adapter/outbound/notification/platform/dispatch/NotificationSchedulerWorker.java +++ b/src/adapter/outbound/notification/src/main/java/dev/caskeleton/adapter/outbound/notification/platform/dispatch/NotificationSchedulerWorker.java @@ -81,10 +81,10 @@ public final class NotificationSchedulerWorker implements AutoCloseable { // by the claim batch size — so a queue of ten and a queue of ten million published the same // number, and the one metric named "queue depth" was the one that could not show a queue // growing. + // The depth alone: the full serving snapshot is several statements and is the health + // endpoint's, not something to run on every poll. metrics.gauge( - NotificationMetricName.QUEUE_DEPTH, - Map.of(), - (double) servingState.currentState().backlogDepth()); + NotificationMetricName.QUEUE_DEPTH, Map.of(), (double) servingState.backlogDepth()); for (RecipientLease lease : claimed) { globalConcurrency.acquireUninterruptibly(); diff --git a/src/adapter/outbound/persistence-jpa/README.md b/src/adapter/outbound/persistence-jpa/README.md index 9269264e..299beed7 100644 --- a/src/adapter/outbound/persistence-jpa/README.md +++ b/src/adapter/outbound/persistence-jpa/README.md @@ -389,6 +389,22 @@ PUBLISHED backlog 를 한 transaction 으로 지우면 그 사이 모든 row loc cadence 측정 필요). retention 한 값은 reaper-local 이라 `@Value` 로 받지만, canonical 6-property 문서는 app-bootstrap `OutboxSettings` / `application.yml` 에 있다. +## liveevent — `JpaLiveEventReplay`와 `LiveEventReaper` + +`app.realtime.replay.enabled=true`이고 vendor가 PostgreSQL이면 replay store와 reaper가 함께 조립된다. 설정은 +`LiveEventReplaySettings`가 `app.realtime.replay.*`를 엄격하게 묶는다. 모르는 키는 기동을 실패시킨다. + +- `retention`: 한 event를 다시 받을 수 있는 시간. 기본값이 없다. 모듈 기본값이 `APP_REALTIME_REPLAY_RETENTION`(1시간)으로 + 채운다. 0이나 음수는 기동 실패다. +- `reaper-interval`: `LiveEventReaper`가 retention이 지난 행을 지우는 간격(기본 10분, 모듈 기본값이 같은 값을 적는다). 0이나 + 음수는 기동 실패다. reaper 자체에는 일정이 없다. 조립 설정이 이 값으로 `reap()`을 등록한다(첫 실행도 한 간격 뒤). + reaper를 직접 만드는 서비스는 일정도 직접 건다. + +reaper는 `BatchedDeletes`로 1,000행씩, 배치마다 따로 commit하고, 배치가 덜 차면 멈춘다. 배치 문장은 `SKIP LOCKED`라 여러 +replica가 같이 돌아도 서로 기다리지 않는다. 트랜잭션 시한은 `SpringTransaction.DEFAULT_TIMEOUT_SECONDS`(30초)다. replay 읽기는 +`retained_until`로 지난 행을 이미 거르므로, reaper가 늦어도 지난 event를 내주지는 않는다. reaper가 없으면 테이블이 끝없이 +자란다(2026-11-13 전에는 지우는 곳이 없었다). + ## operation — `JpaDurableOperationStore` ### claim과 fence (`V23`) diff --git a/src/adapter/outbound/persistence-jpa/src/main/java/dev/caskeleton/adapter/outbound/persistence/autoconfigure/JpaAdapterBeansConfig.java b/src/adapter/outbound/persistence-jpa/src/main/java/dev/caskeleton/adapter/outbound/persistence/autoconfigure/JpaAdapterBeansConfig.java index 193187e5..438f0978 100644 --- a/src/adapter/outbound/persistence-jpa/src/main/java/dev/caskeleton/adapter/outbound/persistence/autoconfigure/JpaAdapterBeansConfig.java +++ b/src/adapter/outbound/persistence-jpa/src/main/java/dev/caskeleton/adapter/outbound/persistence/autoconfigure/JpaAdapterBeansConfig.java @@ -8,6 +8,8 @@ import dev.caskeleton.adapter.outbound.persistence.idempotency.jpa.IdempotencyRe import dev.caskeleton.adapter.outbound.persistence.idempotency.jpa.IdempotencyRecordJpaRepository; import dev.caskeleton.adapter.outbound.persistence.liveevent.JpaLiveEventReplay; import dev.caskeleton.adapter.outbound.persistence.liveevent.LiveEventJpaRepository; +import dev.caskeleton.adapter.outbound.persistence.liveevent.LiveEventReaper; +import dev.caskeleton.adapter.outbound.persistence.liveevent.LiveEventReplaySettings; import dev.caskeleton.adapter.outbound.persistence.lock.DistributedLockPersistenceConfig; import dev.caskeleton.adapter.outbound.persistence.operation.DurableOperationJpaRepository; import dev.caskeleton.adapter.outbound.persistence.operation.JpaDurableOperationStore; @@ -26,7 +28,11 @@ import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty; import org.springframework.context.annotation.Bean; import org.springframework.context.annotation.Configuration; import org.springframework.context.annotation.Import; +import org.springframework.core.env.Environment; import org.springframework.jdbc.core.JdbcOperations; +import org.springframework.scheduling.annotation.EnableScheduling; +import org.springframework.scheduling.annotation.SchedulingConfigurer; +import org.springframework.scheduling.config.FixedDelayTask; import org.springframework.transaction.PlatformTransactionManager; /** @@ -120,11 +126,15 @@ class JpaAdapterBeansConfig { } /** - * Retained live-event history: on when replay is switched on, and only on PostgreSQL (the - * position allocator is an {@code INSERT … ON CONFLICT … RETURNING}). The retention has no - * default on purpose: a replay window nobody chose is one nobody sized. + * Retained live-event history and the reaper that removes it past retention: on when replay is + * switched on, and only on PostgreSQL (the position allocator is an {@code INSERT … ON CONFLICT … + * RETURNING}). The settings are {@link LiveEventReplaySettings}, bound strictly: the retention is + * one hour unless the deployment sets {@code APP_REALTIME_REPLAY_RETENTION} (the module + * defaults), the reaper runs every {@code reaper-interval}. Scheduling is enabled here because + * the reaper is part of the store, as for the idempotency reaper. */ @Configuration(proxyBeanMethods = false) + @EnableScheduling @ConditionalOnProperty(prefix = "app.realtime.replay", name = "enabled", havingValue = "true") @ConditionalOnProperty( prefix = "app.persistence", @@ -133,13 +143,36 @@ class JpaAdapterBeansConfig { matchIfMissing = true) static class LiveEventReplay { + @Bean + LiveEventReplaySettings liveEventReplaySettings(Environment environment) { + return LiveEventReplaySettings.bind(environment); + } + @Bean JpaLiveEventReplay jpaLiveEventReplay( LiveEventJpaRepository repository, JdbcOperations jdbc, Clock clock, - @Value("${app.realtime.replay.retention}") Duration retention) { - return new JpaLiveEventReplay(repository, jdbc, clock, retention); + LiveEventReplaySettings settings) { + return new JpaLiveEventReplay(repository, jdbc, clock, settings.retention()); + } + + @Bean + LiveEventReaper liveEventReaper(LiveEventJpaRepository repository, Clock clock) { + return new LiveEventReaper(repository, clock); + } + + /** + * Runs the reaper every bound {@code reaper-interval}, first one interval after startup. The + * interval is handed over as the settings' value, so it has one default and a zero has already + * failed startup in the settings rather than scheduling a loop. + */ + @Bean + SchedulingConfigurer liveEventReaperSchedule( + LiveEventReaper reaper, LiveEventReplaySettings settings) { + Duration interval = settings.reaperInterval(); + return registrar -> + registrar.addFixedDelayTask(new FixedDelayTask(reaper::reap, interval, interval)); } } diff --git a/src/adapter/outbound/persistence-jpa/src/main/java/dev/caskeleton/adapter/outbound/persistence/experimental/database/TenantDataSourceRegistry.java b/src/adapter/outbound/persistence-jpa/src/main/java/dev/caskeleton/adapter/outbound/persistence/experimental/database/TenantDataSourceRegistry.java index 90737a29..2551c93c 100644 --- a/src/adapter/outbound/persistence-jpa/src/main/java/dev/caskeleton/adapter/outbound/persistence/experimental/database/TenantDataSourceRegistry.java +++ b/src/adapter/outbound/persistence-jpa/src/main/java/dev/caskeleton/adapter/outbound/persistence/experimental/database/TenantDataSourceRegistry.java @@ -23,26 +23,32 @@ import javax.sql.DataSource; public final class TenantDataSourceRegistry implements AutoCloseable { private final Map pools = new LinkedHashMap<>(); + private final Map poolSizes = new LinkedHashMap<>(); private final Function poolFactory; - private final ToIntFunction poolSize; + private final ToIntFunction plannedPoolSize; private final TenantPoolBudget budget; /** * @param poolFactory opens a pool for a tenant from its secret-backed connection profile - * @param poolSize reports a pool's maximum connection count, for the global budget + * @param plannedPoolSize the maximum connection count a tenant's pool will have, read from its + * connection profile before the pool is opened, for the global budget */ public TenantDataSourceRegistry( Function poolFactory, - ToIntFunction poolSize, + ToIntFunction plannedPoolSize, TenantPoolBudget budget) { this.poolFactory = Objects.requireNonNull(poolFactory, "poolFactory"); - this.poolSize = Objects.requireNonNull(poolSize, "poolSize"); + this.plannedPoolSize = Objects.requireNonNull(plannedPoolSize, "plannedPoolSize"); this.budget = Objects.requireNonNull(budget, "budget"); } /** * The tenant's pool, opening it if the budget allows. * + *

The budget is checked against the pool's planned size before the pool is opened: a pool + * opens its first connections as it is built, so one checked afterwards had already taken the + * server past the ceiling, if only until it was closed again. + * * @throws IllegalStateException when the global pool budget is exhausted */ public synchronized DataSource require(TenantId tenant) { @@ -51,9 +57,12 @@ public final class TenantDataSourceRegistry implements AutoCloseable { if (existing != null) { return existing; } - budget.requireCapacity(pools.size(), allocatedConnections()); + budget.requirePoolSlot(pools.size()); + int planned = plannedPoolSize.applyAsInt(tenant); + budget.requireCapacity(pools.size(), allocatedConnections(), planned); DataSource opened = poolFactory.apply(tenant); pools.put(tenant, opened); + poolSizes.put(tenant, planned); return opened; } @@ -72,6 +81,7 @@ public final class TenantDataSourceRegistry implements AutoCloseable { */ public synchronized void evict(TenantId tenant) { DataSource removed = pools.remove(tenant); + poolSizes.remove(tenant); if (removed instanceof AutoCloseable closeable) { TenantDataSourceLifecycle.drainAndClose(closeable); } @@ -84,7 +94,7 @@ public final class TenantDataSourceRegistry implements AutoCloseable { /** How many connections all open pools may hold between them. */ public synchronized int allocatedConnections() { - return pools.values().stream().mapToInt(poolSize).sum(); + return poolSizes.values().stream().mapToInt(Integer::intValue).sum(); } /** The tenants with an open pool. */ @@ -99,5 +109,6 @@ public final class TenantDataSourceRegistry implements AutoCloseable { .map(AutoCloseable.class::cast) .forEach(TenantDataSourceLifecycle::drainAndClose); pools.clear(); + poolSizes.clear(); } } diff --git a/src/adapter/outbound/persistence-jpa/src/main/java/dev/caskeleton/adapter/outbound/persistence/experimental/database/TenantPoolBudget.java b/src/adapter/outbound/persistence-jpa/src/main/java/dev/caskeleton/adapter/outbound/persistence/experimental/database/TenantPoolBudget.java index b3abfb1d..c26fa39d 100644 --- a/src/adapter/outbound/persistence-jpa/src/main/java/dev/caskeleton/adapter/outbound/persistence/experimental/database/TenantPoolBudget.java +++ b/src/adapter/outbound/persistence-jpa/src/main/java/dev/caskeleton/adapter/outbound/persistence/experimental/database/TenantPoolBudget.java @@ -23,19 +23,44 @@ public record TenantPoolBudget(int maxOpenPools, int maxConnectionsAcrossPools) } /** - * Fails when another tenant pool would exceed either ceiling. + * Fails when another tenant pool would exceed the pool count, before that pool is opened. * - * @throws IllegalStateException when the budget is exhausted + * @param openPools how many pools are open now + * @throws IllegalStateException when no other pool fits */ - public void requireCapacity(int openPools, int allocatedConnections) { - if (openPools >= maxOpenPools || allocatedConnections >= maxConnectionsAcrossPools) { + public void requirePoolSlot(int openPools) { + if (openPools >= maxOpenPools) { throw new IllegalStateException("tenant pool budget exhausted"); } } - /** Whether another tenant pool fits inside both ceilings. */ - public boolean hasCapacity(int openPools, int allocatedConnections) { - return openPools < maxOpenPools && allocatedConnections < maxConnectionsAcrossPools; + /** + * Fails when a new tenant pool of {@code newPoolConnections} would exceed either ceiling. + * + *

The new pool's own size is part of the question. Checking only that the pools already open + * stay under the ceiling let a ten-connection pool open with one connection to spare, and the + * total then stood nine above the ceiling the budget exists to keep (round9 review 2026-11-12, + * RR9-01-26). + * + * @param openPools how many pools are open now, the new one not included + * @param allocatedConnections how many connections the open pools may hold between them + * @param newPoolConnections how many the new pool may hold + * @throws IllegalStateException when the budget cannot take the new pool + */ + public void requireCapacity(int openPools, int allocatedConnections, int newPoolConnections) { + if (!hasCapacity(openPools, allocatedConnections, newPoolConnections)) { + throw new IllegalStateException("tenant pool budget exhausted"); + } + } + + /** Whether a new tenant pool of {@code newPoolConnections} fits inside both ceilings. */ + public boolean hasCapacity(int openPools, int allocatedConnections, int newPoolConnections) { + if (newPoolConnections < 1) { + // The size is the server's own configuration, not the caller's input: a state error. + throw new IllegalStateException("a tenant pool holds at least one connection"); + } + return openPools < maxOpenPools + && (long) allocatedConnections + newPoolConnections <= maxConnectionsAcrossPools; } /** The maximum number of tenants that can be open at once. */ diff --git a/src/adapter/outbound/persistence-jpa/src/main/java/dev/caskeleton/adapter/outbound/persistence/hibernate/stateless/HibernateStatelessSessionRunner.java b/src/adapter/outbound/persistence-jpa/src/main/java/dev/caskeleton/adapter/outbound/persistence/hibernate/stateless/HibernateStatelessSessionRunner.java index 00f8aa30..d80927af 100644 --- a/src/adapter/outbound/persistence-jpa/src/main/java/dev/caskeleton/adapter/outbound/persistence/hibernate/stateless/HibernateStatelessSessionRunner.java +++ b/src/adapter/outbound/persistence-jpa/src/main/java/dev/caskeleton/adapter/outbound/persistence/hibernate/stateless/HibernateStatelessSessionRunner.java @@ -1,5 +1,6 @@ package dev.caskeleton.adapter.outbound.persistence.hibernate.stateless; +import java.time.Duration; import java.util.Objects; import java.util.Set; import java.util.function.Function; @@ -19,23 +20,63 @@ import org.hibernate.Transaction; * *

Only registered work names may run, and each carries a declared row cap. An admin bulk path * without a cap is how a maintenance job becomes an outage. + * + *

The transaction has the same time bound as the application's write transactions (30 seconds + * unless the constructor is given another, {@code docs/work/reference/decisions.md} 2.5). The row + * cap can only be checked once the work reports what it touched, so without a bound a runaway + * statement held its locks for as long as it ran before the cap could roll it back. Hibernate hands + * every statement of the work the time left, and the database cancels the one that runs past it + * (round9 review 2026-11-12, RR9-01-25). */ public final class HibernateStatelessSessionRunner { + /** + * The default write-transaction bound, which this runner's transactions share: {@code + * SpringTransaction.DEFAULT_TIMEOUT_SECONDS}. Restated rather than read from it, because the + * {@code hibernate} package does not depend on {@code transaction} (JpaModuleBoundaryTest); + * {@code StatelessRunnerTimeoutTest} holds the two equal. A deployment that moves its write bound + * passes the same value to the three-argument constructor. + */ + public static final Duration DEFAULT_TIMEOUT = Duration.ofSeconds(30); + private final SessionFactory sessionFactory; private final Set registeredWork; + private final int timeoutSeconds; + /** A runner whose transactions are bounded by {@link #DEFAULT_TIMEOUT}. */ public HibernateStatelessSessionRunner( SessionFactory sessionFactory, Set registeredWork) { + this(sessionFactory, registeredWork, DEFAULT_TIMEOUT); + } + + /** + * A runner whose transactions are bounded by {@code timeout}. + * + * @param timeout a whole, positive number of seconds; Hibernate's transaction timeout has no + * finer unit, so a fraction would be dropped without a word + */ + public HibernateStatelessSessionRunner( + SessionFactory sessionFactory, Set registeredWork, Duration timeout) { this.sessionFactory = Objects.requireNonNull(sessionFactory, "sessionFactory"); this.registeredWork = Set.copyOf(Objects.requireNonNull(registeredWork, "registeredWork")); + Objects.requireNonNull(timeout, "timeout"); + if (timeout.isNegative() + || timeout.isZero() + || timeout.toNanosPart() != 0 + || timeout.toSeconds() > Integer.MAX_VALUE) { + throw new IllegalStateException( + "a stateless transaction timeout must be a positive whole number of seconds, was " + + timeout); + } + this.timeoutSeconds = Math.toIntExact(timeout.toSeconds()); } /** * Runs {@code work} in its own stateless session and transaction. * *

The work reports how many rows it touched, and exceeding {@code maxRows} rolls the - * transaction back. + * transaction back. A statement still running when the transaction's time is up is cancelled, and + * the failure rolls the transaction back. * * @param maxRows the row cap for this unit of work * @throws StatelessRowCapExceededException when the work touched more, after rolling back @@ -54,7 +95,9 @@ public final class HibernateStatelessSessionRunner { } try (StatelessSession session = sessionFactory.openStatelessSession()) { - Transaction transaction = session.beginTransaction(); + Transaction transaction = session.getTransaction(); + transaction.setTimeout(timeoutSeconds); + transaction.begin(); try { StatelessWorkResult result = work.apply(session); if (result.affectedRows() > maxRows) { diff --git a/src/adapter/outbound/persistence-jpa/src/main/java/dev/caskeleton/adapter/outbound/persistence/liveevent/JpaLiveEventReplay.java b/src/adapter/outbound/persistence-jpa/src/main/java/dev/caskeleton/adapter/outbound/persistence/liveevent/JpaLiveEventReplay.java index 26e345af..d2762a63 100644 --- a/src/adapter/outbound/persistence-jpa/src/main/java/dev/caskeleton/adapter/outbound/persistence/liveevent/JpaLiveEventReplay.java +++ b/src/adapter/outbound/persistence-jpa/src/main/java/dev/caskeleton/adapter/outbound/persistence/liveevent/JpaLiveEventReplay.java @@ -134,16 +134,6 @@ public final class JpaLiveEventReplay implements LiveEventReplayPort { .toList(); } - /** - * Removes events past their retention. - * - * @return how many went - */ - public int sweepExpired() { - requireCallerTransaction("sweepExpired"); - return repository.sweepExpired(clock.instant()); - } - private static void requireCallerTransaction(String operation) { CallerTransaction.require("live-event " + operation); } diff --git a/src/adapter/outbound/persistence-jpa/src/main/java/dev/caskeleton/adapter/outbound/persistence/liveevent/LiveEventJpaRepository.java b/src/adapter/outbound/persistence-jpa/src/main/java/dev/caskeleton/adapter/outbound/persistence/liveevent/LiveEventJpaRepository.java index cf9ea4ce..9df87a3f 100644 --- a/src/adapter/outbound/persistence-jpa/src/main/java/dev/caskeleton/adapter/outbound/persistence/liveevent/LiveEventJpaRepository.java +++ b/src/adapter/outbound/persistence-jpa/src/main/java/dev/caskeleton/adapter/outbound/persistence/liveevent/LiveEventJpaRepository.java @@ -1,6 +1,7 @@ package dev.caskeleton.adapter.outbound.persistence.liveevent; import dev.caskeleton.adapter.outbound.persistence.liveevent.entity.LiveEventEntity; +import dev.caskeleton.adapter.outbound.persistence.transaction.SpringTransaction; import java.time.Instant; import java.util.List; import org.springframework.data.domain.Limit; @@ -8,6 +9,7 @@ import org.springframework.data.jpa.repository.JpaRepository; import org.springframework.data.jpa.repository.Modifying; import org.springframework.data.jpa.repository.Query; import org.springframework.data.repository.query.Param; +import org.springframework.transaction.annotation.Transactional; /** * Spring Data access for {@code live_event_log}. @@ -66,12 +68,27 @@ public interface LiveEventJpaRepository Long latestRetained(@Param("streamId") String streamId, @Param("now") Instant now); /** - * Removes expired rows. + * Deletes at most {@code limit} rows retained no later than {@code now}, in a transaction of its + * own unless the caller already has one; returns the count. + * + *

Bounded, and skipping rows another replica's reaper is deleting, for the reason the outbox + * purge is bounded: a backlog of expired history is one long transaction holding every row lock + * otherwise. {@code retained_until} is written from the application clock, so it is compared with + * the application clock. * * @param now rows retained no later than this are removed + * @param limit the most rows one statement removes * @return how many went */ + // Bounded by the default transaction bound: a statement stuck behind a lock gives the connection + // back instead of holding it until the pool is empty. + @Transactional(timeout = SpringTransaction.DEFAULT_TIMEOUT_SECONDS) @Modifying - @Query("delete from LiveEventEntity e where e.retainedUntil <= :now") - int sweepExpired(@Param("now") Instant now); + @Query( + value = + "DELETE FROM live_event_log WHERE (stream_id, position) IN (" + + "SELECT stream_id, position FROM live_event_log WHERE retained_until <= :now" + + " FOR UPDATE SKIP LOCKED LIMIT :limit)", + nativeQuery = true) + int deleteExpiredBatch(@Param("now") Instant now, @Param("limit") int limit); } diff --git a/src/adapter/outbound/persistence-jpa/src/main/java/dev/caskeleton/adapter/outbound/persistence/liveevent/LiveEventReaper.java b/src/adapter/outbound/persistence-jpa/src/main/java/dev/caskeleton/adapter/outbound/persistence/liveevent/LiveEventReaper.java new file mode 100644 index 00000000..6e57b633 --- /dev/null +++ b/src/adapter/outbound/persistence-jpa/src/main/java/dev/caskeleton/adapter/outbound/persistence/liveevent/LiveEventReaper.java @@ -0,0 +1,53 @@ +package dev.caskeleton.adapter.outbound.persistence.liveevent; + +import dev.caskeleton.adapter.outbound.persistence.transaction.BatchedDeletes; +import java.time.Clock; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** + * Periodically deletes {@code live_event_log} rows past their retention. + * + *

A replay read already filters on {@code retained_until}, so an expired row is never served; + * without this nothing ever removed one, and the table grew by every event ever published for as + * long as replay was on (round9 review 2026-11-12, RR9-01-03). It is assembled with the replay + * store, as the idempotency and outbox reapers are with theirs, rather than left to each service to + * schedule: a duty that every deployment must remember is one some deployment forgets. + * + *

Deletes in batches of {@link #BATCH_SIZE}, each committed on its own, until a batch comes back + * short. Every replica runs it; the batch skips rows another replica's batch holds. + * + *

It carries no schedule of its own. The replay configuration registers {@link #reap()} at the + * bound {@link LiveEventReplaySettings#reaperInterval()}, so the interval reaches the schedule as a + * value rather than through a bean name an expression looks up. A deployment that builds a reaper + * itself schedules it too. + */ +public class LiveEventReaper { + + /** Rows per delete statement and transaction. */ + static final int BATCH_SIZE = BatchedDeletes.BATCH_SIZE; + + private static final Logger log = LoggerFactory.getLogger(LiveEventReaper.class); + + private final LiveEventJpaRepository repository; + private final Clock clock; + + public LiveEventReaper(LiveEventJpaRepository repository, Clock clock) { + this.repository = repository; + this.clock = clock; + } + + /** + * Deletes every row retained no later than now; returns the count. + * + *

Not transactional itself: each batch is its own transaction. + */ + public int reap() { + int purged = + BatchedDeletes.untilShort(limit -> repository.deleteExpiredBatch(clock.instant(), limit)); + if (purged > 0) { + log.debug("live event reaper purged {} expired row(s)", purged); + } + return purged; + } +} diff --git a/src/adapter/outbound/persistence-jpa/src/main/java/dev/caskeleton/adapter/outbound/persistence/liveevent/LiveEventReplaySettings.java b/src/adapter/outbound/persistence-jpa/src/main/java/dev/caskeleton/adapter/outbound/persistence/liveevent/LiveEventReplaySettings.java new file mode 100644 index 00000000..dd62e36b --- /dev/null +++ b/src/adapter/outbound/persistence-jpa/src/main/java/dev/caskeleton/adapter/outbound/persistence/liveevent/LiveEventReplaySettings.java @@ -0,0 +1,55 @@ +package dev.caskeleton.adapter.outbound.persistence.liveevent; + +import dev.caskeleton.adapter.outbound.support.StrictSettingsBinder; +import java.time.Duration; +import org.springframework.core.env.Environment; + +/** + * The replay store's settings, bound strictly from {@code app.realtime.replay.*}: a misspelt key + * fails startup rather than leaving a default in force. + * + *

The reaper's interval was read only by the {@code @Scheduled} placeholder, with its default + * written twice and nothing to refuse a zero (round9 review 2026-11-12). It is bound here with the + * rest of the store's settings and read from this record by the reaper's schedule. + * + * @param enabled whether the replay store is assembled; the configuration's condition reads it + * @param retention how long an event stays replayable. No default: how long a reconnecting client + * may resume is the deployment's decision, made against what it is willing to store + * @param reaperInterval how often {@link LiveEventReaper} deletes rows past retention; 10 minutes + * unless set, which the module defaults mirror + */ +public record LiveEventReplaySettings( + boolean enabled, Duration retention, Duration reaperInterval) { + + static final String PREFIX = "app.realtime.replay"; + + private static final Duration DEFAULT_REAPER_INTERVAL = Duration.ofMinutes(10); + + public LiveEventReplaySettings { + if (retention == null) { + throw new IllegalArgumentException( + "app.realtime.replay.retention must be set when replay is enabled: how long a client may" + + " resume is the deployment's decision"); + } + if (retention.isNegative() || retention.isZero()) { + throw new IllegalArgumentException( + "app.realtime.replay.retention must be positive, was " + retention); + } + if (reaperInterval == null) { + reaperInterval = DEFAULT_REAPER_INTERVAL; + } else if (reaperInterval.isNegative() || reaperInterval.isZero()) { + throw new IllegalArgumentException( + "app.realtime.replay.reaper-interval must be positive, was " + reaperInterval); + } + } + + /** + * Binds {@link #PREFIX} strictly. + * + * @param environment the environment to bind from + * @return the bound settings + */ + public static LiveEventReplaySettings bind(Environment environment) { + return StrictSettingsBinder.bind(environment, PREFIX, LiveEventReplaySettings.class); + } +} diff --git a/src/adapter/outbound/persistence-jpa/src/main/java/dev/caskeleton/adapter/outbound/persistence/lock/LockSettings.java b/src/adapter/outbound/persistence-jpa/src/main/java/dev/caskeleton/adapter/outbound/persistence/lock/LockSettings.java index 54cc10d6..5d19b24a 100644 --- a/src/adapter/outbound/persistence-jpa/src/main/java/dev/caskeleton/adapter/outbound/persistence/lock/LockSettings.java +++ b/src/adapter/outbound/persistence-jpa/src/main/java/dev/caskeleton/adapter/outbound/persistence/lock/LockSettings.java @@ -5,41 +5,28 @@ import java.time.Duration; import org.springframework.core.env.Environment; /** - * Distributed-lock tuning knobs bound from {@code app.lock.*} (yaml-only defaults, not {@code - * APP_*} env keys). {@code waitTime} = max wait before timeout (default 3s); {@code leaseTtl} = max - * lock hold before auto-expiry (default 30s, must be ≥ {@code waitTime}). See README "lock". + * Distributed-lock tuning bound from {@code app.lock.*} (yaml-only defaults, not {@code APP_*} env + * keys). {@code leaseTtl} = the longest a lock may be held before the provider expires it (default + * 30s); a caller's {@code tryAcquire} may ask for that long or less. See README "lock". + * + *

There is no wait setting. How long to wait is every caller's own argument to {@code + * tryAcquire}; an {@code app.lock.wait-time} used to be bound here, checked against the lease, and + * read by nothing (round9 review 2026-11-12, RR9-01-07). It now fails startup as an unknown key, so + * a deployment that set it learns that it never did anything. */ -public record LockSettings(Duration waitTime, Duration leaseTtl) { +public record LockSettings(Duration leaseTtl) { static final String PREFIX = "app.lock"; - private static final Duration DEFAULT_WAIT_TIME = Duration.ofSeconds(3); private static final Duration DEFAULT_LEASE_TTL = Duration.ofSeconds(30); public LockSettings { - if (waitTime == null) { - waitTime = DEFAULT_WAIT_TIME; - } else if (waitTime.isNegative() || waitTime.isZero()) { - throw new IllegalArgumentException( - "app.lock.wait-time must be positive (> 0), was " + waitTime); - } if (leaseTtl == null) { leaseTtl = DEFAULT_LEASE_TTL; } else if (leaseTtl.isNegative() || leaseTtl.isZero()) { throw new IllegalArgumentException( "app.lock.lease-ttl must be positive (> 0), was " + leaseTtl); } - // Cross-field invariant: a lease shorter than the max wait is nonsensical. - // If the TTL fires before the waitTime elapses, a second holder could acquire - // the lock before the first holder's protected work is complete. - if (leaseTtl.compareTo(waitTime) < 0) { - throw new IllegalArgumentException( - "app.lock.lease-ttl (" - + leaseTtl - + ") must be >= app.lock.wait-time (" - + waitTime - + ") — a lease shorter than the max wait is nonsensical"); - } } /** diff --git a/src/adapter/outbound/persistence-jpa/src/main/resources/config/module-defaults/persistence-jpa.yml b/src/adapter/outbound/persistence-jpa/src/main/resources/config/module-defaults/persistence-jpa.yml index 9b9ff3ce..64a0eb39 100644 --- a/src/adapter/outbound/persistence-jpa/src/main/resources/config/module-defaults/persistence-jpa.yml +++ b/src/adapter/outbound/persistence-jpa/src/main/resources/config/module-defaults/persistence-jpa.yml @@ -199,10 +199,10 @@ app: # true | false (whether to run Flyway migrations automatically at startup) migration-on-startup: ${APP_MIGRATION_ON_STARTUP:true} # feature-distributed-lock-contract D5 — distributed lock acquisition contract (try-lock + - # finite wait + lease TTL). Bound to LockSettings. Plain values, not APP_* env keys; the code - # defaults in LockSettings mirror these. + # finite wait + lease TTL). Bound to LockSettings. A plain value, not an APP_* env key; the code + # default in LockSettings mirrors it. The wait is each caller's tryAcquire argument, so there is + # no wait setting (app.lock.wait-time was read by nothing and is now an unknown key). lock: - wait-time: 3s lease-ttl: 30s realtime: replay: @@ -210,6 +210,10 @@ app: # events into the store and exposes replay/resume semantics. enabled: ${APP_REALTIME_REPLAY_ENABLED:false} retention: ${APP_REALTIME_REPLAY_RETENTION:1h} + # ISO-8601 duration — how often LiveEventReaper deletes rows past retention, 1 000 a batch. + # A plain value like the outbox reaper's interval. Bound strictly by LiveEventReplaySettings, + # whose default mirrors it; zero or negative fails startup. + reaper-interval: PT10M outbox: # feature-domain-event-outbox-contract I11 — the tuning values are literal defaults; # NO env placeholders (spec: 신규 env key 없음). Declared, validated and strictly bound by diff --git a/src/adapter/outbound/persistence-jpa/src/postgresqlIntegrationTest/java/dev/caskeleton/adapter/outbound/persistence/liveevent/JpaLiveEventReplayContractTest.java b/src/adapter/outbound/persistence-jpa/src/postgresqlIntegrationTest/java/dev/caskeleton/adapter/outbound/persistence/liveevent/JpaLiveEventReplayContractTest.java index b8b1b3cf..aa2e38ed 100644 --- a/src/adapter/outbound/persistence-jpa/src/postgresqlIntegrationTest/java/dev/caskeleton/adapter/outbound/persistence/liveevent/JpaLiveEventReplayContractTest.java +++ b/src/adapter/outbound/persistence-jpa/src/postgresqlIntegrationTest/java/dev/caskeleton/adapter/outbound/persistence/liveevent/JpaLiveEventReplayContractTest.java @@ -1,5 +1,7 @@ package dev.caskeleton.adapter.outbound.persistence.liveevent; +import static org.assertj.core.api.Assertions.assertThat; + import dev.caskeleton.adapter.outbound.persistence.liveevent.entity.LiveEventEntity; import dev.caskeleton.adapter.outbound.persistence.platform.JpaPlatformContractSupport; import dev.caskeleton.adapter.outbound.persistence.transaction.SpringTransaction; @@ -13,12 +15,17 @@ import java.nio.charset.StandardCharsets; import java.sql.Connection; import java.sql.SQLException; import java.sql.Statement; +import java.time.Clock; import java.time.Duration; +import java.time.Instant; +import java.time.ZoneOffset; import java.util.Map; +import java.util.UUID; import javax.sql.DataSource; import org.junit.jupiter.api.AfterAll; import org.junit.jupiter.api.BeforeAll; import org.junit.jupiter.api.Tag; +import org.junit.jupiter.api.Test; import org.springframework.context.annotation.AnnotationConfigApplicationContext; import org.springframework.context.annotation.Bean; import org.springframework.context.annotation.Configuration; @@ -98,6 +105,40 @@ class JpaLiveEventReplayContractTest extends LiveEventReplayContract { return RETENTION; } + @Test + void theReaperRemovesRowsPastTheirRetentionAndKeepsTheRest() { + // given + String stream = "reaper-" + UUID.randomUUID(); + Instant start = Instant.parse("2026-11-03T00:00:00Z"); + appendAt(stream, "expired", start); + appendAt(stream, "retained", start.plus(RETENTION).minusSeconds(60)); + LiveEventReaper reaper = + new LiveEventReaper( + context.getBean(LiveEventJpaRepository.class), + Clock.fixed(start.plus(RETENTION).plusSeconds(1), ZoneOffset.UTC)); + + // when + var unused = reaper.reap(); + + // then + assertThat( + new JdbcTemplate(support.dataSource()) + .queryForList( + "SELECT payload FROM live_event_log WHERE stream_id = ?", String.class, stream)) + .containsExactly("retained"); + } + + private void appendAt(String streamId, String payload, Instant at) { + JpaLiveEventReplay writer = + new JpaLiveEventReplay( + context.getBean(LiveEventJpaRepository.class), + new JdbcTemplate(support.dataSource()), + Clock.fixed(at, ZoneOffset.UTC), + RETENTION); + var unused = + context.getBean(TransactionPort.class).inWrite(() -> writer.append(streamId, payload)); + } + private static void execute(String sql) throws SQLException { try (Connection connection = support.connection(); Statement statement = connection.createStatement()) { diff --git a/src/adapter/outbound/persistence-jpa/src/postgresqlIntegrationTest/java/dev/caskeleton/adapter/outbound/persistence/notification/NotificationFixtures.java b/src/adapter/outbound/persistence-jpa/src/postgresqlIntegrationTest/java/dev/caskeleton/adapter/outbound/persistence/notification/NotificationFixtures.java index d88d0217..12625a8f 100644 --- a/src/adapter/outbound/persistence-jpa/src/postgresqlIntegrationTest/java/dev/caskeleton/adapter/outbound/persistence/notification/NotificationFixtures.java +++ b/src/adapter/outbound/persistence-jpa/src/postgresqlIntegrationTest/java/dev/caskeleton/adapter/outbound/persistence/notification/NotificationFixtures.java @@ -49,7 +49,8 @@ final class NotificationFixtures { migration("jpa/notification-platform", "V10__variables_payload_envelope_guard.sql"), migration("jpa/notification-platform", "V11__notification_reconciliation_claim_lease.sql"), migration("jpa/notification-platform", "V12__notification_suppression_target_unique.sql"), - migration("jpa/notification-platform", "V13__notification_request_acceptance.sql")); + migration("jpa/notification-platform", "V13__notification_request_acceptance.sql"), + migration("jpa/notification-platform", "V14__notification_provider_event_open_index.sql")); } /** diff --git a/src/adapter/outbound/persistence-jpa/src/postgresqlIntegrationTest/java/dev/caskeleton/adapter/outbound/persistence/notification/ServingStateContractTest.java b/src/adapter/outbound/persistence-jpa/src/postgresqlIntegrationTest/java/dev/caskeleton/adapter/outbound/persistence/notification/ServingStateContractTest.java index b10deb33..20154219 100644 --- a/src/adapter/outbound/persistence-jpa/src/postgresqlIntegrationTest/java/dev/caskeleton/adapter/outbound/persistence/notification/ServingStateContractTest.java +++ b/src/adapter/outbound/persistence-jpa/src/postgresqlIntegrationTest/java/dev/caskeleton/adapter/outbound/persistence/notification/ServingStateContractTest.java @@ -8,6 +8,7 @@ import dev.caskeleton.application.notification.platform.observation.Notification import java.io.IOException; import java.sql.Connection; import java.sql.PreparedStatement; +import java.sql.ResultSet; import java.sql.SQLException; import java.sql.Statement; import java.sql.Timestamp; @@ -15,6 +16,9 @@ import java.time.Clock; import java.time.Duration; import java.time.Instant; import java.time.ZoneOffset; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.List; import java.util.UUID; import javax.sql.DataSource; import org.junit.jupiter.api.AfterAll; @@ -23,7 +27,12 @@ import org.junit.jupiter.api.BeforeEach; import org.junit.jupiter.api.DisplayName; import org.junit.jupiter.api.Tag; import org.junit.jupiter.api.Test; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.CsvSource; import org.springframework.jdbc.core.JdbcTemplate; +import org.springframework.jdbc.core.PreparedStatementCreator; +import org.springframework.jdbc.core.RowMapper; +import org.springframework.jdbc.core.SqlProvider; import org.springframework.jdbc.datasource.SingleConnectionDataSource; /** @@ -100,6 +109,39 @@ class ServingStateContractTest { assertThat(measure().backlogDepth()).isZero(); } + @Test + void aDeliveryWhoseWindowClosedIsNotBacklog() throws SQLException { + // given + // Due, but past its expiry: the claim refuses it and the expiry sweep retires it unsent. + insertDelivery( + "READY_TO_DISPATCH", NOW.minus(Duration.ofMinutes(5)), null, null, NOW.minusSeconds(1)); + + // when + long depth = measure().backlogDepth(); + + // then + assertThat(depth).isZero(); + } + + @ParameterizedTest(name = "[{index}] lease_until = NOW에서 {0}초 → backlog {1}") + @CsvSource({"-1, 1", "0, 0"}) + @DisplayName("lease는 NOW 전에 끝났을 때만 끝난 것이다: claim과 같은 lease_until < now") + void aLeaseHasRunOutOnlyOnceItEndedBeforeNow(long offsetSeconds, long expectedDepth) + throws SQLException { + // given + insertDelivery( + "READY_TO_DISPATCH", + NOW.minus(Duration.ofMinutes(1)), + "worker-1", + NOW.plusSeconds(offsetSeconds)); + + // when + long depth = measure().backlogDepth(); + + // then + assertThat(depth).isEqualTo(expectedDepth); + } + @Test @DisplayName("a delivery held under a live lease is not backlog") void aDeliveryUnderALiveLeaseIsNotBacklog() throws SQLException { @@ -113,22 +155,24 @@ class ServingStateContractTest { } @Test - @DisplayName("a lease whose holder never came back is counted as stuck") void anExpiredLeaseIsStuck() throws SQLException { + // given insertDelivery( "DISPATCHING", NOW.minus(Duration.ofMinutes(20)), "worker-1", NOW.minus(Duration.ofMinutes(10))); + // when NotificationServingState state = measure(); - assertThat(state.stuckLeases()) - .as("a worker died mid-dispatch, so a provider call may have happened with no record") - .isEqualTo(1); - assertThat(state.backlogDepth()) - .as("an expired lease is claimable again, but this row is DISPATCHING, not due") - .isZero(); + // then + assertThat(state) + .as( + "a worker died mid-dispatch, so a provider call may have happened with no record; a" + + " dispatching row is never claimed, so it is not backlog") + .extracting(NotificationServingState::stuckLeases, NotificationServingState::backlogDepth) + .containsExactly(1L, 0L); } @Test @@ -162,6 +206,108 @@ class ServingStateContractTest { assertThat(state.oldestReconciliationAge()).isEqualTo(Duration.ofHours(2)); } + @Test + void theSchedulersBacklogReadIsOneStatement() throws SQLException { + // given + insertDelivery("READY_TO_DISPATCH", NOW.minus(Duration.ofMinutes(30)), null, null); + + // when + List outcome; + try (Connection connection = support.connection()) { + RecordingJdbc jdbc = new RecordingJdbc(new SingleConnectionDataSource(connection, true)); + long depth = new JdbcNotificationServingState(jdbc, CLOCK).backlogDepth(); + outcome = List.of(depth, jdbc.statements.size()); + } + + // then + assertThat(outcome) + .as("the scheduler asks this on every poll; the full snapshot is four statements") + .containsExactly(1L, 1); + } + + @Test + void theProjectionLagReadsOpenEventsThroughTheirIndex() throws SQLException { + // given + insertProviderEvent("PENDING", NOW.minus(Duration.ofMinutes(45)), false); + + // when + String plan; + try (Connection connection = support.connection()) { + RecordingJdbc jdbc = new RecordingJdbc(new SingleConnectionDataSource(connection, true)); + var unused = new JdbcNotificationServingState(jdbc, CLOCK).currentState(); + RecordedStatement lag = + jdbc.statements.stream() + .filter(statement -> statement.sql().contains("notification_provider_event")) + .findFirst() + .orElseThrow(); + plan = explainWithoutSequentialScans(connection, lag); + } + + // then + assertThat(plan) + .as("the event table is never purged; only the index over the open events bounds this read") + .contains("ix_notification_provider_event_open"); + } + + /** The plan PostgreSQL picks for {@code statement} when it may not scan a table sequentially. */ + private static String explainWithoutSequentialScans( + Connection connection, RecordedStatement statement) throws SQLException { + boolean autoCommit = connection.getAutoCommit(); + connection.setAutoCommit(false); + try { + try (Statement off = connection.createStatement()) { + off.execute("SET LOCAL enable_seqscan = off"); + } + StringBuilder plan = new StringBuilder(); + try (PreparedStatement explain = connection.prepareStatement("EXPLAIN " + statement.sql())) { + for (int index = 0; index < statement.args().size(); index++) { + explain.setObject(index + 1, statement.args().get(index)); + } + try (ResultSet rows = explain.executeQuery()) { + while (rows.next()) { + plan.append(rows.getString(1)).append('\n'); + } + } + } + return plan.toString(); + } finally { + connection.rollback(); + connection.setAutoCommit(autoCommit); + } + } + + /** A statement the serving state ran, as it ran it. */ + private record RecordedStatement(String sql, List args) {} + + /** Plain JDBC that keeps the statements it was given, so a case reads the shipped SQL. */ + private static final class RecordingJdbc extends JdbcTemplate { + + private final List statements = new ArrayList<>(); + + private RecordingJdbc(DataSource dataSource) { + super(dataSource); + } + + @Override + public T queryForObject(String sql, RowMapper rowMapper, Object... args) { + statements.add(new RecordedStatement(sql, Arrays.asList(args))); + return super.queryForObject(sql, rowMapper, args); + } + + @Override + public T queryForObject(String sql, Class requiredType, Object... args) { + statements.add(new RecordedStatement(sql, Arrays.asList(args))); + return super.queryForObject(sql, requiredType, args); + } + + /** The named-parameter backlog read arrives here; its parameters stay inside the creator. */ + @Override + public List query(PreparedStatementCreator creator, RowMapper rowMapper) { + statements.add(new RecordedStatement(((SqlProvider) creator).getSql(), List.of())); + return super.query(creator, rowMapper); + } + } + private NotificationServingState measure() throws SQLException { // try-with-resources: the pool holds four connections, and a measurement that keeps one is a // measurement that starves the next test. @@ -174,6 +320,16 @@ class ServingStateContractTest { private void insertDelivery( String state, Instant nextDispatchAt, String leaseOwner, Instant leaseUntil) throws SQLException { + insertDelivery(state, nextDispatchAt, leaseOwner, leaseUntil, null); + } + + private void insertDelivery( + String state, + Instant nextDispatchAt, + String leaseOwner, + Instant leaseUntil, + Instant expiresAt) + throws SQLException { UUID requestId = UUID.randomUUID(); try (Connection connection = support.connection()) { try (PreparedStatement statement = @@ -197,17 +353,18 @@ class ServingStateContractTest { + " id, notification_id, tenant_id, recipient_ref, routing_plan_json," + " route_cursor, delivery_state, submission_outcome, delivery_outcome," + " evidence_level, attempt_count, next_dispatch_at, lease_owner, lease_until," - + " created_at, updated_at)" + + " expires_at, created_at, updated_at)" + " VALUES (?, ?, 'tenant-a', 'user-1', '[]'::jsonb, 0, ?, 'NOT_SUBMITTED'," - + " 'UNKNOWN', 'PLATFORM_QUEUED', 0, ?, ?, ?, ?, ?)")) { + + " 'UNKNOWN', 'PLATFORM_QUEUED', 0, ?, ?, ?, ?, ?, ?)")) { statement.setObject(1, UUID.randomUUID()); statement.setObject(2, requestId); statement.setString(3, state); statement.setTimestamp(4, Timestamp.from(nextDispatchAt)); statement.setString(5, leaseOwner); statement.setTimestamp(6, leaseUntil == null ? null : Timestamp.from(leaseUntil)); - statement.setTimestamp(7, Timestamp.from(NOW)); + statement.setTimestamp(7, expiresAt == null ? null : Timestamp.from(expiresAt)); statement.setTimestamp(8, Timestamp.from(NOW)); + statement.setTimestamp(9, Timestamp.from(NOW)); statement.executeUpdate(); } } diff --git a/src/adapter/outbound/persistence-jpa/src/postgresqlIntegrationTest/java/dev/caskeleton/adapter/outbound/persistence/platform/HibernateStatelessSessionRunnerIntegrationTest.java b/src/adapter/outbound/persistence-jpa/src/postgresqlIntegrationTest/java/dev/caskeleton/adapter/outbound/persistence/platform/HibernateStatelessSessionRunnerIntegrationTest.java index 3ac505ad..282e467c 100644 --- a/src/adapter/outbound/persistence-jpa/src/postgresqlIntegrationTest/java/dev/caskeleton/adapter/outbound/persistence/platform/HibernateStatelessSessionRunnerIntegrationTest.java +++ b/src/adapter/outbound/persistence-jpa/src/postgresqlIntegrationTest/java/dev/caskeleton/adapter/outbound/persistence/platform/HibernateStatelessSessionRunnerIntegrationTest.java @@ -2,12 +2,16 @@ package dev.caskeleton.adapter.outbound.persistence.platform; import static org.assertj.core.api.Assertions.assertThat; import static org.assertj.core.api.Assertions.assertThatThrownBy; +import static org.assertj.core.api.Assertions.catchThrowable; import dev.caskeleton.adapter.outbound.persistence.hibernate.stateless.HibernateStatelessSessionRunner; import dev.caskeleton.adapter.outbound.persistence.hibernate.stateless.StatelessRowCapExceededException; import dev.caskeleton.adapter.outbound.persistence.hibernate.stateless.StatelessWorkName; import dev.caskeleton.adapter.outbound.persistence.hibernate.stateless.StatelessWorkResult; import dev.caskeleton.adapter.outbound.persistence.testkit.id.SequenceEntity; +import java.time.Duration; +import java.util.List; +import java.util.Locale; import java.util.Set; import org.junit.jupiter.api.AfterAll; import org.junit.jupiter.api.BeforeAll; @@ -104,6 +108,53 @@ class HibernateStatelessSessionRunnerIntegrationTest { .hasMessageContaining("positive row cap"); } + @Test + void workStillRunningWhenTheTransactionTimeIsUpIsCancelledAndRolledBack() { + // given + var runner = + new HibernateStatelessSessionRunner( + jpa.sessionFactory(), Set.of(WORK), Duration.ofSeconds(1)); + long before = countRows(); + + // when + Throwable failure = + catchThrowable( + () -> + runner.execute( + WORK, + 10, + session -> { + session.insert(new SequenceEntity("timed-out")); + var unused = + session + .createNativeQuery("SELECT 1 FROM pg_sleep(5)", Integer.class) + .getSingleResult(); + return StatelessWorkResult.of(1, 1); + })); + + // then + assertThat(List.of(failure != null && causedByTimeout(failure), countRows())) + .as( + "the transaction's bound cancels the statement; nothing the work wrote is kept: %s", + failure) + .containsExactly(true, before); + } + + /** Whether the failure is the database or Hibernate stopping work that ran out of time. */ + private static boolean causedByTimeout(Throwable failure) { + for (Throwable current = failure; current != null; current = current.getCause()) { + String message = String.valueOf(current.getMessage()).toLowerCase(Locale.ROOT); + if (current instanceof org.hibernate.QueryTimeoutException + || current instanceof jakarta.persistence.QueryTimeoutException + || message.contains("canceling statement due to statement timeout") + || message.contains("canceling statement due to user request") + || message.contains("transaction timeout expired")) { + return true; + } + } + return false; + } + private static long countRows() { return jpa.inTransaction( entityManager -> diff --git a/src/adapter/outbound/persistence-jpa/src/postgresqlIntegrationTest/java/dev/caskeleton/adapter/outbound/persistence/platform/experimental/TenantPoolCapacityContractTest.java b/src/adapter/outbound/persistence-jpa/src/postgresqlIntegrationTest/java/dev/caskeleton/adapter/outbound/persistence/platform/experimental/TenantPoolCapacityContractTest.java index b2ef5043..4b53bf5e 100644 --- a/src/adapter/outbound/persistence-jpa/src/postgresqlIntegrationTest/java/dev/caskeleton/adapter/outbound/persistence/platform/experimental/TenantPoolCapacityContractTest.java +++ b/src/adapter/outbound/persistence-jpa/src/postgresqlIntegrationTest/java/dev/caskeleton/adapter/outbound/persistence/platform/experimental/TenantPoolCapacityContractTest.java @@ -2,6 +2,7 @@ package dev.caskeleton.adapter.outbound.persistence.platform.experimental; import static org.assertj.core.api.Assertions.assertThat; import static org.assertj.core.api.Assertions.assertThatThrownBy; +import static org.assertj.core.api.Assertions.catchThrowable; import com.zaxxer.hikari.HikariConfig; import com.zaxxer.hikari.HikariDataSource; @@ -12,6 +13,7 @@ import dev.caskeleton.application.tenant.TenantId; import java.sql.Connection; import java.sql.SQLException; import java.sql.Statement; +import java.util.ArrayList; import java.util.List; import org.junit.jupiter.api.AfterAll; import org.junit.jupiter.api.BeforeAll; @@ -51,9 +53,9 @@ class TenantPoolCapacityContractTest { void budgetRefusesTheTenantThatWouldExceedTheCeiling() { var budget = new TenantPoolBudget(4, 8); - budget.requireCapacity(3, 6); - assertThat(budget.hasCapacity(4, 8)).isFalse(); - assertThatThrownBy(() -> budget.requireCapacity(4, 8)) + budget.requireCapacity(3, 6, 2); + assertThat(budget.hasCapacity(3, 7, 2)).isFalse(); + assertThatThrownBy(() -> budget.requireCapacity(3, 7, 2)) .isInstanceOf(IllegalStateException.class); } @@ -77,30 +79,50 @@ class TenantPoolCapacityContractTest { void openingPastTheBudgetIsRefusedBeforeAPoolIsCreated() { var budget = new TenantPoolBudget(2, 4); try (var registry = - new TenantDataSourceRegistry( - TenantPoolCapacityContractTest::poolFor, - TenantPoolCapacityContractTest::poolSizeOf, - budget)) { + new TenantDataSourceRegistry(this::poolFor, tenant -> POOL_SIZE_PER_TENANT, budget)) { registry.openTenants(2); assertThat(registry.openPools()).isEqualTo(2); assertThat(registry.allocatedConnections()).isEqualTo(4); assertThatThrownBy(() -> registry.openTenants(3)).isInstanceOf(IllegalStateException.class); - assertThat(registry.openPools()) - .as("a refused tenant must not leave a half-built pool behind") - .isEqualTo(2); + assertThat(List.of(registry.openPools(), opened.size())) + .as("a refused tenant must not leave a half-built pool behind, or have one built at all") + .containsExactly(2, 2); } } + @Test + void aTenantWhosePoolWouldPassTheConnectionCeilingIsRefusedBeforeItsPoolOpens() { + // given + // Two pools of two use four of five connections; the third pool would make six. + var budget = new TenantPoolBudget(4, 5); + List outcome; + try (var registry = + new TenantDataSourceRegistry(this::poolFor, tenant -> POOL_SIZE_PER_TENANT, budget)) { + registry.openTenants(2); + + // when + Throwable refused = catchThrowable(() -> registry.require(new TenantId("tenant-2"))); + outcome = + List.of( + refused instanceof IllegalStateException, + registry.openPools(), + registry.allocatedConnections(), + opened.size()); + } + + // then + assertThat(outcome) + .as("the third pool is refused on its planned size, so it never opens a connection") + .containsExactly(true, 2, 4, 2); + } + @Test @DisplayName("every opened tenant pool reaches the real server") void everyOpenedTenantPoolReachesTheServer() throws SQLException { var budget = new TenantPoolBudget(3, 6); try (var registry = - new TenantDataSourceRegistry( - TenantPoolCapacityContractTest::poolFor, - TenantPoolCapacityContractTest::poolSizeOf, - budget)) { + new TenantDataSourceRegistry(this::poolFor, tenant -> POOL_SIZE_PER_TENANT, budget)) { registry.openTenants(3); for (TenantId tenant : List.copyOf(registry.openTenantIds())) { @@ -118,10 +140,7 @@ class TenantPoolCapacityContractTest { void evictingATenantReturnsItsConnectionsToTheBudget() { var budget = new TenantPoolBudget(2, 4); try (var registry = - new TenantDataSourceRegistry( - TenantPoolCapacityContractTest::poolFor, - TenantPoolCapacityContractTest::poolSizeOf, - budget)) { + new TenantDataSourceRegistry(this::poolFor, tenant -> POOL_SIZE_PER_TENANT, budget)) { registry.openTenants(2); TenantId evicted = List.copyOf(registry.openTenantIds()).get(0); @@ -133,18 +152,19 @@ class TenantPoolCapacityContractTest { } } - private static javax.sql.DataSource poolFor(TenantId tenant) { + /** Every pool this test's factory built, so a test can see one was never opened. */ + private final List opened = new ArrayList<>(); + + private javax.sql.DataSource poolFor(TenantId tenant) { var config = new HikariConfig(); config.setJdbcUrl(support.jdbcUrl()); config.setUsername(support.username()); config.setPassword(support.password()); config.setMaximumPoolSize(POOL_SIZE_PER_TENANT); config.setPoolName("tenant-" + tenant.value()); - return new HikariDataSource(config); - } - - private static int poolSizeOf(javax.sql.DataSource pool) { - return ((HikariDataSource) pool).getMaximumPoolSize(); + HikariDataSource pool = new HikariDataSource(config); + opened.add(pool); + return pool; } private static int serverMaxConnections() throws SQLException { diff --git a/src/adapter/outbound/persistence-jpa/src/postgresqlIntegrationTest/java/dev/caskeleton/adapter/outbound/persistence/readiness/PostgreSqlOptionalStreamLifecycle.java b/src/adapter/outbound/persistence-jpa/src/postgresqlIntegrationTest/java/dev/caskeleton/adapter/outbound/persistence/readiness/PostgreSqlOptionalStreamLifecycle.java index f25dfaee..6a47fc8d 100644 --- a/src/adapter/outbound/persistence-jpa/src/postgresqlIntegrationTest/java/dev/caskeleton/adapter/outbound/persistence/readiness/PostgreSqlOptionalStreamLifecycle.java +++ b/src/adapter/outbound/persistence-jpa/src/postgresqlIntegrationTest/java/dev/caskeleton/adapter/outbound/persistence/readiness/PostgreSqlOptionalStreamLifecycle.java @@ -104,12 +104,14 @@ final class PostgreSqlOptionalStreamLifecycle { // CONFLICT instead of a find-then-insert two webhooks could both win. // V13 records whether a stored request was accepted or dropped by a deduplication window, // so a retry with the same idempotency key gets the same receipt (review 05-23c). + // V14 indexes the provider events still waiting for something, so the projection lag reads + // them and not the whole history (round9 review 2026-11-12, RR9-01-14). // // This list is the stream's contract, not a note about its length: the lifecycle below // disables, re-enables and interrupts the stream and asserts the applied set is unchanged // each time. So a migration added to the stream belongs here, and the version that landed // without being added is why the lane failed the first time anybody ran it. - List.of("0", "1", "2", "3", "4", "5", "6", "7", "8", "9", "10", "11", "12", "13"), + List.of("0", "1", "2", "3", "4", "5", "6", "7", "8", "9", "10", "11", "12", "13", "14"), List.of()); } diff --git a/src/adapter/outbound/persistence-jpa/src/postgresqlIntegrationTest/java/dev/caskeleton/adapter/outbound/persistence/readiness/RunbookSql.java b/src/adapter/outbound/persistence-jpa/src/postgresqlIntegrationTest/java/dev/caskeleton/adapter/outbound/persistence/readiness/RunbookSql.java index ba1855cd..6e4f3c8c 100644 --- a/src/adapter/outbound/persistence-jpa/src/postgresqlIntegrationTest/java/dev/caskeleton/adapter/outbound/persistence/readiness/RunbookSql.java +++ b/src/adapter/outbound/persistence-jpa/src/postgresqlIntegrationTest/java/dev/caskeleton/adapter/outbound/persistence/readiness/RunbookSql.java @@ -8,9 +8,13 @@ import java.util.Map; /** * The SQL a runbook gives an operator, read out of the runbook itself. Every lane that runs a - * runbook's SQL reads it through this one class: so far the outbox runbooks and the migration - * runbook's recovery of a failed concurrent index migration (re-review 2026-11-03 round 8, - * RR8-03-03: the class kept an outbox name after the migration runbook began to use it). + * runbook's SQL reads it through this one class: so far the outbox runbooks ({@code + * PostgreSqlOutboxDispositionRunbookIntegrationTest}, {@code + * PostgreSqlOutboxOrderKeyRunbookIntegrationTest}), the migration runbook's recovery of a failed + * concurrent index migration ({@code PostgreSqlConcurrentIndexMigrationIntegrationTest}) and the + * lock-timeout runbook's holder lookup ({@code PostgreSqlDistributedLockIntegrationTest}). + * Re-review 2026-11-03 round 8, RR8-03-03: the class kept an outbox name after the migration + * runbook began to use it; round9 review 2026-11-12, RR9-01-10: this list missed the lock lane. * *

A runbook block is marked {@code } and followed by a {@code sql} * fence. The lanes that run these blocks against real PostgreSQL read them from the document rather diff --git a/src/adapter/outbound/persistence-jpa/src/test/java/dev/caskeleton/adapter/outbound/persistence/autoconfigure/JpaLiveEventReplayActivationTest.java b/src/adapter/outbound/persistence-jpa/src/test/java/dev/caskeleton/adapter/outbound/persistence/autoconfigure/JpaLiveEventReplayActivationTest.java index 98739e4d..a483b2fe 100644 --- a/src/adapter/outbound/persistence-jpa/src/test/java/dev/caskeleton/adapter/outbound/persistence/autoconfigure/JpaLiveEventReplayActivationTest.java +++ b/src/adapter/outbound/persistence-jpa/src/test/java/dev/caskeleton/adapter/outbound/persistence/autoconfigure/JpaLiveEventReplayActivationTest.java @@ -1,18 +1,34 @@ package dev.caskeleton.adapter.outbound.persistence.autoconfigure; import static org.assertj.core.api.Assertions.assertThat; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.ArgumentMatchers.anyInt; import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.verify; import dev.caskeleton.adapter.outbound.persistence.liveevent.JpaLiveEventReplay; import dev.caskeleton.adapter.outbound.persistence.liveevent.LiveEventJpaRepository; +import dev.caskeleton.adapter.outbound.persistence.liveevent.LiveEventReaper; import java.time.Clock; +import java.time.Duration; +import java.time.Instant; +import java.util.ArrayList; +import java.util.List; +import org.junit.jupiter.api.DisplayName; import org.junit.jupiter.api.Test; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.CsvSource; +import org.springframework.boot.context.properties.bind.UnboundConfigurationPropertiesException; import org.springframework.boot.convert.ApplicationConversionService; import org.springframework.boot.test.context.runner.ApplicationContextRunner; +import org.springframework.context.ApplicationContext; import org.springframework.context.annotation.Bean; import org.springframework.context.annotation.Configuration; import org.springframework.context.annotation.Import; import org.springframework.jdbc.core.JdbcOperations; +import org.springframework.scheduling.config.FixedDelayTask; +import org.springframework.scheduling.config.ScheduledTask; +import org.springframework.scheduling.config.ScheduledTaskHolder; class JpaLiveEventReplayActivationTest { @@ -34,6 +50,110 @@ class JpaLiveEventReplayActivationTest { }); } + @Test + void theReaperIsAssembledWithTheReplayStore() { + runner + .withPropertyValues( + "app.realtime.replay.enabled=true", "app.realtime.replay.retention=PT1H") + .run( + context -> + assertThat(context) + .as("without the reaper nothing removes a row past its retention") + .hasSingleBean(LiveEventReaper.class)); + } + + @Test + void noReaperIsAssembledWithoutTheReplayStore() { + runner.run(context -> assertThat(context).doesNotHaveBean(LiveEventReaper.class)); + } + + @Test + void aZeroReaperIntervalFailsStartup() { + runner + .withPropertyValues( + "app.realtime.replay.enabled=true", + "app.realtime.replay.retention=PT1H", + "app.realtime.replay.reaper-interval=PT0S") + .run( + context -> + assertThat(context.getStartupFailure()) + .rootCause() + .hasMessageContaining("app.realtime.replay.reaper-interval must be positive")); + } + + @Test + void aMisspeltReplayKeyFailsStartup() { + runner + .withPropertyValues( + "app.realtime.replay.enabled=true", + "app.realtime.replay.retention=PT1H", + "app.realtime.replay.reaper-intervall=PT1M") + .run( + context -> + assertThat(context.getStartupFailure()) + .rootCause() + .isInstanceOf(UnboundConfigurationPropertiesException.class) + .hasMessageContaining("reaper-intervall")); + } + + @ParameterizedTest(name = "[{index}] reaper-interval {0} → {1}") + @CsvSource( + value = {"UNSET, PT10M", "PT3M, PT3M"}, + nullValues = "UNSET") + @DisplayName("reaper는 묶인 reaper-interval(기본 PT10M)마다 돌고, 첫 회도 그 간격 뒤다") + void theReaperRunsAtTheBoundInterval(String configured, Duration expected) { + // given + ApplicationContextRunner replay = + runner.withPropertyValues( + "app.realtime.replay.enabled=true", "app.realtime.replay.retention=PT1H"); + if (configured != null) { + replay = replay.withPropertyValues("app.realtime.replay.reaper-interval=" + configured); + } + + // when + List> schedules = new ArrayList<>(); + replay.run(context -> schedules.addAll(fixedDelaySchedules(context))); + + // then + assertThat(schedules) + .as("the schedule is the bound settings' interval, its default included") + .containsExactly(List.of(expected, expected)); + } + + @Test + void theScheduledTaskIsTheReaper() { + // given + ApplicationContextRunner replay = + runner.withPropertyValues( + "app.realtime.replay.enabled=true", "app.realtime.replay.retention=PT1H"); + + // when + List reapedThrough = new ArrayList<>(); + replay.run( + context -> { + fixedDelayTasks(context).forEach(task -> task.getRunnable().run()); + reapedThrough.add(context.getBean(LiveEventJpaRepository.class)); + }); + + // then + verify(reapedThrough.getFirst()).deleteExpiredBatch(any(Instant.class), anyInt()); + } + + /** Interval and initial delay of every fixed-delay task the context scheduled. */ + private static List> fixedDelaySchedules(ApplicationContext context) { + return fixedDelayTasks(context).stream() + .map(task -> List.of(task.getIntervalDuration(), task.getInitialDelayDuration())) + .toList(); + } + + private static List fixedDelayTasks(ApplicationContext context) { + return context.getBean(ScheduledTaskHolder.class).getScheduledTasks().stream() + .map(ScheduledTask::getTask) + .filter(FixedDelayTask.class::isInstance) + .map(FixedDelayTask.class::cast) + .toList(); + } + @Test void enabledReplayCapabilityRequiresExplicitRetention() { runner @@ -42,7 +162,8 @@ class JpaLiveEventReplayActivationTest { context -> { assertThat(context).hasFailed(); assertThat(context.getStartupFailure()) - .hasMessageContaining("app.realtime.replay.retention"); + .rootCause() + .hasMessageContaining("app.realtime.replay.retention must be set"); }); } diff --git a/src/adapter/outbound/persistence-jpa/src/test/java/dev/caskeleton/adapter/outbound/persistence/experimental/ExperimentalFailClosedTest.java b/src/adapter/outbound/persistence-jpa/src/test/java/dev/caskeleton/adapter/outbound/persistence/experimental/ExperimentalFailClosedTest.java index a560adf4..74694d1f 100644 --- a/src/adapter/outbound/persistence-jpa/src/test/java/dev/caskeleton/adapter/outbound/persistence/experimental/ExperimentalFailClosedTest.java +++ b/src/adapter/outbound/persistence-jpa/src/test/java/dev/caskeleton/adapter/outbound/persistence/experimental/ExperimentalFailClosedTest.java @@ -2,6 +2,7 @@ package dev.caskeleton.adapter.outbound.persistence.experimental; import static org.assertj.core.api.Assertions.assertThat; import static org.assertj.core.api.Assertions.assertThatThrownBy; +import static org.assertj.core.api.Assertions.catchThrowable; import dev.caskeleton.adapter.outbound.persistence.experimental.database.TenantPoolBudget; import dev.caskeleton.adapter.outbound.persistence.experimental.replica.ConsistencyAwareDataSourceRouter; @@ -103,14 +104,28 @@ class ExperimentalFailClosedTest { void tenantPoolBudgetIsBounded() { var budget = new TenantPoolBudget(2, 20); - assertThat(budget.hasCapacity(1, 10)).isTrue(); - assertThatThrownBy(() -> budget.requireCapacity(2, 10)) + assertThat(budget.hasCapacity(1, 10, 10)).isTrue(); + assertThatThrownBy(() -> budget.requireCapacity(2, 10, 1)) .isInstanceOf(IllegalStateException.class) .hasMessageContaining("tenant pool budget"); - assertThatThrownBy(() -> budget.requireCapacity(1, 20)) + assertThatThrownBy(() -> budget.requireCapacity(1, 20, 1)) .isInstanceOf(IllegalStateException.class); } + @Test + void aNewPoolLargerThanTheConnectionsLeftIsRefused() { + // given + var budget = new TenantPoolBudget(4, 20); + + // when + Throwable refused = catchThrowable(() -> budget.requireCapacity(1, 15, 10)); + + // then + assertThat(refused) + .isInstanceOf(IllegalStateException.class) + .hasMessageContaining("tenant pool budget exhausted"); + } + /** A lag monitor with a fixed answer, including "I do not know". */ private record FixedLagMonitor(Optional replayedThrough, Optional lag) implements ReplicaLagMonitor {} diff --git a/src/adapter/outbound/persistence-jpa/src/test/java/dev/caskeleton/adapter/outbound/persistence/hibernate/stateless/StatelessRunnerTimeoutTest.java b/src/adapter/outbound/persistence-jpa/src/test/java/dev/caskeleton/adapter/outbound/persistence/hibernate/stateless/StatelessRunnerTimeoutTest.java new file mode 100644 index 00000000..1fb7f801 --- /dev/null +++ b/src/adapter/outbound/persistence-jpa/src/test/java/dev/caskeleton/adapter/outbound/persistence/hibernate/stateless/StatelessRunnerTimeoutTest.java @@ -0,0 +1,45 @@ +package dev.caskeleton.adapter.outbound.persistence.hibernate.stateless; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.catchThrowable; +import static org.mockito.Mockito.mock; + +import dev.caskeleton.adapter.outbound.persistence.transaction.SpringTransaction; +import java.time.Duration; +import java.util.Set; +import org.hibernate.SessionFactory; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.ValueSource; + +/** + * The stateless runner's transaction bound is the write bound every other transaction here has. + * + *

The runner restates the value instead of reading it, because the {@code hibernate} package may + * not depend on {@code transaction} (JpaModuleBoundaryTest). A test may depend on both, so it is + * the place the two copies are held to each other (round9 review 2026-11-12, RR9-01-25). + */ +class StatelessRunnerTimeoutTest { + + @Test + void theRunnersDefaultIsSpringTransactionsWriteDefault() { + assertThat(HibernateStatelessSessionRunner.DEFAULT_TIMEOUT) + .isEqualTo(SpringTransaction.Timeouts.DEFAULT.write()); + } + + @ParameterizedTest + @ValueSource(strings = {"PT0S", "PT-1S", "PT1.5S", "PT2147483648S"}) + void aTimeoutThatIsNotAWholePositiveNumberOfSecondsIsTheAssemblysFault(String timeout) { + // when + Throwable refused = + catchThrowable( + () -> + new HibernateStatelessSessionRunner( + mock(SessionFactory.class), Set.of(), Duration.parse(timeout))); + + // then + assertThat(refused) + .isInstanceOf(IllegalStateException.class) + .hasMessageContaining("must be a positive whole number of seconds"); + } +} diff --git a/src/adapter/outbound/persistence-jpa/src/test/java/dev/caskeleton/adapter/outbound/persistence/liveevent/LiveEventReaperTest.java b/src/adapter/outbound/persistence-jpa/src/test/java/dev/caskeleton/adapter/outbound/persistence/liveevent/LiveEventReaperTest.java new file mode 100644 index 00000000..b43a316e --- /dev/null +++ b/src/adapter/outbound/persistence-jpa/src/test/java/dev/caskeleton/adapter/outbound/persistence/liveevent/LiveEventReaperTest.java @@ -0,0 +1,53 @@ +package dev.caskeleton.adapter.outbound.persistence.liveevent; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.times; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.when; + +import java.time.Clock; +import java.time.Instant; +import java.time.ZoneOffset; +import org.junit.jupiter.api.Test; + +/** + * Unit tests for {@link LiveEventReaper}: rows retained no later than the clock's now go, in + * bounded batches until one comes back short. + * + *

The repository is a Spring Data boundary, so Mockito stands in for it. The statement runs + * against PostgreSQL in {@code JpaLiveEventReplayContractTest}. + */ +class LiveEventReaperTest { + + private static final Instant NOW = Instant.parse("2026-11-13T00:00:00Z"); + private static final int BATCH = LiveEventReaper.BATCH_SIZE; + + private final LiveEventJpaRepository repository = mock(LiveEventJpaRepository.class); + private final LiveEventReaper reaper = + new LiveEventReaper(repository, Clock.fixed(NOW, ZoneOffset.UTC)); + + @Test + void reapKeepsDeletingBatchesUntilOneComesBackShortAndReturnsTheTotal() { + // given + when(repository.deleteExpiredBatch(NOW, BATCH)).thenReturn(BATCH, 4); + + // when + int purged = reaper.reap(); + + // then + assertThat(purged).isEqualTo(BATCH + 4); + } + + @Test + void eachBatchIsItsOwnDeleteStatement() { + // given + when(repository.deleteExpiredBatch(NOW, BATCH)).thenReturn(BATCH, 4); + + // when + var unused = reaper.reap(); + + // then + verify(repository, times(2)).deleteExpiredBatch(NOW, BATCH); + } +} diff --git a/src/adapter/outbound/persistence-jpa/src/test/java/dev/caskeleton/adapter/outbound/persistence/lock/DistributedLockPersistenceConfigTest.java b/src/adapter/outbound/persistence-jpa/src/test/java/dev/caskeleton/adapter/outbound/persistence/lock/DistributedLockPersistenceConfigTest.java index 72fe3def..471a36ba 100644 --- a/src/adapter/outbound/persistence-jpa/src/test/java/dev/caskeleton/adapter/outbound/persistence/lock/DistributedLockPersistenceConfigTest.java +++ b/src/adapter/outbound/persistence-jpa/src/test/java/dev/caskeleton/adapter/outbound/persistence/lock/DistributedLockPersistenceConfigTest.java @@ -49,6 +49,20 @@ class DistributedLockPersistenceConfigTest { }); } + @Test + @DisplayName("읽는 곳이 없던 app.lock.wait-time은 모르는 키로 기동을 실패시킨다") + void theRemovedWaitTimeKeyFailsStartup() { + // when + contextRunner + .withPropertyValues("app.lock.wait-time=3s") + .run( + context -> { + // then + assertThat(context).hasFailed(); + assertThat(context.getStartupFailure()).hasStackTraceContaining("wait-time"); + }); + } + @Test void multiInstanceBuildsTheJdbcProviderAndNotTheInProcessProvider() { contextRunner diff --git a/src/adapter/outbound/persistence-jpa/src/test/resources/db/applied-migrations.sha256 b/src/adapter/outbound/persistence-jpa/src/test/resources/db/applied-migrations.sha256 index 32edf890..93a51fe4 100644 --- a/src/adapter/outbound/persistence-jpa/src/test/resources/db/applied-migrations.sha256 +++ b/src/adapter/outbound/persistence-jpa/src/test/resources/db/applied-migrations.sha256 @@ -14,6 +14,7 @@ be51fabd07220f9ef885534ba36ada5f1491baf24cdfb54d25d93a19ddc29926 jpa/notification-platform/V11__notification_reconciliation_claim_lease.sql 43f9429f91c07528154ec01b721649b2c25fa98857ac154c7c08f4655634657f jpa/notification-platform/V12__notification_suppression_target_unique.sql d01aaf59f66834358abf18246faa9a9eb8dfa9f42097ca9d968c44edff564630 jpa/notification-platform/V13__notification_request_acceptance.sql +8a7dc3604d44f1438b39dfec3aa688f792e7363cd3c1b74bf0c799a6b53f9f0d jpa/notification-platform/V14__notification_provider_event_open_index.sql 1ac7cc1c2509868eb0ce202e1917bb36846f98f17ad4922abe3fac0ef128199b jpa/notification-platform/V1__notification_platform_core.sql 0add67aa23580e51712f84fabe9b09e08401ef708cfa1b3292b120fdb5c12fa2 jpa/notification-platform/V2__notification_platform_contact_template_policy.sql f5b98598a59cc4376f5d233b410f8a5ae9f3cd94156291ebdf2d0da5a66f34ef jpa/notification-platform/V3__notification_platform_inbox_admin.sql diff --git a/src/adapter/outbound/persistence-notification/src/main/java/dev/caskeleton/adapter/outbound/persistence/notification/platform/JdbcNotificationServingState.java b/src/adapter/outbound/persistence-notification/src/main/java/dev/caskeleton/adapter/outbound/persistence/notification/platform/JdbcNotificationServingState.java index 5a81e94b..72a52a8e 100644 --- a/src/adapter/outbound/persistence-notification/src/main/java/dev/caskeleton/adapter/outbound/persistence/notification/platform/JdbcNotificationServingState.java +++ b/src/adapter/outbound/persistence-notification/src/main/java/dev/caskeleton/adapter/outbound/persistence/notification/platform/JdbcNotificationServingState.java @@ -1,13 +1,17 @@ package dev.caskeleton.adapter.outbound.persistence.notification.platform; +import dev.caskeleton.adapter.outbound.persistence.notification.platform.repository.RecipientClaimSql; import dev.caskeleton.application.notification.platform.observation.NotificationServingState; import dev.caskeleton.application.notification.platform.observation.NotificationServingStatePort; import java.sql.Timestamp; import java.time.Clock; import java.time.Duration; import java.time.Instant; +import java.util.Map; import java.util.Objects; import org.springframework.jdbc.core.JdbcOperations; +import org.springframework.jdbc.core.namedparam.NamedParameterJdbcOperations; +import org.springframework.jdbc.core.namedparam.NamedParameterJdbcTemplate; /** * Measures the platform's serving state from the durable rows. @@ -17,34 +21,44 @@ import org.springframework.jdbc.core.JdbcOperations; * identical. These are the questions an operator actually asks during an incident: how far behind * are we, how long has the oldest thing waited, and is anything stuck. * - *

One statement per question rather than one joined statement. They are read independently, a - * slow one does not delay the others, and each index the queries rely on is the same index the - * corresponding worker already uses to claim. + *

One statement per question rather than one joined statement. They are read independently and a + * slow one does not delay the others. Every statement is bounded by an index: the backlog and + * stuck-lease counts by the predicates the claim uses, the reconciliation count by the job's due + * time, and the projection lag by {@code ix_notification_provider_event_open} (V14), which holds + * only the events still waiting for something — the provider-event table itself is never purged, so + * the lag query used to read all of it (round9 review 2026-11-12, RR9-01-14). + * + *

The scheduler reads only {@link #backlogDepth()}, on every poll; the full snapshot is for the + * health endpoint. */ public final class JdbcNotificationServingState implements NotificationServingStatePort { /** - * Deliveries that are due and unclaimed. + * Deliveries the claim would take now. * - *

The lease predicate matches the claim's: a row whose lease has expired is available again, - * so it is backlog. Counting it as in-flight is how a stuck worker hides a growing queue. + *

The {@code WHERE} is the claim's own fragment ({@link RecipientClaimSql#CLAIMABLE_AT_NOW}): + * the same states, due, not expired, and a lease that is absent or has run out. A row whose lease + * has expired is available again, so it is backlog; counting it as in-flight is how a stuck + * worker hides a growing queue. An expired delivery is not backlog: the expiry sweep retires it + * unsent. The fragment's parameter is named, so this statement is too. */ private static final String BACKLOG = """ SELECT count(*) AS depth, - COALESCE(EXTRACT(EPOCH FROM (? - min(next_dispatch_at))), 0) AS oldest_seconds + COALESCE(EXTRACT(EPOCH FROM (:now - min(next_dispatch_at))), 0) AS oldest_seconds FROM notification_recipient_delivery - WHERE delivery_state IN ('PENDING', 'READY_TO_DISPATCH', 'RETRY_WAITING') - AND next_dispatch_at IS NOT NULL - AND next_dispatch_at <= ? - AND (lease_until IS NULL OR lease_until <= ?) - """; + WHERE\s""" + + RecipientClaimSql.CLAIMABLE_AT_NOW; /** * Leases whose holder never came back. * - *

Distinct from backlog: these rows are claimable again, but the fact that they had to expire - * means a worker died mid-dispatch, and a provider call may have happened without a record. + *

Distinct from backlog: these rows are still dispatching, which the claim never takes; lease + * recovery requeues or reconciles them one lease after they ran out. A row left from before + * notification V4 (an owner but fence 0) is counted here and not recovered; {@code + * docs/upgrade/2026-11-round9-claim-fence.md}, row 5, says how to hand it to recovery. That a + * lease had to run out means a worker died or stalled mid-dispatch, and a provider call may have + * happened without a record. */ private static final String STUCK_LEASES = """ @@ -56,7 +70,21 @@ public final class JdbcNotificationServingState implements NotificationServingSt AND delivery_state = 'DISPATCHING' """; - /** Provider events stored and not yet applied to a delivery. */ + /** + * The provider events still waiting for something: the lag query's {@code WHERE}, and the + * predicate of the partial index {@code V14} builds for it. One string, held to the migration by + * {@code JdbcNotificationServingStateOpenEventsTest}: a query the index does not cover reads the + * whole table again, which is what the index was made to stop. + */ + static final String OPEN_EVENTS = + "projection_status IN ('PENDING', 'FAILED') OR attempt_id IS NULL"; + + /** + * Provider events stored and not yet applied to a delivery. + * + *

The {@code WHERE} is exactly the predicate of {@code ix_notification_provider_event_open}: + * every row any of the four counts can see, and none of the applied, matched history. + */ private static final String PROJECTION_LAG = """ SELECT @@ -66,7 +94,9 @@ public final class JdbcNotificationServingState implements NotificationServingSt COALESCE(EXTRACT(EPOCH FROM (? - min(received_at) FILTER (WHERE projection_status = 'PENDING'))), 0) AS oldest_seconds FROM notification_provider_event - """; + WHERE + """ + + OPEN_EVENTS; /** Questions to providers that are due and unanswered. */ private static final String RECONCILIATION_LAG = @@ -78,26 +108,36 @@ public final class JdbcNotificationServingState implements NotificationServingSt """; private final JdbcOperations jdbc; + private final NamedParameterJdbcOperations named; private final Clock clock; public JdbcNotificationServingState(JdbcOperations jdbc, Clock clock) { this.jdbc = Objects.requireNonNull(jdbc, "jdbc"); + this.named = new NamedParameterJdbcTemplate(jdbc); this.clock = Objects.requireNonNull(clock, "clock"); } + @Override + public long backlogDepth() { + Measurement backlog = measureBacklog(Timestamp.from(clock.instant())); + return backlog == null ? 0 : backlog.count(); + } + + /** The backlog query alone: the one statement the scheduler's queue-depth gauge needs. */ + private Measurement measureBacklog(Timestamp at) { + return named.queryForObject( + BACKLOG, + Map.of("now", at), + (rows, rowNumber) -> + new Measurement(rows.getLong("depth"), rows.getDouble("oldest_seconds"))); + } + @Override public NotificationServingState currentState() { Instant now = clock.instant(); Timestamp at = Timestamp.from(now); - Measurement backlog = - jdbc.queryForObject( - BACKLOG, - (rows, rowNumber) -> - new Measurement(rows.getLong("depth"), rows.getDouble("oldest_seconds")), - at, - at, - at); + Measurement backlog = measureBacklog(at); Long stuck = jdbc.queryForObject(STUCK_LEASES, Long.class, at); ProjectionMeasurement projection = jdbc.queryForObject( diff --git a/src/adapter/outbound/persistence-notification/src/main/resources/db/migration/jpa/notification-platform/V14__notification_provider_event_open_index.sql b/src/adapter/outbound/persistence-notification/src/main/resources/db/migration/jpa/notification-platform/V14__notification_provider_event_open_index.sql new file mode 100644 index 00000000..2d2416a2 --- /dev/null +++ b/src/adapter/outbound/persistence-notification/src/main/resources/db/migration/jpa/notification-platform/V14__notification_provider_event_open_index.sql @@ -0,0 +1,26 @@ +-- An index over the provider events still waiting for something. +-- +-- The serving-state projection lag counts pending, failed and unmatched events. It had no WHERE and +-- read the whole notification_provider_event table, which nothing ever purges, and the scheduler +-- asked for it on every poll (round9 review 2026-11-12, RR9-01-14). The query now names exactly +-- this predicate, so it reads only these rows; applied, matched events — the history that grows — +-- are outside the index. +-- +-- Built inside this migration's transaction, not CONCURRENTLY, as V21 of the core stream builds its +-- index: a transactional file fails whole and leaves nothing to repair, where a failed concurrent +-- build leaves an INVALID index and a failed history row an operator must clear by hand +-- (docs/runbooks/migration-failed.md). The cost is that the build holds SHARE on +-- notification_provider_event: provider callbacks wait for it, and providers retry a callback that +-- does not get an answer. On a table already large in production, create the same index +-- CONCURRENTLY by hand before deploying; IF NOT EXISTS then makes this migration a no-op. + +-- Bound the wait for the lock (SHARE on notification_provider_event, held until commit). Unbounded, +-- it queues behind any open transaction that touched the table, and every callback insert queues +-- behind the migration. Five seconds, then the migration fails, rolls back whole and the stream +-- stays at V13: deploy again once that transaction has ended (docs/jpa/migration-guide.md "Bound +-- the lock wait"). +SET LOCAL lock_timeout = '5s'; + +CREATE INDEX IF NOT EXISTS ix_notification_provider_event_open + ON notification_provider_event (received_at) + WHERE projection_status IN ('PENDING', 'FAILED') OR attempt_id IS NULL; diff --git a/src/adapter/outbound/persistence-notification/src/test/java/dev/caskeleton/adapter/outbound/persistence/notification/platform/JdbcNotificationServingStateOpenEventsTest.java b/src/adapter/outbound/persistence-notification/src/test/java/dev/caskeleton/adapter/outbound/persistence/notification/platform/JdbcNotificationServingStateOpenEventsTest.java new file mode 100644 index 00000000..0805a0d6 --- /dev/null +++ b/src/adapter/outbound/persistence-notification/src/test/java/dev/caskeleton/adapter/outbound/persistence/notification/platform/JdbcNotificationServingStateOpenEventsTest.java @@ -0,0 +1,42 @@ +package dev.caskeleton.adapter.outbound.persistence.notification.platform; + +import static java.nio.charset.StandardCharsets.UTF_8; +import static org.assertj.core.api.Assertions.assertThat; + +import java.io.IOException; +import java.io.InputStream; +import java.util.Objects; +import org.junit.jupiter.api.Test; + +/** + * The projection-lag query and the partial index built for it name the same events. + * + *

The index is what keeps the lag query, which the scheduler runs on every poll, from reading + * the whole provider-event history. A predicate that drifted on either side would leave the query + * outside the index and scanning the table again, with nothing failing. + */ +class JdbcNotificationServingStateOpenEventsTest { + + private static final String V14 = + "db/migration/jpa/notification-platform/V14__notification_provider_event_open_index.sql"; + + @Test + void theLagQueryAndItsPartialIndexNameTheSameEvents() throws IOException { + // given + String migration; + try (InputStream sql = getClass().getClassLoader().getResourceAsStream(V14)) { + migration = new String(Objects.requireNonNull(sql, V14).readAllBytes(), UTF_8); + } + int index = migration.indexOf("CREATE INDEX"); + + // when + String indexPredicate = + migration + .substring( + migration.indexOf("WHERE ", index) + "WHERE ".length(), migration.lastIndexOf(';')) + .trim(); + + // then + assertThat(indexPredicate).isEqualTo(JdbcNotificationServingState.OPEN_EVENTS); + } +} diff --git a/src/application/notification/src/main/java/dev/caskeleton/application/notification/platform/observation/NotificationServingState.java b/src/application/notification/src/main/java/dev/caskeleton/application/notification/platform/observation/NotificationServingState.java index ce48cf07..7ddece3f 100644 --- a/src/application/notification/src/main/java/dev/caskeleton/application/notification/platform/observation/NotificationServingState.java +++ b/src/application/notification/src/main/java/dev/caskeleton/application/notification/platform/observation/NotificationServingState.java @@ -12,7 +12,7 @@ import java.util.Objects; * being applied reported exactly the same numbers as an idle healthy one. Readiness that cannot * distinguish those two states is not readiness. * - * @param backlogDepth deliveries due now and not yet claimed + * @param backlogDepth deliveries a claim would take now: due, not expired, not under a live lease * @param oldestDueAge how long the oldest of them has been due * @param stuckLeases leases whose holder has not finished and whose expiry has passed * @param pendingProjections provider events stored and not yet applied diff --git a/src/application/notification/src/main/java/dev/caskeleton/application/notification/platform/observation/NotificationServingStatePort.java b/src/application/notification/src/main/java/dev/caskeleton/application/notification/platform/observation/NotificationServingStatePort.java index 79a817aa..f6ae4f15 100644 --- a/src/application/notification/src/main/java/dev/caskeleton/application/notification/platform/observation/NotificationServingStatePort.java +++ b/src/application/notification/src/main/java/dev/caskeleton/application/notification/platform/observation/NotificationServingStatePort.java @@ -6,7 +6,25 @@ public interface NotificationServingStatePort { /** * Measure the backlog, the stuck leases and the projection and reconciliation lag. * + *

Several questions, for the health snapshot. A caller that needs only the queue depth asks + * {@link #backlogDepth()} instead: the scheduler reads the depth on every poll, and asking all of + * them that often ran four statements a poll, one of them over the whole provider-event table + * (round9 review 2026-11-12, RR9-01-14). + * * @return the current state */ NotificationServingState currentState(); + + /** + * How many deliveries a claim would take now — due, not expired, and not under a live lease: the + * {@link NotificationServingState#backlogDepth()} of {@link #currentState()}, measured alone. + * + *

The default takes it from the full snapshot, so an implementation that can only measure + * everything at once still answers; a store that can count the backlog by itself overrides it. + * + * @return the backlog depth + */ + default long backlogDepth() { + return currentState().backlogDepth(); + } }