남은 트랜잭션·정리 결함 일곱. live event log는 지우는 곳이 없어 끝없이 자란다. app.lock.wait-time은 읽는 곳이 없다. 알림 scheduler는 poll마다 상태 쿼리 넷을 돌리고, projection 지연은 닫힌 event까지 읽는다. stateless runner 트랜잭션에 시한이 없다. tenant pool 예산은 새 pool의 크기를 모른 채 열어 연결 합계가 상한을 넘는다. runbook SQL 설명이 lane 넷 중 셋만 말한다. 고른 방법: LiveEventReaper(1000행씩, 트랜잭션 30초)와 공용 BatchedDeletes를 더한다. app.realtime.replay.*는 LiveEventReplaySettings가 엄격하게 묶고, 조립 설정이 SchedulingConfigurer로 reaper를 reaper-interval(기본 10분)마다 등록한다. LockSettings는 lease-ttl 하나만 받는다(모르는 키로 기동 실패). scheduler는 backlogDepth() 한 문장만 읽고, 그 WHERE는 claim과 같은 SQL 조각(RecipientClaimSql.CLAIMABLE_AT_NOW)이다. projection 지연은 V14 부분 인덱스로 읽는다(lock_timeout, V21 선례대로 CONCURRENTLY 아님). runner 기본 30초(SpringTransaction 쓰기 시한과 같음을 시험이 본다), tenant 예산은 계획된 pool 크기로 열기 전에 본다. 버린 대안: V14를 CONCURRENTLY로(Flyway 트랜잭션 안에서 못 돈다, 큰 DB는 안내대로 미리 만든다), reaper @Scheduled에 SpEL로 settings bean 이름 읽기(문자열 bean 조회는 검사가 못 본다). 위험·한계: LockSettings·TenantDataSourceRegistry·TenantPoolBudget의 공개 생성자가 바뀐다. V14는 큰 notification_provider_event에서 쓰기를 잠깐 막는다(안내 4행에 미리 만드는 법과 INVALID 인덱스 확인). backlog 수치는 창이 닫힌 delivery를 더는 세지 않는다. 검증: desktop RED(pr1-red3) 새 시험 실패 포함. GREEN은 PR 본문. Refs: RR9-01-03, RR9-01-07, RR9-01-09, RR9-01-10, RR9-01-14, RR9-01-25, RR9-01-26 BREAKING CHANGE: JpaLiveEventReplay.sweepExpired 삭제, app.realtime.replay 아래 모르는 키와 app.lock.wait-time은 기동 실패, HibernateStatelessSessionRunner 트랜잭션 30초 상한, TenantPoolBudget·TenantDataSourceRegistry 시그니처 변경. docs/upgrade/2026-11-round9-claim-fence.md
This commit is contained in:
@@ -5,7 +5,7 @@
|
||||
# root yet.
|
||||
# Update only after review with:
|
||||
# ./gradlew :adapter:outbound:persistence-jpa:updateJpaApiSurface -PapproveJpaApiSurfaceChange
|
||||
# types: 166
|
||||
# types: 168
|
||||
dev.caskeleton.adapter.outbound.persistence.api.PersistenceOperationName
|
||||
dev.caskeleton.adapter.outbound.persistence.api.capability.CapabilitySupport
|
||||
dev.caskeleton.adapter.outbound.persistence.api.capability.JpaCapability
|
||||
@@ -104,6 +104,8 @@ dev.caskeleton.adapter.outbound.persistence.idempotency.postgresql.IdempotencySc
|
||||
dev.caskeleton.adapter.outbound.persistence.idempotency.postgresql.PostgreSqlIdempotencyStore
|
||||
dev.caskeleton.adapter.outbound.persistence.liveevent.JpaLiveEventReplay
|
||||
dev.caskeleton.adapter.outbound.persistence.liveevent.LiveEventJpaRepository
|
||||
dev.caskeleton.adapter.outbound.persistence.liveevent.LiveEventReaper
|
||||
dev.caskeleton.adapter.outbound.persistence.liveevent.LiveEventReplaySettings
|
||||
dev.caskeleton.adapter.outbound.persistence.liveevent.entity.LiveEventEntity
|
||||
dev.caskeleton.adapter.outbound.persistence.lock.DistributedLockPersistenceConfig
|
||||
dev.caskeleton.adapter.outbound.persistence.lock.LockRegistryDistributedLock
|
||||
|
||||
@@ -1,13 +1,14 @@
|
||||
# 업그레이드 안내 — 2026-11 claim·fence·트랜잭션 시한·정리
|
||||
|
||||
- 범위: durable operation·알림 수신자·fileserver cleanup·멱등성 기록의 lease, fence, 조건부 쓰기, 정리를 고친
|
||||
변경. 포트 시그니처와 기동 조건이 바뀐 곳이 있다. 아래 표가 그 목록이다.
|
||||
- 근거: 2026-11-12 전체 리뷰(`docs/reviews/2026-11-12-round9-review/`)의 claim·fence(B8) 발견
|
||||
- 범위: durable operation·알림 수신자·fileserver cleanup·멱등성 기록·live event·분산 락·stateless runner의 lease,
|
||||
fence, 조건부 쓰기, 정리, 시간 상한을 고친 변경. 포트 시그니처와 기동 조건이 바뀐 곳이 있다. 아래 표가 그 목록이다.
|
||||
- 근거: 2026-11-12 전체 리뷰(`docs/reviews/2026-11-12-round9-review/`)의 claim·fence(B8)와 트랜잭션 시한·정리(B9) 발견
|
||||
|
||||
## 1. 누가 읽나
|
||||
|
||||
이 템플릿을 `upstream` remote로 두고 merge해 받는 파생 서비스가 읽는다. 특히 durable operation worker를 직접 쓴 서비스,
|
||||
알림 recipient store를 직접 구현한 서비스, 알림 provider timeout을 설정한 서비스다.
|
||||
알림 recipient store를 직접 구현한 서비스, `app.lock.wait-time`이나 알림 provider timeout을 설정한 서비스, experimental
|
||||
tenant pool 예산을 쓰는 서비스다.
|
||||
|
||||
## 2. 할 일
|
||||
|
||||
@@ -16,17 +17,22 @@
|
||||
| 1 | durable operation worker는 `claimNext`가 돌려주는 `DurableOperationClaim`을 들고 있다가 `heartbeat(claim, …)`·`reportProgress(claim, …)`·`succeed(claim, …)`·`fail(claim, …)`에 넘긴다. 예전에는 `(operationId, workerId, …)`였다 | 컴파일이 실패한다 | durable operation worker를 쓴 서비스 |
|
||||
| 2 | `DurableOperationStorePort`를 직접 구현했으면 claim마다 오르는 fence를 저장하고, 네 쓰기가 owner와 fence를 함께 확인하게 한다. `DurableOperationStoreContract`의 새 사례(같은 worker id가 다시 claim한 뒤 앞 실행의 쓰기는 거절)로 확인한다 | 재시작한 worker나 한 프로세스의 두 실행이 같은 id를 쓰면 앞 실행의 결과가 넘겨받은 실행 위에 기록된다 | 자기 durable operation store를 둔 서비스 |
|
||||
| 3 | persistence-jpa의 `V23__durable_operation_lease_fence.sql`(`durable_operation.lease_fence BIGINT NOT NULL DEFAULT 0`)이 기본 stream에 더해진다. 같은 번호의 서비스 마이그레이션이 있으면 번호를 옮긴다. `durable_operation`을 건드린 트랜잭션이 열려 있으면 5초 뒤 실패하고 V22에 남는다(다시 배포) | Flyway가 번호 충돌로 기동하지 않는다 | persistence-jpa를 쓰는 모든 서비스 |
|
||||
| 4 | `RecipientLeaseStorePort.expiredDispatching`이 `List<RecipientLease>`(버려질 때의 owner·fence·만료 시각)를 돌려주고, `RecipientDeliveryStorePort`에 `requeueAbandoned(lease, nextDispatchAt)`가 생겼다. 직접 구현했으면 상태가 `DISPATCHING`이고 owner·fence가 같고 lease가 늘지 않았을 때만 `READY_TO_DISPATCH`로 바꾼다(`RecipientDeliveryStoreContract` 새 사례). owner가 없거나 fence가 0인 `DISPATCHING` 행(notification V4 전에 남은 행)은 복구가 더는 고르지 않는다. 그런 행은 `SELECT id FROM notification_recipient_delivery WHERE delivery_state = 'DISPATCHING' AND (lease_owner IS NULL OR lease_fence = 0)`로 찾는다. 배포 전에 `UPDATE notification_recipient_delivery SET lease_owner = COALESCE(lease_owner, 'pre-v4'), lease_fence = lease_fence + 1, lease_until = now() - interval '1 day', version = version + 1, updated_at = now() WHERE delivery_state = 'DISPATCHING' AND (lease_owner IS NULL OR lease_fence = 0)`로 owner를 채우고 fence를 올려 복구 패스에 넘긴다. 복구는 attempt를 보고 정한다: attempt가 없으면 다시 보낼 차례로 돌리고, 끝나지 않은 attempt가 있으면 맞추기로 넘긴다(그 뒤는 `docs/notification/configuration-reference.md` 「발송」의 lease 문단). 손으로 `READY_TO_DISPATCH`로 바꾸지 않는다. 발송 중에 멈춘 행이면 같은 알림이 한 번 더 나간다. owner가 없는 행은 stuck lease 지표에도 잡히지 않는다 | 컴파일이 실패한다. 조건 없이 바꾸면 끝났거나 다시 claim된 delivery가 다시 발송된다 | 알림 store를 직접 구현한 서비스 |
|
||||
| 5 | 알림 provider profile의 `timeout`이 `app.notification.platform.dispatch.lease-duration`보다 짧은지 확인한다. 짧은 것은 필요조건일 뿐이다: lease는 claim 때 시작해서 attempt 기록·렌더링·provider 허가 대기도 lease를 쓴다. lease는 timeout보다 넉넉히 길게 둔다. 이 timeout이 provider 호출 전체를 묶는 것은 SES(API 호출 시간 상한)다. SMTP는 `spring.mail` socket timeout이 명령마다 묶을 뿐이라 이 검사만으로는 호출 전체가 lease 안에 끝난다고 말할 수 없다 | 짧지 않으면 기동이 실패한다(`notification provider profile '…': timeout (…) must be shorter than …lease-duration`). 전에는 provider 호출 하나가 lease보다 길어도 기동했다. 그때 생기는 일은 `docs/notification/configuration-reference.md` 「발송」의 lease 문단이다 | 알림 capability를 켠 서비스 |
|
||||
| 6 | `IdempotencyReaper`를 직접 만들던 코드는 `new IdempotencyReaper(repository)`로 바꾼다(`Clock` 인자 삭제). 만료 비교는 DB 시계(`now()`)이고 1000행씩 지운다 | 컴파일이 실패한다 | reaper를 직접 조립한 서비스 |
|
||||
| 7 | fileserver cleanup 묶음의 `max-items`를 정할 때, 한 묶음이 claim lease(10분)의 첫 8분 안에 시작할 수 있는 만큼만 처리된다는 것을 안다. 나머지는 시도하지 않은 채 queue에 돌아가 다음 묶음이 다시 claim한다. 시도 횟수는 그대로다(바이트 예산이 바닥나 돌려준 항목도 이제 같다. 전에는 시도 한 번으로 세어 8번째에 보류됐다). 직접 구현한 `CleanupQueuePort`는 새 연산 `release`를 구현한다(`CleanupQueuePortContract` 새 사례) | 느린 저장소에서 큰 `max-items`는 그 사유로 도는 항목이 늘어난다. 전에는 lease가 끝난 뒤에도 남은 항목을 처리해 다른 worker와 겹쳤다 | fileserver capability를 켠 서비스 |
|
||||
| 4 | 알림 stream에 `V14__notification_provider_event_open_index.sql`(부분 인덱스 `ix_notification_provider_event_open`)이 더해진다. `notification_provider_event`가 이미 큰 운영 DB는 배포 전에 같은 인덱스를 `CREATE INDEX CONCURRENTLY IF NOT EXISTS ix_notification_provider_event_open ON notification_provider_event (received_at) WHERE projection_status IN ('PENDING', 'FAILED') OR attempt_id IS NULL;`로 먼저 만든다. 미리 만든 인덱스가 실패해 남았으면(`SELECT indisvalid FROM pg_index WHERE indexrelid = 'ix_notification_provider_event_open'::regclass`가 false) `DROP INDEX CONCURRENTLY ix_notification_provider_event_open` 뒤 다시 만든다. `IF NOT EXISTS`는 쓸 수 없는 인덱스도 있는 것으로 보고 넘어간다 | 마이그레이션이 인덱스를 만드는 동안 provider callback 저장이 기다린다(provider가 callback을 다시 보낸다) | 알림 capability를 켠 서비스 |
|
||||
| 5 | `RecipientLeaseStorePort.expiredDispatching`이 `List<RecipientLease>`(버려질 때의 owner·fence·만료 시각)를 돌려주고, `RecipientDeliveryStorePort`에 `requeueAbandoned(lease, nextDispatchAt)`가 생겼다. 직접 구현했으면 상태가 `DISPATCHING`이고 owner·fence가 같고 lease가 늘지 않았을 때만 `READY_TO_DISPATCH`로 바꾼다(`RecipientDeliveryStoreContract` 새 사례). owner가 없거나 fence가 0인 `DISPATCHING` 행(notification V4 전에 남은 행)은 복구가 더는 고르지 않는다. 그런 행은 `SELECT id FROM notification_recipient_delivery WHERE delivery_state = 'DISPATCHING' AND (lease_owner IS NULL OR lease_fence = 0)`로 찾는다. 배포 전에 `UPDATE notification_recipient_delivery SET lease_owner = COALESCE(lease_owner, 'pre-v4'), lease_fence = lease_fence + 1, lease_until = now() - interval '1 day', version = version + 1, updated_at = now() WHERE delivery_state = 'DISPATCHING' AND (lease_owner IS NULL OR lease_fence = 0)`로 owner를 채우고 fence를 올려 복구 패스에 넘긴다. 복구는 attempt를 보고 정한다: attempt가 없으면 다시 보낼 차례로 돌리고, 끝나지 않은 attempt가 있으면 맞추기로 넘긴다(그 뒤는 `docs/notification/configuration-reference.md` 「발송」의 lease 문단). 손으로 `READY_TO_DISPATCH`로 바꾸지 않는다. 발송 중에 멈춘 행이면 같은 알림이 한 번 더 나간다. owner가 없는 행은 stuck lease 지표에도 잡히지 않는다 | 컴파일이 실패한다. 조건 없이 바꾸면 끝났거나 다시 claim된 delivery가 다시 발송된다 | 알림 store를 직접 구현한 서비스 |
|
||||
| 6 | 알림 provider profile의 `timeout`이 `app.notification.platform.dispatch.lease-duration`보다 짧은지 확인한다. 짧은 것은 필요조건일 뿐이다: lease는 claim 때 시작해서 attempt 기록·렌더링·provider 허가 대기도 lease를 쓴다. lease는 timeout보다 넉넉히 길게 둔다. 이 timeout이 provider 호출 전체를 묶는 것은 SES(API 호출 시간 상한)다. SMTP는 `spring.mail` socket timeout이 명령마다 묶을 뿐이라 이 검사만으로는 호출 전체가 lease 안에 끝난다고 말할 수 없다 | 짧지 않으면 기동이 실패한다(`notification provider profile '…': timeout (…) must be shorter than …lease-duration`). 전에는 provider 호출 하나가 lease보다 길어도 기동했다. 그때 생기는 일은 `docs/notification/configuration-reference.md` 「발송」의 lease 문단이다 | 알림 capability를 켠 서비스 |
|
||||
| 7 | `app.lock.wait-time`을 설정에서 지운다. 기다릴 시간은 `tryAcquire(key, waitTime, leaseTtl)`의 인자다 | `app.lock` 아래 모르는 키라 기동이 실패한다. 전에도 읽는 곳이 없었다 | `app.lock.wait-time`을 설정한 서비스 |
|
||||
| 8 | `JpaLiveEventReplay.sweepExpired()`를 부르던 코드를 지운다. replay를 켜면 `LiveEventReaper`가 `app.realtime.replay.reaper-interval`(기본 `PT10M`)마다 `retained_until`이 지난 행을 1000행씩 지운다. `app.realtime.replay.*`는 이제 `LiveEventReplaySettings`가 엄격하게 묶는다: 그 아래 모르는 키, 0 이하의 `retention`·`reaper-interval`은 기동을 실패시킨다 | 컴파일이 실패한다. 잘못 쓴 키는 기동이 실패한다. 전에는 지우는 곳이 없어 `live_event_log`가 끝없이 자랐다 | `app.realtime.replay.enabled=true`인 서비스 |
|
||||
| 9 | `IdempotencyReaper`를 직접 만들던 코드는 `new IdempotencyReaper(repository)`로 바꾼다(`Clock` 인자 삭제). 만료 비교는 DB 시계(`now()`)이고 1000행씩 지운다 | 컴파일이 실패한다 | reaper를 직접 조립한 서비스 |
|
||||
| 10 | `HibernateStatelessSessionRunner`의 트랜잭션은 이제 30초 상한이 있다(`DEFAULT_TIMEOUT`). 더 긴 일이면 셋째 인자로 상한을 넘기거나 일을 나눈다 | 30초를 넘는 문장이 취소되고 일 전체가 롤백된다. 전에는 상한이 없었다 | stateless runner를 쓰는 서비스 |
|
||||
| 11 | experimental `TenantPoolBudget.requireCapacity`·`hasCapacity`가 새 pool의 연결 수를 셋째 인자로 받는다. pool 수만 볼 때는 `requirePoolSlot(openPools)`를 쓴다. `TenantDataSourceRegistry`의 둘째 인자는 이제 열린 pool의 크기를 재는 함수가 아니라 tenant의 계획된 pool 크기(`ToIntFunction<TenantId>`)다. 예산은 pool을 열기 전에 그 크기로 본다 | 컴파일이 실패한다. 전에는 새 pool 크기를 모른 채 열어 연결 합계가 상한을 넘었다 | experimental tenant pool을 쓰는 서비스 |
|
||||
| 12 | fileserver cleanup 묶음의 `max-items`를 정할 때, 한 묶음이 claim lease(10분)의 첫 8분 안에 시작할 수 있는 만큼만 처리된다는 것을 안다. 나머지는 시도하지 않은 채 queue에 돌아가 다음 묶음이 다시 claim한다. 시도 횟수는 그대로다(바이트 예산이 바닥나 돌려준 항목도 이제 같다. 전에는 시도 한 번으로 세어 8번째에 보류됐다). 직접 구현한 `CleanupQueuePort`는 새 연산 `release`를 구현한다(`CleanupQueuePortContract` 새 사례) | 느린 저장소에서 큰 `max-items`는 그 사유로 도는 항목이 늘어난다. 전에는 lease가 끝난 뒤에도 남은 항목을 처리해 다른 worker와 겹쳤다 | fileserver capability를 켠 서비스 |
|
||||
|
||||
## 3. 바뀐 동작 (따라 할 일 없음)
|
||||
|
||||
- reaper 둘(outbox·멱등성)의 트랜잭션은 `SpringTransaction.DEFAULT_TIMEOUT_SECONDS`(30초)로 묶인다.
|
||||
- reaper 셋(outbox·멱등성·live event)의 트랜잭션은 `SpringTransaction.DEFAULT_TIMEOUT_SECONDS`(30초)로 묶인다.
|
||||
`app.persistence-jpa.transaction.*`를 바꿔도 reaper 시한은 그대로다. 한 트랜잭션이 1000행 한 묶음만 지운다.
|
||||
- 저장소가 행에서 만드는 `DurableOperationClaim`·`OperationLease`·`RecipientLease`·`OutboxLease`·`ClaimedOutboxEvent`·
|
||||
`ReconciliationJob`·`AdminOperationClaim`이 값을 거절할 때 `IllegalStateException`을 던진다
|
||||
`ReconciliationJob`·`AdminOperationClaim`과 stateless runner의 timeout이 값을 거절할 때 `IllegalStateException`을 던진다
|
||||
(전에는 `IllegalArgumentException`). 요청에서 오는 값이 아니라 서버 결함이다. 이 IAE를 잡던 코드는 ISE를 잡는다.
|
||||
- 알림 lease 회복은 버려질 때의 lease 그대로일 때만 delivery를 다시 넣는다. 다른 인스턴스가 이미 다시 넣었거나 holder가
|
||||
끝냈으면 아무것도 쓰지 않고 처리 수에 세지 않는다.
|
||||
@@ -36,3 +42,8 @@
|
||||
- 멱등성 reaper는 다른 트랜잭션이 쥔 기록을 건너뛰고, 바깥 `DELETE`에서도 만료를 다시 본다. 같은 키 요청이 만료된 기록을
|
||||
되살리는 동안 돈 reap이 그 claim을 지우지 않는다.
|
||||
- fileserver recovery 대기열 enqueue는 `INSERT … ON CONFLICT` 한 문장이다. 같은 파일을 동시에 넣어도 25P02로 실패하지 않는다.
|
||||
- 알림 scheduler는 poll마다 backlog 한 문장만 읽는다(전에는 넷). health snapshot의 projection 지연은 열린 provider event만
|
||||
읽는다.
|
||||
- 알림 backlog(`backlogDepth`, scheduler의 queue-depth gauge, health 임계값)는 claim이 지금 가져갈 delivery만 센다. claim과
|
||||
같은 SQL 조각이다. 전에는 창이 닫힌(`expires_at`이 지난) delivery도 셌고, lease가 지금 막 끝나는 행(`lease_until = now`)도
|
||||
셌다. 만료가 쌓인 플랫폼은 배포 직후 backlog 수치가 내려간다. 보낼 일이 줄어든 것이 아니라 보낼 수 없는 행을 빼고 센 것이다.
|
||||
|
||||
+3
-3
@@ -81,10 +81,10 @@ public final class NotificationSchedulerWorker implements AutoCloseable {
|
||||
// by the claim batch size — so a queue of ten and a queue of ten million published the same
|
||||
// number, and the one metric named "queue depth" was the one that could not show a queue
|
||||
// growing.
|
||||
// The depth alone: the full serving snapshot is several statements and is the health
|
||||
// endpoint's, not something to run on every poll.
|
||||
metrics.gauge(
|
||||
NotificationMetricName.QUEUE_DEPTH,
|
||||
Map.of(),
|
||||
(double) servingState.currentState().backlogDepth());
|
||||
NotificationMetricName.QUEUE_DEPTH, Map.of(), (double) servingState.backlogDepth());
|
||||
|
||||
for (RecipientLease lease : claimed) {
|
||||
globalConcurrency.acquireUninterruptibly();
|
||||
|
||||
@@ -389,6 +389,22 @@ PUBLISHED backlog 를 한 transaction 으로 지우면 그 사이 모든 row loc
|
||||
cadence 측정 필요). retention 한 값은 reaper-local 이라 `@Value` 로 받지만, canonical 6-property
|
||||
문서는 app-bootstrap `OutboxSettings` / `application.yml` 에 있다.
|
||||
|
||||
## liveevent — `JpaLiveEventReplay`와 `LiveEventReaper`
|
||||
|
||||
`app.realtime.replay.enabled=true`이고 vendor가 PostgreSQL이면 replay store와 reaper가 함께 조립된다. 설정은
|
||||
`LiveEventReplaySettings`가 `app.realtime.replay.*`를 엄격하게 묶는다. 모르는 키는 기동을 실패시킨다.
|
||||
|
||||
- `retention`: 한 event를 다시 받을 수 있는 시간. 기본값이 없다. 모듈 기본값이 `APP_REALTIME_REPLAY_RETENTION`(1시간)으로
|
||||
채운다. 0이나 음수는 기동 실패다.
|
||||
- `reaper-interval`: `LiveEventReaper`가 retention이 지난 행을 지우는 간격(기본 10분, 모듈 기본값이 같은 값을 적는다). 0이나
|
||||
음수는 기동 실패다. reaper 자체에는 일정이 없다. 조립 설정이 이 값으로 `reap()`을 등록한다(첫 실행도 한 간격 뒤).
|
||||
reaper를 직접 만드는 서비스는 일정도 직접 건다.
|
||||
|
||||
reaper는 `BatchedDeletes`로 1,000행씩, 배치마다 따로 commit하고, 배치가 덜 차면 멈춘다. 배치 문장은 `SKIP LOCKED`라 여러
|
||||
replica가 같이 돌아도 서로 기다리지 않는다. 트랜잭션 시한은 `SpringTransaction.DEFAULT_TIMEOUT_SECONDS`(30초)다. replay 읽기는
|
||||
`retained_until`로 지난 행을 이미 거르므로, reaper가 늦어도 지난 event를 내주지는 않는다. reaper가 없으면 테이블이 끝없이
|
||||
자란다(2026-11-13 전에는 지우는 곳이 없었다).
|
||||
|
||||
## operation — `JpaDurableOperationStore`
|
||||
|
||||
### claim과 fence (`V23`)
|
||||
|
||||
+38
-5
@@ -8,6 +8,8 @@ import dev.caskeleton.adapter.outbound.persistence.idempotency.jpa.IdempotencyRe
|
||||
import dev.caskeleton.adapter.outbound.persistence.idempotency.jpa.IdempotencyRecordJpaRepository;
|
||||
import dev.caskeleton.adapter.outbound.persistence.liveevent.JpaLiveEventReplay;
|
||||
import dev.caskeleton.adapter.outbound.persistence.liveevent.LiveEventJpaRepository;
|
||||
import dev.caskeleton.adapter.outbound.persistence.liveevent.LiveEventReaper;
|
||||
import dev.caskeleton.adapter.outbound.persistence.liveevent.LiveEventReplaySettings;
|
||||
import dev.caskeleton.adapter.outbound.persistence.lock.DistributedLockPersistenceConfig;
|
||||
import dev.caskeleton.adapter.outbound.persistence.operation.DurableOperationJpaRepository;
|
||||
import dev.caskeleton.adapter.outbound.persistence.operation.JpaDurableOperationStore;
|
||||
@@ -26,7 +28,11 @@ import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
|
||||
import org.springframework.context.annotation.Bean;
|
||||
import org.springframework.context.annotation.Configuration;
|
||||
import org.springframework.context.annotation.Import;
|
||||
import org.springframework.core.env.Environment;
|
||||
import org.springframework.jdbc.core.JdbcOperations;
|
||||
import org.springframework.scheduling.annotation.EnableScheduling;
|
||||
import org.springframework.scheduling.annotation.SchedulingConfigurer;
|
||||
import org.springframework.scheduling.config.FixedDelayTask;
|
||||
import org.springframework.transaction.PlatformTransactionManager;
|
||||
|
||||
/**
|
||||
@@ -120,11 +126,15 @@ class JpaAdapterBeansConfig {
|
||||
}
|
||||
|
||||
/**
|
||||
* Retained live-event history: on when replay is switched on, and only on PostgreSQL (the
|
||||
* position allocator is an {@code INSERT … ON CONFLICT … RETURNING}). The retention has no
|
||||
* default on purpose: a replay window nobody chose is one nobody sized.
|
||||
* Retained live-event history and the reaper that removes it past retention: on when replay is
|
||||
* switched on, and only on PostgreSQL (the position allocator is an {@code INSERT … ON CONFLICT …
|
||||
* RETURNING}). The settings are {@link LiveEventReplaySettings}, bound strictly: the retention is
|
||||
* one hour unless the deployment sets {@code APP_REALTIME_REPLAY_RETENTION} (the module
|
||||
* defaults), the reaper runs every {@code reaper-interval}. Scheduling is enabled here because
|
||||
* the reaper is part of the store, as for the idempotency reaper.
|
||||
*/
|
||||
@Configuration(proxyBeanMethods = false)
|
||||
@EnableScheduling
|
||||
@ConditionalOnProperty(prefix = "app.realtime.replay", name = "enabled", havingValue = "true")
|
||||
@ConditionalOnProperty(
|
||||
prefix = "app.persistence",
|
||||
@@ -133,13 +143,36 @@ class JpaAdapterBeansConfig {
|
||||
matchIfMissing = true)
|
||||
static class LiveEventReplay {
|
||||
|
||||
@Bean
|
||||
LiveEventReplaySettings liveEventReplaySettings(Environment environment) {
|
||||
return LiveEventReplaySettings.bind(environment);
|
||||
}
|
||||
|
||||
@Bean
|
||||
JpaLiveEventReplay jpaLiveEventReplay(
|
||||
LiveEventJpaRepository repository,
|
||||
JdbcOperations jdbc,
|
||||
Clock clock,
|
||||
@Value("${app.realtime.replay.retention}") Duration retention) {
|
||||
return new JpaLiveEventReplay(repository, jdbc, clock, retention);
|
||||
LiveEventReplaySettings settings) {
|
||||
return new JpaLiveEventReplay(repository, jdbc, clock, settings.retention());
|
||||
}
|
||||
|
||||
@Bean
|
||||
LiveEventReaper liveEventReaper(LiveEventJpaRepository repository, Clock clock) {
|
||||
return new LiveEventReaper(repository, clock);
|
||||
}
|
||||
|
||||
/**
|
||||
* Runs the reaper every bound {@code reaper-interval}, first one interval after startup. The
|
||||
* interval is handed over as the settings' value, so it has one default and a zero has already
|
||||
* failed startup in the settings rather than scheduling a loop.
|
||||
*/
|
||||
@Bean
|
||||
SchedulingConfigurer liveEventReaperSchedule(
|
||||
LiveEventReaper reaper, LiveEventReplaySettings settings) {
|
||||
Duration interval = settings.reaperInterval();
|
||||
return registrar ->
|
||||
registrar.addFixedDelayTask(new FixedDelayTask(reaper::reap, interval, interval));
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
+17
-6
@@ -23,26 +23,32 @@ import javax.sql.DataSource;
|
||||
public final class TenantDataSourceRegistry implements AutoCloseable {
|
||||
|
||||
private final Map<TenantId, DataSource> pools = new LinkedHashMap<>();
|
||||
private final Map<TenantId, Integer> poolSizes = new LinkedHashMap<>();
|
||||
private final Function<TenantId, DataSource> poolFactory;
|
||||
private final ToIntFunction<DataSource> poolSize;
|
||||
private final ToIntFunction<TenantId> plannedPoolSize;
|
||||
private final TenantPoolBudget budget;
|
||||
|
||||
/**
|
||||
* @param poolFactory opens a pool for a tenant from its secret-backed connection profile
|
||||
* @param poolSize reports a pool's maximum connection count, for the global budget
|
||||
* @param plannedPoolSize the maximum connection count a tenant's pool will have, read from its
|
||||
* connection profile before the pool is opened, for the global budget
|
||||
*/
|
||||
public TenantDataSourceRegistry(
|
||||
Function<TenantId, DataSource> poolFactory,
|
||||
ToIntFunction<DataSource> poolSize,
|
||||
ToIntFunction<TenantId> plannedPoolSize,
|
||||
TenantPoolBudget budget) {
|
||||
this.poolFactory = Objects.requireNonNull(poolFactory, "poolFactory");
|
||||
this.poolSize = Objects.requireNonNull(poolSize, "poolSize");
|
||||
this.plannedPoolSize = Objects.requireNonNull(plannedPoolSize, "plannedPoolSize");
|
||||
this.budget = Objects.requireNonNull(budget, "budget");
|
||||
}
|
||||
|
||||
/**
|
||||
* The tenant's pool, opening it if the budget allows.
|
||||
*
|
||||
* <p>The budget is checked against the pool's planned size before the pool is opened: a pool
|
||||
* opens its first connections as it is built, so one checked afterwards had already taken the
|
||||
* server past the ceiling, if only until it was closed again.
|
||||
*
|
||||
* @throws IllegalStateException when the global pool budget is exhausted
|
||||
*/
|
||||
public synchronized DataSource require(TenantId tenant) {
|
||||
@@ -51,9 +57,12 @@ public final class TenantDataSourceRegistry implements AutoCloseable {
|
||||
if (existing != null) {
|
||||
return existing;
|
||||
}
|
||||
budget.requireCapacity(pools.size(), allocatedConnections());
|
||||
budget.requirePoolSlot(pools.size());
|
||||
int planned = plannedPoolSize.applyAsInt(tenant);
|
||||
budget.requireCapacity(pools.size(), allocatedConnections(), planned);
|
||||
DataSource opened = poolFactory.apply(tenant);
|
||||
pools.put(tenant, opened);
|
||||
poolSizes.put(tenant, planned);
|
||||
return opened;
|
||||
}
|
||||
|
||||
@@ -72,6 +81,7 @@ public final class TenantDataSourceRegistry implements AutoCloseable {
|
||||
*/
|
||||
public synchronized void evict(TenantId tenant) {
|
||||
DataSource removed = pools.remove(tenant);
|
||||
poolSizes.remove(tenant);
|
||||
if (removed instanceof AutoCloseable closeable) {
|
||||
TenantDataSourceLifecycle.drainAndClose(closeable);
|
||||
}
|
||||
@@ -84,7 +94,7 @@ public final class TenantDataSourceRegistry implements AutoCloseable {
|
||||
|
||||
/** How many connections all open pools may hold between them. */
|
||||
public synchronized int allocatedConnections() {
|
||||
return pools.values().stream().mapToInt(poolSize).sum();
|
||||
return poolSizes.values().stream().mapToInt(Integer::intValue).sum();
|
||||
}
|
||||
|
||||
/** The tenants with an open pool. */
|
||||
@@ -99,5 +109,6 @@ public final class TenantDataSourceRegistry implements AutoCloseable {
|
||||
.map(AutoCloseable.class::cast)
|
||||
.forEach(TenantDataSourceLifecycle::drainAndClose);
|
||||
pools.clear();
|
||||
poolSizes.clear();
|
||||
}
|
||||
}
|
||||
|
||||
+32
-7
@@ -23,19 +23,44 @@ public record TenantPoolBudget(int maxOpenPools, int maxConnectionsAcrossPools)
|
||||
}
|
||||
|
||||
/**
|
||||
* Fails when another tenant pool would exceed either ceiling.
|
||||
* Fails when another tenant pool would exceed the pool count, before that pool is opened.
|
||||
*
|
||||
* @throws IllegalStateException when the budget is exhausted
|
||||
* @param openPools how many pools are open now
|
||||
* @throws IllegalStateException when no other pool fits
|
||||
*/
|
||||
public void requireCapacity(int openPools, int allocatedConnections) {
|
||||
if (openPools >= maxOpenPools || allocatedConnections >= maxConnectionsAcrossPools) {
|
||||
public void requirePoolSlot(int openPools) {
|
||||
if (openPools >= maxOpenPools) {
|
||||
throw new IllegalStateException("tenant pool budget exhausted");
|
||||
}
|
||||
}
|
||||
|
||||
/** Whether another tenant pool fits inside both ceilings. */
|
||||
public boolean hasCapacity(int openPools, int allocatedConnections) {
|
||||
return openPools < maxOpenPools && allocatedConnections < maxConnectionsAcrossPools;
|
||||
/**
|
||||
* Fails when a new tenant pool of {@code newPoolConnections} would exceed either ceiling.
|
||||
*
|
||||
* <p>The new pool's own size is part of the question. Checking only that the pools already open
|
||||
* stay under the ceiling let a ten-connection pool open with one connection to spare, and the
|
||||
* total then stood nine above the ceiling the budget exists to keep (round9 review 2026-11-12,
|
||||
* RR9-01-26).
|
||||
*
|
||||
* @param openPools how many pools are open now, the new one not included
|
||||
* @param allocatedConnections how many connections the open pools may hold between them
|
||||
* @param newPoolConnections how many the new pool may hold
|
||||
* @throws IllegalStateException when the budget cannot take the new pool
|
||||
*/
|
||||
public void requireCapacity(int openPools, int allocatedConnections, int newPoolConnections) {
|
||||
if (!hasCapacity(openPools, allocatedConnections, newPoolConnections)) {
|
||||
throw new IllegalStateException("tenant pool budget exhausted");
|
||||
}
|
||||
}
|
||||
|
||||
/** Whether a new tenant pool of {@code newPoolConnections} fits inside both ceilings. */
|
||||
public boolean hasCapacity(int openPools, int allocatedConnections, int newPoolConnections) {
|
||||
if (newPoolConnections < 1) {
|
||||
// The size is the server's own configuration, not the caller's input: a state error.
|
||||
throw new IllegalStateException("a tenant pool holds at least one connection");
|
||||
}
|
||||
return openPools < maxOpenPools
|
||||
&& (long) allocatedConnections + newPoolConnections <= maxConnectionsAcrossPools;
|
||||
}
|
||||
|
||||
/** The maximum number of tenants that can be open at once. */
|
||||
|
||||
+45
-2
@@ -1,5 +1,6 @@
|
||||
package dev.caskeleton.adapter.outbound.persistence.hibernate.stateless;
|
||||
|
||||
import java.time.Duration;
|
||||
import java.util.Objects;
|
||||
import java.util.Set;
|
||||
import java.util.function.Function;
|
||||
@@ -19,23 +20,63 @@ import org.hibernate.Transaction;
|
||||
*
|
||||
* <p>Only registered work names may run, and each carries a declared row cap. An admin bulk path
|
||||
* without a cap is how a maintenance job becomes an outage.
|
||||
*
|
||||
* <p>The transaction has the same time bound as the application's write transactions (30 seconds
|
||||
* unless the constructor is given another, {@code docs/work/reference/decisions.md} 2.5). The row
|
||||
* cap can only be checked once the work reports what it touched, so without a bound a runaway
|
||||
* statement held its locks for as long as it ran before the cap could roll it back. Hibernate hands
|
||||
* every statement of the work the time left, and the database cancels the one that runs past it
|
||||
* (round9 review 2026-11-12, RR9-01-25).
|
||||
*/
|
||||
public final class HibernateStatelessSessionRunner {
|
||||
|
||||
/**
|
||||
* The default write-transaction bound, which this runner's transactions share: {@code
|
||||
* SpringTransaction.DEFAULT_TIMEOUT_SECONDS}. Restated rather than read from it, because the
|
||||
* {@code hibernate} package does not depend on {@code transaction} (JpaModuleBoundaryTest);
|
||||
* {@code StatelessRunnerTimeoutTest} holds the two equal. A deployment that moves its write bound
|
||||
* passes the same value to the three-argument constructor.
|
||||
*/
|
||||
public static final Duration DEFAULT_TIMEOUT = Duration.ofSeconds(30);
|
||||
|
||||
private final SessionFactory sessionFactory;
|
||||
private final Set<StatelessWorkName> registeredWork;
|
||||
private final int timeoutSeconds;
|
||||
|
||||
/** A runner whose transactions are bounded by {@link #DEFAULT_TIMEOUT}. */
|
||||
public HibernateStatelessSessionRunner(
|
||||
SessionFactory sessionFactory, Set<StatelessWorkName> registeredWork) {
|
||||
this(sessionFactory, registeredWork, DEFAULT_TIMEOUT);
|
||||
}
|
||||
|
||||
/**
|
||||
* A runner whose transactions are bounded by {@code timeout}.
|
||||
*
|
||||
* @param timeout a whole, positive number of seconds; Hibernate's transaction timeout has no
|
||||
* finer unit, so a fraction would be dropped without a word
|
||||
*/
|
||||
public HibernateStatelessSessionRunner(
|
||||
SessionFactory sessionFactory, Set<StatelessWorkName> registeredWork, Duration timeout) {
|
||||
this.sessionFactory = Objects.requireNonNull(sessionFactory, "sessionFactory");
|
||||
this.registeredWork = Set.copyOf(Objects.requireNonNull(registeredWork, "registeredWork"));
|
||||
Objects.requireNonNull(timeout, "timeout");
|
||||
if (timeout.isNegative()
|
||||
|| timeout.isZero()
|
||||
|| timeout.toNanosPart() != 0
|
||||
|| timeout.toSeconds() > Integer.MAX_VALUE) {
|
||||
throw new IllegalStateException(
|
||||
"a stateless transaction timeout must be a positive whole number of seconds, was "
|
||||
+ timeout);
|
||||
}
|
||||
this.timeoutSeconds = Math.toIntExact(timeout.toSeconds());
|
||||
}
|
||||
|
||||
/**
|
||||
* Runs {@code work} in its own stateless session and transaction.
|
||||
*
|
||||
* <p>The work reports how many rows it touched, and exceeding {@code maxRows} rolls the
|
||||
* transaction back.
|
||||
* transaction back. A statement still running when the transaction's time is up is cancelled, and
|
||||
* the failure rolls the transaction back.
|
||||
*
|
||||
* @param maxRows the row cap for this unit of work
|
||||
* @throws StatelessRowCapExceededException when the work touched more, after rolling back
|
||||
@@ -54,7 +95,9 @@ public final class HibernateStatelessSessionRunner {
|
||||
}
|
||||
|
||||
try (StatelessSession session = sessionFactory.openStatelessSession()) {
|
||||
Transaction transaction = session.beginTransaction();
|
||||
Transaction transaction = session.getTransaction();
|
||||
transaction.setTimeout(timeoutSeconds);
|
||||
transaction.begin();
|
||||
try {
|
||||
StatelessWorkResult<T> result = work.apply(session);
|
||||
if (result.affectedRows() > maxRows) {
|
||||
|
||||
-10
@@ -134,16 +134,6 @@ public final class JpaLiveEventReplay implements LiveEventReplayPort {
|
||||
.toList();
|
||||
}
|
||||
|
||||
/**
|
||||
* Removes events past their retention.
|
||||
*
|
||||
* @return how many went
|
||||
*/
|
||||
public int sweepExpired() {
|
||||
requireCallerTransaction("sweepExpired");
|
||||
return repository.sweepExpired(clock.instant());
|
||||
}
|
||||
|
||||
private static void requireCallerTransaction(String operation) {
|
||||
CallerTransaction.require("live-event " + operation);
|
||||
}
|
||||
|
||||
+20
-3
@@ -1,6 +1,7 @@
|
||||
package dev.caskeleton.adapter.outbound.persistence.liveevent;
|
||||
|
||||
import dev.caskeleton.adapter.outbound.persistence.liveevent.entity.LiveEventEntity;
|
||||
import dev.caskeleton.adapter.outbound.persistence.transaction.SpringTransaction;
|
||||
import java.time.Instant;
|
||||
import java.util.List;
|
||||
import org.springframework.data.domain.Limit;
|
||||
@@ -8,6 +9,7 @@ import org.springframework.data.jpa.repository.JpaRepository;
|
||||
import org.springframework.data.jpa.repository.Modifying;
|
||||
import org.springframework.data.jpa.repository.Query;
|
||||
import org.springframework.data.repository.query.Param;
|
||||
import org.springframework.transaction.annotation.Transactional;
|
||||
|
||||
/**
|
||||
* Spring Data access for {@code live_event_log}.
|
||||
@@ -66,12 +68,27 @@ public interface LiveEventJpaRepository
|
||||
Long latestRetained(@Param("streamId") String streamId, @Param("now") Instant now);
|
||||
|
||||
/**
|
||||
* Removes expired rows.
|
||||
* Deletes at most {@code limit} rows retained no later than {@code now}, in a transaction of its
|
||||
* own unless the caller already has one; returns the count.
|
||||
*
|
||||
* <p>Bounded, and skipping rows another replica's reaper is deleting, for the reason the outbox
|
||||
* purge is bounded: a backlog of expired history is one long transaction holding every row lock
|
||||
* otherwise. {@code retained_until} is written from the application clock, so it is compared with
|
||||
* the application clock.
|
||||
*
|
||||
* @param now rows retained no later than this are removed
|
||||
* @param limit the most rows one statement removes
|
||||
* @return how many went
|
||||
*/
|
||||
// Bounded by the default transaction bound: a statement stuck behind a lock gives the connection
|
||||
// back instead of holding it until the pool is empty.
|
||||
@Transactional(timeout = SpringTransaction.DEFAULT_TIMEOUT_SECONDS)
|
||||
@Modifying
|
||||
@Query("delete from LiveEventEntity e where e.retainedUntil <= :now")
|
||||
int sweepExpired(@Param("now") Instant now);
|
||||
@Query(
|
||||
value =
|
||||
"DELETE FROM live_event_log WHERE (stream_id, position) IN ("
|
||||
+ "SELECT stream_id, position FROM live_event_log WHERE retained_until <= :now"
|
||||
+ " FOR UPDATE SKIP LOCKED LIMIT :limit)",
|
||||
nativeQuery = true)
|
||||
int deleteExpiredBatch(@Param("now") Instant now, @Param("limit") int limit);
|
||||
}
|
||||
|
||||
+53
@@ -0,0 +1,53 @@
|
||||
package dev.caskeleton.adapter.outbound.persistence.liveevent;
|
||||
|
||||
import dev.caskeleton.adapter.outbound.persistence.transaction.BatchedDeletes;
|
||||
import java.time.Clock;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
|
||||
/**
|
||||
* Periodically deletes {@code live_event_log} rows past their retention.
|
||||
*
|
||||
* <p>A replay read already filters on {@code retained_until}, so an expired row is never served;
|
||||
* without this nothing ever removed one, and the table grew by every event ever published for as
|
||||
* long as replay was on (round9 review 2026-11-12, RR9-01-03). It is assembled with the replay
|
||||
* store, as the idempotency and outbox reapers are with theirs, rather than left to each service to
|
||||
* schedule: a duty that every deployment must remember is one some deployment forgets.
|
||||
*
|
||||
* <p>Deletes in batches of {@link #BATCH_SIZE}, each committed on its own, until a batch comes back
|
||||
* short. Every replica runs it; the batch skips rows another replica's batch holds.
|
||||
*
|
||||
* <p>It carries no schedule of its own. The replay configuration registers {@link #reap()} at the
|
||||
* bound {@link LiveEventReplaySettings#reaperInterval()}, so the interval reaches the schedule as a
|
||||
* value rather than through a bean name an expression looks up. A deployment that builds a reaper
|
||||
* itself schedules it too.
|
||||
*/
|
||||
public class LiveEventReaper {
|
||||
|
||||
/** Rows per delete statement and transaction. */
|
||||
static final int BATCH_SIZE = BatchedDeletes.BATCH_SIZE;
|
||||
|
||||
private static final Logger log = LoggerFactory.getLogger(LiveEventReaper.class);
|
||||
|
||||
private final LiveEventJpaRepository repository;
|
||||
private final Clock clock;
|
||||
|
||||
public LiveEventReaper(LiveEventJpaRepository repository, Clock clock) {
|
||||
this.repository = repository;
|
||||
this.clock = clock;
|
||||
}
|
||||
|
||||
/**
|
||||
* Deletes every row retained no later than now; returns the count.
|
||||
*
|
||||
* <p>Not transactional itself: each batch is its own transaction.
|
||||
*/
|
||||
public int reap() {
|
||||
int purged =
|
||||
BatchedDeletes.untilShort(limit -> repository.deleteExpiredBatch(clock.instant(), limit));
|
||||
if (purged > 0) {
|
||||
log.debug("live event reaper purged {} expired row(s)", purged);
|
||||
}
|
||||
return purged;
|
||||
}
|
||||
}
|
||||
+55
@@ -0,0 +1,55 @@
|
||||
package dev.caskeleton.adapter.outbound.persistence.liveevent;
|
||||
|
||||
import dev.caskeleton.adapter.outbound.support.StrictSettingsBinder;
|
||||
import java.time.Duration;
|
||||
import org.springframework.core.env.Environment;
|
||||
|
||||
/**
|
||||
* The replay store's settings, bound strictly from {@code app.realtime.replay.*}: a misspelt key
|
||||
* fails startup rather than leaving a default in force.
|
||||
*
|
||||
* <p>The reaper's interval was read only by the {@code @Scheduled} placeholder, with its default
|
||||
* written twice and nothing to refuse a zero (round9 review 2026-11-12). It is bound here with the
|
||||
* rest of the store's settings and read from this record by the reaper's schedule.
|
||||
*
|
||||
* @param enabled whether the replay store is assembled; the configuration's condition reads it
|
||||
* @param retention how long an event stays replayable. No default: how long a reconnecting client
|
||||
* may resume is the deployment's decision, made against what it is willing to store
|
||||
* @param reaperInterval how often {@link LiveEventReaper} deletes rows past retention; 10 minutes
|
||||
* unless set, which the module defaults mirror
|
||||
*/
|
||||
public record LiveEventReplaySettings(
|
||||
boolean enabled, Duration retention, Duration reaperInterval) {
|
||||
|
||||
static final String PREFIX = "app.realtime.replay";
|
||||
|
||||
private static final Duration DEFAULT_REAPER_INTERVAL = Duration.ofMinutes(10);
|
||||
|
||||
public LiveEventReplaySettings {
|
||||
if (retention == null) {
|
||||
throw new IllegalArgumentException(
|
||||
"app.realtime.replay.retention must be set when replay is enabled: how long a client may"
|
||||
+ " resume is the deployment's decision");
|
||||
}
|
||||
if (retention.isNegative() || retention.isZero()) {
|
||||
throw new IllegalArgumentException(
|
||||
"app.realtime.replay.retention must be positive, was " + retention);
|
||||
}
|
||||
if (reaperInterval == null) {
|
||||
reaperInterval = DEFAULT_REAPER_INTERVAL;
|
||||
} else if (reaperInterval.isNegative() || reaperInterval.isZero()) {
|
||||
throw new IllegalArgumentException(
|
||||
"app.realtime.replay.reaper-interval must be positive, was " + reaperInterval);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Binds {@link #PREFIX} strictly.
|
||||
*
|
||||
* @param environment the environment to bind from
|
||||
* @return the bound settings
|
||||
*/
|
||||
public static LiveEventReplaySettings bind(Environment environment) {
|
||||
return StrictSettingsBinder.bind(environment, PREFIX, LiveEventReplaySettings.class);
|
||||
}
|
||||
}
|
||||
+9
-22
@@ -5,41 +5,28 @@ import java.time.Duration;
|
||||
import org.springframework.core.env.Environment;
|
||||
|
||||
/**
|
||||
* Distributed-lock tuning knobs bound from {@code app.lock.*} (yaml-only defaults, not {@code
|
||||
* APP_*} env keys). {@code waitTime} = max wait before timeout (default 3s); {@code leaseTtl} = max
|
||||
* lock hold before auto-expiry (default 30s, must be ≥ {@code waitTime}). See README "lock".
|
||||
* Distributed-lock tuning bound from {@code app.lock.*} (yaml-only defaults, not {@code APP_*} env
|
||||
* keys). {@code leaseTtl} = the longest a lock may be held before the provider expires it (default
|
||||
* 30s); a caller's {@code tryAcquire} may ask for that long or less. See README "lock".
|
||||
*
|
||||
* <p>There is no wait setting. How long to wait is every caller's own argument to {@code
|
||||
* tryAcquire}; an {@code app.lock.wait-time} used to be bound here, checked against the lease, and
|
||||
* read by nothing (round9 review 2026-11-12, RR9-01-07). It now fails startup as an unknown key, so
|
||||
* a deployment that set it learns that it never did anything.
|
||||
*/
|
||||
public record LockSettings(Duration waitTime, Duration leaseTtl) {
|
||||
public record LockSettings(Duration leaseTtl) {
|
||||
|
||||
static final String PREFIX = "app.lock";
|
||||
|
||||
private static final Duration DEFAULT_WAIT_TIME = Duration.ofSeconds(3);
|
||||
private static final Duration DEFAULT_LEASE_TTL = Duration.ofSeconds(30);
|
||||
|
||||
public LockSettings {
|
||||
if (waitTime == null) {
|
||||
waitTime = DEFAULT_WAIT_TIME;
|
||||
} else if (waitTime.isNegative() || waitTime.isZero()) {
|
||||
throw new IllegalArgumentException(
|
||||
"app.lock.wait-time must be positive (> 0), was " + waitTime);
|
||||
}
|
||||
if (leaseTtl == null) {
|
||||
leaseTtl = DEFAULT_LEASE_TTL;
|
||||
} else if (leaseTtl.isNegative() || leaseTtl.isZero()) {
|
||||
throw new IllegalArgumentException(
|
||||
"app.lock.lease-ttl must be positive (> 0), was " + leaseTtl);
|
||||
}
|
||||
// Cross-field invariant: a lease shorter than the max wait is nonsensical.
|
||||
// If the TTL fires before the waitTime elapses, a second holder could acquire
|
||||
// the lock before the first holder's protected work is complete.
|
||||
if (leaseTtl.compareTo(waitTime) < 0) {
|
||||
throw new IllegalArgumentException(
|
||||
"app.lock.lease-ttl ("
|
||||
+ leaseTtl
|
||||
+ ") must be >= app.lock.wait-time ("
|
||||
+ waitTime
|
||||
+ ") — a lease shorter than the max wait is nonsensical");
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
|
||||
+7
-3
@@ -199,10 +199,10 @@ app:
|
||||
# true | false (whether to run Flyway migrations automatically at startup)
|
||||
migration-on-startup: ${APP_MIGRATION_ON_STARTUP:true}
|
||||
# feature-distributed-lock-contract D5 — distributed lock acquisition contract (try-lock +
|
||||
# finite wait + lease TTL). Bound to LockSettings. Plain values, not APP_* env keys; the code
|
||||
# defaults in LockSettings mirror these.
|
||||
# finite wait + lease TTL). Bound to LockSettings. A plain value, not an APP_* env key; the code
|
||||
# default in LockSettings mirrors it. The wait is each caller's tryAcquire argument, so there is
|
||||
# no wait setting (app.lock.wait-time was read by nothing and is now an unknown key).
|
||||
lock:
|
||||
wait-time: 3s
|
||||
lease-ttl: 30s
|
||||
realtime:
|
||||
replay:
|
||||
@@ -210,6 +210,10 @@ app:
|
||||
# events into the store and exposes replay/resume semantics.
|
||||
enabled: ${APP_REALTIME_REPLAY_ENABLED:false}
|
||||
retention: ${APP_REALTIME_REPLAY_RETENTION:1h}
|
||||
# ISO-8601 duration — how often LiveEventReaper deletes rows past retention, 1 000 a batch.
|
||||
# A plain value like the outbox reaper's interval. Bound strictly by LiveEventReplaySettings,
|
||||
# whose default mirrors it; zero or negative fails startup.
|
||||
reaper-interval: PT10M
|
||||
outbox:
|
||||
# feature-domain-event-outbox-contract I11 — the tuning values are literal defaults;
|
||||
# NO env placeholders (spec: 신규 env key 없음). Declared, validated and strictly bound by
|
||||
|
||||
+41
@@ -1,5 +1,7 @@
|
||||
package dev.caskeleton.adapter.outbound.persistence.liveevent;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
|
||||
import dev.caskeleton.adapter.outbound.persistence.liveevent.entity.LiveEventEntity;
|
||||
import dev.caskeleton.adapter.outbound.persistence.platform.JpaPlatformContractSupport;
|
||||
import dev.caskeleton.adapter.outbound.persistence.transaction.SpringTransaction;
|
||||
@@ -13,12 +15,17 @@ import java.nio.charset.StandardCharsets;
|
||||
import java.sql.Connection;
|
||||
import java.sql.SQLException;
|
||||
import java.sql.Statement;
|
||||
import java.time.Clock;
|
||||
import java.time.Duration;
|
||||
import java.time.Instant;
|
||||
import java.time.ZoneOffset;
|
||||
import java.util.Map;
|
||||
import java.util.UUID;
|
||||
import javax.sql.DataSource;
|
||||
import org.junit.jupiter.api.AfterAll;
|
||||
import org.junit.jupiter.api.BeforeAll;
|
||||
import org.junit.jupiter.api.Tag;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.springframework.context.annotation.AnnotationConfigApplicationContext;
|
||||
import org.springframework.context.annotation.Bean;
|
||||
import org.springframework.context.annotation.Configuration;
|
||||
@@ -98,6 +105,40 @@ class JpaLiveEventReplayContractTest extends LiveEventReplayContract {
|
||||
return RETENTION;
|
||||
}
|
||||
|
||||
@Test
|
||||
void theReaperRemovesRowsPastTheirRetentionAndKeepsTheRest() {
|
||||
// given
|
||||
String stream = "reaper-" + UUID.randomUUID();
|
||||
Instant start = Instant.parse("2026-11-03T00:00:00Z");
|
||||
appendAt(stream, "expired", start);
|
||||
appendAt(stream, "retained", start.plus(RETENTION).minusSeconds(60));
|
||||
LiveEventReaper reaper =
|
||||
new LiveEventReaper(
|
||||
context.getBean(LiveEventJpaRepository.class),
|
||||
Clock.fixed(start.plus(RETENTION).plusSeconds(1), ZoneOffset.UTC));
|
||||
|
||||
// when
|
||||
var unused = reaper.reap();
|
||||
|
||||
// then
|
||||
assertThat(
|
||||
new JdbcTemplate(support.dataSource())
|
||||
.queryForList(
|
||||
"SELECT payload FROM live_event_log WHERE stream_id = ?", String.class, stream))
|
||||
.containsExactly("retained");
|
||||
}
|
||||
|
||||
private void appendAt(String streamId, String payload, Instant at) {
|
||||
JpaLiveEventReplay writer =
|
||||
new JpaLiveEventReplay(
|
||||
context.getBean(LiveEventJpaRepository.class),
|
||||
new JdbcTemplate(support.dataSource()),
|
||||
Clock.fixed(at, ZoneOffset.UTC),
|
||||
RETENTION);
|
||||
var unused =
|
||||
context.getBean(TransactionPort.class).inWrite(() -> writer.append(streamId, payload));
|
||||
}
|
||||
|
||||
private static void execute(String sql) throws SQLException {
|
||||
try (Connection connection = support.connection();
|
||||
Statement statement = connection.createStatement()) {
|
||||
|
||||
+2
-1
@@ -49,7 +49,8 @@ final class NotificationFixtures {
|
||||
migration("jpa/notification-platform", "V10__variables_payload_envelope_guard.sql"),
|
||||
migration("jpa/notification-platform", "V11__notification_reconciliation_claim_lease.sql"),
|
||||
migration("jpa/notification-platform", "V12__notification_suppression_target_unique.sql"),
|
||||
migration("jpa/notification-platform", "V13__notification_request_acceptance.sql"));
|
||||
migration("jpa/notification-platform", "V13__notification_request_acceptance.sql"),
|
||||
migration("jpa/notification-platform", "V14__notification_provider_event_open_index.sql"));
|
||||
}
|
||||
|
||||
/**
|
||||
|
||||
+167
-10
@@ -8,6 +8,7 @@ import dev.caskeleton.application.notification.platform.observation.Notification
|
||||
import java.io.IOException;
|
||||
import java.sql.Connection;
|
||||
import java.sql.PreparedStatement;
|
||||
import java.sql.ResultSet;
|
||||
import java.sql.SQLException;
|
||||
import java.sql.Statement;
|
||||
import java.sql.Timestamp;
|
||||
@@ -15,6 +16,9 @@ import java.time.Clock;
|
||||
import java.time.Duration;
|
||||
import java.time.Instant;
|
||||
import java.time.ZoneOffset;
|
||||
import java.util.ArrayList;
|
||||
import java.util.Arrays;
|
||||
import java.util.List;
|
||||
import java.util.UUID;
|
||||
import javax.sql.DataSource;
|
||||
import org.junit.jupiter.api.AfterAll;
|
||||
@@ -23,7 +27,12 @@ import org.junit.jupiter.api.BeforeEach;
|
||||
import org.junit.jupiter.api.DisplayName;
|
||||
import org.junit.jupiter.api.Tag;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.junit.jupiter.params.ParameterizedTest;
|
||||
import org.junit.jupiter.params.provider.CsvSource;
|
||||
import org.springframework.jdbc.core.JdbcTemplate;
|
||||
import org.springframework.jdbc.core.PreparedStatementCreator;
|
||||
import org.springframework.jdbc.core.RowMapper;
|
||||
import org.springframework.jdbc.core.SqlProvider;
|
||||
import org.springframework.jdbc.datasource.SingleConnectionDataSource;
|
||||
|
||||
/**
|
||||
@@ -100,6 +109,39 @@ class ServingStateContractTest {
|
||||
assertThat(measure().backlogDepth()).isZero();
|
||||
}
|
||||
|
||||
@Test
|
||||
void aDeliveryWhoseWindowClosedIsNotBacklog() throws SQLException {
|
||||
// given
|
||||
// Due, but past its expiry: the claim refuses it and the expiry sweep retires it unsent.
|
||||
insertDelivery(
|
||||
"READY_TO_DISPATCH", NOW.minus(Duration.ofMinutes(5)), null, null, NOW.minusSeconds(1));
|
||||
|
||||
// when
|
||||
long depth = measure().backlogDepth();
|
||||
|
||||
// then
|
||||
assertThat(depth).isZero();
|
||||
}
|
||||
|
||||
@ParameterizedTest(name = "[{index}] lease_until = NOW에서 {0}초 → backlog {1}")
|
||||
@CsvSource({"-1, 1", "0, 0"})
|
||||
@DisplayName("lease는 NOW 전에 끝났을 때만 끝난 것이다: claim과 같은 lease_until < now")
|
||||
void aLeaseHasRunOutOnlyOnceItEndedBeforeNow(long offsetSeconds, long expectedDepth)
|
||||
throws SQLException {
|
||||
// given
|
||||
insertDelivery(
|
||||
"READY_TO_DISPATCH",
|
||||
NOW.minus(Duration.ofMinutes(1)),
|
||||
"worker-1",
|
||||
NOW.plusSeconds(offsetSeconds));
|
||||
|
||||
// when
|
||||
long depth = measure().backlogDepth();
|
||||
|
||||
// then
|
||||
assertThat(depth).isEqualTo(expectedDepth);
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("a delivery held under a live lease is not backlog")
|
||||
void aDeliveryUnderALiveLeaseIsNotBacklog() throws SQLException {
|
||||
@@ -113,22 +155,24 @@ class ServingStateContractTest {
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("a lease whose holder never came back is counted as stuck")
|
||||
void anExpiredLeaseIsStuck() throws SQLException {
|
||||
// given
|
||||
insertDelivery(
|
||||
"DISPATCHING",
|
||||
NOW.minus(Duration.ofMinutes(20)),
|
||||
"worker-1",
|
||||
NOW.minus(Duration.ofMinutes(10)));
|
||||
|
||||
// when
|
||||
NotificationServingState state = measure();
|
||||
|
||||
assertThat(state.stuckLeases())
|
||||
.as("a worker died mid-dispatch, so a provider call may have happened with no record")
|
||||
.isEqualTo(1);
|
||||
assertThat(state.backlogDepth())
|
||||
.as("an expired lease is claimable again, but this row is DISPATCHING, not due")
|
||||
.isZero();
|
||||
// then
|
||||
assertThat(state)
|
||||
.as(
|
||||
"a worker died mid-dispatch, so a provider call may have happened with no record; a"
|
||||
+ " dispatching row is never claimed, so it is not backlog")
|
||||
.extracting(NotificationServingState::stuckLeases, NotificationServingState::backlogDepth)
|
||||
.containsExactly(1L, 0L);
|
||||
}
|
||||
|
||||
@Test
|
||||
@@ -162,6 +206,108 @@ class ServingStateContractTest {
|
||||
assertThat(state.oldestReconciliationAge()).isEqualTo(Duration.ofHours(2));
|
||||
}
|
||||
|
||||
@Test
|
||||
void theSchedulersBacklogReadIsOneStatement() throws SQLException {
|
||||
// given
|
||||
insertDelivery("READY_TO_DISPATCH", NOW.minus(Duration.ofMinutes(30)), null, null);
|
||||
|
||||
// when
|
||||
List<Object> outcome;
|
||||
try (Connection connection = support.connection()) {
|
||||
RecordingJdbc jdbc = new RecordingJdbc(new SingleConnectionDataSource(connection, true));
|
||||
long depth = new JdbcNotificationServingState(jdbc, CLOCK).backlogDepth();
|
||||
outcome = List.of(depth, jdbc.statements.size());
|
||||
}
|
||||
|
||||
// then
|
||||
assertThat(outcome)
|
||||
.as("the scheduler asks this on every poll; the full snapshot is four statements")
|
||||
.containsExactly(1L, 1);
|
||||
}
|
||||
|
||||
@Test
|
||||
void theProjectionLagReadsOpenEventsThroughTheirIndex() throws SQLException {
|
||||
// given
|
||||
insertProviderEvent("PENDING", NOW.minus(Duration.ofMinutes(45)), false);
|
||||
|
||||
// when
|
||||
String plan;
|
||||
try (Connection connection = support.connection()) {
|
||||
RecordingJdbc jdbc = new RecordingJdbc(new SingleConnectionDataSource(connection, true));
|
||||
var unused = new JdbcNotificationServingState(jdbc, CLOCK).currentState();
|
||||
RecordedStatement lag =
|
||||
jdbc.statements.stream()
|
||||
.filter(statement -> statement.sql().contains("notification_provider_event"))
|
||||
.findFirst()
|
||||
.orElseThrow();
|
||||
plan = explainWithoutSequentialScans(connection, lag);
|
||||
}
|
||||
|
||||
// then
|
||||
assertThat(plan)
|
||||
.as("the event table is never purged; only the index over the open events bounds this read")
|
||||
.contains("ix_notification_provider_event_open");
|
||||
}
|
||||
|
||||
/** The plan PostgreSQL picks for {@code statement} when it may not scan a table sequentially. */
|
||||
private static String explainWithoutSequentialScans(
|
||||
Connection connection, RecordedStatement statement) throws SQLException {
|
||||
boolean autoCommit = connection.getAutoCommit();
|
||||
connection.setAutoCommit(false);
|
||||
try {
|
||||
try (Statement off = connection.createStatement()) {
|
||||
off.execute("SET LOCAL enable_seqscan = off");
|
||||
}
|
||||
StringBuilder plan = new StringBuilder();
|
||||
try (PreparedStatement explain = connection.prepareStatement("EXPLAIN " + statement.sql())) {
|
||||
for (int index = 0; index < statement.args().size(); index++) {
|
||||
explain.setObject(index + 1, statement.args().get(index));
|
||||
}
|
||||
try (ResultSet rows = explain.executeQuery()) {
|
||||
while (rows.next()) {
|
||||
plan.append(rows.getString(1)).append('\n');
|
||||
}
|
||||
}
|
||||
}
|
||||
return plan.toString();
|
||||
} finally {
|
||||
connection.rollback();
|
||||
connection.setAutoCommit(autoCommit);
|
||||
}
|
||||
}
|
||||
|
||||
/** A statement the serving state ran, as it ran it. */
|
||||
private record RecordedStatement(String sql, List<Object> args) {}
|
||||
|
||||
/** Plain JDBC that keeps the statements it was given, so a case reads the shipped SQL. */
|
||||
private static final class RecordingJdbc extends JdbcTemplate {
|
||||
|
||||
private final List<RecordedStatement> statements = new ArrayList<>();
|
||||
|
||||
private RecordingJdbc(DataSource dataSource) {
|
||||
super(dataSource);
|
||||
}
|
||||
|
||||
@Override
|
||||
public <T> T queryForObject(String sql, RowMapper<T> rowMapper, Object... args) {
|
||||
statements.add(new RecordedStatement(sql, Arrays.asList(args)));
|
||||
return super.queryForObject(sql, rowMapper, args);
|
||||
}
|
||||
|
||||
@Override
|
||||
public <T> T queryForObject(String sql, Class<T> requiredType, Object... args) {
|
||||
statements.add(new RecordedStatement(sql, Arrays.asList(args)));
|
||||
return super.queryForObject(sql, requiredType, args);
|
||||
}
|
||||
|
||||
/** The named-parameter backlog read arrives here; its parameters stay inside the creator. */
|
||||
@Override
|
||||
public <T> List<T> query(PreparedStatementCreator creator, RowMapper<T> rowMapper) {
|
||||
statements.add(new RecordedStatement(((SqlProvider) creator).getSql(), List.of()));
|
||||
return super.query(creator, rowMapper);
|
||||
}
|
||||
}
|
||||
|
||||
private NotificationServingState measure() throws SQLException {
|
||||
// try-with-resources: the pool holds four connections, and a measurement that keeps one is a
|
||||
// measurement that starves the next test.
|
||||
@@ -174,6 +320,16 @@ class ServingStateContractTest {
|
||||
private void insertDelivery(
|
||||
String state, Instant nextDispatchAt, String leaseOwner, Instant leaseUntil)
|
||||
throws SQLException {
|
||||
insertDelivery(state, nextDispatchAt, leaseOwner, leaseUntil, null);
|
||||
}
|
||||
|
||||
private void insertDelivery(
|
||||
String state,
|
||||
Instant nextDispatchAt,
|
||||
String leaseOwner,
|
||||
Instant leaseUntil,
|
||||
Instant expiresAt)
|
||||
throws SQLException {
|
||||
UUID requestId = UUID.randomUUID();
|
||||
try (Connection connection = support.connection()) {
|
||||
try (PreparedStatement statement =
|
||||
@@ -197,17 +353,18 @@ class ServingStateContractTest {
|
||||
+ " id, notification_id, tenant_id, recipient_ref, routing_plan_json,"
|
||||
+ " route_cursor, delivery_state, submission_outcome, delivery_outcome,"
|
||||
+ " evidence_level, attempt_count, next_dispatch_at, lease_owner, lease_until,"
|
||||
+ " created_at, updated_at)"
|
||||
+ " expires_at, created_at, updated_at)"
|
||||
+ " VALUES (?, ?, 'tenant-a', 'user-1', '[]'::jsonb, 0, ?, 'NOT_SUBMITTED',"
|
||||
+ " 'UNKNOWN', 'PLATFORM_QUEUED', 0, ?, ?, ?, ?, ?)")) {
|
||||
+ " 'UNKNOWN', 'PLATFORM_QUEUED', 0, ?, ?, ?, ?, ?, ?)")) {
|
||||
statement.setObject(1, UUID.randomUUID());
|
||||
statement.setObject(2, requestId);
|
||||
statement.setString(3, state);
|
||||
statement.setTimestamp(4, Timestamp.from(nextDispatchAt));
|
||||
statement.setString(5, leaseOwner);
|
||||
statement.setTimestamp(6, leaseUntil == null ? null : Timestamp.from(leaseUntil));
|
||||
statement.setTimestamp(7, Timestamp.from(NOW));
|
||||
statement.setTimestamp(7, expiresAt == null ? null : Timestamp.from(expiresAt));
|
||||
statement.setTimestamp(8, Timestamp.from(NOW));
|
||||
statement.setTimestamp(9, Timestamp.from(NOW));
|
||||
statement.executeUpdate();
|
||||
}
|
||||
}
|
||||
|
||||
+51
@@ -2,12 +2,16 @@ package dev.caskeleton.adapter.outbound.persistence.platform;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
import static org.assertj.core.api.Assertions.assertThatThrownBy;
|
||||
import static org.assertj.core.api.Assertions.catchThrowable;
|
||||
|
||||
import dev.caskeleton.adapter.outbound.persistence.hibernate.stateless.HibernateStatelessSessionRunner;
|
||||
import dev.caskeleton.adapter.outbound.persistence.hibernate.stateless.StatelessRowCapExceededException;
|
||||
import dev.caskeleton.adapter.outbound.persistence.hibernate.stateless.StatelessWorkName;
|
||||
import dev.caskeleton.adapter.outbound.persistence.hibernate.stateless.StatelessWorkResult;
|
||||
import dev.caskeleton.adapter.outbound.persistence.testkit.id.SequenceEntity;
|
||||
import java.time.Duration;
|
||||
import java.util.List;
|
||||
import java.util.Locale;
|
||||
import java.util.Set;
|
||||
import org.junit.jupiter.api.AfterAll;
|
||||
import org.junit.jupiter.api.BeforeAll;
|
||||
@@ -104,6 +108,53 @@ class HibernateStatelessSessionRunnerIntegrationTest {
|
||||
.hasMessageContaining("positive row cap");
|
||||
}
|
||||
|
||||
@Test
|
||||
void workStillRunningWhenTheTransactionTimeIsUpIsCancelledAndRolledBack() {
|
||||
// given
|
||||
var runner =
|
||||
new HibernateStatelessSessionRunner(
|
||||
jpa.sessionFactory(), Set.of(WORK), Duration.ofSeconds(1));
|
||||
long before = countRows();
|
||||
|
||||
// when
|
||||
Throwable failure =
|
||||
catchThrowable(
|
||||
() ->
|
||||
runner.execute(
|
||||
WORK,
|
||||
10,
|
||||
session -> {
|
||||
session.insert(new SequenceEntity("timed-out"));
|
||||
var unused =
|
||||
session
|
||||
.createNativeQuery("SELECT 1 FROM pg_sleep(5)", Integer.class)
|
||||
.getSingleResult();
|
||||
return StatelessWorkResult.of(1, 1);
|
||||
}));
|
||||
|
||||
// then
|
||||
assertThat(List.of(failure != null && causedByTimeout(failure), countRows()))
|
||||
.as(
|
||||
"the transaction's bound cancels the statement; nothing the work wrote is kept: %s",
|
||||
failure)
|
||||
.containsExactly(true, before);
|
||||
}
|
||||
|
||||
/** Whether the failure is the database or Hibernate stopping work that ran out of time. */
|
||||
private static boolean causedByTimeout(Throwable failure) {
|
||||
for (Throwable current = failure; current != null; current = current.getCause()) {
|
||||
String message = String.valueOf(current.getMessage()).toLowerCase(Locale.ROOT);
|
||||
if (current instanceof org.hibernate.QueryTimeoutException
|
||||
|| current instanceof jakarta.persistence.QueryTimeoutException
|
||||
|| message.contains("canceling statement due to statement timeout")
|
||||
|| message.contains("canceling statement due to user request")
|
||||
|| message.contains("transaction timeout expired")) {
|
||||
return true;
|
||||
}
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
private static long countRows() {
|
||||
return jpa.inTransaction(
|
||||
entityManager ->
|
||||
|
||||
+44
-24
@@ -2,6 +2,7 @@ package dev.caskeleton.adapter.outbound.persistence.platform.experimental;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
import static org.assertj.core.api.Assertions.assertThatThrownBy;
|
||||
import static org.assertj.core.api.Assertions.catchThrowable;
|
||||
|
||||
import com.zaxxer.hikari.HikariConfig;
|
||||
import com.zaxxer.hikari.HikariDataSource;
|
||||
@@ -12,6 +13,7 @@ import dev.caskeleton.application.tenant.TenantId;
|
||||
import java.sql.Connection;
|
||||
import java.sql.SQLException;
|
||||
import java.sql.Statement;
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
import org.junit.jupiter.api.AfterAll;
|
||||
import org.junit.jupiter.api.BeforeAll;
|
||||
@@ -51,9 +53,9 @@ class TenantPoolCapacityContractTest {
|
||||
void budgetRefusesTheTenantThatWouldExceedTheCeiling() {
|
||||
var budget = new TenantPoolBudget(4, 8);
|
||||
|
||||
budget.requireCapacity(3, 6);
|
||||
assertThat(budget.hasCapacity(4, 8)).isFalse();
|
||||
assertThatThrownBy(() -> budget.requireCapacity(4, 8))
|
||||
budget.requireCapacity(3, 6, 2);
|
||||
assertThat(budget.hasCapacity(3, 7, 2)).isFalse();
|
||||
assertThatThrownBy(() -> budget.requireCapacity(3, 7, 2))
|
||||
.isInstanceOf(IllegalStateException.class);
|
||||
}
|
||||
|
||||
@@ -77,30 +79,50 @@ class TenantPoolCapacityContractTest {
|
||||
void openingPastTheBudgetIsRefusedBeforeAPoolIsCreated() {
|
||||
var budget = new TenantPoolBudget(2, 4);
|
||||
try (var registry =
|
||||
new TenantDataSourceRegistry(
|
||||
TenantPoolCapacityContractTest::poolFor,
|
||||
TenantPoolCapacityContractTest::poolSizeOf,
|
||||
budget)) {
|
||||
new TenantDataSourceRegistry(this::poolFor, tenant -> POOL_SIZE_PER_TENANT, budget)) {
|
||||
registry.openTenants(2);
|
||||
assertThat(registry.openPools()).isEqualTo(2);
|
||||
assertThat(registry.allocatedConnections()).isEqualTo(4);
|
||||
|
||||
assertThatThrownBy(() -> registry.openTenants(3)).isInstanceOf(IllegalStateException.class);
|
||||
assertThat(registry.openPools())
|
||||
.as("a refused tenant must not leave a half-built pool behind")
|
||||
.isEqualTo(2);
|
||||
assertThat(List.of(registry.openPools(), opened.size()))
|
||||
.as("a refused tenant must not leave a half-built pool behind, or have one built at all")
|
||||
.containsExactly(2, 2);
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
void aTenantWhosePoolWouldPassTheConnectionCeilingIsRefusedBeforeItsPoolOpens() {
|
||||
// given
|
||||
// Two pools of two use four of five connections; the third pool would make six.
|
||||
var budget = new TenantPoolBudget(4, 5);
|
||||
List<Object> outcome;
|
||||
try (var registry =
|
||||
new TenantDataSourceRegistry(this::poolFor, tenant -> POOL_SIZE_PER_TENANT, budget)) {
|
||||
registry.openTenants(2);
|
||||
|
||||
// when
|
||||
Throwable refused = catchThrowable(() -> registry.require(new TenantId("tenant-2")));
|
||||
outcome =
|
||||
List.of(
|
||||
refused instanceof IllegalStateException,
|
||||
registry.openPools(),
|
||||
registry.allocatedConnections(),
|
||||
opened.size());
|
||||
}
|
||||
|
||||
// then
|
||||
assertThat(outcome)
|
||||
.as("the third pool is refused on its planned size, so it never opens a connection")
|
||||
.containsExactly(true, 2, 4, 2);
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("every opened tenant pool reaches the real server")
|
||||
void everyOpenedTenantPoolReachesTheServer() throws SQLException {
|
||||
var budget = new TenantPoolBudget(3, 6);
|
||||
try (var registry =
|
||||
new TenantDataSourceRegistry(
|
||||
TenantPoolCapacityContractTest::poolFor,
|
||||
TenantPoolCapacityContractTest::poolSizeOf,
|
||||
budget)) {
|
||||
new TenantDataSourceRegistry(this::poolFor, tenant -> POOL_SIZE_PER_TENANT, budget)) {
|
||||
registry.openTenants(3);
|
||||
|
||||
for (TenantId tenant : List.copyOf(registry.openTenantIds())) {
|
||||
@@ -118,10 +140,7 @@ class TenantPoolCapacityContractTest {
|
||||
void evictingATenantReturnsItsConnectionsToTheBudget() {
|
||||
var budget = new TenantPoolBudget(2, 4);
|
||||
try (var registry =
|
||||
new TenantDataSourceRegistry(
|
||||
TenantPoolCapacityContractTest::poolFor,
|
||||
TenantPoolCapacityContractTest::poolSizeOf,
|
||||
budget)) {
|
||||
new TenantDataSourceRegistry(this::poolFor, tenant -> POOL_SIZE_PER_TENANT, budget)) {
|
||||
registry.openTenants(2);
|
||||
TenantId evicted = List.copyOf(registry.openTenantIds()).get(0);
|
||||
|
||||
@@ -133,18 +152,19 @@ class TenantPoolCapacityContractTest {
|
||||
}
|
||||
}
|
||||
|
||||
private static javax.sql.DataSource poolFor(TenantId tenant) {
|
||||
/** Every pool this test's factory built, so a test can see one was never opened. */
|
||||
private final List<HikariDataSource> opened = new ArrayList<>();
|
||||
|
||||
private javax.sql.DataSource poolFor(TenantId tenant) {
|
||||
var config = new HikariConfig();
|
||||
config.setJdbcUrl(support.jdbcUrl());
|
||||
config.setUsername(support.username());
|
||||
config.setPassword(support.password());
|
||||
config.setMaximumPoolSize(POOL_SIZE_PER_TENANT);
|
||||
config.setPoolName("tenant-" + tenant.value());
|
||||
return new HikariDataSource(config);
|
||||
}
|
||||
|
||||
private static int poolSizeOf(javax.sql.DataSource pool) {
|
||||
return ((HikariDataSource) pool).getMaximumPoolSize();
|
||||
HikariDataSource pool = new HikariDataSource(config);
|
||||
opened.add(pool);
|
||||
return pool;
|
||||
}
|
||||
|
||||
private static int serverMaxConnections() throws SQLException {
|
||||
|
||||
+3
-1
@@ -104,12 +104,14 @@ final class PostgreSqlOptionalStreamLifecycle {
|
||||
// CONFLICT instead of a find-then-insert two webhooks could both win.
|
||||
// V13 records whether a stored request was accepted or dropped by a deduplication window,
|
||||
// so a retry with the same idempotency key gets the same receipt (review 05-23c).
|
||||
// V14 indexes the provider events still waiting for something, so the projection lag reads
|
||||
// them and not the whole history (round9 review 2026-11-12, RR9-01-14).
|
||||
//
|
||||
// This list is the stream's contract, not a note about its length: the lifecycle below
|
||||
// disables, re-enables and interrupts the stream and asserts the applied set is unchanged
|
||||
// each time. So a migration added to the stream belongs here, and the version that landed
|
||||
// without being added is why the lane failed the first time anybody ran it.
|
||||
List.of("0", "1", "2", "3", "4", "5", "6", "7", "8", "9", "10", "11", "12", "13"),
|
||||
List.of("0", "1", "2", "3", "4", "5", "6", "7", "8", "9", "10", "11", "12", "13", "14"),
|
||||
List.of());
|
||||
}
|
||||
|
||||
|
||||
+7
-3
@@ -8,9 +8,13 @@ import java.util.Map;
|
||||
|
||||
/**
|
||||
* The SQL a runbook gives an operator, read out of the runbook itself. Every lane that runs a
|
||||
* runbook's SQL reads it through this one class: so far the outbox runbooks and the migration
|
||||
* runbook's recovery of a failed concurrent index migration (re-review 2026-11-03 round 8,
|
||||
* RR8-03-03: the class kept an outbox name after the migration runbook began to use it).
|
||||
* runbook's SQL reads it through this one class: so far the outbox runbooks ({@code
|
||||
* PostgreSqlOutboxDispositionRunbookIntegrationTest}, {@code
|
||||
* PostgreSqlOutboxOrderKeyRunbookIntegrationTest}), the migration runbook's recovery of a failed
|
||||
* concurrent index migration ({@code PostgreSqlConcurrentIndexMigrationIntegrationTest}) and the
|
||||
* lock-timeout runbook's holder lookup ({@code PostgreSqlDistributedLockIntegrationTest}).
|
||||
* Re-review 2026-11-03 round 8, RR8-03-03: the class kept an outbox name after the migration
|
||||
* runbook began to use it; round9 review 2026-11-12, RR9-01-10: this list missed the lock lane.
|
||||
*
|
||||
* <p>A runbook block is marked {@code <!-- runbook-sql: id -->} and followed by a {@code sql}
|
||||
* fence. The lanes that run these blocks against real PostgreSQL read them from the document rather
|
||||
|
||||
+122
-1
@@ -1,18 +1,34 @@
|
||||
package dev.caskeleton.adapter.outbound.persistence.autoconfigure;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
import static org.mockito.ArgumentMatchers.any;
|
||||
import static org.mockito.ArgumentMatchers.anyInt;
|
||||
import static org.mockito.Mockito.mock;
|
||||
import static org.mockito.Mockito.verify;
|
||||
|
||||
import dev.caskeleton.adapter.outbound.persistence.liveevent.JpaLiveEventReplay;
|
||||
import dev.caskeleton.adapter.outbound.persistence.liveevent.LiveEventJpaRepository;
|
||||
import dev.caskeleton.adapter.outbound.persistence.liveevent.LiveEventReaper;
|
||||
import java.time.Clock;
|
||||
import java.time.Duration;
|
||||
import java.time.Instant;
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
import org.junit.jupiter.api.DisplayName;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.junit.jupiter.params.ParameterizedTest;
|
||||
import org.junit.jupiter.params.provider.CsvSource;
|
||||
import org.springframework.boot.context.properties.bind.UnboundConfigurationPropertiesException;
|
||||
import org.springframework.boot.convert.ApplicationConversionService;
|
||||
import org.springframework.boot.test.context.runner.ApplicationContextRunner;
|
||||
import org.springframework.context.ApplicationContext;
|
||||
import org.springframework.context.annotation.Bean;
|
||||
import org.springframework.context.annotation.Configuration;
|
||||
import org.springframework.context.annotation.Import;
|
||||
import org.springframework.jdbc.core.JdbcOperations;
|
||||
import org.springframework.scheduling.config.FixedDelayTask;
|
||||
import org.springframework.scheduling.config.ScheduledTask;
|
||||
import org.springframework.scheduling.config.ScheduledTaskHolder;
|
||||
|
||||
class JpaLiveEventReplayActivationTest {
|
||||
|
||||
@@ -34,6 +50,110 @@ class JpaLiveEventReplayActivationTest {
|
||||
});
|
||||
}
|
||||
|
||||
@Test
|
||||
void theReaperIsAssembledWithTheReplayStore() {
|
||||
runner
|
||||
.withPropertyValues(
|
||||
"app.realtime.replay.enabled=true", "app.realtime.replay.retention=PT1H")
|
||||
.run(
|
||||
context ->
|
||||
assertThat(context)
|
||||
.as("without the reaper nothing removes a row past its retention")
|
||||
.hasSingleBean(LiveEventReaper.class));
|
||||
}
|
||||
|
||||
@Test
|
||||
void noReaperIsAssembledWithoutTheReplayStore() {
|
||||
runner.run(context -> assertThat(context).doesNotHaveBean(LiveEventReaper.class));
|
||||
}
|
||||
|
||||
@Test
|
||||
void aZeroReaperIntervalFailsStartup() {
|
||||
runner
|
||||
.withPropertyValues(
|
||||
"app.realtime.replay.enabled=true",
|
||||
"app.realtime.replay.retention=PT1H",
|
||||
"app.realtime.replay.reaper-interval=PT0S")
|
||||
.run(
|
||||
context ->
|
||||
assertThat(context.getStartupFailure())
|
||||
.rootCause()
|
||||
.hasMessageContaining("app.realtime.replay.reaper-interval must be positive"));
|
||||
}
|
||||
|
||||
@Test
|
||||
void aMisspeltReplayKeyFailsStartup() {
|
||||
runner
|
||||
.withPropertyValues(
|
||||
"app.realtime.replay.enabled=true",
|
||||
"app.realtime.replay.retention=PT1H",
|
||||
"app.realtime.replay.reaper-intervall=PT1M")
|
||||
.run(
|
||||
context ->
|
||||
assertThat(context.getStartupFailure())
|
||||
.rootCause()
|
||||
.isInstanceOf(UnboundConfigurationPropertiesException.class)
|
||||
.hasMessageContaining("reaper-intervall"));
|
||||
}
|
||||
|
||||
@ParameterizedTest(name = "[{index}] reaper-interval {0} → {1}")
|
||||
@CsvSource(
|
||||
value = {"UNSET, PT10M", "PT3M, PT3M"},
|
||||
nullValues = "UNSET")
|
||||
@DisplayName("reaper는 묶인 reaper-interval(기본 PT10M)마다 돌고, 첫 회도 그 간격 뒤다")
|
||||
void theReaperRunsAtTheBoundInterval(String configured, Duration expected) {
|
||||
// given
|
||||
ApplicationContextRunner replay =
|
||||
runner.withPropertyValues(
|
||||
"app.realtime.replay.enabled=true", "app.realtime.replay.retention=PT1H");
|
||||
if (configured != null) {
|
||||
replay = replay.withPropertyValues("app.realtime.replay.reaper-interval=" + configured);
|
||||
}
|
||||
|
||||
// when
|
||||
List<List<Duration>> schedules = new ArrayList<>();
|
||||
replay.run(context -> schedules.addAll(fixedDelaySchedules(context)));
|
||||
|
||||
// then
|
||||
assertThat(schedules)
|
||||
.as("the schedule is the bound settings' interval, its default included")
|
||||
.containsExactly(List.of(expected, expected));
|
||||
}
|
||||
|
||||
@Test
|
||||
void theScheduledTaskIsTheReaper() {
|
||||
// given
|
||||
ApplicationContextRunner replay =
|
||||
runner.withPropertyValues(
|
||||
"app.realtime.replay.enabled=true", "app.realtime.replay.retention=PT1H");
|
||||
|
||||
// when
|
||||
List<LiveEventJpaRepository> reapedThrough = new ArrayList<>();
|
||||
replay.run(
|
||||
context -> {
|
||||
fixedDelayTasks(context).forEach(task -> task.getRunnable().run());
|
||||
reapedThrough.add(context.getBean(LiveEventJpaRepository.class));
|
||||
});
|
||||
|
||||
// then
|
||||
verify(reapedThrough.getFirst()).deleteExpiredBatch(any(Instant.class), anyInt());
|
||||
}
|
||||
|
||||
/** Interval and initial delay of every fixed-delay task the context scheduled. */
|
||||
private static List<List<Duration>> fixedDelaySchedules(ApplicationContext context) {
|
||||
return fixedDelayTasks(context).stream()
|
||||
.map(task -> List.of(task.getIntervalDuration(), task.getInitialDelayDuration()))
|
||||
.toList();
|
||||
}
|
||||
|
||||
private static List<FixedDelayTask> fixedDelayTasks(ApplicationContext context) {
|
||||
return context.getBean(ScheduledTaskHolder.class).getScheduledTasks().stream()
|
||||
.map(ScheduledTask::getTask)
|
||||
.filter(FixedDelayTask.class::isInstance)
|
||||
.map(FixedDelayTask.class::cast)
|
||||
.toList();
|
||||
}
|
||||
|
||||
@Test
|
||||
void enabledReplayCapabilityRequiresExplicitRetention() {
|
||||
runner
|
||||
@@ -42,7 +162,8 @@ class JpaLiveEventReplayActivationTest {
|
||||
context -> {
|
||||
assertThat(context).hasFailed();
|
||||
assertThat(context.getStartupFailure())
|
||||
.hasMessageContaining("app.realtime.replay.retention");
|
||||
.rootCause()
|
||||
.hasMessageContaining("app.realtime.replay.retention must be set");
|
||||
});
|
||||
}
|
||||
|
||||
|
||||
+18
-3
@@ -2,6 +2,7 @@ package dev.caskeleton.adapter.outbound.persistence.experimental;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
import static org.assertj.core.api.Assertions.assertThatThrownBy;
|
||||
import static org.assertj.core.api.Assertions.catchThrowable;
|
||||
|
||||
import dev.caskeleton.adapter.outbound.persistence.experimental.database.TenantPoolBudget;
|
||||
import dev.caskeleton.adapter.outbound.persistence.experimental.replica.ConsistencyAwareDataSourceRouter;
|
||||
@@ -103,14 +104,28 @@ class ExperimentalFailClosedTest {
|
||||
void tenantPoolBudgetIsBounded() {
|
||||
var budget = new TenantPoolBudget(2, 20);
|
||||
|
||||
assertThat(budget.hasCapacity(1, 10)).isTrue();
|
||||
assertThatThrownBy(() -> budget.requireCapacity(2, 10))
|
||||
assertThat(budget.hasCapacity(1, 10, 10)).isTrue();
|
||||
assertThatThrownBy(() -> budget.requireCapacity(2, 10, 1))
|
||||
.isInstanceOf(IllegalStateException.class)
|
||||
.hasMessageContaining("tenant pool budget");
|
||||
assertThatThrownBy(() -> budget.requireCapacity(1, 20))
|
||||
assertThatThrownBy(() -> budget.requireCapacity(1, 20, 1))
|
||||
.isInstanceOf(IllegalStateException.class);
|
||||
}
|
||||
|
||||
@Test
|
||||
void aNewPoolLargerThanTheConnectionsLeftIsRefused() {
|
||||
// given
|
||||
var budget = new TenantPoolBudget(4, 20);
|
||||
|
||||
// when
|
||||
Throwable refused = catchThrowable(() -> budget.requireCapacity(1, 15, 10));
|
||||
|
||||
// then
|
||||
assertThat(refused)
|
||||
.isInstanceOf(IllegalStateException.class)
|
||||
.hasMessageContaining("tenant pool budget exhausted");
|
||||
}
|
||||
|
||||
/** A lag monitor with a fixed answer, including "I do not know". */
|
||||
private record FixedLagMonitor(Optional<Instant> replayedThrough, Optional<Duration> lag)
|
||||
implements ReplicaLagMonitor {}
|
||||
|
||||
+45
@@ -0,0 +1,45 @@
|
||||
package dev.caskeleton.adapter.outbound.persistence.hibernate.stateless;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
import static org.assertj.core.api.Assertions.catchThrowable;
|
||||
import static org.mockito.Mockito.mock;
|
||||
|
||||
import dev.caskeleton.adapter.outbound.persistence.transaction.SpringTransaction;
|
||||
import java.time.Duration;
|
||||
import java.util.Set;
|
||||
import org.hibernate.SessionFactory;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.junit.jupiter.params.ParameterizedTest;
|
||||
import org.junit.jupiter.params.provider.ValueSource;
|
||||
|
||||
/**
|
||||
* The stateless runner's transaction bound is the write bound every other transaction here has.
|
||||
*
|
||||
* <p>The runner restates the value instead of reading it, because the {@code hibernate} package may
|
||||
* not depend on {@code transaction} (JpaModuleBoundaryTest). A test may depend on both, so it is
|
||||
* the place the two copies are held to each other (round9 review 2026-11-12, RR9-01-25).
|
||||
*/
|
||||
class StatelessRunnerTimeoutTest {
|
||||
|
||||
@Test
|
||||
void theRunnersDefaultIsSpringTransactionsWriteDefault() {
|
||||
assertThat(HibernateStatelessSessionRunner.DEFAULT_TIMEOUT)
|
||||
.isEqualTo(SpringTransaction.Timeouts.DEFAULT.write());
|
||||
}
|
||||
|
||||
@ParameterizedTest
|
||||
@ValueSource(strings = {"PT0S", "PT-1S", "PT1.5S", "PT2147483648S"})
|
||||
void aTimeoutThatIsNotAWholePositiveNumberOfSecondsIsTheAssemblysFault(String timeout) {
|
||||
// when
|
||||
Throwable refused =
|
||||
catchThrowable(
|
||||
() ->
|
||||
new HibernateStatelessSessionRunner(
|
||||
mock(SessionFactory.class), Set.of(), Duration.parse(timeout)));
|
||||
|
||||
// then
|
||||
assertThat(refused)
|
||||
.isInstanceOf(IllegalStateException.class)
|
||||
.hasMessageContaining("must be a positive whole number of seconds");
|
||||
}
|
||||
}
|
||||
+53
@@ -0,0 +1,53 @@
|
||||
package dev.caskeleton.adapter.outbound.persistence.liveevent;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
import static org.mockito.Mockito.mock;
|
||||
import static org.mockito.Mockito.times;
|
||||
import static org.mockito.Mockito.verify;
|
||||
import static org.mockito.Mockito.when;
|
||||
|
||||
import java.time.Clock;
|
||||
import java.time.Instant;
|
||||
import java.time.ZoneOffset;
|
||||
import org.junit.jupiter.api.Test;
|
||||
|
||||
/**
|
||||
* Unit tests for {@link LiveEventReaper}: rows retained no later than the clock's now go, in
|
||||
* bounded batches until one comes back short.
|
||||
*
|
||||
* <p>The repository is a Spring Data boundary, so Mockito stands in for it. The statement runs
|
||||
* against PostgreSQL in {@code JpaLiveEventReplayContractTest}.
|
||||
*/
|
||||
class LiveEventReaperTest {
|
||||
|
||||
private static final Instant NOW = Instant.parse("2026-11-13T00:00:00Z");
|
||||
private static final int BATCH = LiveEventReaper.BATCH_SIZE;
|
||||
|
||||
private final LiveEventJpaRepository repository = mock(LiveEventJpaRepository.class);
|
||||
private final LiveEventReaper reaper =
|
||||
new LiveEventReaper(repository, Clock.fixed(NOW, ZoneOffset.UTC));
|
||||
|
||||
@Test
|
||||
void reapKeepsDeletingBatchesUntilOneComesBackShortAndReturnsTheTotal() {
|
||||
// given
|
||||
when(repository.deleteExpiredBatch(NOW, BATCH)).thenReturn(BATCH, 4);
|
||||
|
||||
// when
|
||||
int purged = reaper.reap();
|
||||
|
||||
// then
|
||||
assertThat(purged).isEqualTo(BATCH + 4);
|
||||
}
|
||||
|
||||
@Test
|
||||
void eachBatchIsItsOwnDeleteStatement() {
|
||||
// given
|
||||
when(repository.deleteExpiredBatch(NOW, BATCH)).thenReturn(BATCH, 4);
|
||||
|
||||
// when
|
||||
var unused = reaper.reap();
|
||||
|
||||
// then
|
||||
verify(repository, times(2)).deleteExpiredBatch(NOW, BATCH);
|
||||
}
|
||||
}
|
||||
+14
@@ -49,6 +49,20 @@ class DistributedLockPersistenceConfigTest {
|
||||
});
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("읽는 곳이 없던 app.lock.wait-time은 모르는 키로 기동을 실패시킨다")
|
||||
void theRemovedWaitTimeKeyFailsStartup() {
|
||||
// when
|
||||
contextRunner
|
||||
.withPropertyValues("app.lock.wait-time=3s")
|
||||
.run(
|
||||
context -> {
|
||||
// then
|
||||
assertThat(context).hasFailed();
|
||||
assertThat(context.getStartupFailure()).hasStackTraceContaining("wait-time");
|
||||
});
|
||||
}
|
||||
|
||||
@Test
|
||||
void multiInstanceBuildsTheJdbcProviderAndNotTheInProcessProvider() {
|
||||
contextRunner
|
||||
|
||||
@@ -14,6 +14,7 @@
|
||||
be51fabd07220f9ef885534ba36ada5f1491baf24cdfb54d25d93a19ddc29926 jpa/notification-platform/V11__notification_reconciliation_claim_lease.sql
|
||||
43f9429f91c07528154ec01b721649b2c25fa98857ac154c7c08f4655634657f jpa/notification-platform/V12__notification_suppression_target_unique.sql
|
||||
d01aaf59f66834358abf18246faa9a9eb8dfa9f42097ca9d968c44edff564630 jpa/notification-platform/V13__notification_request_acceptance.sql
|
||||
8a7dc3604d44f1438b39dfec3aa688f792e7363cd3c1b74bf0c799a6b53f9f0d jpa/notification-platform/V14__notification_provider_event_open_index.sql
|
||||
1ac7cc1c2509868eb0ce202e1917bb36846f98f17ad4922abe3fac0ef128199b jpa/notification-platform/V1__notification_platform_core.sql
|
||||
0add67aa23580e51712f84fabe9b09e08401ef708cfa1b3292b120fdb5c12fa2 jpa/notification-platform/V2__notification_platform_contact_template_policy.sql
|
||||
f5b98598a59cc4376f5d233b410f8a5ae9f3cd94156291ebdf2d0da5a66f34ef jpa/notification-platform/V3__notification_platform_inbox_admin.sql
|
||||
|
||||
+64
-24
@@ -1,13 +1,17 @@
|
||||
package dev.caskeleton.adapter.outbound.persistence.notification.platform;
|
||||
|
||||
import dev.caskeleton.adapter.outbound.persistence.notification.platform.repository.RecipientClaimSql;
|
||||
import dev.caskeleton.application.notification.platform.observation.NotificationServingState;
|
||||
import dev.caskeleton.application.notification.platform.observation.NotificationServingStatePort;
|
||||
import java.sql.Timestamp;
|
||||
import java.time.Clock;
|
||||
import java.time.Duration;
|
||||
import java.time.Instant;
|
||||
import java.util.Map;
|
||||
import java.util.Objects;
|
||||
import org.springframework.jdbc.core.JdbcOperations;
|
||||
import org.springframework.jdbc.core.namedparam.NamedParameterJdbcOperations;
|
||||
import org.springframework.jdbc.core.namedparam.NamedParameterJdbcTemplate;
|
||||
|
||||
/**
|
||||
* Measures the platform's serving state from the durable rows.
|
||||
@@ -17,34 +21,44 @@ import org.springframework.jdbc.core.JdbcOperations;
|
||||
* identical. These are the questions an operator actually asks during an incident: how far behind
|
||||
* are we, how long has the oldest thing waited, and is anything stuck.
|
||||
*
|
||||
* <p>One statement per question rather than one joined statement. They are read independently, a
|
||||
* slow one does not delay the others, and each index the queries rely on is the same index the
|
||||
* corresponding worker already uses to claim.
|
||||
* <p>One statement per question rather than one joined statement. They are read independently and a
|
||||
* slow one does not delay the others. Every statement is bounded by an index: the backlog and
|
||||
* stuck-lease counts by the predicates the claim uses, the reconciliation count by the job's due
|
||||
* time, and the projection lag by {@code ix_notification_provider_event_open} (V14), which holds
|
||||
* only the events still waiting for something — the provider-event table itself is never purged, so
|
||||
* the lag query used to read all of it (round9 review 2026-11-12, RR9-01-14).
|
||||
*
|
||||
* <p>The scheduler reads only {@link #backlogDepth()}, on every poll; the full snapshot is for the
|
||||
* health endpoint.
|
||||
*/
|
||||
public final class JdbcNotificationServingState implements NotificationServingStatePort {
|
||||
|
||||
/**
|
||||
* Deliveries that are due and unclaimed.
|
||||
* Deliveries the claim would take now.
|
||||
*
|
||||
* <p>The lease predicate matches the claim's: a row whose lease has expired is available again,
|
||||
* so it is backlog. Counting it as in-flight is how a stuck worker hides a growing queue.
|
||||
* <p>The {@code WHERE} is the claim's own fragment ({@link RecipientClaimSql#CLAIMABLE_AT_NOW}):
|
||||
* the same states, due, not expired, and a lease that is absent or has run out. A row whose lease
|
||||
* has expired is available again, so it is backlog; counting it as in-flight is how a stuck
|
||||
* worker hides a growing queue. An expired delivery is not backlog: the expiry sweep retires it
|
||||
* unsent. The fragment's parameter is named, so this statement is too.
|
||||
*/
|
||||
private static final String BACKLOG =
|
||||
"""
|
||||
SELECT count(*) AS depth,
|
||||
COALESCE(EXTRACT(EPOCH FROM (? - min(next_dispatch_at))), 0) AS oldest_seconds
|
||||
COALESCE(EXTRACT(EPOCH FROM (:now - min(next_dispatch_at))), 0) AS oldest_seconds
|
||||
FROM notification_recipient_delivery
|
||||
WHERE delivery_state IN ('PENDING', 'READY_TO_DISPATCH', 'RETRY_WAITING')
|
||||
AND next_dispatch_at IS NOT NULL
|
||||
AND next_dispatch_at <= ?
|
||||
AND (lease_until IS NULL OR lease_until <= ?)
|
||||
""";
|
||||
WHERE\s"""
|
||||
+ RecipientClaimSql.CLAIMABLE_AT_NOW;
|
||||
|
||||
/**
|
||||
* Leases whose holder never came back.
|
||||
*
|
||||
* <p>Distinct from backlog: these rows are claimable again, but the fact that they had to expire
|
||||
* means a worker died mid-dispatch, and a provider call may have happened without a record.
|
||||
* <p>Distinct from backlog: these rows are still dispatching, which the claim never takes; lease
|
||||
* recovery requeues or reconciles them one lease after they ran out. A row left from before
|
||||
* notification V4 (an owner but fence 0) is counted here and not recovered; {@code
|
||||
* docs/upgrade/2026-11-round9-claim-fence.md}, row 5, says how to hand it to recovery. That a
|
||||
* lease had to run out means a worker died or stalled mid-dispatch, and a provider call may have
|
||||
* happened without a record.
|
||||
*/
|
||||
private static final String STUCK_LEASES =
|
||||
"""
|
||||
@@ -56,7 +70,21 @@ public final class JdbcNotificationServingState implements NotificationServingSt
|
||||
AND delivery_state = 'DISPATCHING'
|
||||
""";
|
||||
|
||||
/** Provider events stored and not yet applied to a delivery. */
|
||||
/**
|
||||
* The provider events still waiting for something: the lag query's {@code WHERE}, and the
|
||||
* predicate of the partial index {@code V14} builds for it. One string, held to the migration by
|
||||
* {@code JdbcNotificationServingStateOpenEventsTest}: a query the index does not cover reads the
|
||||
* whole table again, which is what the index was made to stop.
|
||||
*/
|
||||
static final String OPEN_EVENTS =
|
||||
"projection_status IN ('PENDING', 'FAILED') OR attempt_id IS NULL";
|
||||
|
||||
/**
|
||||
* Provider events stored and not yet applied to a delivery.
|
||||
*
|
||||
* <p>The {@code WHERE} is exactly the predicate of {@code ix_notification_provider_event_open}:
|
||||
* every row any of the four counts can see, and none of the applied, matched history.
|
||||
*/
|
||||
private static final String PROJECTION_LAG =
|
||||
"""
|
||||
SELECT
|
||||
@@ -66,7 +94,9 @@ public final class JdbcNotificationServingState implements NotificationServingSt
|
||||
COALESCE(EXTRACT(EPOCH FROM (? - min(received_at)
|
||||
FILTER (WHERE projection_status = 'PENDING'))), 0) AS oldest_seconds
|
||||
FROM notification_provider_event
|
||||
""";
|
||||
WHERE
|
||||
"""
|
||||
+ OPEN_EVENTS;
|
||||
|
||||
/** Questions to providers that are due and unanswered. */
|
||||
private static final String RECONCILIATION_LAG =
|
||||
@@ -78,26 +108,36 @@ public final class JdbcNotificationServingState implements NotificationServingSt
|
||||
""";
|
||||
|
||||
private final JdbcOperations jdbc;
|
||||
private final NamedParameterJdbcOperations named;
|
||||
private final Clock clock;
|
||||
|
||||
public JdbcNotificationServingState(JdbcOperations jdbc, Clock clock) {
|
||||
this.jdbc = Objects.requireNonNull(jdbc, "jdbc");
|
||||
this.named = new NamedParameterJdbcTemplate(jdbc);
|
||||
this.clock = Objects.requireNonNull(clock, "clock");
|
||||
}
|
||||
|
||||
@Override
|
||||
public long backlogDepth() {
|
||||
Measurement backlog = measureBacklog(Timestamp.from(clock.instant()));
|
||||
return backlog == null ? 0 : backlog.count();
|
||||
}
|
||||
|
||||
/** The backlog query alone: the one statement the scheduler's queue-depth gauge needs. */
|
||||
private Measurement measureBacklog(Timestamp at) {
|
||||
return named.queryForObject(
|
||||
BACKLOG,
|
||||
Map.of("now", at),
|
||||
(rows, rowNumber) ->
|
||||
new Measurement(rows.getLong("depth"), rows.getDouble("oldest_seconds")));
|
||||
}
|
||||
|
||||
@Override
|
||||
public NotificationServingState currentState() {
|
||||
Instant now = clock.instant();
|
||||
Timestamp at = Timestamp.from(now);
|
||||
|
||||
Measurement backlog =
|
||||
jdbc.queryForObject(
|
||||
BACKLOG,
|
||||
(rows, rowNumber) ->
|
||||
new Measurement(rows.getLong("depth"), rows.getDouble("oldest_seconds")),
|
||||
at,
|
||||
at,
|
||||
at);
|
||||
Measurement backlog = measureBacklog(at);
|
||||
Long stuck = jdbc.queryForObject(STUCK_LEASES, Long.class, at);
|
||||
ProjectionMeasurement projection =
|
||||
jdbc.queryForObject(
|
||||
|
||||
+26
@@ -0,0 +1,26 @@
|
||||
-- An index over the provider events still waiting for something.
|
||||
--
|
||||
-- The serving-state projection lag counts pending, failed and unmatched events. It had no WHERE and
|
||||
-- read the whole notification_provider_event table, which nothing ever purges, and the scheduler
|
||||
-- asked for it on every poll (round9 review 2026-11-12, RR9-01-14). The query now names exactly
|
||||
-- this predicate, so it reads only these rows; applied, matched events — the history that grows —
|
||||
-- are outside the index.
|
||||
--
|
||||
-- Built inside this migration's transaction, not CONCURRENTLY, as V21 of the core stream builds its
|
||||
-- index: a transactional file fails whole and leaves nothing to repair, where a failed concurrent
|
||||
-- build leaves an INVALID index and a failed history row an operator must clear by hand
|
||||
-- (docs/runbooks/migration-failed.md). The cost is that the build holds SHARE on
|
||||
-- notification_provider_event: provider callbacks wait for it, and providers retry a callback that
|
||||
-- does not get an answer. On a table already large in production, create the same index
|
||||
-- CONCURRENTLY by hand before deploying; IF NOT EXISTS then makes this migration a no-op.
|
||||
|
||||
-- Bound the wait for the lock (SHARE on notification_provider_event, held until commit). Unbounded,
|
||||
-- it queues behind any open transaction that touched the table, and every callback insert queues
|
||||
-- behind the migration. Five seconds, then the migration fails, rolls back whole and the stream
|
||||
-- stays at V13: deploy again once that transaction has ended (docs/jpa/migration-guide.md "Bound
|
||||
-- the lock wait").
|
||||
SET LOCAL lock_timeout = '5s';
|
||||
|
||||
CREATE INDEX IF NOT EXISTS ix_notification_provider_event_open
|
||||
ON notification_provider_event (received_at)
|
||||
WHERE projection_status IN ('PENDING', 'FAILED') OR attempt_id IS NULL;
|
||||
+42
@@ -0,0 +1,42 @@
|
||||
package dev.caskeleton.adapter.outbound.persistence.notification.platform;
|
||||
|
||||
import static java.nio.charset.StandardCharsets.UTF_8;
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.io.InputStream;
|
||||
import java.util.Objects;
|
||||
import org.junit.jupiter.api.Test;
|
||||
|
||||
/**
|
||||
* The projection-lag query and the partial index built for it name the same events.
|
||||
*
|
||||
* <p>The index is what keeps the lag query, which the scheduler runs on every poll, from reading
|
||||
* the whole provider-event history. A predicate that drifted on either side would leave the query
|
||||
* outside the index and scanning the table again, with nothing failing.
|
||||
*/
|
||||
class JdbcNotificationServingStateOpenEventsTest {
|
||||
|
||||
private static final String V14 =
|
||||
"db/migration/jpa/notification-platform/V14__notification_provider_event_open_index.sql";
|
||||
|
||||
@Test
|
||||
void theLagQueryAndItsPartialIndexNameTheSameEvents() throws IOException {
|
||||
// given
|
||||
String migration;
|
||||
try (InputStream sql = getClass().getClassLoader().getResourceAsStream(V14)) {
|
||||
migration = new String(Objects.requireNonNull(sql, V14).readAllBytes(), UTF_8);
|
||||
}
|
||||
int index = migration.indexOf("CREATE INDEX");
|
||||
|
||||
// when
|
||||
String indexPredicate =
|
||||
migration
|
||||
.substring(
|
||||
migration.indexOf("WHERE ", index) + "WHERE ".length(), migration.lastIndexOf(';'))
|
||||
.trim();
|
||||
|
||||
// then
|
||||
assertThat(indexPredicate).isEqualTo(JdbcNotificationServingState.OPEN_EVENTS);
|
||||
}
|
||||
}
|
||||
+1
-1
@@ -12,7 +12,7 @@ import java.util.Objects;
|
||||
* being applied reported exactly the same numbers as an idle healthy one. Readiness that cannot
|
||||
* distinguish those two states is not readiness.
|
||||
*
|
||||
* @param backlogDepth deliveries due now and not yet claimed
|
||||
* @param backlogDepth deliveries a claim would take now: due, not expired, not under a live lease
|
||||
* @param oldestDueAge how long the oldest of them has been due
|
||||
* @param stuckLeases leases whose holder has not finished and whose expiry has passed
|
||||
* @param pendingProjections provider events stored and not yet applied
|
||||
|
||||
+18
@@ -6,7 +6,25 @@ public interface NotificationServingStatePort {
|
||||
/**
|
||||
* Measure the backlog, the stuck leases and the projection and reconciliation lag.
|
||||
*
|
||||
* <p>Several questions, for the health snapshot. A caller that needs only the queue depth asks
|
||||
* {@link #backlogDepth()} instead: the scheduler reads the depth on every poll, and asking all of
|
||||
* them that often ran four statements a poll, one of them over the whole provider-event table
|
||||
* (round9 review 2026-11-12, RR9-01-14).
|
||||
*
|
||||
* @return the current state
|
||||
*/
|
||||
NotificationServingState currentState();
|
||||
|
||||
/**
|
||||
* How many deliveries a claim would take now — due, not expired, and not under a live lease: the
|
||||
* {@link NotificationServingState#backlogDepth()} of {@link #currentState()}, measured alone.
|
||||
*
|
||||
* <p>The default takes it from the full snapshot, so an implementation that can only measure
|
||||
* everything at once still answers; a store that can count the backlog by itself overrides it.
|
||||
*
|
||||
* @return the backlog depth
|
||||
*/
|
||||
default long backlogDepth() {
|
||||
return currentState().backlogDepth();
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user