diff --git a/docs/architecture/jpa-api-surface.txt b/docs/architecture/jpa-api-surface.txt index e2d3f24a..0063f347 100644 --- a/docs/architecture/jpa-api-surface.txt +++ b/docs/architecture/jpa-api-surface.txt @@ -5,7 +5,7 @@ # root yet. # Update only after review with: # ./gradlew :adapter:outbound:persistence-jpa:updateJpaApiSurface -PapproveJpaApiSurfaceChange -# types: 165 +# types: 166 dev.caskeleton.adapter.outbound.persistence.api.PersistenceOperationName dev.caskeleton.adapter.outbound.persistence.api.capability.CapabilitySupport dev.caskeleton.adapter.outbound.persistence.api.capability.JpaCapability @@ -169,5 +169,6 @@ dev.caskeleton.adapter.outbound.persistence.springdata.SafeSortField dev.caskeleton.adapter.outbound.persistence.springdata.SafeSortMapper dev.caskeleton.adapter.outbound.persistence.springdata.SafeSortRegistry dev.caskeleton.adapter.outbound.persistence.springdata.ScrollPolicy +dev.caskeleton.adapter.outbound.persistence.transaction.BatchedDeletes dev.caskeleton.adapter.outbound.persistence.transaction.CallerTransaction dev.caskeleton.adapter.outbound.persistence.transaction.SpringTransaction diff --git a/docs/ledger/estimates.jsonl b/docs/ledger/estimates.jsonl index a506c749..8efac6e1 100644 --- a/docs/ledger/estimates.jsonl +++ b/docs/ledger/estimates.jsonl @@ -15,3 +15,4 @@ {"schema": 1, "kind": "actual", "task": "w2-be-ci-perf", "transcript": "a6b6b25f-a4f7-46c2-a746-5ab20a588c78", "minutes": 663, "active_minutes": 417, "calls": 464, "output_tokens": 438927, "cache_read_tokens": 256417797, "max_context": 911512, "within": "over", "result": "partial", "commits": ["ecc14004", "6325b518", "86488256", "c7aff0f4", "7054d179", "575aa1f7", "deeb1126", "96caf997", "cfc80ae8", "000a59ed", "5a95db45", "d237bacc", "7284b581", "14053060", "7dbf11ca", "2c180b90", "24dde586", "aeb60339", "083c6910"], "why": "시간이 p80을 넘었다: desktop 공유 잠금 대기자 10여 개(flock이 순서 없이 깨움)로 측정·검증이 2시간 넘게 밀렸고, runner 1대 대기열에서 늦게 풀린 job이 맨 뒤로 가 수동 실행 결과가 몇 시간씩 늦었다. 예측에 없던 결정: Gitea needs+if Blocked 재현(일회용 workflow 셋), Trivy가 처음 읽은 lockfile 라이선스와 이미지가 jar 라이선스를 못 읽는 것에 맞춘 판정 재설계. 결과는 partial: runner 초록은 NFS(소유자 sudo)·compose(수정 뒤 runner 미확인) 빼고 달성, 속도 목표 넷은 미달(캐시 효과는 병합 뒤, PG 컨테이너 159개 공유는 후속).", "recorded_at": "2026-11-12T08:34+09:00"} {"schema": 1, "kind": "actual", "task": "w2-be-ci-perf", "correction": true, "transcript": "a6b6b25f-a4f7-46c2-a746-5ab20a588c78", "minutes": 1266, "active_minutes": 555, "calls": 556, "output_tokens": 534063, "cache_read_tokens": 301335415, "max_context": 963079, "within": "over", "result": "partial", "commits": ["ecc14004", "6325b518", "86488256", "c7aff0f4", "7054d179", "575aa1f7", "deeb1126", "96caf997", "cfc80ae8", "000a59ed", "5a95db45", "d237bacc", "7284b581", "14053060", "7dbf11ca", "2c180b90", "24dde586", "aeb60339", "083c6910", "335fc967", "9682c52c", "3e06a809", "d17f8bcf", "7db40c9b"], "why": "앞 actual 줄을 고친다. 그 뒤 총괄 지시 둘(새 main 위로, main.yml concurrency와 취소 증거)과 main 두 번 병합으로 작업 중 138분·호출 92가 더 들었다. 대부분 runner 대기열(PR run이 한 번에 2~4시간 뒤로 밀림)과 desktop 공유 잠금 대기였다. 결과는 여전히 partial: runner PR 130 초록·release gate 125 초록, NFS는 소유자 sudo, 속도 목표 넷은 미달(PR 13.5→12.0분/목표 8, release gate 45.6→32.5분/목표 35는 이미지 job 포함), 캐시 효과는 병합 뒤 측정.", "recorded_at": "2026-11-12T18:36+09:00"} {"schema": 1, "kind": "estimate", "task": "w2-be-ci-perf-2", "transcript": "a6b6b25f-a4f7-46c2-a746-5ab20a588c78", "started_at": "2026-11-12T18:45+09:00", "base": "e7be517e", "task_kind": "improve", "size": "large", "minutes": {"p30": 150, "p50": 240, "p80": 400}, "calls": {"p30": 120, "p50": 200, "p80": 320}, "output_tokens": {"p30": 100000, "p50": 170000, "p80": 280000}, "basis": "직전 라운드 w2-be-ci-perf 실측 555분(p50 144의 3.9배, 대부분 runner·desktop 잠금 대기). 이번 판단 5개(B2·NFS·C2·job 합치기·main 줄 순서), desktop main 측정 1회(30분)+pr 1회, runner nightly 1회(2시간)+release 1회+PR 2회. 대기가 시간을 정한다", "recorded_at": "2026-11-12T18:58+09:00"} +{"schema": 1, "kind": "estimate", "task": "r9-group-b", "transcript": "0ef672ab-db3e-427a-9f53-b2e918462534", "started_at": "2026-11-13T07:00:00+09:00", "base": "d3a90479", "task_kind": "improve", "size": "large", "minutes": {"p30": 190, "p50": 286, "p80": 424}, "calls": {"p30": 378, "p50": 567, "p80": 819}, "output_tokens": {"p30": 348000, "p50": 522000, "p80": 754000}, "basis": "발견 42건·PR 3·결정 약 15·desktop 실행 약 20회(Testcontainers lane)·코드 약 3000줄; 원값 360/540/800분·600/900/1300호출·600k/900k/1300k에 보정 0.53/0.63/0.58 곱함; 선례 w2-be-ci-perf 555분", "recorded_at": "2026-11-13T06:52+09:00"} diff --git a/docs/manual/04-persistence.md b/docs/manual/04-persistence.md index 040425ef..88720690 100644 --- a/docs/manual/04-persistence.md +++ b/docs/manual/04-persistence.md @@ -91,6 +91,17 @@ store는 자기 트랜잭션을 열지 않는다. 유스케이스가 연 트랜 예: `DurableOperationStoreContract`(`InMemoryDurableOperationStore`, `JpaDurableOperationStoreContractTest`). - 여러 인스턴스가 집는 작업 → `FOR UPDATE SKIP LOCKED`는 **트랜잭션 안에서**, 그리고 lease(만료 시각)와 fence 값을 함께 저장한다. 예: `ReconciliationJobStorePort.claimDue`(application-notification). + - 연장·완료·되돌리기는 그 claim의 owner와 fence를 `WHERE`에 넣은 조건부 UPDATE 하나다. fence는 claim마다 + 오르므로 같은 worker id가 다시 claim해도 앞 실행의 쓰기는 0행이다(예: `DurableOperationClaim`). + - `WHERE id IN (SELECT … LIMIT n)`으로 고른 행을 바꿀 때는 subquery를 `FOR UPDATE SKIP LOCKED`로 잠가 남이 쥔 행을 + 고르지 않는다. 상태 조건도 바깥 `UPDATE … WHERE`에 한 번 더 둔다. 잠금 절이 빠지면 READ COMMITTED는 잠금을 기다린 + 행에 바깥 조건만 다시 보고 subquery 안 조건은 보지 않기 때문이다(예: `RecipientClaimSql.EXPIRE_OVERDUE`). 잠그지 않는 + subquery라면 바깥 조건이 유일한 방어다(예: inbox `markAllRead`). + - 만료는 그 시각을 쓴 시계로 본다. row lease(outbox, 알림 수신자, durable operation, fileserver cleanup, + Mongo migration lock)는 앱 `Clock`으로 쓰고 앱 `Clock`으로 비교한다. 시험이 시계를 옮길 수 있게 하려는 + 것이고, 대신 인스턴스 사이 시계 차이가 가장 짧은 lease보다 훨씬 작아야 한다(NTP 동기화가 배포 조건이다). + DB가 쓴 시각은 DB 시계로(멱등성 `expires_at`은 `clock_timestamp()`로 쓰고 `now()`로 지운다), Redis가 쓴 + 시각은 Redis `TIME`으로 본다. 한 저장소 안에서 두 시계를 섞지 않는다. - 수정 충돌 → 버전 열(낙관적 잠금). 예: `DurableOperationEntity`(persistence-jpa)의 `@Version` 열 `row_version`. - **예외**: DB가 볼 수 없는 규칙(외부 시스템 상태)은 분산 락(`DistributedLockPort`)이나 멱등성 키(05장). - 동시성 테스트는 실 DB lane에서 동시 요청 N개 → 행 1개를 단언한다. diff --git a/docs/notification/configuration-reference.md b/docs/notification/configuration-reference.md index 1b298505..ca79ccd6 100644 --- a/docs/notification/configuration-reference.md +++ b/docs/notification/configuration-reference.md @@ -26,7 +26,7 @@ support leaf의 `ModuleDefaultsEnvironmentPostProcessor`가 가장 낮은 우선 | 속성 | 환경 변수 | 기본값 | 범위 | |---|---|---|---| | `dispatch.claim-batch-size` | `APP_NOTIFICATION_PLATFORM_CLAIM_BATCH_SIZE` | `50` | 1..1000 | -| `dispatch.lease-duration` | `APP_NOTIFICATION_PLATFORM_LEASE_DURATION` | `2m` | 양수, 유한 | +| `dispatch.lease-duration` | `APP_NOTIFICATION_PLATFORM_LEASE_DURATION` | `2m` | 양수, 유한. `poll-interval`과 켜진 provider profile의 `timeout`보다 길어야 기동한다 | | `dispatch.poll-interval` | `APP_NOTIFICATION_PLATFORM_POLL_INTERVAL` | `1s` | 양수, 유한 | | `dispatch.max-global-concurrency` | `APP_NOTIFICATION_PLATFORM_MAX_CONCURRENCY` | `64` | 양수 | | `dispatch.max-additional-attempts` | `APP_NOTIFICATION_PLATFORM_MAX_ADDITIONAL_ATTEMPTS` | `4` | 0 이상 | @@ -35,8 +35,11 @@ support leaf의 `ModuleDefaultsEnvironmentPostProcessor`가 가장 낮은 우선 모든 값에 상한이 있다. 「무제한」은 받는 설정이 아니다. -lease는 provider 호출과 그 시간 상한보다 길어야 한다. 그보다 짧으면, 살아 있는 worker가 아직 기다리는 전달을 두 번째 -worker가 claim하고, 수신자는 알림을 두 번 받는다. +lease는 provider 호출 한 번이 delivery를 쥐는 시간보다 길어야 한다. 그보다 짧으면 아직 기다리는 호출이 멈춘 lease로 +보인다. lease가 끝난 뒤 lease만큼 더 지나도 호출이 끝나지 않았으면, lease 복구가 그 delivery를 버려진 것으로 보고 맞추기로 +넘긴다. 기본 제공 provider(SES·SMTP)는 결과 조회가 없어 운영자 확인(`RECONCILIATION_REQUIRED`)으로 넘어간다. 그 뒤에 +끝난 호출도 자기 결과를 기록한다(owner와 fence가 그대로다). 두 번째 worker가 다시 보내지는 않는다. claim은 +`DISPATCHING`인 delivery를 가져가지 않는다. ## 콜백 diff --git a/docs/registries/env-keys.yaml b/docs/registries/env-keys.yaml index 3df26fc2..99a9b670 100644 --- a/docs/registries/env-keys.yaml +++ b/docs/registries/env-keys.yaml @@ -5063,7 +5063,8 @@ env_keys: required_test: adapter-contract:notification-dispatch-bounds - name: APP_NOTIFICATION_PLATFORM_LEASE_DURATION - # source: NTF-025 — provider 호출과 그 timeout보다 길어야 한다. 아니면 살아 있는 worker의 전달을 다른 worker가 claim한다 + # source: NTF-025 — poll-interval과 켜진 provider profile의 timeout보다 길어야 기동한다(2026-11-13부터 검사). + # 짧으면 살아 있는 worker의 전달을 다른 worker가 claim한다. 넉넉히 길게 둔다: lease는 claim 때 시작한다 type: duration default: 2m allowed_values: null diff --git a/docs/upgrade/2026-11-round9-claim-fence.md b/docs/upgrade/2026-11-round9-claim-fence.md new file mode 100644 index 00000000..99ae53f9 --- /dev/null +++ b/docs/upgrade/2026-11-round9-claim-fence.md @@ -0,0 +1,38 @@ +# 업그레이드 안내 — 2026-11 claim·fence·트랜잭션 시한·정리 + +- 범위: durable operation·알림 수신자·fileserver cleanup·멱등성 기록의 lease, fence, 조건부 쓰기, 정리를 고친 + 변경. 포트 시그니처와 기동 조건이 바뀐 곳이 있다. 아래 표가 그 목록이다. +- 근거: 2026-11-12 전체 리뷰(`docs/reviews/2026-11-12-round9-review/`)의 claim·fence(B8) 발견 + +## 1. 누가 읽나 + +이 템플릿을 `upstream` remote로 두고 merge해 받는 파생 서비스가 읽는다. 특히 durable operation worker를 직접 쓴 서비스, +알림 recipient store를 직접 구현한 서비스, 알림 provider timeout을 설정한 서비스다. + +## 2. 할 일 + +| # | 할 일 | 안 하면 | 해당 서비스 | +| --- | --- | --- | --- | +| 1 | durable operation worker는 `claimNext`가 돌려주는 `DurableOperationClaim`을 들고 있다가 `heartbeat(claim, …)`·`reportProgress(claim, …)`·`succeed(claim, …)`·`fail(claim, …)`에 넘긴다. 예전에는 `(operationId, workerId, …)`였다 | 컴파일이 실패한다 | durable operation worker를 쓴 서비스 | +| 2 | `DurableOperationStorePort`를 직접 구현했으면 claim마다 오르는 fence를 저장하고, 네 쓰기가 owner와 fence를 함께 확인하게 한다. `DurableOperationStoreContract`의 새 사례(같은 worker id가 다시 claim한 뒤 앞 실행의 쓰기는 거절)로 확인한다 | 재시작한 worker나 한 프로세스의 두 실행이 같은 id를 쓰면 앞 실행의 결과가 넘겨받은 실행 위에 기록된다 | 자기 durable operation store를 둔 서비스 | +| 3 | persistence-jpa의 `V23__durable_operation_lease_fence.sql`(`durable_operation.lease_fence BIGINT NOT NULL DEFAULT 0`)이 기본 stream에 더해진다. 같은 번호의 서비스 마이그레이션이 있으면 번호를 옮긴다. `durable_operation`을 건드린 트랜잭션이 열려 있으면 5초 뒤 실패하고 V22에 남는다(다시 배포) | Flyway가 번호 충돌로 기동하지 않는다 | persistence-jpa를 쓰는 모든 서비스 | +| 4 | `RecipientLeaseStorePort.expiredDispatching`이 `List`(버려질 때의 owner·fence·만료 시각)를 돌려주고, `RecipientDeliveryStorePort`에 `requeueAbandoned(lease, nextDispatchAt)`가 생겼다. 직접 구현했으면 상태가 `DISPATCHING`이고 owner·fence가 같고 lease가 늘지 않았을 때만 `READY_TO_DISPATCH`로 바꾼다(`RecipientDeliveryStoreContract` 새 사례). owner가 없거나 fence가 0인 `DISPATCHING` 행(notification V4 전에 남은 행)은 복구가 더는 고르지 않는다. 그런 행은 `SELECT id FROM notification_recipient_delivery WHERE delivery_state = 'DISPATCHING' AND (lease_owner IS NULL OR lease_fence = 0)`로 찾는다. 배포 전에 `UPDATE notification_recipient_delivery SET lease_owner = COALESCE(lease_owner, 'pre-v4'), lease_fence = lease_fence + 1, lease_until = now() - interval '1 day', version = version + 1, updated_at = now() WHERE delivery_state = 'DISPATCHING' AND (lease_owner IS NULL OR lease_fence = 0)`로 owner를 채우고 fence를 올려 복구 패스에 넘긴다. 복구는 attempt를 보고 정한다: attempt가 없으면 다시 보낼 차례로 돌리고, 끝나지 않은 attempt가 있으면 맞추기로 넘긴다(그 뒤는 `docs/notification/configuration-reference.md` 「발송」의 lease 문단). 손으로 `READY_TO_DISPATCH`로 바꾸지 않는다. 발송 중에 멈춘 행이면 같은 알림이 한 번 더 나간다. owner가 없는 행은 stuck lease 지표에도 잡히지 않는다 | 컴파일이 실패한다. 조건 없이 바꾸면 끝났거나 다시 claim된 delivery가 다시 발송된다 | 알림 store를 직접 구현한 서비스 | +| 5 | 알림 provider profile의 `timeout`이 `app.notification.platform.dispatch.lease-duration`보다 짧은지 확인한다. 짧은 것은 필요조건일 뿐이다: lease는 claim 때 시작해서 attempt 기록·렌더링·provider 허가 대기도 lease를 쓴다. lease는 timeout보다 넉넉히 길게 둔다. 이 timeout이 provider 호출 전체를 묶는 것은 SES(API 호출 시간 상한)다. SMTP는 `spring.mail` socket timeout이 명령마다 묶을 뿐이라 이 검사만으로는 호출 전체가 lease 안에 끝난다고 말할 수 없다 | 짧지 않으면 기동이 실패한다(`notification provider profile '…': timeout (…) must be shorter than …lease-duration`). 전에는 provider 호출 하나가 lease보다 길어도 기동했다. 그때 생기는 일은 `docs/notification/configuration-reference.md` 「발송」의 lease 문단이다 | 알림 capability를 켠 서비스 | +| 6 | `IdempotencyReaper`를 직접 만들던 코드는 `new IdempotencyReaper(repository)`로 바꾼다(`Clock` 인자 삭제). 만료 비교는 DB 시계(`now()`)이고 1000행씩 지운다 | 컴파일이 실패한다 | reaper를 직접 조립한 서비스 | +| 7 | fileserver cleanup 묶음의 `max-items`를 정할 때, 한 묶음이 claim lease(10분)의 첫 8분 안에 시작할 수 있는 만큼만 처리된다는 것을 안다. 나머지는 시도하지 않은 채 queue에 돌아가 다음 묶음이 다시 claim한다. 시도 횟수는 그대로다(바이트 예산이 바닥나 돌려준 항목도 이제 같다. 전에는 시도 한 번으로 세어 8번째에 보류됐다). 직접 구현한 `CleanupQueuePort`는 새 연산 `release`를 구현한다(`CleanupQueuePortContract` 새 사례) | 느린 저장소에서 큰 `max-items`는 그 사유로 도는 항목이 늘어난다. 전에는 lease가 끝난 뒤에도 남은 항목을 처리해 다른 worker와 겹쳤다 | fileserver capability를 켠 서비스 | + +## 3. 바뀐 동작 (따라 할 일 없음) + +- reaper 둘(outbox·멱등성)의 트랜잭션은 `SpringTransaction.DEFAULT_TIMEOUT_SECONDS`(30초)로 묶인다. + `app.persistence-jpa.transaction.*`를 바꿔도 reaper 시한은 그대로다. 한 트랜잭션이 1000행 한 묶음만 지운다. +- 저장소가 행에서 만드는 `DurableOperationClaim`·`OperationLease`·`RecipientLease`·`OutboxLease`·`ClaimedOutboxEvent`· + `ReconciliationJob`·`AdminOperationClaim`이 값을 거절할 때 `IllegalStateException`을 던진다 + (전에는 `IllegalArgumentException`). 요청에서 오는 값이 아니라 서버 결함이다. 이 IAE를 잡던 코드는 ISE를 잡는다. +- 알림 lease 회복은 버려질 때의 lease 그대로일 때만 delivery를 다시 넣는다. 다른 인스턴스가 이미 다시 넣었거나 holder가 + 끝냈으면 아무것도 쓰지 않고 처리 수에 세지 않는다. +- 알림 만료 sweep(`EXPIRE_OVERDUE`)은 다른 트랜잭션이 쥔 행을 건너뛰고, 바깥 `UPDATE`에서도 상태·만료를 다시 본다. + inbox 전체 읽음(`markAllRead`)도 바깥 `UPDATE`에서 상태를 다시 본다. 기다리는 사이 claim된 delivery나 보관된 inbox 항목을 + 바꾸지 않는다. +- 멱등성 reaper는 다른 트랜잭션이 쥔 기록을 건너뛰고, 바깥 `DELETE`에서도 만료를 다시 본다. 같은 키 요청이 만료된 기록을 + 되살리는 동안 돈 reap이 그 claim을 지우지 않는다. +- fileserver recovery 대기열 enqueue는 `INSERT … ON CONFLICT` 한 문장이다. 같은 파일을 동시에 넣어도 25P02로 실패하지 않는다. diff --git a/src/adapter/outbound/notification/src/main/java/dev/caskeleton/adapter/outbound/notification/platform/autoconfigure/NotificationPlatformSettings.java b/src/adapter/outbound/notification/src/main/java/dev/caskeleton/adapter/outbound/notification/platform/autoconfigure/NotificationPlatformSettings.java index 15e9eb5d..19a86e65 100644 --- a/src/adapter/outbound/notification/src/main/java/dev/caskeleton/adapter/outbound/notification/platform/autoconfigure/NotificationPlatformSettings.java +++ b/src/adapter/outbound/notification/src/main/java/dev/caskeleton/adapter/outbound/notification/platform/autoconfigure/NotificationPlatformSettings.java @@ -37,6 +37,8 @@ public record NotificationPlatformSettings( providers = providers == null ? Map.of() : Map.copyOf(providers); boolean callbacksEnabled = callbacks.enabled(); providers.forEach((id, provider) -> provider.validate(id, callbacksEnabled)); + Duration leaseDuration = dispatch.leaseDuration(); + providers.forEach((id, provider) -> provider.requireCallFitsInLease(id, leaseDuration)); } /** Source-compatible constructor for callers that rely on the original settings shape. */ @@ -264,6 +266,40 @@ public record NotificationPlatformSettings( } } + /** + * Fail the boot when one call of this profile alone would outlast the dispatch lease. + * + *

A worker holds a delivery's lease while it waits on the provider, and the profile's + * timeout is how long that wait can be. A lease that ends first shows a live call as a stuck + * lease; if the call is still running one lease later, lease recovery takes the delivery for + * abandoned and hands it to reconciliation, and the shipped families (SES, SMTP) have no status + * query, so it is parked for an operator. A call that finishes after that still records its + * result, since it holds the same owner and fence. No second worker claims it: the claim never + * takes a dispatching delivery. Each value is fine on its own, so only the pair can be checked + * (round9 review 2026-11-12, RR9-01-24). + * + *

A necessary condition, not a sufficient one: the lease starts at the claim, so writing the + * attempt, rendering and waiting for a provider permit spend it before the call begins. A + * deployment keeps the lease well above the timeout rather than just above it. + * + * @param profileId the profile id + * @param leaseDuration {@code app.notification.platform.dispatch.lease-duration} + */ + void requireCallFitsInLease(String profileId, Duration leaseDuration) { + if (!enabled) { + return; + } + require( + timeout.compareTo(leaseDuration) < 0, + profileId, + "timeout (" + + timeout + + ") must be shorter than app.notification.platform.dispatch.lease-duration (" + + leaseDuration + + "): a provider call that outlasts its lease shows as a stuck lease, and one lease" + + " later lease recovery hands its delivery to reconciliation while it may still run"); + } + private static void require(boolean condition, String profileId, String message) { if (!condition) { throw new IllegalArgumentException( diff --git a/src/adapter/outbound/notification/src/main/java/dev/caskeleton/adapter/outbound/notification/platform/dispatch/NotificationDispatchProperties.java b/src/adapter/outbound/notification/src/main/java/dev/caskeleton/adapter/outbound/notification/platform/dispatch/NotificationDispatchProperties.java index 7cafab9b..cf00636b 100644 --- a/src/adapter/outbound/notification/src/main/java/dev/caskeleton/adapter/outbound/notification/platform/dispatch/NotificationDispatchProperties.java +++ b/src/adapter/outbound/notification/src/main/java/dev/caskeleton/adapter/outbound/notification/platform/dispatch/NotificationDispatchProperties.java @@ -42,8 +42,9 @@ public record NotificationDispatchProperties( requirePositive(estimatedDispatchDuration, "estimatedDispatchDuration"); requirePositive(maxQueueAge, "maxQueueAge"); if (leaseDuration.compareTo(pollInterval) <= 0) { - // A lease shorter than the poll interval expires before the worker can renew it, so two - // workers would routinely claim the same job. + // A sanity bound, not a renewal rule (see RecipientLeaseStorePort.renew). The lease is how + // long one dispatch may hold a delivery and the poll interval only how long an idle worker + // sleeps, so a lease no longer than that nap is too short for any real send. throw new IllegalArgumentException("leaseDuration must exceed pollInterval"); } } diff --git a/src/adapter/outbound/notification/src/test/java/dev/caskeleton/adapter/outbound/notification/platform/autoconfigure/NotificationPlatformSettingsTest.java b/src/adapter/outbound/notification/src/test/java/dev/caskeleton/adapter/outbound/notification/platform/autoconfigure/NotificationPlatformSettingsTest.java index 05182ad6..91a446d6 100644 --- a/src/adapter/outbound/notification/src/test/java/dev/caskeleton/adapter/outbound/notification/platform/autoconfigure/NotificationPlatformSettingsTest.java +++ b/src/adapter/outbound/notification/src/test/java/dev/caskeleton/adapter/outbound/notification/platform/autoconfigure/NotificationPlatformSettingsTest.java @@ -31,6 +31,39 @@ class NotificationPlatformSettingsTest { .hasMessageContaining("unknown provider type '" + removedType + "'"); } + @Test + void aProfileWhoseTimeoutIsNotShorterThanTheLeaseFailsStartup() { + // given + Duration lease = NotificationPlatformSettings.Dispatch.defaults().leaseDuration(); + NotificationPlatformSettings.Provider slow = + new NotificationPlatformSettings.Provider( + "SMTP", true, false, "PRODUCTION", "smtp-key", lease, 8, 20, null); + + // when + Throwable failure = catchThrowable(() -> properties(slow)); + + // then + assertThat(failure) + .isInstanceOf(IllegalArgumentException.class) + .hasMessageContaining("notification provider profile 'profile'") + .hasMessageContaining("must be shorter than app.notification.platform.dispatch.lease"); + } + + @Test + void aDisabledProfileIsNotHeldToTheLease() { + // given + Duration lease = NotificationPlatformSettings.Dispatch.defaults().leaseDuration(); + NotificationPlatformSettings.Provider off = + new NotificationPlatformSettings.Provider( + "SMTP", false, false, "PRODUCTION", "smtp-key", lease.multipliedBy(2), 8, 20, null); + + // when + NotificationPlatformSettings settings = properties(off); + + // then + assertThat(settings.providers()).containsOnlyKeys("profile"); + } + @Test void enabledSesRequiresItsTypedTransportIdentity() { assertThatThrownBy( diff --git a/src/adapter/outbound/persistence-fileserver/src/main/java/dev/caskeleton/adapter/outbound/persistence/fileserver/JpaCleanupQueue.java b/src/adapter/outbound/persistence-fileserver/src/main/java/dev/caskeleton/adapter/outbound/persistence/fileserver/JpaCleanupQueue.java index 8912277d..56eed895 100644 --- a/src/adapter/outbound/persistence-fileserver/src/main/java/dev/caskeleton/adapter/outbound/persistence/fileserver/JpaCleanupQueue.java +++ b/src/adapter/outbound/persistence-fileserver/src/main/java/dev/caskeleton/adapter/outbound/persistence/fileserver/JpaCleanupQueue.java @@ -118,6 +118,12 @@ public class JpaCleanupQueue implements CleanupQueuePort { settle(item, STATUS_FAILED, nextAttemptAt, reasonCode, clock.instant()); } + @Override + public void release(CleanupItem item, Instant nextAttemptAt) { + CallerTransaction.require("cleanup queue release"); + items.releaseClaim(item.cleanupId(), item.claimToken(), nextAttemptAt, clock.instant()); + } + @Override public void markAbandoned(CleanupItem item, String reasonCode) { CallerTransaction.require("cleanup queue markAbandoned"); diff --git a/src/adapter/outbound/persistence-fileserver/src/main/java/dev/caskeleton/adapter/outbound/persistence/fileserver/JpaRecoveryQueue.java b/src/adapter/outbound/persistence-fileserver/src/main/java/dev/caskeleton/adapter/outbound/persistence/fileserver/JpaRecoveryQueue.java index e56faffe..caa3ce2b 100644 --- a/src/adapter/outbound/persistence-fileserver/src/main/java/dev/caskeleton/adapter/outbound/persistence/fileserver/JpaRecoveryQueue.java +++ b/src/adapter/outbound/persistence-fileserver/src/main/java/dev/caskeleton/adapter/outbound/persistence/fileserver/JpaRecoveryQueue.java @@ -7,7 +7,6 @@ import dev.caskeleton.application.fileserver.api.FileId; import dev.caskeleton.application.fileserver.recovery.ReconciliationStatus; import dev.caskeleton.application.fileserver.recovery.RecoveryQueuePort; import java.time.Clock; -import java.time.Instant; import java.util.List; import java.util.UUID; import org.springframework.data.domain.Limit; @@ -15,10 +14,11 @@ import org.springframework.data.domain.Limit; /** * Durable recovery queue over {@code fs_recovery_item}. * - *

An enqueue is an upsert: the same file reported twice updates the open item rather than adding - * a second one. Reconciliation runs on a schedule and re-raises whatever it still cannot settle, so - * an append-only queue would grow one row per sweep per unresolved file and bury the distinct - * problems under repetitions of the same one. + *

An enqueue is an upsert, one {@code INSERT … ON CONFLICT} statement: the same file reported + * twice — in sequence or in two concurrent transactions — updates the open item rather than adding + * a second one or failing on the open-item index. Reconciliation runs on a schedule and re-raises + * whatever it still cannot settle, so an append-only queue would grow one row per sweep per + * unresolved file and bury the distinct problems under repetitions of the same one. * *

Resolution keeps the outcome rather than deleting the row. {@code UNRESOLVED} and {@code * QUARANTINE_REQUIRED} are the two answers a human has to act on, and both are worthless if the @@ -26,8 +26,6 @@ import org.springframework.data.domain.Limit; */ public class JpaRecoveryQueue implements RecoveryQueuePort { - private static final String STATUS_PENDING = "PENDING"; - private final FileserverRecoveryRepository items; private final Clock clock; @@ -39,12 +37,7 @@ public class JpaRecoveryQueue implements RecoveryQueuePort { @Override public void enqueue(FileId fileId, String reasonCode) { CallerTransaction.require("recovery queue enqueue"); - Instant now = clock.instant(); - if (items.refreshPending(fileId.value(), reasonCode, now) > 0) { - return; - } - items.save( - new RecoveryItemEntity(UUID.randomUUID(), fileId.value(), reasonCode, STATUS_PENDING, now)); + items.upsertPending(UUID.randomUUID(), fileId.value(), reasonCode, clock.instant()); } @Override diff --git a/src/adapter/outbound/persistence-fileserver/src/main/java/dev/caskeleton/adapter/outbound/persistence/fileserver/entity/RecoveryItemEntity.java b/src/adapter/outbound/persistence-fileserver/src/main/java/dev/caskeleton/adapter/outbound/persistence/fileserver/entity/RecoveryItemEntity.java index 020901fe..5d4cd9e8 100644 --- a/src/adapter/outbound/persistence-fileserver/src/main/java/dev/caskeleton/adapter/outbound/persistence/fileserver/entity/RecoveryItemEntity.java +++ b/src/adapter/outbound/persistence-fileserver/src/main/java/dev/caskeleton/adapter/outbound/persistence/fileserver/entity/RecoveryItemEntity.java @@ -52,17 +52,6 @@ public class RecoveryItemEntity extends AssignedIdEntity { return recoveryId; } - public RecoveryItemEntity( - UUID recoveryId, UUID fileId, String reasonCode, String status, Instant createdAt) { - this.recoveryId = recoveryId; - this.fileId = fileId; - this.reasonCode = reasonCode; - this.status = status; - this.attempt = 0; - this.createdAt = createdAt; - this.updatedAt = createdAt; - } - public UUID getRecoveryId() { return recoveryId; } diff --git a/src/adapter/outbound/persistence-fileserver/src/main/java/dev/caskeleton/adapter/outbound/persistence/fileserver/repository/FileserverCleanupRepository.java b/src/adapter/outbound/persistence-fileserver/src/main/java/dev/caskeleton/adapter/outbound/persistence/fileserver/repository/FileserverCleanupRepository.java index 0e302f72..80105229 100644 --- a/src/adapter/outbound/persistence-fileserver/src/main/java/dev/caskeleton/adapter/outbound/persistence/fileserver/repository/FileserverCleanupRepository.java +++ b/src/adapter/outbound/persistence-fileserver/src/main/java/dev/caskeleton/adapter/outbound/persistence/fileserver/repository/FileserverCleanupRepository.java @@ -63,6 +63,32 @@ public interface FileserverCleanupRepository extends JpaRepositoryThe item goes back to {@code PENDING} when it was never tried and to {@code FAILED} when an + * earlier attempt was — the two statuses a claim takes from — and keeps its counter and its last + * error. + */ + @Modifying(clearAutomatically = true, flushAutomatically = true) + @Query( + """ + update CleanupItemEntity c + set c.status = case when c.attempt = 0 then 'PENDING' else 'FAILED' end, + c.nextAttemptAt = :nextAttemptAt, + c.claimOwner = null, + c.claimToken = null, + c.leaseUntil = null, + c.updatedAt = :now + where c.cleanupId = :cleanupId + and c.claimToken = :token + """) + int releaseClaim( + @Param("cleanupId") UUID cleanupId, + @Param("token") UUID token, + @Param("nextAttemptAt") Instant nextAttemptAt, + @Param("now") Instant now); + /** * Returns items whose claim has expired, for a reaper to take over. * diff --git a/src/adapter/outbound/persistence-fileserver/src/main/java/dev/caskeleton/adapter/outbound/persistence/fileserver/repository/FileserverRecoveryRepository.java b/src/adapter/outbound/persistence-fileserver/src/main/java/dev/caskeleton/adapter/outbound/persistence/fileserver/repository/FileserverRecoveryRepository.java index 88f8f885..49792a40 100644 --- a/src/adapter/outbound/persistence-fileserver/src/main/java/dev/caskeleton/adapter/outbound/persistence/fileserver/repository/FileserverRecoveryRepository.java +++ b/src/adapter/outbound/persistence-fileserver/src/main/java/dev/caskeleton/adapter/outbound/persistence/fileserver/repository/FileserverRecoveryRepository.java @@ -22,23 +22,34 @@ public interface FileserverRecoveryRepository extends JpaRepository findPending(Limit limit); /** - * Re-raises an open item instead of adding a second one. + * Opens an item for the file, or re-raises the open one instead of adding a second, in one + * statement. * *

Reconciliation is retried on a schedule, so the same file reaches the queue repeatedly. One * open item per file keeps the queue a worklist rather than a failure log; the newest reason wins * because it describes the most recent evidence. + * + *

One {@code INSERT … ON CONFLICT} on the partial unique index {@code uq_fs_recovery_open} + * rather than an update followed by an insert when it matched nothing. Two enqueues of the same + * file in concurrent transactions both saw no open row, both inserted, and the second hit the + * index — a unique violation that aborts its transaction ({@code 25P02}) and fails the sweep that + * reported it. Here the second waits for the first to commit and then updates the row it wrote. */ @Modifying(clearAutomatically = true, flushAutomatically = true) @Query( - """ - update RecoveryItemEntity r - set r.reasonCode = :reasonCode, - r.attempt = r.attempt + 1, - r.updatedAt = :now - where r.fileId = :fileId - and r.status = 'PENDING' - """) - int refreshPending( + value = + """ + INSERT INTO fs_recovery_item + (recovery_id, file_id, reason_code, status, attempt, created_at, updated_at) + VALUES (:recoveryId, :fileId, :reasonCode, 'PENDING', 0, :now, :now) + ON CONFLICT (file_id) WHERE status = 'PENDING' + DO UPDATE SET reason_code = EXCLUDED.reason_code, + attempt = fs_recovery_item.attempt + 1, + updated_at = EXCLUDED.updated_at + """, + nativeQuery = true) + int upsertPending( + @Param("recoveryId") UUID recoveryId, @Param("fileId") UUID fileId, @Param("reasonCode") String reasonCode, @Param("now") Instant now); diff --git a/src/adapter/outbound/persistence-jpa/AGENTS.md b/src/adapter/outbound/persistence-jpa/AGENTS.md index e34653d4..26c31492 100644 --- a/src/adapter/outbound/persistence-jpa/AGENTS.md +++ b/src/adapter/outbound/persistence-jpa/AGENTS.md @@ -88,6 +88,9 @@ lock TTL)과 계약 표(트랜잭션 모드, SQLState → 오류 코드, vendor, `lease_token`을 올린다(`V17`). `markPublished`/`markFailed`/`markDead`는 `event_id` + `IN_FLIGHT` + owner + token에 대한 조건부 UPDATE 하나이고 `APPLIED`/`STALE`을 돌려준다. 완료는 행을 읽어서 덮어쓰지 않는다. V2 outbox 세대는 2026-09-25에 지웠다(`docs/capability-status.md`). +- durable operation fencing: `claimNext`가 claim의 행 lock 아래에서 `lease_fence`를 올리고(`V23`) `DurableOperationClaim` + 으로 돌려준다. heartbeat·progress·succeed·fail은 operation id + `RUNNING` + owner + fence에 대한 조건부 UPDATE 하나다. + 같은 worker id가 다시 claim하면 앞 실행의 쓰기는 0행이다. - `TransactionPort` 구현(`SpringTransaction`) — 애플리케이션의 트랜잭션 의도와 Spring `PlatformTransactionManager` 사이의 다리. - 감사 metadata 기반 + actor 이음매(`audit/AuditableEntity`, `audit/AuditContextPort`, `audit/DomainContextAuditContext`) — diff --git a/src/adapter/outbound/persistence-jpa/README.md b/src/adapter/outbound/persistence-jpa/README.md index 403e60e6..9269264e 100644 --- a/src/adapter/outbound/persistence-jpa/README.md +++ b/src/adapter/outbound/persistence-jpa/README.md @@ -389,6 +389,23 @@ PUBLISHED backlog 를 한 transaction 으로 지우면 그 사이 모든 row loc cadence 측정 필요). retention 한 값은 reaper-local 이라 `@Value` 로 받지만, canonical 6-property 문서는 app-bootstrap `OutboxSettings` / `application.yml` 에 있다. +## operation — `JpaDurableOperationStore` + +### claim과 fence (`V23`) + +lease가 지난 실행은 아직 살아서 일을 끝내는 중일 수 있다. 예전 heartbeat·progress·succeed·fail은 +`lease_owner = :workerId`만 확인해서, 같은 worker id가 다시 claim하면(재시작한 worker, 한 프로세스의 두 실행) +앞 실행의 결과가 넘겨받은 실행 위에 기록됐다(round9 리뷰 2026-11-12, RR9-01-02). outbox(`lease_token`), +알림 수신자(`lease_fence`), reconciliation(`claim_fence`), fileserver cleanup(`claim_token`)과 같은 모양으로 맞췄다. + +- **claim**: `claimNext(workerId, …)`가 row lock(`FOR UPDATE SKIP LOCKED`)을 쥔 상태에서 `lease_owner`, + `lease_fence = lease_fence + 1`을 기록하고 `DurableOperationClaim(submission, workerId, fence)`을 돌려준다. +- **쓰기**: heartbeat·progress·succeed·fail은 한 문장의 조건부 UPDATE다 — + `WHERE operation_id = ? AND state = 'RUNNING' AND lease_owner = ? AND lease_fence = ?`(heartbeat는 + `lease_expires_at > now`도). 1행이면 적용, 0행이면 그 claim은 더 이상 이 operation을 쥐지 않는다. +- fence는 줄지 않는다. reclaim·cancel·expire는 owner를 비우고 fence는 그대로 둔다. `V23` 전에 쓴 행은 0이고, + claim은 1 이상을 나눠 주므로 fence 0을 쥔 worker는 없다. + ## JPA 운영 capability 후보 `src/adapter/outbound/persistence-jpa/release/readiness-cards.yaml`이 capability와 독립 schema stream의 @@ -533,7 +550,7 @@ provider 선택(flag → bean → registry)의 정본은 설정 코드 `lock/Dis 반환 핸들은 `lock::unlock` 이다. lease TTL 이 `close()` 전에 만료된 `JdbcLockRegistry` 의 경우, 내부 `JdbcLock.unlock()` 이 `ConcurrentModificationException` 을 던진다(row 가 이미 회수됨). 이 어댑터는 여기서 일부러 잡지 않는다 — metered `distributedLockProvider` 데코레이터 -(컴포지션 루트)가 SI-LOCK-C5 계약을 소유한다: 로그 + `lock.lease.expired` metric 후 `close()` 에서 +(이 모듈의 `DistributedLockPersistenceConfig`가 감싼 `MeteredDistributedLock`)가 SI-LOCK-C5 계약을 소유한다: 로그 + `lock.lease.expired` metric 후 `close()` 에서 정상 return 해, 만료가 호출자의 보호작업 예외를 가리지 않게 한다. in-process `DefaultLockRegistry` 경로는 만료가 없어 그 `close()` 가 이 예외를 던질 수 없다. @@ -554,11 +571,11 @@ provider 선택(flag → bean → registry)의 정본은 설정 코드 `lock/Dis `DistributedLock.tryLock(waitTime, leaseTtl)` 로 전달한다. ### `LockSettings` -`app.lock.*` 에서 바인딩되는 yaml-only 기본값이다(`config/module-defaults/persistence-jpa.yml`). -새 `APP_*` env 키가 아니므로 `env-keys.yaml` 엔트리가 필요 없다. `DistributedLockPersistenceConfig`의 -`@EnableConfigurationProperties`가 바인딩한다. cross-field invariant: `leaseTtl >= waitTime` -이어야 한다 — TTL 이 waitTime 보다 먼저 만료되면 첫 holder 의 보호작업이 끝나기 전에 두 번째 -holder 가 락을 잡을 수 있다. +`app.lock.lease-ttl` 하나를 바인딩하는 yaml-only 기본값이다(`config/module-defaults/persistence-jpa.yml`). +새 `APP_*` env 키가 아니므로 `env-keys.yaml` 엔트리가 필요 없다. `DistributedLockPersistenceConfig#lockSettings`가 +`StrictSettingsBinder`로 바인딩해, `app.lock` 아래 모르는 키는 기동을 실패시킨다. 기다릴 시간은 호출자가 +`tryAcquire(key, waitTime, leaseTtl)`마다 넘긴다. 예전 `app.lock.wait-time`은 lease와 비교만 하고 읽는 곳이 +없었다(round9 리뷰 2026-11-12, RR9-01-07). 지금은 모르는 키라 설정해 두면 기동이 실패한다. ### `MeteredDistributedLock` (app-bootstrap에서 이동) - **`DistributedLockPort`를 감싸는 얇은 Micrometer 데코레이터다.** `tryAcquire` 호출마다 diff --git a/src/adapter/outbound/persistence-jpa/src/main/java/dev/caskeleton/adapter/outbound/persistence/autoconfigure/JpaAdapterBeansConfig.java b/src/adapter/outbound/persistence-jpa/src/main/java/dev/caskeleton/adapter/outbound/persistence/autoconfigure/JpaAdapterBeansConfig.java index 18964a00..193187e5 100644 --- a/src/adapter/outbound/persistence-jpa/src/main/java/dev/caskeleton/adapter/outbound/persistence/autoconfigure/JpaAdapterBeansConfig.java +++ b/src/adapter/outbound/persistence-jpa/src/main/java/dev/caskeleton/adapter/outbound/persistence/autoconfigure/JpaAdapterBeansConfig.java @@ -154,8 +154,8 @@ class JpaAdapterBeansConfig { static class IdempotencyExpiry { @Bean - IdempotencyReaper idempotencyReaper(IdempotencyRecordJpaRepository repository, Clock clock) { - return new IdempotencyReaper(repository, clock); + IdempotencyReaper idempotencyReaper(IdempotencyRecordJpaRepository repository) { + return new IdempotencyReaper(repository); } } } diff --git a/src/adapter/outbound/persistence-jpa/src/main/java/dev/caskeleton/adapter/outbound/persistence/idempotency/jpa/IdempotencyReaper.java b/src/adapter/outbound/persistence-jpa/src/main/java/dev/caskeleton/adapter/outbound/persistence/idempotency/jpa/IdempotencyReaper.java index 3e754839..053baef9 100644 --- a/src/adapter/outbound/persistence-jpa/src/main/java/dev/caskeleton/adapter/outbound/persistence/idempotency/jpa/IdempotencyReaper.java +++ b/src/adapter/outbound/persistence-jpa/src/main/java/dev/caskeleton/adapter/outbound/persistence/idempotency/jpa/IdempotencyReaper.java @@ -1,32 +1,41 @@ package dev.caskeleton.adapter.outbound.persistence.idempotency.jpa; -import java.time.Clock; +import dev.caskeleton.adapter.outbound.persistence.transaction.BatchedDeletes; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import org.springframework.scheduling.annotation.Scheduled; -import org.springframework.transaction.annotation.Transactional; /** * Periodically deletes idempotency rows whose {@code expires_at} has passed. The PostgreSQL store * moves {@code expires_at} forward on every completion and failure, and a claim is reset in place * rather than re-inserted, so this is the only thing that bounds table growth. + * + *

Deletes in batches of {@link #BATCH_SIZE}, each committed on its own, until a batch comes back + * short — the outbox reaper's shape. Expiry is the database's clock, not this process's: the store + * wrote {@code expires_at} from {@code clock_timestamp()} (see {@link + * IdempotencyRecordJpaRepository#deleteExpiredBatch}). */ public class IdempotencyReaper { + /** Rows per delete statement and transaction. */ + static final int BATCH_SIZE = BatchedDeletes.BATCH_SIZE; + private static final Logger log = LoggerFactory.getLogger(IdempotencyReaper.class); private final IdempotencyRecordJpaRepository repository; - private final Clock clock; - public IdempotencyReaper(IdempotencyRecordJpaRepository repository, Clock clock) { + public IdempotencyReaper(IdempotencyRecordJpaRepository repository) { this.repository = repository; - this.clock = clock; } + /** + * Deletes every record whose expiry has passed by the database clock; returns the count. + * + *

Not transactional itself: each batch is its own transaction. + */ @Scheduled(fixedDelayString = "${app.capabilities.idempotency.reaper-interval:PT10M}") - @Transactional public int reap() { - int purged = repository.deleteExpired(clock.instant()); + int purged = BatchedDeletes.untilShort(repository::deleteExpiredBatch); if (purged > 0) { log.debug("idempotency reaper purged {} expired record(s)", purged); } diff --git a/src/adapter/outbound/persistence-jpa/src/main/java/dev/caskeleton/adapter/outbound/persistence/idempotency/jpa/IdempotencyRecordJpaRepository.java b/src/adapter/outbound/persistence-jpa/src/main/java/dev/caskeleton/adapter/outbound/persistence/idempotency/jpa/IdempotencyRecordJpaRepository.java index 22a07ca7..0a8183d7 100644 --- a/src/adapter/outbound/persistence-jpa/src/main/java/dev/caskeleton/adapter/outbound/persistence/idempotency/jpa/IdempotencyRecordJpaRepository.java +++ b/src/adapter/outbound/persistence-jpa/src/main/java/dev/caskeleton/adapter/outbound/persistence/idempotency/jpa/IdempotencyRecordJpaRepository.java @@ -1,12 +1,13 @@ package dev.caskeleton.adapter.outbound.persistence.idempotency.jpa; import dev.caskeleton.adapter.outbound.persistence.idempotency.jpa.entity.IdempotencyRecordEntity; -import java.time.Instant; +import dev.caskeleton.adapter.outbound.persistence.transaction.SpringTransaction; import java.util.UUID; import org.springframework.data.jpa.repository.Modifying; import org.springframework.data.jpa.repository.Query; import org.springframework.data.repository.Repository; import org.springframework.data.repository.query.Param; +import org.springframework.transaction.annotation.Transactional; /** * Spring Data repository for {@link IdempotencyRecordEntity}: the reaper's expiry purge, and @@ -18,8 +19,34 @@ import org.springframework.data.repository.query.Param; */ public interface IdempotencyRecordJpaRepository extends Repository { - /** Reaper / TTL boundary (§E): purge every record whose expiry is at/before {@code now}. */ + /** + * Reaper / TTL boundary (§E): deletes at most {@code limit} records whose expiry has passed, in a + * transaction of its own unless the caller already has one; returns the count. + * + *

Expiry is judged by the database clock ({@code now()}) because that is the clock that wrote + * it: the PostgreSQL store sets {@code expires_at} from {@code clock_timestamp()}. Compared with + * the application's clock instead, a host whose clock runs ahead purged a record whose lease or + * replay window was still open. + * + *

Bounded for the reason the outbox purge is: a backlog of expired rows is what a reaper + * outage leaves behind, and one unbounded {@code DELETE} over it is a long transaction holding + * every deleted row's lock. + * + *

A same-key request reclaims an expired record in place (the store's reset). The delete must + * not take that claim: the next retry would run the work again. Two things stop it. The subquery + * locks the rows it picks and skips rows another transaction holds, so a record being reclaimed + * is not picked. The expiry is also a condition of the {@code DELETE} itself, so if the lock + * clause is ever lost — and a delete waits on the row and, under READ COMMITTED, re-checks only + * its own predicate — the record it waited for is still kept once it is no longer expired. + */ + @Transactional(timeout = SpringTransaction.DEFAULT_TIMEOUT_SECONDS) @Modifying - @Query("delete from IdempotencyRecordEntity e where e.expiresAt <= :now") - int deleteExpired(@Param("now") Instant now); + @Query( + value = + "DELETE FROM idempotency_record WHERE id IN (" + + "SELECT id FROM idempotency_record WHERE expires_at <= now()" + + " LIMIT :limit FOR UPDATE SKIP LOCKED)" + + " AND expires_at <= now()", + nativeQuery = true) + int deleteExpiredBatch(@Param("limit") int limit); } diff --git a/src/adapter/outbound/persistence-jpa/src/main/java/dev/caskeleton/adapter/outbound/persistence/operation/DurableOperationJpaRepository.java b/src/adapter/outbound/persistence-jpa/src/main/java/dev/caskeleton/adapter/outbound/persistence/operation/DurableOperationJpaRepository.java index 3d17b6ab..38924d6b 100644 --- a/src/adapter/outbound/persistence-jpa/src/main/java/dev/caskeleton/adapter/outbound/persistence/operation/DurableOperationJpaRepository.java +++ b/src/adapter/outbound/persistence-jpa/src/main/java/dev/caskeleton/adapter/outbound/persistence/operation/DurableOperationJpaRepository.java @@ -17,10 +17,11 @@ import org.springframework.data.repository.query.Param; * claim-then-check has every worker fetch the same head row and all but one retry, and a plain * {@code FOR UPDATE} makes them queue behind each other so the pool serialises. * - *

Every state-changing statement carries the owning worker in its {@code WHERE} clause and - * returns the affected count. A worker whose lease lapsed while it was still working must not be - * able to record a result over the worker that took over; the update simply matches nothing, and - * the caller learns it lost the lease from the count. + *

Every state-changing statement carries the owning worker and the fence of its claim in its + * {@code WHERE} clause and returns the affected count. A run whose lease lapsed while it was still + * working must not be able to record a result over the run that took over — even when both runs use + * the same worker id, which only the fence tells apart; the update simply matches nothing, and the + * caller learns it lost the lease from the count. */ public interface DurableOperationJpaRepository extends Repository { @@ -79,6 +80,8 @@ public interface DurableOperationJpaRepository extends RepositoryRunnable is PENDING, or RUNNING whose lease has lapsed. The second half is the crash * recovery path: without it an operation claimed by a worker that then died is never picked up * again, and the queue stops draining while every row still reads RUNNING. + * + * @return one row of {@code [operation_id, lease_fence]} when a row was claimed, none otherwise */ @Modifying(clearAutomatically = true, flushAutomatically = true) @Query( @@ -90,6 +93,7 @@ public interface DurableOperationJpaRepository extends Repository claimNext( + List claimNext( @Param("workerId") String workerId, @Param("leaseExpiresAt") Instant leaseExpiresAt, @Param("now") Instant now); @@ -119,12 +123,14 @@ public interface DurableOperationJpaRepository extends Repository :now """, nativeQuery = true) int heartbeat( @Param("operationId") String operationId, @Param("workerId") String workerId, + @Param("fence") long fence, @Param("leaseExpiresAt") Instant leaseExpiresAt, @Param("now") Instant now); @@ -141,11 +147,13 @@ public interface DurableOperationJpaRepository extends Repository claimNext( + public Optional claimNext( String workerId, Duration leaseDuration, Instant now) { requireCallerTransaction("claimNext"); - List claimed = repository.claimNext(workerId, now.plus(leaseDuration), now); + List claimed = repository.claimNext(workerId, now.plus(leaseDuration), now); if (claimed.isEmpty()) { return Optional.empty(); } + String operationId = (String) claimed.get(0)[0]; + long fence = ((Number) claimed.get(0)[1]).longValue(); return repository - .findById(claimed.get(0)) + .findById(operationId) .map( entity -> - new DurableOperationSubmission( - new DurableOperationId(entity.getOperationId()), - entity.getOperationName(), - entity.getPrincipal(), - entity.getTenant().isEmpty() ? null : entity.getTenant(), - new RequestFingerprint(entity.getRequestHash()), - entity.getPayload(), - entity.getSubmittedAt(), - Duration.between(entity.getSubmittedAt(), entity.getExpiresAt()))); + new DurableOperationClaim( + new DurableOperationSubmission( + new DurableOperationId(entity.getOperationId()), + entity.getOperationName(), + entity.getPrincipal(), + entity.getTenant().isEmpty() ? null : entity.getTenant(), + new RequestFingerprint(entity.getRequestHash()), + entity.getPayload(), + entity.getSubmittedAt(), + Duration.between(entity.getSubmittedAt(), entity.getExpiresAt())), + workerId, + fence)); } @Override - public boolean heartbeat( - DurableOperationId operationId, String workerId, Duration leaseDuration, Instant now) { + public boolean heartbeat(DurableOperationClaim claim, Duration leaseDuration, Instant now) { requireCallerTransaction("heartbeat"); - return repository.heartbeat(operationId.value(), workerId, now.plus(leaseDuration), now) == 1; + return repository.heartbeat( + claim.operationId().value(), + claim.workerId(), + claim.fence(), + now.plus(leaseDuration), + now) + == 1; } @Override - public boolean reportProgress( - DurableOperationId operationId, String workerId, OperationProgressSnapshot progress) { + public boolean reportProgress(DurableOperationClaim claim, OperationProgressSnapshot progress) { requireCallerTransaction("reportProgress"); return repository.reportProgress( - operationId.value(), - workerId, + claim.operationId().value(), + claim.workerId(), + claim.fence(), progress.completedUnits(), progress.totalUnits().orElse(null), progress.phase().orElse(null)) @@ -124,25 +135,24 @@ public class JpaDurableOperationStore implements DurableOperationStorePort { } @Override - public boolean succeed( - DurableOperationId operationId, - String workerId, - String resultReference, - Instant completedAt) { + public boolean succeed(DurableOperationClaim claim, String resultReference, Instant completedAt) { requireCallerTransaction("succeed"); - return repository.succeed(operationId.value(), workerId, resultReference, completedAt) == 1; + return repository.succeed( + claim.operationId().value(), + claim.workerId(), + claim.fence(), + resultReference, + completedAt) + == 1; } @Override - public boolean fail( - DurableOperationId operationId, - String workerId, - OperationFailure failure, - Instant completedAt) { + public boolean fail(DurableOperationClaim claim, OperationFailure failure, Instant completedAt) { requireCallerTransaction("fail"); return repository.fail( - operationId.value(), - workerId, + claim.operationId().value(), + claim.workerId(), + claim.fence(), failure.code(), failure.detail(), failure.retryable(), diff --git a/src/adapter/outbound/persistence-jpa/src/main/java/dev/caskeleton/adapter/outbound/persistence/outbox/jpa/JpaOutboxStore.java b/src/adapter/outbound/persistence-jpa/src/main/java/dev/caskeleton/adapter/outbound/persistence/outbox/jpa/JpaOutboxStore.java index e23e34ca..9d8158b1 100644 --- a/src/adapter/outbound/persistence-jpa/src/main/java/dev/caskeleton/adapter/outbound/persistence/outbox/jpa/JpaOutboxStore.java +++ b/src/adapter/outbound/persistence-jpa/src/main/java/dev/caskeleton/adapter/outbound/persistence/outbox/jpa/JpaOutboxStore.java @@ -378,7 +378,7 @@ public class JpaOutboxStore OutboxEventStatus.valueOf(entity.getStatus()), entity.getAttemptCount(), entity.getDeferredAttemptCount()); - } catch (IllegalArgumentException exception) { + } catch (IllegalArgumentException | IllegalStateException exception) { throw new IllegalStateException( "invalid canonical outbox metadata for eventId=" + entity.getEventId(), exception); } diff --git a/src/adapter/outbound/persistence-jpa/src/main/java/dev/caskeleton/adapter/outbound/persistence/outbox/jpa/OutboxEventJpaRepository.java b/src/adapter/outbound/persistence-jpa/src/main/java/dev/caskeleton/adapter/outbound/persistence/outbox/jpa/OutboxEventJpaRepository.java index 9f4d6d0c..65de4485 100644 --- a/src/adapter/outbound/persistence-jpa/src/main/java/dev/caskeleton/adapter/outbound/persistence/outbox/jpa/OutboxEventJpaRepository.java +++ b/src/adapter/outbound/persistence-jpa/src/main/java/dev/caskeleton/adapter/outbound/persistence/outbox/jpa/OutboxEventJpaRepository.java @@ -1,6 +1,7 @@ package dev.caskeleton.adapter.outbound.persistence.outbox.jpa; import dev.caskeleton.adapter.outbound.persistence.outbox.jpa.entity.OutboxEventEntity; +import dev.caskeleton.adapter.outbound.persistence.transaction.SpringTransaction; import java.time.Instant; import java.util.List; import java.util.Optional; @@ -34,9 +35,10 @@ public interface OutboxEventJpaRepository extends RepositoryBounded because a backlog of published rows is exactly what a relay outage leaves behind, * and one unbounded {@code DELETE} over it is a long transaction holding every deleted row's - * lock. The key subquery is plain SQL that PostgreSQL and H2 both run. + * lock. The key subquery is plain SQL that PostgreSQL and H2 both run. The transaction has the + * default bound, as the other reapers' do. */ - @Transactional + @Transactional(timeout = SpringTransaction.DEFAULT_TIMEOUT_SECONDS) @Modifying @Query( value = diff --git a/src/adapter/outbound/persistence-jpa/src/main/java/dev/caskeleton/adapter/outbound/persistence/outbox/jpa/OutboxReaper.java b/src/adapter/outbound/persistence-jpa/src/main/java/dev/caskeleton/adapter/outbound/persistence/outbox/jpa/OutboxReaper.java index e9042775..a90f66cf 100644 --- a/src/adapter/outbound/persistence-jpa/src/main/java/dev/caskeleton/adapter/outbound/persistence/outbox/jpa/OutboxReaper.java +++ b/src/adapter/outbound/persistence-jpa/src/main/java/dev/caskeleton/adapter/outbound/persistence/outbox/jpa/OutboxReaper.java @@ -1,5 +1,6 @@ package dev.caskeleton.adapter.outbound.persistence.outbox.jpa; +import dev.caskeleton.adapter.outbound.persistence.transaction.BatchedDeletes; import java.time.Clock; import java.time.Duration; import java.time.Instant; @@ -18,7 +19,7 @@ public class OutboxReaper { /** Rows per delete statement and transaction. */ // ponytail: fixed batch size; make it a setting if a deployment's backlog needs another value. - static final int BATCH_SIZE = 1_000; + static final int BATCH_SIZE = BatchedDeletes.BATCH_SIZE; private static final Logger log = LoggerFactory.getLogger(OutboxReaper.class); @@ -41,12 +42,7 @@ public class OutboxReaper { @Scheduled(fixedDelayString = "${app.outbox.reaper-interval:PT10M}") public int reap() { Instant cutoff = clock.instant().minus(retention); - int purged = 0; - int deleted; - do { - deleted = repository.deletePublishedBatch(cutoff, BATCH_SIZE); - purged += deleted; - } while (deleted == BATCH_SIZE); + int purged = BatchedDeletes.untilShort(limit -> repository.deletePublishedBatch(cutoff, limit)); if (purged > 0) { log.debug("outbox reaper purged {} published row(s)", purged); } diff --git a/src/adapter/outbound/persistence-jpa/src/main/java/dev/caskeleton/adapter/outbound/persistence/transaction/BatchedDeletes.java b/src/adapter/outbound/persistence-jpa/src/main/java/dev/caskeleton/adapter/outbound/persistence/transaction/BatchedDeletes.java new file mode 100644 index 00000000..4889a740 --- /dev/null +++ b/src/adapter/outbound/persistence-jpa/src/main/java/dev/caskeleton/adapter/outbound/persistence/transaction/BatchedDeletes.java @@ -0,0 +1,36 @@ +package dev.caskeleton.adapter.outbound.persistence.transaction; + +import java.util.function.IntUnaryOperator; + +/** + * Deletes a backlog in bounded batches, one transaction each, until a batch comes back short. + * + *

Every reaper here has the same reason to: a backlog of expired rows is what a reaper outage + * leaves behind, and one unbounded {@code DELETE} over it is a long transaction holding every + * deleted row's lock. The outbox, idempotency and live-event reapers each held their own copy of + * this loop; one copy is the one that cannot drift. + */ +public final class BatchedDeletes { + + /** The most rows one batch deletes. */ + public static final int BATCH_SIZE = 1_000; + + private BatchedDeletes() {} + + /** + * Runs {@code batch} until it deletes fewer than {@link #BATCH_SIZE} rows. + * + * @param batch deletes at most its argument's number of rows, in a transaction of its own, and + * returns how many it deleted + * @return how many rows all the batches deleted + */ + public static int untilShort(IntUnaryOperator batch) { + int purged = 0; + int deleted; + do { + deleted = batch.applyAsInt(BATCH_SIZE); + purged += deleted; + } while (deleted == BATCH_SIZE); + return purged; + } +} diff --git a/src/adapter/outbound/persistence-jpa/src/main/java/dev/caskeleton/adapter/outbound/persistence/transaction/SpringTransaction.java b/src/adapter/outbound/persistence-jpa/src/main/java/dev/caskeleton/adapter/outbound/persistence/transaction/SpringTransaction.java index aad85fd9..10475d66 100644 --- a/src/adapter/outbound/persistence-jpa/src/main/java/dev/caskeleton/adapter/outbound/persistence/transaction/SpringTransaction.java +++ b/src/adapter/outbound/persistence-jpa/src/main/java/dev/caskeleton/adapter/outbound/persistence/transaction/SpringTransaction.java @@ -115,6 +115,15 @@ public final class SpringTransaction implements TransactionPort { return execute(requiresNewTemplate, action); } + /** + * The default bound, in whole seconds, of every transaction this leaf opens: {@link + * Timeouts#DEFAULT}, and the reapers' own transactions, whose {@code @Transactional(timeout)} + * needs a compile-time constant. A deployment's {@code app.persistence-jpa.transaction.*} moves + * the first and not the second; a reaper deletes one bounded batch per transaction, so the + * default is ample for it whatever the application's units of work need. + */ + public static final int DEFAULT_TIMEOUT_SECONDS = 30; + /** * How long one {@link SpringTransaction} write or read transaction may run (review 04-10). * @@ -140,7 +149,9 @@ public final class SpringTransaction implements TransactionPort { /** 30 seconds each; see the class comment for why. */ public static final Timeouts DEFAULT = - new Timeouts(Duration.ofSeconds(30), Duration.ofSeconds(30)); + new Timeouts( + Duration.ofSeconds(DEFAULT_TIMEOUT_SECONDS), + Duration.ofSeconds(DEFAULT_TIMEOUT_SECONDS)); public Timeouts { requireWholePositiveSeconds(write, "write"); diff --git a/src/adapter/outbound/persistence-jpa/src/main/resources/db/migration/postgresql/V23__durable_operation_lease_fence.sql b/src/adapter/outbound/persistence-jpa/src/main/resources/db/migration/postgresql/V23__durable_operation_lease_fence.sql new file mode 100644 index 00000000..1aef212e --- /dev/null +++ b/src/adapter/outbound/persistence-jpa/src/main/resources/db/migration/postgresql/V23__durable_operation_lease_fence.sql @@ -0,0 +1,24 @@ +-- A fence on the durable operation lease. +-- +-- Until now a heartbeat, a progress report and a completion checked only lease_owner. A worker that +-- restarts under the same id, or one process running two executions under one id, claims the +-- operation again, and the earlier run — still alive and still finishing — then matches the same +-- lease_owner and records its result over the run that took over. The outbox (lease_token), the +-- notification recipient lease (lease_fence), the reconciliation claim (claim_fence) and the +-- fileserver claim (claim_token) all fence; this was the one store that did not (round9 review +-- 2026-11-12, RR9-01-02). +-- +-- Every claim increments lease_fence and returns it; every later write names the fence it holds in +-- its WHERE clause. NOT NULL DEFAULT 0 is a constant default, so PostgreSQL 11+ records it in the +-- catalog and rewrites no row; rows written before this keep 0 until their next claim, and no +-- worker holds fence 0 because a claim hands out 1 or more. + +-- Bound the wait for the lock this migration takes (ACCESS EXCLUSIVE on durable_operation, held +-- until it commits). Unbounded, it queues behind any open transaction that has touched the table, +-- and every submit, poll and claim queues behind the migration. Five seconds, then the migration +-- fails, rolls back whole and the database stays at V22: deploy again once that transaction has +-- ended (docs/jpa/migration-guide.md "Bound the lock wait"). +SET LOCAL lock_timeout = '5s'; + +ALTER TABLE durable_operation + ADD COLUMN lease_fence BIGINT NOT NULL DEFAULT 0; diff --git a/src/adapter/outbound/persistence-jpa/src/postgresqlIntegrationTest/java/dev/caskeleton/adapter/outbound/persistence/idempotency/postgresql/IdempotencyReaperIntegrationTest.java b/src/adapter/outbound/persistence-jpa/src/postgresqlIntegrationTest/java/dev/caskeleton/adapter/outbound/persistence/idempotency/postgresql/IdempotencyReaperIntegrationTest.java new file mode 100644 index 00000000..ec92e02d --- /dev/null +++ b/src/adapter/outbound/persistence-jpa/src/postgresqlIntegrationTest/java/dev/caskeleton/adapter/outbound/persistence/idempotency/postgresql/IdempotencyReaperIntegrationTest.java @@ -0,0 +1,205 @@ +package dev.caskeleton.adapter.outbound.persistence.idempotency.postgresql; + +import static org.assertj.core.api.Assertions.assertThat; + +import dev.caskeleton.adapter.outbound.persistence.idempotency.jpa.IdempotencyReaper; +import dev.caskeleton.adapter.outbound.persistence.idempotency.jpa.IdempotencyRecordJpaRepository; +import dev.caskeleton.adapter.outbound.persistence.platform.JpaPlatformContractSupport; +import jakarta.persistence.EntityManagerFactory; +import java.sql.Connection; +import java.sql.Statement; +import java.time.Duration; +import java.util.List; +import java.util.Properties; +import java.util.concurrent.ExecutorService; +import java.util.concurrent.Executors; +import java.util.concurrent.Future; +import java.util.concurrent.TimeUnit; +import org.flywaydb.core.Flyway; +import org.junit.jupiter.api.AfterAll; +import org.junit.jupiter.api.BeforeAll; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Tag; +import org.junit.jupiter.api.Test; +import org.springframework.data.jpa.repository.support.JpaRepositoryFactory; +import org.springframework.jdbc.core.JdbcTemplate; +import org.springframework.orm.jpa.JpaTransactionManager; +import org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean; +import org.springframework.orm.jpa.SharedEntityManagerCreator; +import org.springframework.orm.jpa.vendor.HibernateJpaVendorAdapter; +import org.springframework.transaction.TransactionManager; +import org.springframework.transaction.annotation.AnnotationTransactionAttributeSource; +import org.springframework.transaction.interceptor.TransactionInterceptor; + +/** + * The idempotency reaper's delete against a real PostgreSQL: expiry is the database clock's, and a + * backlog larger than one batch is drained. + * + *

The store writes {@code expires_at} from {@code clock_timestamp()}, so the rows here are + * written the same way, relative to the database's {@code now()}. A reaper that compared them with + * the application's clock purged a live record whenever the host clock ran ahead of the database's + * (round9 review 2026-11-12, RR9-01-04); the reaper now has no clock of its own to compare with. + * + *

The repository is the production shape: its own {@code @Transactional} opens each batch's + * transaction, through the same interceptor Spring Data installs, and no test transaction wraps the + * reap. + */ +@Tag("jpa-contract") +class IdempotencyReaperIntegrationTest { + + private static JpaPlatformContractSupport support; + private static JdbcTemplate jdbc; + private static LocalContainerEntityManagerFactoryBean persistence; + private static IdempotencyReaper reaper; + + @BeforeAll + static void start() { + support = JpaPlatformContractSupport.start(); + Flyway.configure() + .dataSource(support.dataSource()) + .locations("classpath:db/migration/postgresql") + .table("flyway_schema_history") + .baselineOnMigrate(false) + .outOfOrder(false) + .load() + .migrate(); + jdbc = new JdbcTemplate(support.dataSource()); + persistence = new LocalContainerEntityManagerFactoryBean(); + persistence.setDataSource(support.dataSource()); + persistence.setJpaVendorAdapter(new HibernateJpaVendorAdapter()); + persistence.setPackagesToScan( + "dev.caskeleton.adapter.outbound.persistence.idempotency.jpa.entity"); + Properties properties = new Properties(); + properties.setProperty("hibernate.jdbc.time_zone", "UTC"); + persistence.setJpaProperties(properties); + persistence.afterPropertiesSet(); + EntityManagerFactory entityManagers = persistence.getObject(); + JpaTransactionManager transactionManager = new JpaTransactionManager(entityManagers); + JpaRepositoryFactory repositories = + new JpaRepositoryFactory( + SharedEntityManagerCreator.createSharedEntityManager(entityManagers)); + repositories.addRepositoryProxyPostProcessor( + (proxy, information) -> + proxy.addAdvice( + new TransactionInterceptor( + (TransactionManager) transactionManager, + new AnnotationTransactionAttributeSource()))); + reaper = + new IdempotencyReaper(repositories.getRepository(IdempotencyRecordJpaRepository.class)); + } + + @AfterAll + static void stop() { + if (persistence != null) { + persistence.destroy(); + } + if (support != null) { + support.close(); + } + } + + @BeforeEach + void empty() { + jdbc.update("delete from idempotency_record"); + } + + @Test + void aRecordTheDatabaseClockHasNotExpiredSurvivesTheReap() { + // given + insert("live", 1, "now() + interval '1 hour'"); + insert("expired", 1, "now() - interval '1 second'"); + + // when + reap(); + + // then + assertThat(jdbc.queryForList("select idempotency_key from idempotency_record", String.class)) + .containsExactly("live-0"); + } + + @Test + void backlogLargerThanOneBatchIsDrainedInOneReap() { + // given + insert("expired", 2_501, "now() - interval '1 minute'"); + + // when + int purged = reap(); + + // then + assertThat(List.of(purged, count())).containsExactly(2_501, 0); + } + + @Test + void aRecordReclaimedWhileTheReapRunsSurvivesIt() throws Exception { + // given + insert("reset", 1, "now() - interval '1 second'"); + try (Connection reclaim = support.dataSource().getConnection()) { + reclaim.setAutoCommit(false); + try (Statement statement = reclaim.createStatement()) { + // What the store's reset does to an expired record a same-key request finds: it holds the + // row and claims it again in place. + statement.executeUpdate( + "update idempotency_record set status = 'CLAIMED'," + + " expires_at = now() + interval '1 hour' where idempotency_key = 'reset-0'"); + } + try (ExecutorService executor = Executors.newVirtualThreadPerTaskExecutor()) { + Future reaping = executor.submit(() -> reap()); + awaitDoneOrBlocked(reaping, Duration.ofSeconds(5)); + reclaim.commit(); + + // when + reaping.get(30, TimeUnit.SECONDS); + } + } + + // then + assertThat(jdbc.queryForList("select status from idempotency_record", String.class)) + .as("the reap deleted the claim a same-key request had just taken") + .containsExactly("CLAIMED"); + } + + private static int reap() { + return reaper.reap(); + } + + /** + * Returns once the reap has finished or is waiting on a row lock, whichever comes first, and + * fails the test when neither happened in time: committing the reclaim before the reap had run + * would pass without the race this test exists for. + */ + private static void awaitDoneOrBlocked(Future reaping, Duration within) + throws InterruptedException { + long deadline = System.nanoTime() + within.toNanos(); + while (!reaping.isDone()) { + Integer waiting = + jdbc.queryForObject( + "select count(*) from pg_locks where not granted and locktype = 'transactionid'", + Integer.class); + if (waiting != null && waiting > 0) { + return; + } + if (System.nanoTime() >= deadline) { + throw new AssertionError( + "the reap neither finished nor waited on the reclaimed row within " + within); + } + Thread.sleep(20); + } + } + + private static int count() { + Integer rows = jdbc.queryForObject("select count(*) from idempotency_record", Integer.class); + return rows == null ? -1 : rows; + } + + private static void insert(String keyPrefix, int rows, String expiresAtSql) { + jdbc.update( + "insert into idempotency_record (id, principal, idempotency_key, use_case_name," + + " request_hash, status, created_at, expires_at)" + + " select gen_random_uuid(), 'alice', ? || '-' || n, 'orders.create'," + + " repeat('a', 64), 'COMPLETED', now(), " + + expiresAtSql + + " from generate_series(0, ? - 1) as n", + keyPrefix, + rows); + } +} diff --git a/src/adapter/outbound/persistence-jpa/src/postgresqlIntegrationTest/java/dev/caskeleton/adapter/outbound/persistence/idempotency/postgresql/PostgreSqlIdempotencyStoreContractTest.java b/src/adapter/outbound/persistence-jpa/src/postgresqlIntegrationTest/java/dev/caskeleton/adapter/outbound/persistence/idempotency/postgresql/PostgreSqlIdempotencyStoreContractTest.java index cac6ff47..8825edf8 100644 --- a/src/adapter/outbound/persistence-jpa/src/postgresqlIntegrationTest/java/dev/caskeleton/adapter/outbound/persistence/idempotency/postgresql/PostgreSqlIdempotencyStoreContractTest.java +++ b/src/adapter/outbound/persistence-jpa/src/postgresqlIntegrationTest/java/dev/caskeleton/adapter/outbound/persistence/idempotency/postgresql/PostgreSqlIdempotencyStoreContractTest.java @@ -7,7 +7,6 @@ import dev.caskeleton.adapter.outbound.persistence.transaction.SpringTransaction import dev.caskeleton.application.idempotency.IdempotencyStorePort; import dev.caskeleton.application.idempotency.testkit.IdempotencyStoreContract; import jakarta.persistence.EntityManagerFactory; -import java.time.Clock; import java.time.Duration; import java.util.Properties; import java.util.function.Supplier; @@ -85,7 +84,7 @@ class PostgreSqlIdempotencyStoreContractTest extends IdempotencyStoreContract { new JpaRepositoryFactory( SharedEntityManagerCreator.createSharedEntityManager(entityManagers)) .getRepository(IdempotencyRecordJpaRepository.class); - reaper = new IdempotencyReaper(repository, Clock.systemUTC()); + reaper = new IdempotencyReaper(repository); reaperTransactions = new TransactionTemplate(new JpaTransactionManager(entityManagers)); } diff --git a/src/adapter/outbound/persistence-jpa/src/postgresqlIntegrationTest/java/dev/caskeleton/adapter/outbound/persistence/notification/NotificationJpaStack.java b/src/adapter/outbound/persistence-jpa/src/postgresqlIntegrationTest/java/dev/caskeleton/adapter/outbound/persistence/notification/NotificationJpaStack.java index 8408e920..72b52075 100644 --- a/src/adapter/outbound/persistence-jpa/src/postgresqlIntegrationTest/java/dev/caskeleton/adapter/outbound/persistence/notification/NotificationJpaStack.java +++ b/src/adapter/outbound/persistence-jpa/src/postgresqlIntegrationTest/java/dev/caskeleton/adapter/outbound/persistence/notification/NotificationJpaStack.java @@ -3,7 +3,10 @@ package dev.caskeleton.adapter.outbound.persistence.notification; import dev.caskeleton.adapter.outbound.persistence.platform.JpaPlatformContractSupport; import jakarta.persistence.EntityManager; import jakarta.persistence.EntityManagerFactory; +import java.time.Duration; +import java.time.Instant; import java.util.Map; +import java.util.concurrent.Future; import java.util.function.Supplier; import org.flywaydb.core.Flyway; import org.springframework.data.jpa.repository.support.JpaRepositoryFactory; @@ -69,6 +72,33 @@ public final class NotificationJpaStack implements AutoCloseable { return new JdbcTemplate(database.dataSource()); } + /** + * Waits until {@code work} has finished or some session of this database is blocked on a lock. + * + *

A race case holds one transaction open and starts a second statement that may or may not + * need the first's row lock; this is how it knows the second has either finished or queued behind + * the first before it lets the first commit, without sleeping a guessed interval. + */ + public void awaitDoneOrBlocked(Future work, Duration deadline) { + Instant giveUp = Instant.now().plus(deadline); + JdbcTemplate jdbc = jdbc(); + while (Instant.now().isBefore(giveUp)) { + if (work.isDone()) { + return; + } + Integer waiting = + jdbc.queryForObject( + "SELECT count(*) FROM pg_stat_activity" + + " WHERE datname = current_database() AND wait_event_type = 'Lock'", + Integer.class); + if (waiting != null && waiting > 0) { + return; + } + Thread.onSpinWait(); + } + throw new IllegalStateException("work neither finished nor waited on a lock in " + deadline); + } + /** * Deletes every row of the delivery tables, children first, so a contract case starts empty. * diff --git a/src/adapter/outbound/persistence-jpa/src/postgresqlIntegrationTest/java/dev/caskeleton/adapter/outbound/persistence/notification/PostgreSqlRecipientLeaseStoreContractTest.java b/src/adapter/outbound/persistence-jpa/src/postgresqlIntegrationTest/java/dev/caskeleton/adapter/outbound/persistence/notification/PostgreSqlRecipientLeaseStoreContractTest.java index fe4134f7..a4bec546 100644 --- a/src/adapter/outbound/persistence-jpa/src/postgresqlIntegrationTest/java/dev/caskeleton/adapter/outbound/persistence/notification/PostgreSqlRecipientLeaseStoreContractTest.java +++ b/src/adapter/outbound/persistence-jpa/src/postgresqlIntegrationTest/java/dev/caskeleton/adapter/outbound/persistence/notification/PostgreSqlRecipientLeaseStoreContractTest.java @@ -15,11 +15,17 @@ import dev.caskeleton.application.notification.platform.dispatch.NotificationReq import dev.caskeleton.application.notification.platform.dispatch.RecipientDeliveryRecord; import dev.caskeleton.application.notification.platform.dispatch.RecipientDeliveryStorePort; import dev.caskeleton.application.notification.platform.dispatch.RecipientLeaseStorePort; +import dev.caskeleton.application.notification.platform.dispatch.testkit.ContractClock; import dev.caskeleton.application.notification.platform.dispatch.testkit.NotificationRows; import dev.caskeleton.application.notification.platform.dispatch.testkit.RecipientLeaseStoreContract; import java.time.Duration; import java.util.List; import java.util.Optional; +import java.util.concurrent.CompletableFuture; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.ExecutorService; +import java.util.concurrent.Executors; +import java.util.concurrent.TimeUnit; import java.util.function.Supplier; import org.junit.jupiter.api.AfterAll; import org.junit.jupiter.api.BeforeAll; @@ -114,6 +120,72 @@ class PostgreSqlRecipientLeaseStoreContractTest extends RecipientLeaseStoreContr .isGreaterThan(before); } + @Test + void anExpirySweepRacingAClaimLeavesTheClaimedJobDispatching() throws Exception { + // given + NotificationRequestRecord request = + NotificationRows.request(new TenantId("tenant-a"), "idem-race", NOW); + RecipientDeliveryRecord job = + NotificationRows.job( + request, + RecipientDeliveryState.READY_TO_DISPATCH, + Optional.of(NOW), + Optional.of(NOW.plusSeconds(10)), + NOW); + inTransaction(() -> requests().insert(request, List.of(job))); + CountDownLatch claimed = new CountDownLatch(1); + CountDownLatch release = new CountDownLatch(1); + ExecutorService pool = Executors.newFixedThreadPool(2); + + // when + List outcome; + try { + // The claim holds its row lock, uncommitted, while the sweep runs at an instant past the + // job's expiry. The sweep used to pick the id from the committed READY row, wait for the + // lock, and then retire the row the claim had just made DISPATCHING. + CompletableFuture claim = + CompletableFuture.runAsync( + () -> + inTransaction( + () -> { + var unused = leases().claim("worker-a", 10, Duration.ofMinutes(5)); + claimed.countDown(); + awaitOrFail(release); + return null; + }), + pool); + awaitOrFail(claimed); + ((ContractClock) clock()).moveTo(NOW.plusSeconds(11)); + CompletableFuture sweep = + CompletableFuture.supplyAsync( + () -> inTransaction(() -> leases().expireOverdue(10)), pool); + stack.awaitDoneOrBlocked(sweep, Duration.ofSeconds(30)); + release.countDown(); + claim.get(30, TimeUnit.SECONDS); + outcome = List.of(sweep.get(30, TimeUnit.SECONDS), stateOf(job.id())); + } finally { + pool.shutdownNow(); + } + + // then + assertThat(outcome).containsExactly(0, RecipientDeliveryState.DISPATCHING); + } + + private RecipientDeliveryState stateOf(RecipientDeliveryId id) { + return deliveries().find(id).orElseThrow().state(); + } + + private static void awaitOrFail(CountDownLatch latch) { + try { + if (!latch.await(30, TimeUnit.SECONDS)) { + throw new IllegalStateException("the other transaction did not arrive within 30 seconds"); + } + } catch (InterruptedException interrupted) { + Thread.currentThread().interrupt(); + throw new IllegalStateException("interrupted while waiting", interrupted); + } + } + private static long versionOf(RecipientDeliveryId id) { return stack .jdbc() diff --git a/src/adapter/outbound/persistence-jpa/src/postgresqlIntegrationTest/java/dev/caskeleton/adapter/outbound/persistence/notification/platform/repository/InboxBulkReadVersionContractTest.java b/src/adapter/outbound/persistence-jpa/src/postgresqlIntegrationTest/java/dev/caskeleton/adapter/outbound/persistence/notification/platform/repository/InboxBulkReadVersionContractTest.java index 76863346..4851dadc 100644 --- a/src/adapter/outbound/persistence-jpa/src/postgresqlIntegrationTest/java/dev/caskeleton/adapter/outbound/persistence/notification/platform/repository/InboxBulkReadVersionContractTest.java +++ b/src/adapter/outbound/persistence-jpa/src/postgresqlIntegrationTest/java/dev/caskeleton/adapter/outbound/persistence/notification/platform/repository/InboxBulkReadVersionContractTest.java @@ -5,9 +5,17 @@ import static org.assertj.core.api.Assertions.catchThrowable; import dev.caskeleton.adapter.outbound.persistence.notification.NotificationJpaStack; import dev.caskeleton.adapter.outbound.persistence.notification.platform.entity.InboxItemEntity; +import java.sql.Connection; +import java.sql.PreparedStatement; import java.sql.Timestamp; +import java.time.Duration; import java.time.Instant; +import java.util.List; import java.util.UUID; +import java.util.concurrent.CompletableFuture; +import java.util.concurrent.ExecutorService; +import java.util.concurrent.Executors; +import java.util.concurrent.TimeUnit; import org.junit.jupiter.api.AfterAll; import org.junit.jupiter.api.BeforeAll; import org.junit.jupiter.api.DisplayName; @@ -74,6 +82,49 @@ class InboxBulkReadVersionContractTest { .isEqualTo("READ"); } + @Test + void anItemArchivedWhileTheBulkReadWaitsForItStaysArchived() throws Exception { + // given + UUID id = insertUnreadItem(); + ExecutorService pool = Executors.newSingleThreadExecutor(); + + // when + List outcome; + try (Connection archiving = stack.jdbc().getDataSource().getConnection()) { + // The archive holds the row uncommitted while mark-all-read runs. The bulk read picks the id + // from the committed unread row, waits for the lock, and used to mark READ the item the + // user had just archived — putting it back in the inbox. + archiving.setAutoCommit(false); + try (PreparedStatement archive = + archiving.prepareStatement( + "UPDATE notification_inbox_item SET state = 'ARCHIVED', version = version + 1" + + " WHERE id = ?")) { + archive.setObject(1, id); + archive.executeUpdate(); + } + CompletableFuture bulkRead = + CompletableFuture.supplyAsync( + () -> + stack.inTransaction( + () -> items.markAllRead("tenant-a", "user-1", null, null, 100, NOW)), + pool); + stack.awaitDoneOrBlocked(bulkRead, Duration.ofSeconds(30)); + archiving.commit(); + outcome = List.of(bulkRead.get(30, TimeUnit.SECONDS), stateOf(id)); + } finally { + pool.shutdownNow(); + } + + // then + assertThat(outcome).containsExactly(0, "ARCHIVED"); + } + + private static String stateOf(UUID id) { + return stack + .jdbc() + .queryForObject("SELECT state FROM notification_inbox_item WHERE id = ?", String.class, id); + } + private static UUID insertUnreadItem() { UUID id = UUID.randomUUID(); stack diff --git a/src/adapter/outbound/persistence-jpa/src/postgresqlIntegrationTest/java/dev/caskeleton/adapter/outbound/persistence/operation/DurableOperationSchemaConstraintTest.java b/src/adapter/outbound/persistence-jpa/src/postgresqlIntegrationTest/java/dev/caskeleton/adapter/outbound/persistence/operation/DurableOperationSchemaConstraintTest.java new file mode 100644 index 00000000..501e79c7 --- /dev/null +++ b/src/adapter/outbound/persistence-jpa/src/postgresqlIntegrationTest/java/dev/caskeleton/adapter/outbound/persistence/operation/DurableOperationSchemaConstraintTest.java @@ -0,0 +1,214 @@ +package dev.caskeleton.adapter.outbound.persistence.operation; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.catchThrowable; + +import dev.caskeleton.adapter.outbound.persistence.platform.JpaPlatformContractSupport; +import java.io.IOException; +import java.io.InputStream; +import java.nio.charset.StandardCharsets; +import java.sql.Connection; +import java.sql.PreparedStatement; +import java.sql.SQLException; +import java.sql.Statement; +import java.sql.Timestamp; +import java.time.Duration; +import java.time.Instant; +import org.junit.jupiter.api.AfterAll; +import org.junit.jupiter.api.BeforeAll; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Tag; +import org.junit.jupiter.api.Test; + +/** + * The {@code durable_operation} table's own constraints, against a real PostgreSQL. + * + *

What the store does is {@link JpaDurableOperationStoreContractTest}: the shipped adapter + * through the shared {@code DurableOperationStoreContract}. This class holds only what that + * contract cannot reach — the constraints a second writer meets when it bypasses the adapter. A + * {@code CHECK} constraint a fake does not enforce is a comment, so each one is shown refusing a + * row here, by writing the row directly. + * + *

It replaces a class that re-typed the adapter's SQL into the test and ran that copy (round9 + * review 2026-11-12, handed over from RR9 area 01): the copy passed whatever the shipped statements + * did, and seven of its cases repeated the contract. + */ +@Tag("jpa-contract") +class DurableOperationSchemaConstraintTest { + + private static final Instant NOW = Instant.parse("2026-08-25T09:00:00Z"); + + private static JpaPlatformContractSupport support; + + @BeforeAll + static void startServer() { + support = JpaPlatformContractSupport.start(); + } + + @AfterAll + static void stopServer() { + if (support != null) { + support.close(); + } + } + + @BeforeEach + void migrate() throws SQLException, IOException { + try (Connection connection = support.connection(); + Statement statement = connection.createStatement()) { + statement.execute("DROP SCHEMA public CASCADE"); + statement.execute("CREATE SCHEMA public"); + statement.execute(migration("V11__durable_operation.sql")); + statement.execute(migration("V23__durable_operation_lease_fence.sql")); + } + } + + @Test + void identicalScopeIsRefusedByTheUniqueConstraint() throws SQLException { + // given + insertPending("op-1", "hash-1"); + + // when + // The constraint is the arbiter the adapter's ON CONFLICT names, not a prior read: two + // identical submissions can arrive at the same instant, and a read-then-insert lets both in. + Throwable refused = catchThrowable(() -> insertPending("op-2", "hash-1")); + + // then + assertThat(refused) + .isInstanceOf(SQLException.class) + .hasMessageContaining("uq_durable_operation_scope"); + } + + @Test + void succeededRowWithoutAResultIsRefused() throws SQLException { + // given + insertPending("op-1", "hash-1"); + + // when + Throwable refused = + catchThrowable( + () -> + execute( + "UPDATE durable_operation SET state = 'SUCCEEDED', completed_at = now()" + + " WHERE operation_id = 'op-1'")); + + // then + assertThat(refused) + .isInstanceOf(SQLException.class) + .hasMessageContaining("ck_durable_operation_succeeded_has_result"); + } + + @Test + void failedRowWithoutAReasonIsRefused() throws SQLException { + // given + insertPending("op-1", "hash-1"); + + // when + Throwable refused = + catchThrowable( + () -> + execute( + "UPDATE durable_operation SET state = 'FAILED', completed_at = now()" + + " WHERE operation_id = 'op-1'")); + + // then + assertThat(refused) + .isInstanceOf(SQLException.class) + .hasMessageContaining("ck_durable_operation_failed_has_reason"); + } + + @Test + void runningRowWithoutALeaseIsRefused() throws SQLException { + // given + insertPending("op-1", "hash-1"); + + // when + Throwable refused = + catchThrowable( + () -> + execute( + "UPDATE durable_operation SET state = 'RUNNING' WHERE operation_id = 'op-1'")); + + // then + assertThat(refused) + .isInstanceOf(SQLException.class) + .hasMessageContaining("ck_durable_operation_running_has_lease"); + } + + @Test + void stateOutsideThePublishedVocabularyIsRefused() throws SQLException { + // given + insertPending("op-1", "hash-1"); + + // when + // A worker that could write its own state would extend the published contract by writing a + // row, and a client branching on status would meet a value that is in no version of the API. + Throwable refused = + catchThrowable( + () -> + execute( + "UPDATE durable_operation SET state = 'ALMOST_DONE'" + + " WHERE operation_id = 'op-1'")); + + // then + assertThat(refused) + .isInstanceOf(SQLException.class) + .hasMessageContaining("ck_durable_operation_state"); + } + + @Test + void expiryNotAfterSubmissionIsRefused() throws SQLException { + // given + insertPending("op-1", "hash-1"); + + // when + Throwable refused = + catchThrowable( + () -> + execute( + "UPDATE durable_operation SET expires_at = submitted_at" + + " WHERE operation_id = 'op-1'")); + + // then + assertThat(refused) + .isInstanceOf(SQLException.class) + .hasMessageContaining("ck_durable_operation_expiry_after_submission"); + } + + private void insertPending(String operationId, String hash) throws SQLException { + try (Connection connection = support.connection(); + PreparedStatement statement = + connection.prepareStatement( + """ + INSERT INTO durable_operation + (operation_id, tenant, principal, operation_name, request_hash, payload, + state, submitted_at, expires_at) + VALUES (?, '', 'alice', 'transfers.create', ?, '{}', 'PENDING', ?, ?) + """)) { + statement.setString(1, operationId); + statement.setString(2, hash); + statement.setTimestamp(3, Timestamp.from(NOW)); + statement.setTimestamp(4, Timestamp.from(NOW.plus(Duration.ofDays(1)))); + statement.executeUpdate(); + } + } + + private void execute(String sql) throws SQLException { + try (Connection connection = support.connection(); + Statement statement = connection.createStatement()) { + statement.execute(sql); + } + } + + private static String migration(String file) throws IOException { + try (InputStream resource = + DurableOperationSchemaConstraintTest.class + .getClassLoader() + .getResourceAsStream("db/migration/postgresql/" + file)) { + if (resource == null) { + throw new IOException("the durable operation migration " + file + " is missing"); + } + return new String(resource.readAllBytes(), StandardCharsets.UTF_8); + } + } +} diff --git a/src/adapter/outbound/persistence-jpa/src/postgresqlIntegrationTest/java/dev/caskeleton/adapter/outbound/persistence/operation/DurableOperationStoreContractTest.java b/src/adapter/outbound/persistence-jpa/src/postgresqlIntegrationTest/java/dev/caskeleton/adapter/outbound/persistence/operation/DurableOperationStoreContractTest.java deleted file mode 100644 index 0e3b7a3d..00000000 --- a/src/adapter/outbound/persistence-jpa/src/postgresqlIntegrationTest/java/dev/caskeleton/adapter/outbound/persistence/operation/DurableOperationStoreContractTest.java +++ /dev/null @@ -1,474 +0,0 @@ -package dev.caskeleton.adapter.outbound.persistence.operation; - -import static org.assertj.core.api.Assertions.assertThat; -import static org.assertj.core.api.Assertions.assertThatThrownBy; - -import dev.caskeleton.adapter.outbound.persistence.platform.JpaPlatformContractSupport; -import java.io.IOException; -import java.io.InputStream; -import java.nio.charset.StandardCharsets; -import java.sql.Connection; -import java.sql.PreparedStatement; -import java.sql.ResultSet; -import java.sql.SQLException; -import java.sql.Statement; -import java.sql.Timestamp; -import java.time.Duration; -import java.time.Instant; -import java.util.ArrayList; -import java.util.List; -import java.util.Optional; -import org.junit.jupiter.api.AfterAll; -import org.junit.jupiter.api.BeforeAll; -import org.junit.jupiter.api.BeforeEach; -import org.junit.jupiter.api.DisplayName; -import org.junit.jupiter.api.Tag; -import org.junit.jupiter.api.Test; - -/** - * The durable operation store against a real PostgreSQL. - * - *

Every claim in this store is about concurrency or about a constraint, and neither survives - * being tested against a fake. {@code FOR UPDATE SKIP LOCKED} does not exist in H2's PostgreSQL - * mode with the semantics that matter, a {@code CHECK} constraint a fake does not enforce is a - * comment, and a lease race has no meaning where only one connection exists. - * - *

A dead worker is written the only way that is faithful here: the row a crash leaves behind — a - * lease nobody will renew — and then recovery runs against it. That is precisely what a killed - * process is from the database's side. - */ -@Tag("jpa-contract") -class DurableOperationStoreContractTest { - - private static final Instant NOW = Instant.parse("2026-08-25T09:00:00Z"); - private static final Duration LEASE = Duration.ofMinutes(2); - - private static JpaPlatformContractSupport support; - - @BeforeAll - static void startServer() { - support = JpaPlatformContractSupport.start(); - } - - @AfterAll - static void stopServer() { - if (support != null) { - support.close(); - } - } - - @BeforeEach - void migrate() throws SQLException, IOException { - try (Connection connection = support.connection(); - Statement statement = connection.createStatement()) { - statement.execute("DROP SCHEMA public CASCADE"); - statement.execute("CREATE SCHEMA public"); - statement.execute(migration()); - } - } - - @Test - @DisplayName("a lease left behind by a dead worker is reclaimed by another") - void expiredLeaseCanBeReclaimedAfterWorkerCrash() throws SQLException { - submit("op-1", "hash-1"); - claim("worker-a", NOW); - - // The worker is gone. Nothing renews the lease, and nothing else may touch the operation until - // it lapses — that exclusivity is the point of the lease. - assertThat(claim("worker-b", NOW.plusSeconds(30))).isEmpty(); - - assertThat(claim("worker-b", NOW.plus(LEASE).plusSeconds(1))).contains("op-1"); - assertThat(state("op-1")).isEqualTo("RUNNING"); - assertThat(leaseOwner("op-1")).isEqualTo("worker-b"); - } - - @Test - @DisplayName("a reclaimed operation keeps the moment it first started") - void reclaimKeepsTheOriginalStartTime() throws SQLException { - submit("op-1", "hash-1"); - claim("worker-a", NOW); - - claim("worker-b", NOW.plus(LEASE).plusSeconds(1)); - - // COALESCE, not an unconditional assignment. Overwriting it would make every retry look like a - // fresh start and hide how long the operation has really been running. - assertThat(startedAt("op-1")).isEqualTo(NOW); - } - - @Test - @DisplayName("two workers claiming at once take different operations") - void concurrentWorkersDoNotTakeTheSameOperation() throws Exception { - submit("op-1", "hash-1"); - submit("op-2", "hash-2"); - - List claimed = new ArrayList<>(); - try (Connection first = support.connection(); - Connection second = support.connection()) { - first.setAutoCommit(false); - second.setAutoCommit(false); - // Interleaved on purpose and inside open transactions: with plain FOR UPDATE the second - // claim blocks here until the first commits, and with no locking at all both take op-1. - claimed.add(claimOn(first, "worker-a", NOW).orElseThrow()); - claimed.add(claimOn(second, "worker-b", NOW).orElseThrow()); - first.commit(); - second.commit(); - } - - assertThat(claimed).containsExactlyInAnyOrder("op-1", "op-2"); - } - - @Test - @DisplayName("a worker whose lease lapsed cannot record a result over its successor") - void aStaleWorkerCannotOverwriteTheNewOwner() throws SQLException { - submit("op-1", "hash-1"); - claim("worker-a", NOW); - claim("worker-b", NOW.plus(LEASE).plusSeconds(1)); - - // worker-a is alive and still finishing. Letting it report here is how one operation ends up - // with two answers, and which one survives would come down to timing. - assertThat(succeed("op-1", "worker-a", "/results/a")).isZero(); - assertThat(succeed("op-1", "worker-b", "/results/b")).isOne(); - assertThat(resultReference("op-1")).isEqualTo("/results/b"); - } - - @Test - @DisplayName("a heartbeat from a worker that lost the lease fails rather than extending it") - void heartbeatFromAStaleWorkerFails() throws SQLException { - submit("op-1", "hash-1"); - claim("worker-a", NOW); - claim("worker-b", NOW.plus(LEASE).plusSeconds(1)); - - assertThat(heartbeat("op-1", "worker-a", NOW.plus(LEASE).plusSeconds(2))).isZero(); - assertThat(heartbeat("op-1", "worker-b", NOW.plus(LEASE).plusSeconds(2))).isOne(); - } - - @Test - @DisplayName("cancel works once and never reverses a finished operation") - void cancelIsIdempotentAndNeverReversesTerminal() throws SQLException { - submit("op-1", "hash-1"); - claim("worker-a", NOW); - succeed("op-1", "worker-a", "/results/a"); - - // The work happened. Recording it as CANCELED would replace the only record of a real outcome - // with a claim that it never occurred. - assertThat(cancel("op-1", "alice")).isZero(); - assertThat(state("op-1")).isEqualTo("SUCCEEDED"); - - submit("op-2", "hash-2"); - assertThat(cancel("op-2", "alice")).isOne(); - assertThat(cancel("op-2", "alice")).isZero(); - assertThat(state("op-2")).isEqualTo("CANCELED"); - } - - @Test - @DisplayName("another principal cannot cancel an operation they did not submit") - void cancelIsScopedToThePrincipal() throws SQLException { - submit("op-1", "hash-1"); - - assertThat(cancel("op-1", "mallory")).isZero(); - assertThat(state("op-1")).isEqualTo("PENDING"); - } - - @Test - @DisplayName("a cancelled operation is not handed to a worker") - void cancelledOperationsAreNotClaimable() throws SQLException { - submit("op-1", "hash-1"); - cancel("op-1", "alice"); - - assertThat(claim("worker-a", NOW)).isEmpty(); - } - - @Test - @DisplayName("an identical resubmission is refused by the unique constraint") - void resubmissionCollidesOnScope() throws SQLException { - submit("op-1", "hash-1"); - - // The constraint is the arbiter rather than a prior read: two identical submissions can arrive - // at the same instant, and a read-then-insert lets both through. - assertThatThrownBy(() -> submit("op-2", "hash-1")).isInstanceOf(SQLException.class); - } - - @Test - @DisplayName("a succeeded row with no result is refused by the database") - void succeededRowRequiresAResult() throws SQLException { - submit("op-1", "hash-1"); - - assertThatThrownBy( - () -> - execute( - "UPDATE durable_operation SET state = 'SUCCEEDED', completed_at = now()," - + " lease_owner = NULL WHERE operation_id = 'op-1'")) - .isInstanceOf(SQLException.class) - .hasMessageContaining("ck_durable_operation_succeeded_has_result"); - } - - @Test - @DisplayName("a running row with no lease is refused by the database") - void runningRowRequiresALease() { - assertThatThrownBy( - () -> { - submit("op-1", "hash-1"); - execute("UPDATE durable_operation SET state = 'RUNNING' WHERE operation_id = 'op-1'"); - }) - .isInstanceOf(SQLException.class) - .hasMessageContaining("ck_durable_operation_running_has_lease"); - } - - @Test - @DisplayName("a state outside the published vocabulary is refused") - void stateVocabularyIsClosed() { - // A worker that could write its own state would extend the published contract by writing a - // row, and a client branching on status would meet a value that is in no version of the API. - assertThatThrownBy( - () -> { - submit("op-1", "hash-1"); - execute( - "UPDATE durable_operation SET state = 'ALMOST_DONE' WHERE operation_id = 'op-1'"); - }) - .isInstanceOf(SQLException.class) - .hasMessageContaining("ck_durable_operation_state"); - } - - @Test - @DisplayName("an aged-out operation is expired, and a finished one is left alone") - void expirySweepLeavesRealOutcomesAlone() throws SQLException { - submit("op-1", "hash-1"); - submit("op-2", "hash-2"); - claim("worker-a", NOW); - - List expired = expireStale(NOW.plus(Duration.ofDays(2))); - - assertThat(expired).containsExactlyInAnyOrder("op-1", "op-2"); - submit("op-3", "hash-3"); - claim("worker-b", NOW); - succeed("op-3", "worker-b", "/results/c"); - assertThat(expireStale(NOW.plus(Duration.ofDays(2)))).isEmpty(); - assertThat(state("op-3")).isEqualTo("SUCCEEDED"); - } - - @Test - @DisplayName("reclaim returns lapsed leases to the queue") - void reclaimReturnsLapsedLeases() throws SQLException { - submit("op-1", "hash-1"); - claim("worker-a", NOW); - - assertThat(reclaim(NOW.plusSeconds(30))).isEmpty(); - assertThat(reclaim(NOW.plus(LEASE).plusSeconds(1))).containsExactly("op-1"); - assertThat(state("op-1")).isEqualTo("PENDING"); - assertThat(leaseOwner("op-1")).isNull(); - } - - private void submit(String operationId, String hash) throws SQLException { - try (Connection connection = support.connection(); - PreparedStatement statement = - connection.prepareStatement( - """ - INSERT INTO durable_operation - (operation_id, tenant, principal, operation_name, request_hash, payload, - state, submitted_at, expires_at) - VALUES (?, '', 'alice', 'transfers.create', ?, '{}', 'PENDING', ?, ?) - """)) { - statement.setString(1, operationId); - statement.setString(2, hash); - statement.setTimestamp(3, Timestamp.from(NOW)); - statement.setTimestamp(4, Timestamp.from(NOW.plus(Duration.ofDays(1)))); - statement.executeUpdate(); - } - } - - private Optional claim(String workerId, Instant now) throws SQLException { - try (Connection connection = support.connection()) { - return claimOn(connection, workerId, now); - } - } - - private Optional claimOn(Connection connection, String workerId, Instant now) - throws SQLException { - try (PreparedStatement statement = - connection.prepareStatement( - """ - UPDATE durable_operation - SET state = 'RUNNING', - lease_owner = ?, - lease_acquired_at = ?, - lease_expires_at = ?, - started_at = COALESCE(started_at, ?), - row_version = row_version + 1 - WHERE operation_id = ( - SELECT operation_id - FROM durable_operation - WHERE (state = 'PENDING' - OR (state = 'RUNNING' AND lease_expires_at <= ?)) - AND expires_at > ? - ORDER BY submitted_at - FOR UPDATE SKIP LOCKED - LIMIT 1) - RETURNING operation_id - """)) { - statement.setString(1, workerId); - statement.setTimestamp(2, Timestamp.from(now)); - statement.setTimestamp(3, Timestamp.from(now.plus(LEASE))); - statement.setTimestamp(4, Timestamp.from(now)); - statement.setTimestamp(5, Timestamp.from(now)); - statement.setTimestamp(6, Timestamp.from(now)); - try (ResultSet rows = statement.executeQuery()) { - return rows.next() ? Optional.of(rows.getString(1)) : Optional.empty(); - } - } - } - - private int succeed(String operationId, String workerId, String resultReference) - throws SQLException { - try (Connection connection = support.connection(); - PreparedStatement statement = - connection.prepareStatement( - """ - UPDATE durable_operation - SET state = 'SUCCEEDED', result_reference = ?, completed_at = ?, - lease_owner = NULL, lease_acquired_at = NULL, lease_expires_at = NULL, - row_version = row_version + 1 - WHERE operation_id = ? AND state = 'RUNNING' AND lease_owner = ? - """)) { - statement.setString(1, resultReference); - statement.setTimestamp(2, Timestamp.from(NOW.plusSeconds(10))); - statement.setString(3, operationId); - statement.setString(4, workerId); - return statement.executeUpdate(); - } - } - - private int heartbeat(String operationId, String workerId, Instant now) throws SQLException { - try (Connection connection = support.connection(); - PreparedStatement statement = - connection.prepareStatement( - """ - UPDATE durable_operation - SET lease_expires_at = ?, row_version = row_version + 1 - WHERE operation_id = ? AND state = 'RUNNING' AND lease_owner = ? - AND lease_expires_at > ? - """)) { - statement.setTimestamp(1, Timestamp.from(now.plus(LEASE))); - statement.setString(2, operationId); - statement.setString(3, workerId); - statement.setTimestamp(4, Timestamp.from(now)); - return statement.executeUpdate(); - } - } - - private int cancel(String operationId, String principal) throws SQLException { - try (Connection connection = support.connection(); - PreparedStatement statement = - connection.prepareStatement( - """ - UPDATE durable_operation - SET state = 'CANCELED', completed_at = ?, lease_owner = NULL, - lease_acquired_at = NULL, lease_expires_at = NULL, - row_version = row_version + 1 - WHERE operation_id = ? AND principal = ? AND state IN ('PENDING', 'RUNNING') - """)) { - statement.setTimestamp(1, Timestamp.from(NOW.plusSeconds(5))); - statement.setString(2, operationId); - statement.setString(3, principal); - return statement.executeUpdate(); - } - } - - private List reclaim(Instant now) throws SQLException { - return idsFrom( - """ - UPDATE durable_operation - SET state = 'PENDING', lease_owner = NULL, lease_acquired_at = NULL, - lease_expires_at = NULL, row_version = row_version + 1 - WHERE state = 'RUNNING' AND lease_expires_at <= ? - RETURNING operation_id - """, - now); - } - - private List expireStale(Instant now) throws SQLException { - return idsFrom( - """ - UPDATE durable_operation - SET state = 'EXPIRED', completed_at = ?, lease_owner = NULL, - lease_acquired_at = NULL, lease_expires_at = NULL, - row_version = row_version + 1 - WHERE state IN ('PENDING', 'RUNNING') AND expires_at <= ? - RETURNING operation_id - """, - now, - now); - } - - private List idsFrom(String sql, Instant... arguments) throws SQLException { - try (Connection connection = support.connection(); - PreparedStatement statement = connection.prepareStatement(sql)) { - for (int index = 0; index < arguments.length; index++) { - statement.setTimestamp(index + 1, Timestamp.from(arguments[index])); - } - List ids = new ArrayList<>(); - try (ResultSet rows = statement.executeQuery()) { - while (rows.next()) { - ids.add(rows.getString(1)); - } - } - return ids; - } - } - - private String state(String operationId) throws SQLException { - return column(operationId, "state"); - } - - private String leaseOwner(String operationId) throws SQLException { - return column(operationId, "lease_owner"); - } - - private String resultReference(String operationId) throws SQLException { - return column(operationId, "result_reference"); - } - - private Instant startedAt(String operationId) throws SQLException { - try (Connection connection = support.connection(); - PreparedStatement statement = - connection.prepareStatement( - "SELECT started_at FROM durable_operation WHERE operation_id = ?")) { - statement.setString(1, operationId); - try (ResultSet rows = statement.executeQuery()) { - rows.next(); - return rows.getTimestamp(1).toInstant(); - } - } - } - - private String column(String operationId, String name) throws SQLException { - try (Connection connection = support.connection(); - PreparedStatement statement = - connection.prepareStatement( - "SELECT " + name + " FROM durable_operation WHERE operation_id = ?")) { - statement.setString(1, operationId); - try (ResultSet rows = statement.executeQuery()) { - rows.next(); - return rows.getString(1); - } - } - } - - private void execute(String sql) throws SQLException { - try (Connection connection = support.connection(); - Statement statement = connection.createStatement()) { - statement.execute(sql); - } - } - - private static String migration() throws IOException { - try (InputStream resource = - DurableOperationStoreContractTest.class - .getClassLoader() - .getResourceAsStream("db/migration/postgresql/V11__durable_operation.sql")) { - if (resource == null) { - throw new IOException("the durable operation migration is missing from the classpath"); - } - return new String(resource.readAllBytes(), StandardCharsets.UTF_8); - } - } -} diff --git a/src/adapter/outbound/persistence-jpa/src/postgresqlIntegrationTest/java/dev/caskeleton/adapter/outbound/persistence/operation/JpaDurableOperationStoreContractTest.java b/src/adapter/outbound/persistence-jpa/src/postgresqlIntegrationTest/java/dev/caskeleton/adapter/outbound/persistence/operation/JpaDurableOperationStoreContractTest.java index 9cd7d16a..22fa4f0f 100644 --- a/src/adapter/outbound/persistence-jpa/src/postgresqlIntegrationTest/java/dev/caskeleton/adapter/outbound/persistence/operation/JpaDurableOperationStoreContractTest.java +++ b/src/adapter/outbound/persistence-jpa/src/postgresqlIntegrationTest/java/dev/caskeleton/adapter/outbound/persistence/operation/JpaDurableOperationStoreContractTest.java @@ -1,9 +1,15 @@ package dev.caskeleton.adapter.outbound.persistence.operation; +import static org.assertj.core.api.Assertions.assertThat; + import dev.caskeleton.adapter.outbound.persistence.operation.entity.DurableOperationEntity; import dev.caskeleton.adapter.outbound.persistence.platform.JpaPlatformContractSupport; import dev.caskeleton.adapter.outbound.persistence.transaction.SpringTransaction; +import dev.caskeleton.application.idempotency.RequestFingerprint; +import dev.caskeleton.application.operation.DurableOperationClaim; +import dev.caskeleton.application.operation.DurableOperationId; import dev.caskeleton.application.operation.DurableOperationStorePort; +import dev.caskeleton.application.operation.DurableOperationSubmission; import dev.caskeleton.application.operation.testkit.DurableOperationStoreContract; import dev.caskeleton.application.transaction.TransactionPort; import jakarta.persistence.EntityManagerFactory; @@ -13,13 +19,22 @@ import java.nio.charset.StandardCharsets; import java.sql.Connection; import java.sql.SQLException; import java.sql.Statement; +import java.time.Duration; +import java.time.Instant; +import java.util.List; import java.util.Map; +import java.util.concurrent.CompletableFuture; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.ExecutorService; +import java.util.concurrent.Executors; +import java.util.concurrent.TimeUnit; import java.util.function.Supplier; import javax.sql.DataSource; import org.junit.jupiter.api.AfterAll; import org.junit.jupiter.api.BeforeAll; import org.junit.jupiter.api.BeforeEach; import org.junit.jupiter.api.Tag; +import org.junit.jupiter.api.Test; import org.springframework.context.annotation.AnnotationConfigApplicationContext; import org.springframework.context.annotation.Bean; import org.springframework.context.annotation.Configuration; @@ -32,9 +47,10 @@ import org.springframework.transaction.PlatformTransactionManager; /** * The shipped adapter, through Spring Data and a Spring transaction, against real PostgreSQL. * - *

The raw-SQL {@link DurableOperationStoreContractTest} proves the schema; this proves the code - * that talks to it. A resubmission only breaks where PostgreSQL aborts the transaction after a - * constraint violation, so no fake can stand in here. + *

{@link DurableOperationSchemaConstraintTest} proves the table's constraints; this proves the + * code that talks to it. A resubmission only breaks where PostgreSQL aborts the transaction after a + * constraint violation, and two claims only race where two transactions hold row locks at once, so + * no fake can stand in here. */ @Tag("jpa-contract") class JpaDurableOperationStoreContractTest extends DurableOperationStoreContract { @@ -45,7 +61,8 @@ class JpaDurableOperationStoreContractTest extends DurableOperationStoreContract @BeforeAll static void start() throws SQLException, IOException { support = JpaPlatformContractSupport.start(); - execute(migration()); + execute(migration("V11__durable_operation.sql")); + execute(migration("V23__durable_operation_lease_fence.sql")); context = new AnnotationConfigApplicationContext(); context.getBeanFactory().registerSingleton("dataSource", support.dataSource()); context.register(AdapterConfiguration.class); @@ -67,6 +84,74 @@ class JpaDurableOperationStoreContractTest extends DurableOperationStoreContract execute("TRUNCATE durable_operation"); } + @Test + void twoWorkersClaimingAtOnceTakeDifferentOperations() throws Exception { + // given + Instant now = Instant.parse("2026-08-25T09:00:00Z"); + inTransaction(() -> store().submit(submission("op-1", "a", now))); + inTransaction(() -> store().submit(submission("op-2", "b", now.plusSeconds(1)))); + CountDownLatch firstClaimed = new CountDownLatch(1); + CountDownLatch secondDone = new CountDownLatch(1); + ExecutorService pool = Executors.newSingleThreadExecutor(); + + // when + List claimed; + try { + // The first claim holds its row lock until the second has claimed. With plain FOR UPDATE the + // second would block behind it; with no lock at all both would take op-1. + CompletableFuture first = + CompletableFuture.supplyAsync( + () -> + inTransaction( + () -> { + DurableOperationClaim claim = claimNow("worker-a", now); + firstClaimed.countDown(); + awaitOrFail(secondDone); + return claim; + }), + pool); + awaitOrFail(firstClaimed); + DurableOperationClaim second = inTransaction(() -> claimNow("worker-b", now)); + secondDone.countDown(); + claimed = + List.of( + first.get(30, TimeUnit.SECONDS).operationId().value(), second.operationId().value()); + } finally { + pool.shutdownNow(); + } + + // then + assertThat(claimed).containsExactlyInAnyOrder("op-1", "op-2"); + } + + private DurableOperationClaim claimNow(String workerId, Instant now) { + return store().claimNext(workerId, Duration.ofMinutes(2), now).orElseThrow(); + } + + private static void awaitOrFail(CountDownLatch latch) { + try { + if (!latch.await(30, TimeUnit.SECONDS)) { + throw new IllegalStateException("the other claim did not arrive within 30 seconds"); + } + } catch (InterruptedException interrupted) { + Thread.currentThread().interrupt(); + throw new IllegalStateException("interrupted while waiting for the other claim", interrupted); + } + } + + private static DurableOperationSubmission submission( + String operationId, String hashLetter, Instant submittedAt) { + return new DurableOperationSubmission( + new DurableOperationId(operationId), + "reports.generate", + "alice", + null, + new RequestFingerprint(hashLetter.repeat(64)), + "{}", + submittedAt, + Duration.ofDays(1)); + } + @Override protected DurableOperationStorePort store() { return context.getBean(JpaDurableOperationStore.class); @@ -84,13 +169,13 @@ class JpaDurableOperationStoreContractTest extends DurableOperationStoreContract } } - private static String migration() throws IOException { + private static String migration(String file) throws IOException { try (InputStream resource = JpaDurableOperationStoreContractTest.class .getClassLoader() - .getResourceAsStream("db/migration/postgresql/V11__durable_operation.sql")) { + .getResourceAsStream("db/migration/postgresql/" + file)) { if (resource == null) { - throw new IOException("the durable operation migration is missing from the classpath"); + throw new IOException("the durable operation migration " + file + " is missing"); } return new String(resource.readAllBytes(), StandardCharsets.UTF_8); } diff --git a/src/adapter/outbound/persistence-jpa/src/postgresqlIntegrationTest/java/dev/caskeleton/adapter/outbound/persistence/readiness/FileserverPostgreSqlStack.java b/src/adapter/outbound/persistence-jpa/src/postgresqlIntegrationTest/java/dev/caskeleton/adapter/outbound/persistence/readiness/FileserverPostgreSqlStack.java index a6e77b62..068f1dc0 100644 --- a/src/adapter/outbound/persistence-jpa/src/postgresqlIntegrationTest/java/dev/caskeleton/adapter/outbound/persistence/readiness/FileserverPostgreSqlStack.java +++ b/src/adapter/outbound/persistence-jpa/src/postgresqlIntegrationTest/java/dev/caskeleton/adapter/outbound/persistence/readiness/FileserverPostgreSqlStack.java @@ -96,6 +96,36 @@ final class FileserverPostgreSqlStack implements AutoCloseable { return fileId; } + /** + * Waits until some session of this database is blocked on a lock, so a case can hold one + * transaction open until a second one is queued behind it without sleeping a guessed interval. + */ + void awaitASessionWaitingOnALock(Duration deadline) { + Instant giveUp = Instant.now().plus(deadline); + while (Instant.now().isBefore(giveUp)) { + Integer waiting = + jdbc.queryForObject( + "select count(*) from pg_stat_activity" + + " where datname = current_database() and wait_event_type = 'Lock'", + Integer.class); + if (waiting != null && waiting > 0) { + return; + } + Thread.onSpinWait(); + } + throw new IllegalStateException("no session waited on a lock within " + deadline); + } + + /** Counts the rows of {@code fs_recovery_item} for one file, open and resolved. */ + int recoveryItemsOf(FileId fileId) { + Integer rows = + jdbc.queryForObject( + "select count(*) from fs_recovery_item where file_id = ?", + Integer.class, + fileId.value()); + return rows == null ? -1 : rows; + } + /** Deletes every Fileserver row, children first. */ void empty() { jdbc.execute( diff --git a/src/adapter/outbound/persistence-jpa/src/postgresqlIntegrationTest/java/dev/caskeleton/adapter/outbound/persistence/readiness/PostgreSqlMigrationIntegrationTest.java b/src/adapter/outbound/persistence-jpa/src/postgresqlIntegrationTest/java/dev/caskeleton/adapter/outbound/persistence/readiness/PostgreSqlMigrationIntegrationTest.java index bf5117a8..0439e9d5 100644 --- a/src/adapter/outbound/persistence-jpa/src/postgresqlIntegrationTest/java/dev/caskeleton/adapter/outbound/persistence/readiness/PostgreSqlMigrationIntegrationTest.java +++ b/src/adapter/outbound/persistence-jpa/src/postgresqlIntegrationTest/java/dev/caskeleton/adapter/outbound/persistence/readiness/PostgreSqlMigrationIntegrationTest.java @@ -50,7 +50,7 @@ class PostgreSqlMigrationIntegrationTest { assertThat(appliedVersions(postgres, "flyway_schema_history")) .containsExactly( "1", "3", "4", "5", "6", "9", "10", "11", "12", "13", "14", "15", "16", "17", "18", - "19", "20", "21", "22"); + "19", "20", "21", "22", "23"); assertThat( singleValue( postgres, diff --git a/src/adapter/outbound/persistence-jpa/src/postgresqlIntegrationTest/java/dev/caskeleton/adapter/outbound/persistence/readiness/PostgreSqlNotificationBackgroundPassIntegrationTest.java b/src/adapter/outbound/persistence-jpa/src/postgresqlIntegrationTest/java/dev/caskeleton/adapter/outbound/persistence/readiness/PostgreSqlNotificationBackgroundPassIntegrationTest.java index 74d09587..09606e87 100644 --- a/src/adapter/outbound/persistence-jpa/src/postgresqlIntegrationTest/java/dev/caskeleton/adapter/outbound/persistence/readiness/PostgreSqlNotificationBackgroundPassIntegrationTest.java +++ b/src/adapter/outbound/persistence-jpa/src/postgresqlIntegrationTest/java/dev/caskeleton/adapter/outbound/persistence/readiness/PostgreSqlNotificationBackgroundPassIntegrationTest.java @@ -70,6 +70,7 @@ import org.flywaydb.core.Flyway; import org.junit.jupiter.api.AfterAll; import org.junit.jupiter.api.BeforeAll; import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.DisplayName; import org.junit.jupiter.api.Tag; import org.junit.jupiter.api.Test; import org.springframework.context.annotation.AnnotationConfigApplicationContext; @@ -189,7 +190,25 @@ class PostgreSqlNotificationBackgroundPassIntegrationTest { } @Test - @org.junit.jupiter.api.DisplayName("요청 저장은 collapse 와 dedup DROP 판정을 함께 남겨 같은 키 재조회가 같은 답을 준다") + @DisplayName("fence 없이 DISPATCHING으로 남은 옛 행이 있어도 복구 패스는 다른 행을 되돌린다") + void aDispatchingRowNoClaimWroteDoesNotStopTheRecoveryPass() { + // given + UUID legacy = insertDelivery("DISPATCHING", Instant.now().minus(Duration.ofMinutes(20)), null); + jdbc.update("update notification_recipient_delivery set lease_fence = 0 where id = ?", legacy); + UUID abandoned = + insertDelivery("DISPATCHING", Instant.now().minus(Duration.ofMinutes(10)), null); + + // when + recovery().recoverOnce(); + + // then + assertThat(List.of(deliveryState(legacy), deliveryState(abandoned))) + .as("a row before notification V4's fence is no lease the requeue can match on") + .containsExactly("DISPATCHING", "READY_TO_DISPATCH"); + } + + @Test + @DisplayName("요청 저장은 collapse 와 dedup DROP 판정을 함께 남겨 같은 키 재조회가 같은 답을 준다") void aStoredRequestKeepsItsCollapseAndItsDropVerdict() { // given JpaNotificationRequestStore requests = @@ -414,13 +433,15 @@ class PostgreSqlNotificationBackgroundPassIntegrationTest { "insert into notification_recipient_delivery (id, notification_id, tenant_id," + " recipient_ref, routing_plan_json, route_cursor, delivery_state, submission_outcome," + " delivery_outcome, evidence_level, attempt_count, next_dispatch_at, lease_owner," - + " lease_until, expires_at, created_at, updated_at)" + + " lease_fence, lease_until, expires_at, created_at, updated_at)" + " values (?, ?, 'tenant-a', 'user-1', '[]'::jsonb, 0, ?, 'NOT_SUBMITTED', 'UNKNOWN'," - + " 'PLATFORM_QUEUED', 0, now(), ?, ?, ?, now(), now())", + + " 'PLATFORM_QUEUED', 0, now(), ?, ?, ?, ?, now(), now())", delivery, request, state, leaseUntil == null ? null : "dead-worker", + // A claim writes the owner and raises the fence together, so a leased row has both. + leaseUntil == null ? 0L : 1L, leaseUntil == null ? null : Timestamp.from(leaseUntil), expiresAt == null ? null : Timestamp.from(expiresAt)); return delivery; diff --git a/src/adapter/outbound/persistence-jpa/src/postgresqlIntegrationTest/java/dev/caskeleton/adapter/outbound/persistence/readiness/PostgreSqlRecoveryQueueContractTest.java b/src/adapter/outbound/persistence-jpa/src/postgresqlIntegrationTest/java/dev/caskeleton/adapter/outbound/persistence/readiness/PostgreSqlRecoveryQueueContractTest.java index 9ba78957..a4c34b3c 100644 --- a/src/adapter/outbound/persistence-jpa/src/postgresqlIntegrationTest/java/dev/caskeleton/adapter/outbound/persistence/readiness/PostgreSqlRecoveryQueueContractTest.java +++ b/src/adapter/outbound/persistence-jpa/src/postgresqlIntegrationTest/java/dev/caskeleton/adapter/outbound/persistence/readiness/PostgreSqlRecoveryQueueContractTest.java @@ -1,16 +1,25 @@ package dev.caskeleton.adapter.outbound.persistence.readiness; +import static org.assertj.core.api.Assertions.assertThat; + import dev.caskeleton.adapter.outbound.persistence.fileserver.JpaRecoveryQueue; import dev.caskeleton.application.fileserver.api.FileId; import dev.caskeleton.application.fileserver.api.StorageNamespace; import dev.caskeleton.application.fileserver.recovery.RecoveryQueuePort; import dev.caskeleton.application.fileserver.testkit.RecoveryQueuePortContract; import java.time.Duration; +import java.util.List; +import java.util.concurrent.CompletableFuture; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.ExecutorService; +import java.util.concurrent.Executors; +import java.util.concurrent.TimeUnit; import java.util.function.Supplier; import org.junit.jupiter.api.AfterAll; import org.junit.jupiter.api.BeforeAll; import org.junit.jupiter.api.BeforeEach; import org.junit.jupiter.api.Tag; +import org.junit.jupiter.api.Test; /** * {@link JpaRecoveryQueue} held to the contract the in-memory queue passes, on real PostgreSQL @@ -38,6 +47,66 @@ class PostgreSqlRecoveryQueueContractTest extends RecoveryQueuePortContract { stack.empty(); } + @Test + void theSameFileEnqueuedInTwoConcurrentTransactionsLeavesOneOpenItem() throws Exception { + // given + FileId file = recordedFile(); + CountDownLatch firstEnqueued = new CountDownLatch(1); + CountDownLatch secondQueued = new CountDownLatch(1); + ExecutorService pool = Executors.newFixedThreadPool(2); + + // when + List outcome; + try { + // The first transaction holds its new row uncommitted until the second has queued behind the + // open-item index. Update-then-insert made the second insert too and fail on that index. + CompletableFuture first = + CompletableFuture.runAsync( + () -> + inTransaction( + () -> { + queue().enqueue(file, "FIRST_REPORT"); + firstEnqueued.countDown(); + awaitOrFail(secondQueued); + return null; + }), + pool); + awaitOrFail(firstEnqueued); + CompletableFuture second = + CompletableFuture.runAsync( + () -> + inTransaction( + () -> { + queue().enqueue(file, "SECOND_REPORT"); + return null; + }), + pool); + stack.awaitASessionWaitingOnALock(Duration.ofSeconds(30)); + secondQueued.countDown(); + first.get(30, TimeUnit.SECONDS); + second.get(30, TimeUnit.SECONDS); + outcome = + List.of(stack.recoveryItemsOf(file), inTransaction(() -> queue().pending(10)).size()); + } finally { + pool.shutdownNow(); + } + + // then + assertThat(outcome).containsExactly(1, 1); + } + + private static void awaitOrFail(CountDownLatch latch) { + try { + if (!latch.await(30, TimeUnit.SECONDS)) { + throw new IllegalStateException("the other enqueue did not arrive within 30 seconds"); + } + } catch (InterruptedException interrupted) { + Thread.currentThread().interrupt(); + throw new IllegalStateException( + "interrupted while waiting for the other enqueue", interrupted); + } + } + @Override protected RecoveryQueuePort queue() { return stack.bean(JpaRecoveryQueue.class); diff --git a/src/adapter/outbound/persistence-jpa/src/test/java/dev/caskeleton/adapter/outbound/persistence/idempotency/jpa/IdempotencyReaperTest.java b/src/adapter/outbound/persistence-jpa/src/test/java/dev/caskeleton/adapter/outbound/persistence/idempotency/jpa/IdempotencyReaperTest.java index 72235d67..e80b313d 100644 --- a/src/adapter/outbound/persistence-jpa/src/test/java/dev/caskeleton/adapter/outbound/persistence/idempotency/jpa/IdempotencyReaperTest.java +++ b/src/adapter/outbound/persistence-jpa/src/test/java/dev/caskeleton/adapter/outbound/persistence/idempotency/jpa/IdempotencyReaperTest.java @@ -2,23 +2,61 @@ package dev.caskeleton.adapter.outbound.persistence.idempotency.jpa; import static org.assertj.core.api.Assertions.assertThat; import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.times; +import static org.mockito.Mockito.verify; import static org.mockito.Mockito.when; -import java.time.Clock; -import java.time.Instant; -import java.time.ZoneOffset; import org.junit.jupiter.api.Test; +/** + * Unit tests for {@link IdempotencyReaper}: the delete runs in bounded batches until one comes back + * short, as the outbox reaper's does. + * + *

The repository is a Spring Data boundary, so Mockito stands in for it. The statement itself — + * expiry judged by the database clock — runs against PostgreSQL in {@code + * IdempotencyReaperIntegrationTest}. + */ class IdempotencyReaperTest { - private static final Instant NOW = Instant.parse("2026-06-09T12:00:00Z"); + private static final int BATCH = IdempotencyReaper.BATCH_SIZE; + + private final IdempotencyRecordJpaRepository repository = + mock(IdempotencyRecordJpaRepository.class); + private final IdempotencyReaper reaper = new IdempotencyReaper(repository); @Test - void reapDeletesRecordsExpiredAsOfNow() { - IdempotencyRecordJpaRepository repo = mock(IdempotencyRecordJpaRepository.class); - when(repo.deleteExpired(NOW)).thenReturn(3); - IdempotencyReaper reaper = new IdempotencyReaper(repo, Clock.fixed(NOW, ZoneOffset.UTC)); + void reapKeepsDeletingBatchesUntilOneComesBackShortAndReturnsTheTotal() { + // given + when(repository.deleteExpiredBatch(BATCH)).thenReturn(BATCH, BATCH, 3); - assertThat(reaper.reap()).isEqualTo(3); + // when + int purged = reaper.reap(); + + // then + assertThat(purged).isEqualTo(2 * BATCH + 3); + } + + @Test + void eachBatchIsItsOwnDeleteStatement() { + // given + when(repository.deleteExpiredBatch(BATCH)).thenReturn(BATCH, BATCH, 3); + + // when + var unused = reaper.reap(); + + // then + verify(repository, times(3)).deleteExpiredBatch(BATCH); + } + + @Test + void reapReturnsZeroWhenNothingHasExpired() { + // given + when(repository.deleteExpiredBatch(BATCH)).thenReturn(0); + + // when + int purged = reaper.reap(); + + // then + assertThat(purged).isZero(); } } diff --git a/src/adapter/outbound/persistence-jpa/src/test/resources/db/applied-migrations.sha256 b/src/adapter/outbound/persistence-jpa/src/test/resources/db/applied-migrations.sha256 index 1eb57d14..32edf890 100644 --- a/src/adapter/outbound/persistence-jpa/src/test/resources/db/applied-migrations.sha256 +++ b/src/adapter/outbound/persistence-jpa/src/test/resources/db/applied-migrations.sha256 @@ -42,6 +42,7 @@ a0306e5b0fbcaf43984d256362dc4244aec57a0ce5031e4f5263be3762fe1caf postgresql/V19 25db296e55444636c8a017d9b736eecf4a839a9caf1733c0c4a5f37bae5a3161 postgresql/V20__outbox_event_deferred_attempt_count.sql 21744ea1aab2b29fbb3417b9a5c4389522199413048cc7035aa649ffcdd852cf postgresql/V21__outbox_aggregate_sequence.sql 20da5649a29bf2305f0e04ab14003b21b7f120d867439fb01f1658c98179dba0 postgresql/V22__outbox_event_last_failure_code.sql +9492ace50ab828144792b77c72d92fce50bec29a028ebc917f7e55facf58d85a postgresql/V23__durable_operation_lease_fence.sql 8ab106731d4d04d3719da0692069947736dc490ca46aa5a3ea9bdf65a5d956dc postgresql/V1__idempotency_record.sql 6b62a01a4eeb792b81015b4810ecc6f3f021e0a17dea50c871493f7694d4f0b9 postgresql/V3__outbox_event.sql c57ab8a35a10079190ad8a563be58fae4147c9406c5bb752debb6998324ccdf6 postgresql/V4__int_lock.sql diff --git a/src/adapter/outbound/persistence-notification/AGENTS.md b/src/adapter/outbound/persistence-notification/AGENTS.md index 94fa03ad..cb9ae8b1 100644 --- a/src/adapter/outbound/persistence-notification/AGENTS.md +++ b/src/adapter/outbound/persistence-notification/AGENTS.md @@ -8,7 +8,8 @@ - 패키지 루트: `dev.caskeleton.adapter.outbound.persistence.notification`. 패키지 규칙은 `persistence-fileserver`와 같아서 두 capability leaf를 같은 방식으로 읽는다: - `platform.entity` — JPA 행만. - - `platform.repository` — Spring Data 인터페이스와 그것이 돌리는 SQL 글(`RecipientClaimSql`). + - `platform.repository` — Spring Data 인터페이스와 그것이 돌리는 SQL 글(`RecipientClaimSql`). `platform`의 store도 + 이 SQL 조각을 쓴다. - `platform` — store 어댑터(`Jpa*`, `Jdbc*`, inbox의 `JpaNotificationInbox`와 그 outbox 도우미), `NotificationRecordMapper`, payload codec. - `configuration` — store를 bean으로 등록하는 자동 구성 창구. leaf 루트에는 scan 설정과 schema stream이 있다. diff --git a/src/adapter/outbound/persistence-notification/src/main/java/dev/caskeleton/adapter/outbound/persistence/notification/platform/JpaRecipientDeliveryStore.java b/src/adapter/outbound/persistence-notification/src/main/java/dev/caskeleton/adapter/outbound/persistence/notification/platform/JpaRecipientDeliveryStore.java index 23fdb2dc..a2da23c2 100644 --- a/src/adapter/outbound/persistence-notification/src/main/java/dev/caskeleton/adapter/outbound/persistence/notification/platform/JpaRecipientDeliveryStore.java +++ b/src/adapter/outbound/persistence-notification/src/main/java/dev/caskeleton/adapter/outbound/persistence/notification/platform/JpaRecipientDeliveryStore.java @@ -99,6 +99,23 @@ public final class JpaRecipientDeliveryStore implements RecipientDeliveryStorePo return reread(written, record.id()); } + @Override + public Optional requeueAbandoned( + RecipientLease abandoned, Instant nextDispatchAt) { + CallerTransaction.require("recipient delivery requeueAbandoned"); + Objects.requireNonNull(abandoned, "abandoned"); + Objects.requireNonNull(nextDispatchAt, "nextDispatchAt"); + int written = + recipients.requeueAbandoned( + abandoned.recipientDeliveryId().value(), + abandoned.owner(), + abandoned.fence(), + abandoned.leaseUntil(), + nextDispatchAt, + clock.instant()); + return reread(written, abandoned.recipientDeliveryId()); + } + @Override public Optional transitionHeldBy( RecipientDeliveryId id, diff --git a/src/adapter/outbound/persistence-notification/src/main/java/dev/caskeleton/adapter/outbound/persistence/notification/platform/JpaRecipientLeaseStore.java b/src/adapter/outbound/persistence-notification/src/main/java/dev/caskeleton/adapter/outbound/persistence/notification/platform/JpaRecipientLeaseStore.java index 65784bac..98cd4bef 100644 --- a/src/adapter/outbound/persistence-notification/src/main/java/dev/caskeleton/adapter/outbound/persistence/notification/platform/JpaRecipientLeaseStore.java +++ b/src/adapter/outbound/persistence-notification/src/main/java/dev/caskeleton/adapter/outbound/persistence/notification/platform/JpaRecipientLeaseStore.java @@ -5,9 +5,11 @@ import dev.caskeleton.adapter.outbound.persistence.transaction.CallerTransaction import dev.caskeleton.application.notification.platform.api.RecipientDeliveryId; import dev.caskeleton.application.notification.platform.dispatch.RecipientLease; import dev.caskeleton.application.notification.platform.dispatch.RecipientLeaseStorePort; +import java.sql.Timestamp; import java.time.Clock; import java.time.Duration; import java.time.Instant; +import java.time.OffsetDateTime; import java.util.List; import java.util.Objects; import java.util.Optional; @@ -109,13 +111,38 @@ public final class JpaRecipientLeaseStore implements RecipientLeaseStorePort { } @Override - public List expiredDispatching(int limit, Duration olderThan) { + public List expiredDispatching(int limit, Duration olderThan) { Objects.requireNonNull(olderThan, "olderThan"); if (limit < 1) { throw new IllegalArgumentException("limit"); } return recipients.selectExpiredDispatching(clock.instant().minus(olderThan), limit).stream() - .map(RecipientDeliveryId::new) + .map( + row -> + new RecipientLease( + new RecipientDeliveryId((UUID) row[0]), + (String) row[1], + ((Number) row[2]).longValue(), + instantOf(row[3]))) .toList(); } + + /** + * A {@code timestamptz} column of a native row. Hibernate hands it back as an {@link + * OffsetDateTime} or a {@link Timestamp} depending on its JDBC type mapping, never null here + * because the select only returns rows that have a lease. + */ + private static Instant instantOf(Object column) { + if (column instanceof Instant instant) { + return instant; + } + if (column instanceof OffsetDateTime offset) { + return offset.toInstant(); + } + if (column instanceof Timestamp timestamp) { + return timestamp.toInstant(); + } + throw new IllegalStateException( + "lease_until came back as " + (column == null ? "null" : column.getClass().getName())); + } } diff --git a/src/adapter/outbound/persistence-notification/src/main/java/dev/caskeleton/adapter/outbound/persistence/notification/platform/repository/InboxItemJpaRepository.java b/src/adapter/outbound/persistence-notification/src/main/java/dev/caskeleton/adapter/outbound/persistence/notification/platform/repository/InboxItemJpaRepository.java index 15428027..e7ccfb32 100644 --- a/src/adapter/outbound/persistence-notification/src/main/java/dev/caskeleton/adapter/outbound/persistence/notification/platform/repository/InboxItemJpaRepository.java +++ b/src/adapter/outbound/persistence-notification/src/main/java/dev/caskeleton/adapter/outbound/persistence/notification/platform/repository/InboxItemJpaRepository.java @@ -60,18 +60,16 @@ public interface InboxItemJpaRepository extends Repository{@code read_at IS NULL} in the predicate is what makes the update idempotent under * concurrency: a second caller updates zero rows instead of overwriting the first caller's * timestamp. - */ - /** - * Marks everything up to and including a cursor position as read. * *

The cutoff is a tuple, matching the list's own {@code (created_at DESC, id DESC)} ordering. * It used to be a timestamp alone, and the list's order is not a total order on {@code @@ -95,14 +93,23 @@ public interface InboxItemJpaRepository extends RepositoryThe lease store contract ({@code RecipientLeaseStoreContract}, run on PostgreSQL by {@code * PostgreSqlRecipientLeaseStoreContractTest}) now drives this statement through the repository and * the store, so no test retypes it or runs it beside the adapter (review 2026-11-01 04-14). The - * named parameters are Spring Data's. + * named parameters are bound by Spring Data for the repository statements, and by a {@code + * NamedParameterJdbcTemplate} where the serving state reads {@link #CLAIMABLE_AT_NOW}. */ public final class RecipientClaimSql { + /** + * The states a delivery waits in to be sent, which the claim takes from, the expiry retires from + * and the serving state counts as backlog. One list so the statements cannot disagree about what + * is waiting. + */ + public static final String CLAIMABLE_STATES = "('PENDING', 'READY_TO_DISPATCH', 'RETRY_WAITING')"; + + /** + * Which deliveries the claim takes at {@code :now}: due, in a waiting state, not expired, and + * with no lease or one that has run out. The serving state counts its backlog with this same + * fragment, so the backlog is exactly what a claim would take. + */ + public static final String CLAIMABLE_AT_NOW = + "next_dispatch_at <= :now" + + " AND delivery_state IN " + + CLAIMABLE_STATES + + " AND (expires_at IS NULL OR expires_at > :now)" + + " AND (lease_until IS NULL OR lease_until < :now)"; + /** * Claims due deliveries and returns their fences. * @@ -28,10 +48,8 @@ public final class RecipientClaimSql { public static final String CLAIM_BATCH = "WITH claimable AS (" + " SELECT id FROM notification_recipient_delivery" - + " WHERE next_dispatch_at <= :now" - + " AND delivery_state IN ('PENDING', 'READY_TO_DISPATCH', 'RETRY_WAITING')" - + " AND (expires_at IS NULL OR expires_at > :now)" - + " AND (lease_until IS NULL OR lease_until < :now)" + + " WHERE " + + CLAIMABLE_AT_NOW + " ORDER BY next_dispatch_at, id" + " FOR UPDATE SKIP LOCKED" + " LIMIT :batchSize" @@ -51,7 +69,18 @@ public final class RecipientClaimSql { + " WHERE d.id = claimable.id" + " RETURNING d.id, d.lease_fence"; - /** Retires deliveries whose window closed before anyone claimed them. */ + /** + * Retires deliveries whose window closed before anyone claimed them. + * + *

A delivery a worker claimed while this statement ran must not be retired to {@code EXPIRED} + * under the worker dispatching it, which is what the unlocked subquery did (round9 review + * 2026-11-12, RR9-01-13). Two things stop it. The subquery locks the rows it picks and skips rows + * another transaction holds, so a delivery being claimed is not picked. The state and expiry + * conditions are also on the {@code UPDATE} itself: under READ COMMITTED PostgreSQL re-checks + * only the outer {@code WHERE} against a row it had to wait for, so if the lock clause is ever + * lost the claimed row is still left alone. A delivery being claimed right now is not expiring + * either: the claim refuses expired rows in the same statement. + */ public static final String EXPIRE_OVERDUE = "UPDATE notification_recipient_delivery SET" + " delivery_state = 'EXPIRED'," @@ -61,11 +90,17 @@ public final class RecipientClaimSql { + " updated_at = :now" + " WHERE id IN (" + " SELECT id FROM notification_recipient_delivery" - + " WHERE delivery_state IN ('PENDING', 'READY_TO_DISPATCH', 'RETRY_WAITING')" + + " WHERE delivery_state IN " + + CLAIMABLE_STATES + " AND expires_at IS NOT NULL" + " AND expires_at <= :now" + " ORDER BY expires_at" - + " LIMIT :batchSize)"; + + " FOR UPDATE SKIP LOCKED" + + " LIMIT :batchSize)" + + " AND delivery_state IN " + + CLAIMABLE_STATES + + " AND expires_at IS NOT NULL" + + " AND expires_at <= :now"; private RecipientClaimSql() {} } diff --git a/src/adapter/outbound/persistence-notification/src/main/java/dev/caskeleton/adapter/outbound/persistence/notification/platform/repository/RecipientDeliveryJpaRepository.java b/src/adapter/outbound/persistence-notification/src/main/java/dev/caskeleton/adapter/outbound/persistence/notification/platform/repository/RecipientDeliveryJpaRepository.java index b5f1ee18..1240cf0f 100644 --- a/src/adapter/outbound/persistence-notification/src/main/java/dev/caskeleton/adapter/outbound/persistence/notification/platform/repository/RecipientDeliveryJpaRepository.java +++ b/src/adapter/outbound/persistence-notification/src/main/java/dev/caskeleton/adapter/outbound/persistence/notification/platform/repository/RecipientDeliveryJpaRepository.java @@ -183,13 +183,48 @@ public interface RecipientDeliveryJpaRepository extends RepositoryOnly rows a claim wrote: an owner and a fence of at least one. The requeue matches on both, + * and a row without them — one left dispatching before notification V4 added the fence — is no + * lease anyone holds. Read here it would be a lease the application refuses to build, and the one + * row would fail every recovery pass after it. + * + * @return one row per delivery: id, lease_owner, lease_fence, lease_until + */ @Query( value = - "SELECT id FROM notification_recipient_delivery " + "SELECT id, lease_owner, lease_fence, lease_until FROM notification_recipient_delivery " + "WHERE delivery_state = 'DISPATCHING' AND lease_until < :threshold " + + "AND lease_owner IS NOT NULL AND lease_fence > 0 " + "ORDER BY lease_until LIMIT :batchSize", nativeQuery = true) - List selectExpiredDispatching( + List selectExpiredDispatching( @Param("threshold") Instant threshold, @Param("batchSize") int batchSize); + + /** + * Returns an abandoned dispatch to the queue while it is still the one recovery found. + * + *

Owner, fence, state and expiry are all conditions. A new claim raises the fence, a finished + * holder moves the state, another instance's recovery moves the state, and a holder that renewed + * in time moves the expiry past what recovery saw; in each case the job is no longer abandoned + * and this matches nothing. + */ + @Modifying(clearAutomatically = true, flushAutomatically = true) + @Query( + value = + "UPDATE notification_recipient_delivery " + + "SET delivery_state = 'READY_TO_DISPATCH', next_dispatch_at = :nextDispatchAt, " + + "version = version + 1, updated_at = :now " + + "WHERE id = :id AND lease_owner = :owner AND lease_fence = :fence " + + "AND delivery_state = 'DISPATCHING' AND lease_until <= :leaseUntil", + nativeQuery = true) + int requeueAbandoned( + @Param("id") UUID id, + @Param("owner") String owner, + @Param("fence") long fence, + @Param("leaseUntil") Instant leaseUntil, + @Param("nextDispatchAt") Instant nextDispatchAt, + @Param("now") Instant now); } diff --git a/src/application/core/src/main/java/dev/caskeleton/application/operation/DurableOperationClaim.java b/src/application/core/src/main/java/dev/caskeleton/application/operation/DurableOperationClaim.java new file mode 100644 index 00000000..80bdbe0c --- /dev/null +++ b/src/application/core/src/main/java/dev/caskeleton/application/operation/DurableOperationClaim.java @@ -0,0 +1,45 @@ +package dev.caskeleton.application.operation; + +import java.util.Objects; + +/** + * One worker's claim on one durable operation, as returned by {@link + * DurableOperationStorePort#claimNext}. + * + *

The {@code fence} is what tells one run from the next. Every claim increments it, and a + * heartbeat, progress report or completion names the fence it holds, so a run that stalled past its + * lease finds nothing left to change instead of overwriting the run that took over. The worker id + * alone cannot do this: a worker that restarts with the same id, or one process running two + * executions under one id, would otherwise have its earlier run accepted as the current one. + * + *

Kept apart from {@link DurableOperationSubmission} on purpose, as the outbox keeps its lease + * apart from the claimed event: the work to execute is the submission, and the claim is store + * bookkeeping the work has no use for. + * + *

A store builds it from a row, never from a request, so a value it refuses is a server fault: + * {@link IllegalStateException}, not the {@code IllegalArgumentException} the web layer would + * answer with a 400 (manual 06). + * + * @param submission the claimed work + * @param workerId the worker that holds the claim + * @param fence the fencing token this claim was granted; at least 1 + */ +public record DurableOperationClaim( + DurableOperationSubmission submission, String workerId, long fence) { + + public DurableOperationClaim { + Objects.requireNonNull(submission, "submission"); + if (workerId == null || workerId.isBlank()) { + throw new IllegalStateException( + "a claim needs an owner; an anonymous one cannot be verified when the work completes"); + } + if (fence < 1) { + throw new IllegalStateException("fence must be >= 1, was " + fence); + } + } + + /** The claimed operation. */ + public DurableOperationId operationId() { + return submission.operationId(); + } +} diff --git a/src/application/core/src/main/java/dev/caskeleton/application/operation/DurableOperationStorePort.java b/src/application/core/src/main/java/dev/caskeleton/application/operation/DurableOperationStorePort.java index 440d3fb2..66c718b2 100644 --- a/src/application/core/src/main/java/dev/caskeleton/application/operation/DurableOperationStorePort.java +++ b/src/application/core/src/main/java/dev/caskeleton/application/operation/DurableOperationStorePort.java @@ -11,9 +11,12 @@ import java.util.Optional; *

A port because the application owns what an operation is and nothing about how it is stored. * The JPA adapter behind it is one implementation; an in-memory one serves tests. * - *

Every method that changes a running operation takes the {@code workerId} and verifies it. A - * worker whose lease lapsed may still be alive and still finishing its work, and letting it report - * a result after another worker took over is how an operation ends up with two answers. + *

Every method that changes a running operation takes the {@link DurableOperationClaim} that + * {@link #claimNext} returned and verifies both its worker and its fence. A worker whose lease + * lapsed may still be alive and still finishing its work, and letting it report a result after + * another run took over is how an operation ends up with two answers. The worker id alone does not + * tell the runs apart: the same id may claim the operation again after a restart, so the fence — + * incremented by every claim — is what the store compares. */ public interface DurableOperationStorePort { @@ -43,48 +46,43 @@ public interface DurableOperationStorePort { * *

Runnable means PENDING, or RUNNING with a lapsed lease. The second case is the recovery * path: without it, an operation claimed by a worker that then died is never picked up again. + * Every claim gets a fence higher than any earlier claim of the same operation, whoever made it. * * @param workerId who is claiming * @param leaseDuration how long the claim holds without a heartbeat * @param now the current instant + * @return the claim the worker names on every later write, or empty when nothing is runnable */ - Optional claimNext( - String workerId, Duration leaseDuration, Instant now); + Optional claimNext(String workerId, Duration leaseDuration, Instant now); /** - * Extends a lease the worker still holds. + * Extends a lease the claim still holds. * - * @return false when the lease is no longer this worker's, which means it must stop + * @return false when the lease lapsed or another claim (by any worker, this one included) took + * the operation over, which means this run must stop */ - boolean heartbeat( - DurableOperationId operationId, String workerId, Duration leaseDuration, Instant now); + boolean heartbeat(DurableOperationClaim claim, Duration leaseDuration, Instant now); /** - * Publishes progress for an operation this worker holds. + * Publishes progress for an operation the claim holds. * - * @return false when the lease is no longer this worker's + * @return false when another claim took the operation over or it is no longer running */ - boolean reportProgress( - DurableOperationId operationId, String workerId, OperationProgressSnapshot progress); + boolean reportProgress(DurableOperationClaim claim, OperationProgressSnapshot progress); /** * Records success. * - * @return false when the lease is no longer this worker's, so the result is not recorded + * @return false when another claim took the operation over, so the result is not recorded */ - boolean succeed( - DurableOperationId operationId, String workerId, String resultReference, Instant completedAt); + boolean succeed(DurableOperationClaim claim, String resultReference, Instant completedAt); /** * Records failure. * - * @return false when the lease is no longer this worker's + * @return false when another claim took the operation over, so the failure is not recorded */ - boolean fail( - DurableOperationId operationId, - String workerId, - OperationFailure failure, - Instant completedAt); + boolean fail(DurableOperationClaim claim, OperationFailure failure, Instant completedAt); /** * Requests cancellation. diff --git a/src/application/core/src/main/java/dev/caskeleton/application/operation/OperationLease.java b/src/application/core/src/main/java/dev/caskeleton/application/operation/OperationLease.java index 86328d84..618f23c1 100644 --- a/src/application/core/src/main/java/dev/caskeleton/application/operation/OperationLease.java +++ b/src/application/core/src/main/java/dev/caskeleton/application/operation/OperationLease.java @@ -11,6 +11,10 @@ import java.util.Objects; * ever with no worker behind it — the failure that makes a queue stop draining while every * dashboard reports it busy. * + *

A store builds it from a row, never from a request, so a value it refuses is a server fault: + * {@link IllegalStateException}, not the {@code IllegalArgumentException} the web layer would + * answer with a 400 (manual 06). + * * @param workerId who holds it * @param acquiredAt when it was taken * @param expiresAt when another worker may take it @@ -22,11 +26,11 @@ public record OperationLease(String workerId, Instant acquiredAt, Instant expire Objects.requireNonNull(acquiredAt, "acquiredAt"); Objects.requireNonNull(expiresAt, "expiresAt"); if (workerId.isBlank()) { - throw new IllegalArgumentException( + throw new IllegalStateException( "a lease needs an owner; an anonymous one cannot be verified when the work completes"); } if (!expiresAt.isAfter(acquiredAt)) { - throw new IllegalArgumentException("a lease that expires when taken is not a lease"); + throw new IllegalStateException("a lease that expires when taken is not a lease"); } } diff --git a/src/application/core/src/main/java/dev/caskeleton/application/outbox/ClaimedOutboxEvent.java b/src/application/core/src/main/java/dev/caskeleton/application/outbox/ClaimedOutboxEvent.java index bafa1b5e..05c0062e 100644 --- a/src/application/core/src/main/java/dev/caskeleton/application/outbox/ClaimedOutboxEvent.java +++ b/src/application/core/src/main/java/dev/caskeleton/application/outbox/ClaimedOutboxEvent.java @@ -12,6 +12,10 @@ import java.util.Objects; * publishes the bytes that were validated rather than a re-rendering of them. The accessors below * name the few identities the relay and its diagnostics read without reaching into the envelope. * + *

A store builds it from a row, never from a request, so a count it refuses is a server fault: + * {@link IllegalStateException}, not the {@code IllegalArgumentException} the web layer would + * answer with a 400 (manual 06). + * * @param event the appended event * @param status the row's lifecycle status; {@link OutboxEventStatus#IN_FLIGHT} right after a claim * @param attemptCount claims made, counting the claim that returned this row; the retry delay grows @@ -30,10 +34,10 @@ public record ClaimedOutboxEvent( Objects.requireNonNull(event, "event must not be null"); Objects.requireNonNull(status, "status must not be null"); if (attemptCount < 0) { - throw new IllegalArgumentException("attemptCount must be >= 0, was " + attemptCount); + throw new IllegalStateException("attemptCount must be >= 0, was " + attemptCount); } if (deferredAttemptCount < 0 || deferredAttemptCount > attemptCount) { - throw new IllegalArgumentException( + throw new IllegalStateException( "deferredAttemptCount must be between 0 and attemptCount (" + attemptCount + "), was " diff --git a/src/application/core/src/main/java/dev/caskeleton/application/outbox/OutboxLease.java b/src/application/core/src/main/java/dev/caskeleton/application/outbox/OutboxLease.java index eed397ab..ae783bc1 100644 --- a/src/application/core/src/main/java/dev/caskeleton/application/outbox/OutboxLease.java +++ b/src/application/core/src/main/java/dev/caskeleton/application/outbox/OutboxLease.java @@ -13,6 +13,10 @@ import java.util.Objects; *

Kept apart from {@link ClaimedOutboxEvent} on purpose: the broker adapter receives the event, * and the claim is store bookkeeping it has no use for. * + *

A store builds it from a row, never from a request, so a value it refuses is a server fault: + * {@link IllegalStateException}, not the {@code IllegalArgumentException} the web layer would + * answer with a 400 (manual 06). + * * @param event the claimed row * @param owner the relay instance that holds the claim * @param token the fencing token this claim was granted @@ -22,10 +26,10 @@ public record OutboxLease(ClaimedOutboxEvent event, String owner, long token) { public OutboxLease { Objects.requireNonNull(event, "event must not be null"); if (owner == null || owner.isBlank()) { - throw new IllegalArgumentException("owner must not be blank"); + throw new IllegalStateException("owner must not be blank"); } if (token < 1) { - throw new IllegalArgumentException("token must be >= 1, was " + token); + throw new IllegalStateException("token must be >= 1, was " + token); } } diff --git a/src/application/core/src/test/java/dev/caskeleton/application/operation/CancelDurableOperationUseCaseTest.java b/src/application/core/src/test/java/dev/caskeleton/application/operation/CancelDurableOperationUseCaseTest.java index 628ee3ef..8c845cac 100644 --- a/src/application/core/src/test/java/dev/caskeleton/application/operation/CancelDurableOperationUseCaseTest.java +++ b/src/application/core/src/test/java/dev/caskeleton/application/operation/CancelDurableOperationUseCaseTest.java @@ -39,8 +39,9 @@ class CancelDurableOperationUseCaseTest { void cancelOfAFinishedOperationReportsNoChange() { // given operations.submit(submission(null)); - operations.claimNext("worker-a", Duration.ofMinutes(1), NOW); - operations.succeed(ID, "worker-a", "/results/1", NOW); + DurableOperationClaim claim = + operations.claimNext("worker-a", Duration.ofMinutes(1), NOW).orElseThrow(); + operations.succeed(claim, "/results/1", NOW); // when Optional canceled = diff --git a/src/application/core/src/test/java/dev/caskeleton/application/operation/StoreBuiltRecordsTest.java b/src/application/core/src/test/java/dev/caskeleton/application/operation/StoreBuiltRecordsTest.java new file mode 100644 index 00000000..13f6efaf --- /dev/null +++ b/src/application/core/src/test/java/dev/caskeleton/application/operation/StoreBuiltRecordsTest.java @@ -0,0 +1,80 @@ +package dev.caskeleton.application.operation; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.catchThrowable; + +import dev.caskeleton.application.idempotency.RequestFingerprint; +import dev.caskeleton.application.outbox.ClaimedOutboxEvent; +import dev.caskeleton.application.outbox.OutboxEventStatus; +import dev.caskeleton.application.outbox.OutboxLease; +import dev.caskeleton.application.outbox.testkit.OutboxEvents; +import java.time.Duration; +import java.time.Instant; +import java.util.stream.Stream; +import org.assertj.core.api.ThrowableAssert.ThrowingCallable; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.Arguments; +import org.junit.jupiter.params.provider.MethodSource; + +/** The claims and leases a store builds from its rows refuse a bad value as each record says. */ +class StoreBuiltRecordsTest { + + private static final Instant NOW = Instant.parse("2026-11-13T09:00:00Z"); + + static Stream refusedValues() { + DurableOperationSubmission submission = + new DurableOperationSubmission( + new DurableOperationId("op-1"), + "reports.generate", + "alice", + "tenant-a", + new RequestFingerprint("a".repeat(64)), + "{}", + NOW, + Duration.ofHours(1)); + return Stream.of( + Arguments.of( + "owner 없는 claim", + (ThrowingCallable) () -> new DurableOperationClaim(submission, " ", 1), + "a claim needs an owner"), + Arguments.of( + "fence 0인 claim", + (ThrowingCallable) () -> new DurableOperationClaim(submission, "worker-1", 0), + "fence must be >= 1, was 0"), + Arguments.of( + "owner 없는 lease", + (ThrowingCallable) () -> new OperationLease(" ", NOW, NOW.plusSeconds(30)), + "a lease needs an owner"), + Arguments.of( + "잡는 순간 끝나는 lease", + (ThrowingCallable) () -> new OperationLease("worker-1", NOW, NOW), + "a lease that expires when taken is not a lease"), + Arguments.of( + "owner 없는 outbox lease", + (ThrowingCallable) () -> new OutboxLease(OutboxEvents.claimed("evt-1", NOW, 1), " ", 1), + "owner must not be blank"), + Arguments.of( + "token 0인 outbox lease", + (ThrowingCallable) + () -> new OutboxLease(OutboxEvents.claimed("evt-1", NOW, 1), "relay-1", 0), + "token must be >= 1, was 0"), + Arguments.of( + "미룬 claim이 claim보다 많은 outbox 행", + (ThrowingCallable) + () -> + new ClaimedOutboxEvent( + OutboxEvents.event("evt-2", NOW), OutboxEventStatus.IN_FLIGHT, 1, 2), + "deferredAttemptCount must be between 0 and attemptCount (1), was 2")); + } + + @ParameterizedTest(name = "{0}") + @MethodSource("refusedValues") + void aValueAStoreCannotHaveBuiltIsAServerFault( + String name, ThrowingCallable build, String message) { + // when + Throwable refused = catchThrowable(build); + + // then + assertThat(refused).isInstanceOf(IllegalStateException.class).hasMessageContaining(message); + } +} diff --git a/src/application/core/src/test/java/dev/caskeleton/application/outbox/ClaimedOutboxEventContractTest.java b/src/application/core/src/test/java/dev/caskeleton/application/outbox/ClaimedOutboxEventContractTest.java index a5a6a3fa..b22ab546 100644 --- a/src/application/core/src/test/java/dev/caskeleton/application/outbox/ClaimedOutboxEventContractTest.java +++ b/src/application/core/src/test/java/dev/caskeleton/application/outbox/ClaimedOutboxEventContractTest.java @@ -49,7 +49,8 @@ class ClaimedOutboxEventContractTest { // then assertThat(thrown) - .isInstanceOf(IllegalArgumentException.class) + .as("a store builds the claim from its row, so a bad count is the store's fault") + .isInstanceOf(IllegalStateException.class) .hasMessage("attemptCount must be >= 0, was -1"); } } diff --git a/src/application/core/src/testFixtures/java/dev/caskeleton/application/operation/testkit/DurableOperationStoreContract.java b/src/application/core/src/testFixtures/java/dev/caskeleton/application/operation/testkit/DurableOperationStoreContract.java index 4be958d2..38f896aa 100644 --- a/src/application/core/src/testFixtures/java/dev/caskeleton/application/operation/testkit/DurableOperationStoreContract.java +++ b/src/application/core/src/testFixtures/java/dev/caskeleton/application/operation/testkit/DurableOperationStoreContract.java @@ -4,6 +4,7 @@ import static org.assertj.core.api.Assertions.assertThat; import dev.caskeleton.application.idempotency.RequestFingerprint; import dev.caskeleton.application.operation.DurableOperation; +import dev.caskeleton.application.operation.DurableOperationClaim; import dev.caskeleton.application.operation.DurableOperationId; import dev.caskeleton.application.operation.DurableOperationState; import dev.caskeleton.application.operation.DurableOperationStorePort; @@ -25,6 +26,10 @@ import org.junit.jupiter.api.Test; * *

Every call goes through {@link #inTransaction}, because the real store refuses writes without * a caller transaction and because a resubmission must leave that transaction usable. + * + *

A lease write names the {@link DurableOperationClaim} it holds. The fence cases are why: the + * same worker id claiming an operation again must fence out its own earlier run, which a check on + * the worker id alone lets through. */ public abstract class DurableOperationStoreContract { @@ -33,6 +38,9 @@ public abstract class DurableOperationStoreContract { private static final Duration TTL = Duration.ofDays(1); private static final String HASH_A = "a".repeat(64); private static final String HASH_B = "b".repeat(64); + private static final String HASH_C = "c".repeat(64); + private static final OperationFailure TIMED_OUT = + new OperationFailure("DEPENDENCY_TIMEOUT", "timed out", true); /** The store under test, empty at the start of each test. */ protected abstract DurableOperationStorePort store(); @@ -182,10 +190,36 @@ public abstract class DurableOperationStoreContract { submit(submission); // when - Optional claimed = claim("worker-a", NOW); + Optional claimed = claim("worker-a", NOW); // then - assertThat(claimed).contains(submission); + assertThat(claimed).contains(new DurableOperationClaim(submission, "worker-a", 1)); + } + + @Test + void everyClaimOfAnOperationGetsAHigherFence() { + // given + submit(submission("op-1", null, HASH_A)); + DurableOperationClaim first = claim("worker-a", NOW).orElseThrow(); + + // when + DurableOperationClaim second = claim("worker-a", NOW.plus(LEASE).plusSeconds(1)).orElseThrow(); + + // then + assertThat(second.fence()).isGreaterThan(first.fence()); + } + + @Test + void canceledOperationIsNotHandedToAWorker() { + // given + submit(submission("op-1", null, HASH_A)); + cancel("op-1", "alice", NOW.plusSeconds(5)); + + // when + Optional claimed = claim("worker-a", NOW.plusSeconds(6)); + + // then + assertThat(claimed).isEmpty(); } @Test @@ -195,7 +229,7 @@ public abstract class DurableOperationStoreContract { claim("worker-a", NOW); // when - Optional claimed = claim("worker-b", NOW.plusSeconds(30)); + Optional claimed = claim("worker-b", NOW.plusSeconds(30)); // then assertThat(claimed).isEmpty(); @@ -224,11 +258,11 @@ public abstract class DurableOperationStoreContract { void heartbeatFromTheLeaseHolderExtendsTheLease() { // given submit(submission("op-1", null, HASH_A)); - claim("worker-a", NOW); + DurableOperationClaim held = claim("worker-a", NOW).orElseThrow(); Instant beat = NOW.plusSeconds(60); // when - inTransaction(() -> store().heartbeat(id("op-1"), "worker-a", LEASE, beat)); + inTransaction(() -> store().heartbeat(held, LEASE, beat)); // then assertThat(find("op-1", "alice", beat).orElseThrow().lease()) @@ -239,13 +273,12 @@ public abstract class DurableOperationStoreContract { void heartbeatFromAWorkerThatLostTheLeaseIsRefused() { // given submit(submission("op-1", null, HASH_A)); - claim("worker-a", NOW); + DurableOperationClaim stale = claim("worker-a", NOW).orElseThrow(); Instant later = NOW.plus(LEASE).plusSeconds(1); claim("worker-b", later); // when - boolean extended = - inTransaction(() -> store().heartbeat(id("op-1"), "worker-a", LEASE, later.plusSeconds(1))); + boolean extended = inTransaction(() -> store().heartbeat(stale, LEASE, later.plusSeconds(1))); // then assertThat(extended).isFalse(); @@ -255,12 +288,12 @@ public abstract class DurableOperationStoreContract { void progressFromTheLeaseHolderIsPublished() { // given submit(submission("op-1", null, HASH_A)); - claim("worker-a", NOW); + DurableOperationClaim held = claim("worker-a", NOW).orElseThrow(); OperationProgressSnapshot progress = new OperationProgressSnapshot(3, Optional.of(10L), Optional.of("extracting")); // when - inTransaction(() -> store().reportProgress(id("op-1"), "worker-a", progress)); + inTransaction(() -> store().reportProgress(held, progress)); // then assertThat(find("op-1", "alice", NOW).orElseThrow().progress()).contains(progress); @@ -270,12 +303,13 @@ public abstract class DurableOperationStoreContract { void progressFromAnotherWorkerIsRefused() { // given submit(submission("op-1", null, HASH_A)); - claim("worker-a", NOW); + DurableOperationClaim held = claim("worker-a", NOW).orElseThrow(); + DurableOperationClaim impostor = + new DurableOperationClaim(held.submission(), "worker-b", held.fence()); // when boolean published = - inTransaction( - () -> store().reportProgress(id("op-1"), "worker-b", OperationProgressSnapshot.of(1))); + inTransaction(() -> store().reportProgress(impostor, OperationProgressSnapshot.of(1))); // then assertThat(published).isFalse(); @@ -285,11 +319,11 @@ public abstract class DurableOperationStoreContract { void successFromTheLeaseHolderIsRecorded() { // given submit(submission("op-1", null, HASH_A)); - claim("worker-a", NOW); + DurableOperationClaim held = claim("worker-a", NOW).orElseThrow(); Instant done = NOW.plusSeconds(10); // when - inTransaction(() -> store().succeed(id("op-1"), "worker-a", "/results/a", done)); + inTransaction(() -> store().succeed(held, "/results/a", done)); // then assertThat(find("op-1", "alice", done).orElseThrow()) @@ -309,28 +343,106 @@ public abstract class DurableOperationStoreContract { void staleWorkerCannotRecordAResultOverItsSuccessor() { // given submit(submission("op-1", null, HASH_A)); - claim("worker-a", NOW); + DurableOperationClaim stale = claim("worker-a", NOW).orElseThrow(); Instant later = NOW.plus(LEASE).plusSeconds(1); claim("worker-b", later); // when boolean recorded = - inTransaction( - () -> store().succeed(id("op-1"), "worker-a", "/results/a", later.plusSeconds(1))); + inTransaction(() -> store().succeed(stale, "/results/a", later.plusSeconds(1))); // then assertThat(recorded).isFalse(); } + @Test + void heartbeatOfAnEarlierRunOfTheSameWorkerIsRefused() { + // given + submit(submission("op-1", null, HASH_A)); + DurableOperationClaim earlier = claim("worker-a", NOW).orElseThrow(); + Instant later = NOW.plus(LEASE).plusSeconds(1); + claim("worker-a", later); + + // when + boolean accepted = inTransaction(() -> store().heartbeat(earlier, LEASE, later.plusSeconds(1))); + + // then + assertThat(accepted).isFalse(); + } + + @Test + void progressOfAnEarlierRunOfTheSameWorkerIsRefused() { + // given + submit(submission("op-1", null, HASH_A)); + DurableOperationClaim earlier = claim("worker-a", NOW).orElseThrow(); + Instant later = NOW.plus(LEASE).plusSeconds(1); + claim("worker-a", later); + + // when + boolean accepted = + inTransaction(() -> store().reportProgress(earlier, OperationProgressSnapshot.of(1))); + + // then + assertThat(accepted).isFalse(); + } + + @Test + void successOfAnEarlierRunOfTheSameWorkerIsRefused() { + // given + submit(submission("op-1", null, HASH_A)); + DurableOperationClaim earlier = claim("worker-a", NOW).orElseThrow(); + Instant later = NOW.plus(LEASE).plusSeconds(1); + claim("worker-a", later); + + // when + boolean accepted = + inTransaction(() -> store().succeed(earlier, "/results/earlier", later.plusSeconds(1))); + + // then + assertThat(accepted).isFalse(); + } + + @Test + void failureOfAnEarlierRunOfTheSameWorkerIsRefused() { + // given + submit(submission("op-1", null, HASH_A)); + DurableOperationClaim earlier = claim("worker-a", NOW).orElseThrow(); + Instant later = NOW.plus(LEASE).plusSeconds(1); + claim("worker-a", later); + + // when + boolean accepted = inTransaction(() -> store().fail(earlier, TIMED_OUT, later.plusSeconds(1))); + + // then + assertThat(accepted).isFalse(); + } + + @Test + void resultOfTheRunThatTookOverSurvivesItsSameWorkerPredecessor() { + // given + submit(submission("op-1", null, HASH_A)); + DurableOperationClaim earlier = claim("worker-a", NOW).orElseThrow(); + Instant later = NOW.plus(LEASE).plusSeconds(1); + DurableOperationClaim current = claim("worker-a", later).orElseThrow(); + inTransaction(() -> store().succeed(current, "/results/current", later.plusSeconds(1))); + + // when + inTransaction(() -> store().succeed(earlier, "/results/earlier", later.plusSeconds(2))); + + // then + assertThat(find("op-1", "alice", later.plusSeconds(2)).orElseThrow().resultReference()) + .contains("/results/current"); + } + @Test void failureFromTheLeaseHolderIsRecorded() { // given submit(submission("op-1", null, HASH_A)); - claim("worker-a", NOW); + DurableOperationClaim held = claim("worker-a", NOW).orElseThrow(); OperationFailure failure = new OperationFailure("DEPENDENCY_TIMEOUT", "timed out", true); // when - inTransaction(() -> store().fail(id("op-1"), "worker-a", failure, NOW.plusSeconds(10))); + inTransaction(() -> store().fail(held, failure, NOW.plusSeconds(10))); // then assertThat(find("op-1", "alice", NOW.plusSeconds(10)).orElseThrow()) @@ -369,8 +481,8 @@ public abstract class DurableOperationStoreContract { void cancelNeverReversesASuccess() { // given submit(submission("op-1", null, HASH_A)); - claim("worker-a", NOW); - inTransaction(() -> store().succeed(id("op-1"), "worker-a", "/results/a", NOW.plusSeconds(10))); + DurableOperationClaim held = claim("worker-a", NOW).orElseThrow(); + inTransaction(() -> store().succeed(held, "/results/a", NOW.plusSeconds(10))); // when cancel("op-1", "alice", NOW.plusSeconds(20)); @@ -407,19 +519,35 @@ public abstract class DurableOperationStoreContract { } @Test - void expirySweepExpiresAgedOutWorkAndLeavesFinishedWorkAlone() { + void reclaimLeavesALiveLeaseAlone() { // given submit(submission("op-1", null, HASH_A)); claim("worker-a", NOW); - inTransaction(() -> store().succeed(id("op-1"), "worker-a", "/results/a", NOW.plusSeconds(10))); + + // when + List reclaimed = + inTransaction(() -> store().reclaimExpiredLeases(NOW.plusSeconds(30))); + + // then + assertThat(reclaimed).isEmpty(); + } + + @Test + void expirySweepExpiresAgedOutWorkAndLeavesFinishedWorkAlone() { + // given + submit(submission("op-1", null, HASH_A)); + DurableOperationClaim held = claim("worker-a", NOW).orElseThrow(); + inTransaction(() -> store().succeed(held, "/results/a", NOW.plusSeconds(10))); submit(submission("op-2", null, HASH_B)); + submit(submission("op-3", null, HASH_C)); + claim("worker-b", NOW.plusSeconds(20)); // when List expired = inTransaction(() -> store().expireStaleOperations(NOW.plus(TTL).plus(TTL))); // then - assertThat(expired).containsExactly(id("op-2")); + assertThat(expired).containsExactlyInAnyOrder(id("op-2"), id("op-3")); } private DurableOperationSubmitResult submit(DurableOperationSubmission submission) { @@ -430,7 +558,7 @@ public abstract class DurableOperationStoreContract { return inTransaction(() -> store().find(id(operationId), principal, now)); } - private Optional claim(String workerId, Instant now) { + private Optional claim(String workerId, Instant now) { return inTransaction(() -> store().claimNext(workerId, LEASE, now)); } diff --git a/src/application/core/src/testFixtures/java/dev/caskeleton/application/operation/testkit/InMemoryDurableOperationStore.java b/src/application/core/src/testFixtures/java/dev/caskeleton/application/operation/testkit/InMemoryDurableOperationStore.java index 019193c3..3c76c99a 100644 --- a/src/application/core/src/testFixtures/java/dev/caskeleton/application/operation/testkit/InMemoryDurableOperationStore.java +++ b/src/application/core/src/testFixtures/java/dev/caskeleton/application/operation/testkit/InMemoryDurableOperationStore.java @@ -1,6 +1,7 @@ package dev.caskeleton.application.operation.testkit; import dev.caskeleton.application.operation.DurableOperation; +import dev.caskeleton.application.operation.DurableOperationClaim; import dev.caskeleton.application.operation.DurableOperationId; import dev.caskeleton.application.operation.DurableOperationState; import dev.caskeleton.application.operation.DurableOperationStorePort; @@ -26,8 +27,10 @@ import java.util.concurrent.ConcurrentHashMap; *

Written to the same rules rather than to whatever makes the tests pass: a submission scope * that already exists returns the existing operation, a lease is checked on every write, and a * terminal operation is never reopened. A fake that was laxer than the adapter would let the use - * case tests certify behaviour the real store refuses. {@link DurableOperationStoreContract} holds - * this fake and the adapter to the same cases. + * case tests certify behaviour the real store refuses. Every claim increments the operation's + * fence, and a write is accepted only from the claim holding the current fence, as the adapter's + * {@code lease_fence} column does. {@link DurableOperationStoreContract} holds this fake and the + * adapter to the same cases. * *

Built over a {@link DirectTransactionPort}, every write made inside one of its transactions is * undone when that transaction rolls back, as the database undoes it; built without one, writes @@ -41,6 +44,7 @@ public final class InMemoryDurableOperationStore implements DurableOperationStor private final Map byScope = new ConcurrentHashMap<>(); private final Map submissions = new ConcurrentHashMap<>(); + private final Map fences = new ConcurrentHashMap<>(); private final DirectTransactionPort transactions; /** A store whose writes stay as they are made. */ @@ -103,7 +107,7 @@ public final class InMemoryDurableOperationStore implements DurableOperationStor } @Override - public Optional claimNext( + public Optional claimNext( String workerId, Duration leaseDuration, Instant now) { return byId.values().stream() .filter( @@ -126,19 +130,22 @@ public final class InMemoryDurableOperationStore implements DurableOperationStor Optional.empty(), Optional.of(new OperationLease(workerId, now, now.plus(leaseDuration)))); write(claimed.operationId(), claimed); - return submissions.get(claimed.operationId()); + long previous = fences.getOrDefault(claimed.operationId(), 0L); + fences.put(claimed.operationId(), previous + 1); + onRollback(() -> fences.put(claimed.operationId(), previous)); + return new DurableOperationClaim( + submissions.get(claimed.operationId()), workerId, previous + 1); }); } @Override - public boolean heartbeat( - DurableOperationId operationId, String workerId, Duration leaseDuration, Instant now) { - DurableOperation operation = byId.get(operationId); - if (!holdsLease(operation, workerId, now)) { + public boolean heartbeat(DurableOperationClaim claim, Duration leaseDuration, Instant now) { + DurableOperation operation = byId.get(claim.operationId()); + if (!holdsLease(operation, claim, now)) { return false; } write( - operationId, + claim.operationId(), replace( operation, operation.state(), @@ -152,14 +159,13 @@ public final class InMemoryDurableOperationStore implements DurableOperationStor } @Override - public boolean reportProgress( - DurableOperationId operationId, String workerId, OperationProgressSnapshot progress) { - DurableOperation operation = byId.get(operationId); - if (operation == null || !ownedBy(operation, workerId)) { + public boolean reportProgress(DurableOperationClaim claim, OperationProgressSnapshot progress) { + DurableOperation operation = byId.get(claim.operationId()); + if (!heldBy(operation, claim)) { return false; } write( - operationId, + claim.operationId(), replace( operation, operation.state(), @@ -173,17 +179,13 @@ public final class InMemoryDurableOperationStore implements DurableOperationStor } @Override - public boolean succeed( - DurableOperationId operationId, - String workerId, - String resultReference, - Instant completedAt) { - DurableOperation operation = byId.get(operationId); - if (operation == null || !ownedBy(operation, workerId)) { + public boolean succeed(DurableOperationClaim claim, String resultReference, Instant completedAt) { + DurableOperation operation = byId.get(claim.operationId()); + if (!heldBy(operation, claim)) { return false; } write( - operationId, + claim.operationId(), replace( operation, DurableOperationState.SUCCEEDED, @@ -197,17 +199,13 @@ public final class InMemoryDurableOperationStore implements DurableOperationStor } @Override - public boolean fail( - DurableOperationId operationId, - String workerId, - OperationFailure failure, - Instant completedAt) { - DurableOperation operation = byId.get(operationId); - if (operation == null || !ownedBy(operation, workerId)) { + public boolean fail(DurableOperationClaim claim, OperationFailure failure, Instant completedAt) { + DurableOperation operation = byId.get(claim.operationId()); + if (!heldBy(operation, claim)) { return false; } write( - operationId, + claim.operationId(), replace( operation, DurableOperationState.FAILED, @@ -316,6 +314,7 @@ public final class InMemoryDurableOperationStore implements DurableOperationStor byId.clear(); byScope.clear(); submissions.clear(); + fences.clear(); } // Same as the adapter: a poll after the TTL sees EXPIRED even before the sweeper runs. @@ -334,13 +333,15 @@ public final class InMemoryDurableOperationStore implements DurableOperationStor Optional.empty()); } - private static boolean holdsLease(DurableOperation operation, String workerId, Instant now) { - return operation != null && ownedBy(operation, workerId) && !operation.leaseExpiredAt(now); + private boolean holdsLease(DurableOperation operation, DurableOperationClaim claim, Instant now) { + return heldBy(operation, claim) && !operation.leaseExpiredAt(now); } - private static boolean ownedBy(DurableOperation operation, String workerId) { - return operation.state() == DurableOperationState.RUNNING - && operation.lease().map(lease -> lease.workerId().equals(workerId)).orElse(false); + private boolean heldBy(DurableOperation operation, DurableOperationClaim claim) { + return operation != null + && operation.state() == DurableOperationState.RUNNING + && operation.lease().map(lease -> lease.workerId().equals(claim.workerId())).orElse(false) + && fences.getOrDefault(claim.operationId(), 0L) == claim.fence(); } private static DurableOperation replace( diff --git a/src/application/fileserver/src/main/java/dev/caskeleton/application/fileserver/cleanup/CleanupQueuePort.java b/src/application/fileserver/src/main/java/dev/caskeleton/application/fileserver/cleanup/CleanupQueuePort.java index 4577fec0..ad90e67d 100644 --- a/src/application/fileserver/src/main/java/dev/caskeleton/application/fileserver/cleanup/CleanupQueuePort.java +++ b/src/application/fileserver/src/main/java/dev/caskeleton/application/fileserver/cleanup/CleanupQueuePort.java @@ -44,6 +44,16 @@ public interface CleanupQueuePort { /** Records a failed attempt and when to try again, if {@code item}'s claim is still current. */ void markFailed(CleanupItem item, String reasonCode, Instant nextAttemptAt); + /** + * Gives a claimed item back unattempted, if {@code item}'s claim is still current. + * + *

For an item the batch claimed and then did not start — its byte budget or its claim's lease + * ran out first. The claim ends and the item is claimable again at {@code nextAttemptAt}; the + * attempt counter does not move, because nothing was tried. Recorded as a failure instead, every + * such release spent an attempt, and a busy queue's tail was abandoned without ever being tried. + */ + void release(CleanupItem item, Instant nextAttemptAt); + /** * Parks an item for good, if {@code item}'s claim is still current. * diff --git a/src/application/fileserver/src/main/java/dev/caskeleton/application/fileserver/cleanup/DefaultCleanupService.java b/src/application/fileserver/src/main/java/dev/caskeleton/application/fileserver/cleanup/DefaultCleanupService.java index d28a7d40..71033678 100644 --- a/src/application/fileserver/src/main/java/dev/caskeleton/application/fileserver/cleanup/DefaultCleanupService.java +++ b/src/application/fileserver/src/main/java/dev/caskeleton/application/fileserver/cleanup/DefaultCleanupService.java @@ -49,6 +49,20 @@ public final class DefaultCleanupService implements CleanupService { */ public static final Duration CLAIM_LEASE = Duration.ofMinutes(10); + /** + * No claimed item is started once less than this is left of the batch's claim lease. + * + *

A batch claims up to {@code maxItems} under one lease and works them one after another, so + * the last item starts long after the claim. Past the lease, the reaper may already have handed + * that item to another worker, and both delete it and settle its quota. The outbox relay checks + * its batch against its lease when it is assembled and the notification scheduler claims only + * what it can start; cleanup has no bound on one delete to check against, so it stops starting + * items instead, and releases the rest for the next batch the way the byte budget does (round9 + * review 2026-11-12, RR9-01-23). Two minutes is the time one delete against slow storage is + * allowed to finish inside its lease. + */ + static final Duration LAST_START_BEFORE_LEASE_ENDS = Duration.ofMinutes(2); + private final CleanupQueuePort queue; private final CleanupContentPort contentGateway; private final FileMetadataPort metadataStore; @@ -95,6 +109,7 @@ public final class DefaultCleanupService implements CleanupService { // claiming means the recovered items are eligible in this same pass. transactions.inWrite(() -> queue.reclaimExpiredClaims(now, maxItems)); List due = transactions.inWrite(() -> queue.claimDue(now, maxItems, CLAIM_LEASE)); + Instant lastStart = now.plus(CLAIM_LEASE).minus(LAST_START_BEFORE_LEASE_ENDS); int deleted = 0; int skippedActiveLease = 0; @@ -113,10 +128,16 @@ public final class DefaultCleanupService implements CleanupService { failed++; continue; } + if (!clock.instant().isBefore(lastStart)) { + // The claim's lease is nearly spent; an item started now could still be running when the + // reaper hands it to another worker. Given back untried, like the byte budget's. + release(item, now); + continue; + } if (reclaimed >= maxBytes) { - // The byte budget is spent; the remaining claims are released for the next batch rather - // than being marked done, so nothing is silently dropped. - markFailed(item, "BATCH_BYTE_BUDGET_EXHAUSTED", now); + // The byte budget is spent; the remaining claims are given back untried rather than + // being marked done, so nothing is silently dropped and no attempt is spent on them. + release(item, now); continue; } Outcome outcome = process(item, now); @@ -330,6 +351,11 @@ public final class DefaultCleanupService implements CleanupService { }); } + /** Gives a claimed item back untried, in a write transaction of its own. */ + private void release(CleanupItem item, Instant nextAttemptAt) { + transactions.inWrite(() -> queue.release(item, nextAttemptAt)); + } + /** * Reschedules an item, or parks it once this failure spends its retry budget. * diff --git a/src/application/fileserver/src/test/java/dev/caskeleton/application/fileserver/cleanup/CleanupServiceTest.java b/src/application/fileserver/src/test/java/dev/caskeleton/application/fileserver/cleanup/CleanupServiceTest.java index dd0bd88d..ead9afb2 100644 --- a/src/application/fileserver/src/test/java/dev/caskeleton/application/fileserver/cleanup/CleanupServiceTest.java +++ b/src/application/fileserver/src/test/java/dev/caskeleton/application/fileserver/cleanup/CleanupServiceTest.java @@ -26,7 +26,10 @@ import dev.caskeleton.application.fileserver.testkit.ReadyFileFixture; import dev.caskeleton.application.fileserver.testkit.RecordingQuotaReclaimGateway; import java.time.Clock; import java.time.Duration; +import java.time.Instant; +import java.time.ZoneId; import java.time.ZoneOffset; +import java.util.List; import java.util.OptionalLong; import java.util.UUID; import org.junit.jupiter.api.DisplayName; @@ -321,7 +324,91 @@ class CleanupServiceTest { CleanupBatchResult result = cleanup.runBatch(100, 1); assertThat(result.deleted()).isEqualTo(1); - assertThat(queue.failed().values()).contains("BATCH_BYTE_BUDGET_EXHAUSTED"); + assertThat(List.of(queue.released().size(), queue.failed().size())) + .as("the deferred item is given back untried; recorded as failed it spent an attempt") + .containsExactly(1, 0); + } + + @Test + void itemsTheClaimLeaseNoLongerCoversAreReleasedNotStarted() { + // given + // Each staging delete takes just over half the time an item may still start in (the lease less + // its last two minutes): the first two items run, the third is handed back to the queue. + MovableClock slowStorage = new MovableClock(FileserverFixtures.NOW); + Duration perDelete = + DefaultCleanupService.CLAIM_LEASE + .minus(DefaultCleanupService.LAST_START_BEFORE_LEASE_ENDS) + .dividedBy(2) + .plusSeconds(1); + content.afterEachStagingDiscard(() -> slowStorage.advance(perDelete)); + CleanupService timed = + new DefaultCleanupService( + queue, + content, + metadata, + sessions, + quota, + Duration.ofMinutes(5), + FileserverMetricsPort.noop(), + transactions, + slowStorage); + for (int item = 0; item < 3; item++) { + cancelledStagingItem(); + } + + // when + CleanupBatchResult result = timed.runBatch(100, 1L << 30); + + // then + assertThat(List.of(result.deleted(), queue.released().size(), queue.failed().size())) + .as( + "the third item comes back untried rather than failed; that a release leaves the" + + " attempt counter where it was is the queue contract's") + .containsExactly(2, 1, 0); + } + + private void cancelledStagingItem() { + UploadId uploadId = UploadId.of(UUID.randomUUID()); + FileId fileId = FileId.of(UUID.randomUUID()); + sessions.create( + new UploadSessionDraft( + uploadId, + fileId, + UploadMode.SINGLE_SHOT, + OptionalLong.of(SIZE), + FileserverFixtures.NOW.plusSeconds(3600))); + sessions.terminate(uploadId); + content.storeStaging(uploadId); + queue.enqueue(CleanupRequest.forStaging(CleanupType.CANCELLED_STAGING, fileId, uploadId)); + } + + /** A clock the slow-storage gateway moves forward. */ + private static final class MovableClock extends Clock { + + private Instant now; + + private MovableClock(Instant start) { + this.now = start; + } + + private void advance(Duration duration) { + now = now.plus(duration); + } + + @Override + public ZoneId getZone() { + return ZoneOffset.UTC; + } + + @Override + public Clock withZone(ZoneId zone) { + return this; + } + + @Override + public Instant instant() { + return now; + } } @Test diff --git a/src/application/fileserver/src/testFixtures/java/dev/caskeleton/application/fileserver/testkit/CleanupQueuePortContract.java b/src/application/fileserver/src/testFixtures/java/dev/caskeleton/application/fileserver/testkit/CleanupQueuePortContract.java index 0db36011..b9cc169b 100644 --- a/src/application/fileserver/src/testFixtures/java/dev/caskeleton/application/fileserver/testkit/CleanupQueuePortContract.java +++ b/src/application/fileserver/src/testFixtures/java/dev/caskeleton/application/fileserver/testkit/CleanupQueuePortContract.java @@ -119,6 +119,40 @@ public abstract class CleanupQueuePortContract { }); } + @Test + void aReleasedItemIsClaimableAgainWithItsAttemptUnchanged() { + // given + enqueue(contentCleanup()); + CleanupItem item = claim(due(), 10).get(0); + settle(() -> queue().release(item, due())); + + // when + List again = claim(due(), 10); + + // then + assertThat(again) + .singleElement() + .extracting(CleanupItem::cleanupId, CleanupItem::attempt) + .containsExactly(item.cleanupId(), 0); + } + + @Test + void aReleaseUnderAReclaimedTokenChangesNothing() { + // given + enqueue(contentCleanup()); + CleanupItem stale = claim(due(), 10).get(0); + settle(() -> queue().release(stale, due())); + var unused = claim(due(), 10); + + // when + settle(() -> queue().release(stale, due())); + + // then + assertThat(claim(due(), 10)) + .as("the stale release would hand the current claim's item to a second worker") + .isEmpty(); + } + @Test @DisplayName("완료된 항목은 다시 claim되지 않는다") void aDoneItemIsNeverClaimedAgain() { diff --git a/src/application/fileserver/src/testFixtures/java/dev/caskeleton/application/fileserver/testkit/FakeCleanupContentGateway.java b/src/application/fileserver/src/testFixtures/java/dev/caskeleton/application/fileserver/testkit/FakeCleanupContentGateway.java index 8a485a3c..3b842068 100644 --- a/src/application/fileserver/src/testFixtures/java/dev/caskeleton/application/fileserver/testkit/FakeCleanupContentGateway.java +++ b/src/application/fileserver/src/testFixtures/java/dev/caskeleton/application/fileserver/testkit/FakeCleanupContentGateway.java @@ -7,6 +7,7 @@ import dev.caskeleton.application.fileserver.api.content.DeleteResult; import dev.caskeleton.application.fileserver.cleanup.CleanupContentPort; import java.util.LinkedHashSet; import java.util.List; +import java.util.Objects; import java.util.Set; /** Tracks which physical objects still exist so a test can prove nothing live was removed. */ @@ -16,6 +17,7 @@ public final class FakeCleanupContentGateway implements CleanupContentPort { private final Set stagingObjects = new LinkedHashSet<>(); private final Set quarantined = new LinkedHashSet<>(); private final List preconditions = new java.util.ArrayList<>(); + private Runnable afterStagingDiscard = () -> {}; public void store(ContentKey key) { objects.add(key); @@ -46,9 +48,20 @@ public final class FakeCleanupContentGateway implements CleanupContentPort { return DeleteResult.removed(precondition.expectedSize().orElse(0)); } + /** + * Runs {@code hook} after every staging discard, for a test that needs the storage to be slow — + * advancing its clock — or to fail mid-batch. + * + * @param hook what a discard does after removing the staging object + */ + public void afterEachStagingDiscard(Runnable hook) { + afterStagingDiscard = Objects.requireNonNull(hook, "hook"); + } + @Override public void discardStaging(UploadId uploadId) { stagingObjects.remove(uploadId); + afterStagingDiscard.run(); } public void quarantine(ContentKey key) { diff --git a/src/application/fileserver/src/testFixtures/java/dev/caskeleton/application/fileserver/testkit/InMemoryCleanupQueue.java b/src/application/fileserver/src/testFixtures/java/dev/caskeleton/application/fileserver/testkit/InMemoryCleanupQueue.java index 50edd600..0dc81d84 100644 --- a/src/application/fileserver/src/testFixtures/java/dev/caskeleton/application/fileserver/testkit/InMemoryCleanupQueue.java +++ b/src/application/fileserver/src/testFixtures/java/dev/caskeleton/application/fileserver/testkit/InMemoryCleanupQueue.java @@ -35,6 +35,7 @@ public final class InMemoryCleanupQueue implements CleanupQueuePort { private final Map rows = new LinkedHashMap<>(); private final List done = new ArrayList<>(); private final Map failed = new LinkedHashMap<>(); + private final List released = new ArrayList<>(); private final Map abandoned = new LinkedHashMap<>(); private final List reclaimSweeps = new ArrayList<>(); private final List claimSweeps = new ArrayList<>(); @@ -108,6 +109,15 @@ public final class InMemoryCleanupQueue implements CleanupQueuePort { } } + @Override + public void release(CleanupItem item, Instant nextAttemptAt) { + Row row = rows.get(item.cleanupId()); + if (row != null && item.claimToken().equals(row.claimToken)) { + row.release(nextAttemptAt); + released.add(item); + } + } + @Override public void markAbandoned(CleanupItem item, String reasonCode) { if (settle(item, Status.ABANDONED, clock.instant())) { @@ -151,6 +161,11 @@ public final class InMemoryCleanupQueue implements CleanupQueuePort { return Map.copyOf(failed); } + /** Items the caller gave back untried, in order. */ + public List released() { + return List.copyOf(released); + } + /** Items the caller parked for good, with the reason given. */ public Map abandoned() { return Map.copyOf(abandoned); @@ -180,6 +195,13 @@ public final class InMemoryCleanupQueue implements CleanupQueuePort { this.nextAttemptAt = nextAttemptAt; } + private void release(Instant nextAttemptAt) { + this.status = attempt == 0 ? Status.PENDING : Status.FAILED; + this.nextAttemptAt = nextAttemptAt; + this.claimToken = null; + this.leaseUntil = null; + } + private void settle(Status status, Instant nextAttemptAt) { this.status = status; this.attempt++; diff --git a/src/application/notification/src/main/java/dev/caskeleton/application/notification/platform/admin/AdminOperationClaim.java b/src/application/notification/src/main/java/dev/caskeleton/application/notification/platform/admin/AdminOperationClaim.java index bd902930..40006855 100644 --- a/src/application/notification/src/main/java/dev/caskeleton/application/notification/platform/admin/AdminOperationClaim.java +++ b/src/application/notification/src/main/java/dev/caskeleton/application/notification/platform/admin/AdminOperationClaim.java @@ -11,6 +11,10 @@ import java.util.Optional; * the work is done; and the same id with a different command is a mistake worth reporting rather * than an idempotent repeat. * + *

A store builds it from a row, never from a request, so a value it refuses is a server fault: + * {@link IllegalStateException}, not the {@code IllegalArgumentException} the web layer would + * answer with a 400 (manual 06). + * * @param outcome what the caller may do * @param existing the earlier result, when there is one */ @@ -37,7 +41,7 @@ public record AdminOperationClaim(Outcome outcome, OptionalA requeue is one write conditional on the abandoned lease recovery found (owner, fence, state, + * expiry), not a read followed by an unconditional transition. The pass runs on every instance, and + * between finding a job and writing it another instance's recovery may have requeued it and a new + * worker claimed it, or the holder may have finished; the unconditional write put such a job back + * to {@code READY_TO_DISPATCH} and it was dispatched again (round9 review 2026-11-12, RR9-01-12). A + * requeue that matches nothing is not counted as handled. + * *

Every write opens its own transaction; the pass itself runs on a bare scheduler thread. The * stores behind these ports are modifying JPA queries, which throw outside a transaction — so the * pass used to fail on its first statement, every time, and the scheduler only logged it. It lives @@ -110,14 +116,14 @@ public final class LeaseRecoveryService { // Retiring expired deliveries rides along with recovery because both answer the same operator // question — "why is this row still here?" — and neither needs a scheduler of its own. int handled = transactions.inWrite(() -> leases.expireOverdue(batchSize)); - List abandoned = leases.expiredDispatching(batchSize, staleAfter); - for (var recipientDeliveryId : abandoned) { + List abandoned = leases.expiredDispatching(batchSize, staleAfter); + for (var lease : abandoned) { + RecipientDeliveryId recipientDeliveryId = lease.recipientDeliveryId(); var attemptsOfDelivery = attempts.attemptsOf(recipientDeliveryId); if (attemptsOfDelivery.isEmpty()) { // Crash before the attempt row: no provider call can have happened, so this is the one // safe requeue. Leaving it DISPATCHING stranded the delivery permanently. - requeue(recipientDeliveryId); - handled++; + handled += requeue(lease); continue; } if (attemptsOfDelivery.stream().allMatch(attempt -> attempt.completedAt().isPresent())) { @@ -127,8 +133,7 @@ public final class LeaseRecoveryService { // the delivery, and dispatch derives the next step from exactly those facts — an accepted // submission stops as COMPLETED, an ambiguous one goes to reconciliation — so handing it // back to dispatch cannot send a second copy of something the provider accepted. - requeue(recipientDeliveryId); - handled++; + handled += requeue(lease); continue; } for (var attempt : attemptsOfDelivery) { @@ -141,25 +146,25 @@ public final class LeaseRecoveryService { // identically, so every abandoned attempt had to be reconciled. The pre-call row the // dispatcher writes never lands here: it records the start as unknown, because it is // still the stored state while the provider call runs. - requeue(recipientDeliveryId); + handled += requeue(lease); } else { // Outside any transaction: reconciliation asks the provider over the network and opens // its own boundaries for what it writes. reconciliation.reconcile(attempt.id()); + handled++; } - handled++; } } } return handled; } - private void requeue(RecipientDeliveryId recipientDeliveryId) { - transactions.inWrite( - () -> - deliveries.transition( - recipientDeliveryId, - RecipientDeliveryState.READY_TO_DISPATCH, - java.util.Optional.of(clock.instant()))); + /** Requeues the abandoned dispatch; 1 when it was still abandoned and moved, 0 otherwise. */ + private int requeue(RecipientLease abandoned) { + return transactions + .inWrite(() -> deliveries.requeueAbandoned(abandoned, clock.instant())) + .isPresent() + ? 1 + : 0; } } diff --git a/src/application/notification/src/main/java/dev/caskeleton/application/notification/platform/dispatch/RecipientDeliveryStorePort.java b/src/application/notification/src/main/java/dev/caskeleton/application/notification/platform/dispatch/RecipientDeliveryStorePort.java index 101c9f35..c82fa5aa 100644 --- a/src/application/notification/src/main/java/dev/caskeleton/application/notification/platform/dispatch/RecipientDeliveryStorePort.java +++ b/src/application/notification/src/main/java/dev/caskeleton/application/notification/platform/dispatch/RecipientDeliveryStorePort.java @@ -49,4 +49,28 @@ public interface RecipientDeliveryStorePort { RecipientDeliveryState state, Optional nextDispatchAt, RecipientLease lease); + + /** + * Returns a job a dead worker left dispatching to the queue, only while it is still the abandoned + * dispatch recovery found. + * + *

One conditional write on the abandoned lease — same owner, same fence, still {@code + * DISPATCHING}, lease not extended past what recovery saw — rather than a read followed by an + * unconditional transition. Recovery runs on every instance, and between finding the job and + * writing it another instance may have requeued it and a new worker claimed it, or the original + * holder may have finished; an unconditional write put such a job back to {@code + * READY_TO_DISPATCH} and it was dispatched again. + * + *

The target state is the operation's meaning, as {@code EXPIRED} is {@code + * RecipientLeaseStorePort.expireOverdue}'s, so it is not an argument. What recovery decides is + * which operation applies: this one when no attempt may have reached the provider, the + * reconciliation path otherwise. + * + * @param abandoned the lease as {@link RecipientLeaseStorePort#expiredDispatching} returned it + * @param nextDispatchAt when the job becomes due again + * @return the requeued job, or empty when it is no longer that abandoned dispatch and nothing was + * written + */ + Optional requeueAbandoned( + RecipientLease abandoned, Instant nextDispatchAt); } diff --git a/src/application/notification/src/main/java/dev/caskeleton/application/notification/platform/dispatch/RecipientLease.java b/src/application/notification/src/main/java/dev/caskeleton/application/notification/platform/dispatch/RecipientLease.java index 0b9962a3..5660e4d3 100644 --- a/src/application/notification/src/main/java/dev/caskeleton/application/notification/platform/dispatch/RecipientLease.java +++ b/src/application/notification/src/main/java/dev/caskeleton/application/notification/platform/dispatch/RecipientLease.java @@ -16,6 +16,10 @@ import java.util.Objects; * that no longer exists and updates nothing. It travels with the lease rather than being re-read, * because re-reading is the race. * + *

A store builds it from a row, never from a request, so a value it refuses is a server fault: + * {@link IllegalStateException}, not the {@code IllegalArgumentException} the web layer would + * answer with a 400 (manual 06). + * * @param recipientDeliveryId the job this lease is for * @param owner the holder, including the process incarnation that claimed it * @param fence the claim generation; a lower fence has been superseded @@ -29,11 +33,11 @@ public record RecipientLease( Objects.requireNonNull(owner, "owner"); Objects.requireNonNull(leaseUntil, "leaseUntil"); if (owner.isBlank()) { - throw new IllegalArgumentException("owner"); + throw new IllegalStateException("a lease needs an owner"); } if (fence < 1) { // A zero fence is the value an unclaimed row holds; a lease can never legitimately carry it. - throw new IllegalArgumentException("fence"); + throw new IllegalStateException("fence must be >= 1, was " + fence); } } } diff --git a/src/application/notification/src/main/java/dev/caskeleton/application/notification/platform/dispatch/RecipientLeaseStorePort.java b/src/application/notification/src/main/java/dev/caskeleton/application/notification/platform/dispatch/RecipientLeaseStorePort.java index 634fabd7..da251a5d 100644 --- a/src/application/notification/src/main/java/dev/caskeleton/application/notification/platform/dispatch/RecipientLeaseStorePort.java +++ b/src/application/notification/src/main/java/dev/caskeleton/application/notification/platform/dispatch/RecipientLeaseStorePort.java @@ -1,6 +1,5 @@ package dev.caskeleton.application.notification.platform.dispatch; -import dev.caskeleton.application.notification.platform.api.RecipientDeliveryId; import java.time.Duration; import java.util.List; import java.util.Optional; @@ -25,6 +24,9 @@ public interface RecipientLeaseStorePort { /** * Extend a lease that is still being worked on. * + *

The template's dispatcher never calls it: a dispatch runs inside the lease it claimed. It is + * here for a deployment whose own dispatch outlasts one lease. + * * @return the extended lease, or empty when this holder no longer owns the job */ Optional renew(RecipientLease lease, Duration leaseDuration); @@ -40,8 +42,24 @@ public interface RecipientLeaseStorePort { /** Whether this lease is still the current one, for checking before an external side effect. */ boolean stillHeld(RecipientLease lease); - /** Jobs left {@code DISPATCHING} by a worker that died, for reconciliation. */ - List expiredDispatching(int limit, Duration olderThan); + /** + * Jobs left {@code DISPATCHING} by a worker that died, for recovery. + * + *

Each comes back as the lease it was abandoned under — owner, fence and expiry as they stood + * when it was found. Recovery decides what to do from attempts it reads afterwards, and the lease + * is what lets its write say "only if this is still that abandoned dispatch": another instance's + * recovery, a new claim or the holder itself may have moved the job in between. + * + *

Only jobs a claim leased: an owner and a fence of at least one. A {@code DISPATCHING} job + * without them is no lease anyone holds — one left before the fence existed, or one moved there + * without a claim — and is not returned; offered, it would be a lease nobody can build, and the + * pass would fail on it every time. + * + * @param limit how many to return + * @param olderThan how long a lease must have been expired before the job counts as abandoned + * @return the abandoned leases, oldest expiry first + */ + List expiredDispatching(int limit, Duration olderThan); /** * Retire deliveries whose window closed before anyone claimed them. diff --git a/src/application/notification/src/main/java/dev/caskeleton/application/notification/platform/dispatch/ReconciliationJob.java b/src/application/notification/src/main/java/dev/caskeleton/application/notification/platform/dispatch/ReconciliationJob.java index c8545b62..e0d3e401 100644 --- a/src/application/notification/src/main/java/dev/caskeleton/application/notification/platform/dispatch/ReconciliationJob.java +++ b/src/application/notification/src/main/java/dev/caskeleton/application/notification/platform/dispatch/ReconciliationJob.java @@ -24,6 +24,10 @@ import java.util.UUID; * the store's writes after the claim match on it, so an instance that held the job past its lease * cannot reschedule or remove it once another instance has claimed it since. * + *

A store builds it from a row, never from a request, so a value it refuses is a server fault: + * {@link IllegalStateException}, not the {@code IllegalArgumentException} the web layer would + * answer with a 400 (manual 06). + * * @param id the job identity * @param attemptId the attempt whose outcome is unknown * @param providerProfileId the profile that must answer, since capability is per profile @@ -55,10 +59,10 @@ public record ReconciliationJob( Objects.requireNonNull(createdAt, "createdAt"); Objects.requireNonNull(updatedAt, "updatedAt"); if (attempts < 0) { - throw new IllegalArgumentException("attempts must not be negative"); + throw new IllegalStateException("attempts must not be negative"); } if (claimFence < 0) { - throw new IllegalArgumentException("claimFence must not be negative"); + throw new IllegalStateException("claimFence must not be negative"); } } } diff --git a/src/application/notification/src/test/java/dev/caskeleton/application/notification/platform/dispatch/LeaseRecoveryServiceTest.java b/src/application/notification/src/test/java/dev/caskeleton/application/notification/platform/dispatch/LeaseRecoveryServiceTest.java index 7ff29358..ed508150 100644 --- a/src/application/notification/src/test/java/dev/caskeleton/application/notification/platform/dispatch/LeaseRecoveryServiceTest.java +++ b/src/application/notification/src/test/java/dev/caskeleton/application/notification/platform/dispatch/LeaseRecoveryServiceTest.java @@ -186,6 +186,40 @@ class LeaseRecoveryServiceTest { assertThat(recovery.recoverOnce()).isEqualTo(3); } + @Test + void aDeliveryItsHolderFinishedAfterRecoveryFoundItIsNotRequeued() { + // given + RecipientDeliveryId id = jobOf(NOW, Optional.empty()); + RecipientLease lease = queue.claim("worker-a", 10, Duration.ofSeconds(30)).getFirst(); + workerDied(); + // The holder completes the job between recovery finding it abandoned and recovery writing it + // — the window another instance or a slow worker uses in production. + attempts.beforeAttemptsOf( + delivery -> { + var unused = + queue.transitionHeldBy( + delivery, RecipientDeliveryState.COMPLETED, Optional.empty(), lease); + }); + LeaseRecoveryService racing = + new LeaseRecoveryService( + queue, + attempts, + queue, + reconciled::add, + new DirectTransactionPort(), + CLOCK, + Duration.ofMinutes(1), + 10); + + // when + racing.recoverOnce(); + + // then + assertThat(stateOf(id)) + .as("the holder was only slow; recovery's requeue must not reopen what it finished") + .isEqualTo(RecipientDeliveryState.COMPLETED); + } + private static dev.caskeleton.application.notification.platform.provider.ProviderExecutionEvidence notStarted() { return dev.caskeleton.application.notification.platform.provider.ProviderExecutionEvidence diff --git a/src/application/notification/src/test/java/dev/caskeleton/application/notification/platform/dispatch/StoreBuiltRecordsTest.java b/src/application/notification/src/test/java/dev/caskeleton/application/notification/platform/dispatch/StoreBuiltRecordsTest.java new file mode 100644 index 00000000..dd33283f --- /dev/null +++ b/src/application/notification/src/test/java/dev/caskeleton/application/notification/platform/dispatch/StoreBuiltRecordsTest.java @@ -0,0 +1,76 @@ +package dev.caskeleton.application.notification.platform.dispatch; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.catchThrowable; + +import dev.caskeleton.application.notification.platform.admin.AdminOperationClaim; +import dev.caskeleton.application.notification.platform.api.DeliveryAttemptId; +import dev.caskeleton.application.notification.platform.api.ProviderProfileId; +import dev.caskeleton.application.notification.platform.api.RecipientDeliveryId; +import java.time.Instant; +import java.util.Optional; +import java.util.UUID; +import java.util.stream.Stream; +import org.assertj.core.api.ThrowableAssert.ThrowingCallable; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.Arguments; +import org.junit.jupiter.params.provider.MethodSource; + +/** + * The leases, jobs and claims a notification store builds from its rows refuse a bad value as each + * record says. + */ +class StoreBuiltRecordsTest { + + private static final Instant NOW = Instant.parse("2026-11-13T09:00:00Z"); + + static Stream refusedValues() { + RecipientDeliveryId delivery = new RecipientDeliveryId(UUID.randomUUID()); + return Stream.of( + Arguments.of( + "owner 없는 수신자 lease", + (ThrowingCallable) () -> new RecipientLease(delivery, " ", 1, NOW), + "a lease needs an owner"), + Arguments.of( + "fence 0인 수신자 lease", + (ThrowingCallable) () -> new RecipientLease(delivery, "worker-1", 0, NOW), + "fence must be >= 1, was 0"), + Arguments.of( + "시도 수가 음수인 맞추기 job", + (ThrowingCallable) () -> job(-1, 0), + "attempts must not be negative"), + Arguments.of( + "fence가 음수인 맞추기 job", + (ThrowingCallable) () -> job(0, -1), + "claimFence must not be negative"), + Arguments.of( + "결과 없는 운영자 재생", + (ThrowingCallable) + () -> new AdminOperationClaim(AdminOperationClaim.Outcome.REPLAY, Optional.empty()), + "a replay must carry the result it is replaying")); + } + + @ParameterizedTest(name = "{0}") + @MethodSource("refusedValues") + void aValueAStoreCannotHaveBuiltIsAServerFault( + String name, ThrowingCallable build, String message) { + // when + Throwable refused = catchThrowable(build); + + // then + assertThat(refused).isInstanceOf(IllegalStateException.class).hasMessageContaining(message); + } + + private static ReconciliationJob job(int attempts, long claimFence) { + return new ReconciliationJob( + UUID.randomUUID(), + new DeliveryAttemptId(UUID.randomUUID()), + new ProviderProfileId("email-main"), + NOW, + attempts, + Optional.empty(), + NOW, + NOW, + claimFence); + } +} diff --git a/src/application/notification/src/testFixtures/java/dev/caskeleton/application/notification/platform/dispatch/testkit/InMemoryDeliveryAttemptStore.java b/src/application/notification/src/testFixtures/java/dev/caskeleton/application/notification/platform/dispatch/testkit/InMemoryDeliveryAttemptStore.java index ae257085..65bc4d4d 100644 --- a/src/application/notification/src/testFixtures/java/dev/caskeleton/application/notification/platform/dispatch/testkit/InMemoryDeliveryAttemptStore.java +++ b/src/application/notification/src/testFixtures/java/dev/caskeleton/application/notification/platform/dispatch/testkit/InMemoryDeliveryAttemptStore.java @@ -24,6 +24,7 @@ import java.util.Map; import java.util.Objects; import java.util.Optional; import java.util.Set; +import java.util.function.Consumer; /** * The fake of the attempt store and the two read models built on it, held to {@link @@ -47,6 +48,8 @@ public final class InMemoryDeliveryAttemptStore /** Stored attempts, in insertion order, as their rows read back. */ private final Map rows = new LinkedHashMap<>(); + private volatile Consumer beforeAttemptsOf = id -> {}; + private final Map requestIdHashes = new LinkedHashMap<>(); private final Map engagement = new LinkedHashMap<>(); private final Map suppression = new LinkedHashMap<>(); @@ -140,7 +143,23 @@ public final class InMemoryDeliveryAttemptStore } @Override - public synchronized List attemptsOf( + public List attemptsOf(RecipientDeliveryId recipientDeliveryId) { + // Outside the monitor: the hook writes to the recipient queue, as another worker would. + beforeAttemptsOf.accept(recipientDeliveryId); + return attemptsFor(recipientDeliveryId); + } + + /** + * Runs {@code hook} before every {@link #attemptsOf} read, for a test that needs another worker + * to act between a caller finding a delivery and reading its attempts. + * + * @param hook what happens, given the delivery being read + */ + public void beforeAttemptsOf(Consumer hook) { + beforeAttemptsOf = Objects.requireNonNull(hook, "hook"); + } + + private synchronized List attemptsFor( RecipientDeliveryId recipientDeliveryId) { return rows.values().stream() .filter(row -> row.recipientDeliveryId().equals(recipientDeliveryId)) @@ -155,7 +174,7 @@ public final class InMemoryDeliveryAttemptStore @Override public synchronized int nextAttemptNo(RecipientDeliveryId recipientDeliveryId) { - return attemptsOf(recipientDeliveryId).stream() + return attemptsFor(recipientDeliveryId).stream() .mapToInt(DeliveryAttemptRecord::attemptNo) .max() .orElse(0) diff --git a/src/application/notification/src/testFixtures/java/dev/caskeleton/application/notification/platform/dispatch/testkit/InMemoryRecipientQueue.java b/src/application/notification/src/testFixtures/java/dev/caskeleton/application/notification/platform/dispatch/testkit/InMemoryRecipientQueue.java index 89517a47..7714a522 100644 --- a/src/application/notification/src/testFixtures/java/dev/caskeleton/application/notification/platform/dispatch/testkit/InMemoryRecipientQueue.java +++ b/src/application/notification/src/testFixtures/java/dev/caskeleton/application/notification/platform/dispatch/testkit/InMemoryRecipientQueue.java @@ -160,7 +160,7 @@ public final class InMemoryRecipientQueue } @Override - public List expiredDispatching(int limit, Duration olderThan) { + public List expiredDispatching(int limit, Duration olderThan) { Objects.requireNonNull(olderThan, "olderThan"); if (limit < 1) { throw new IllegalArgumentException("limit"); @@ -170,9 +170,18 @@ public final class InMemoryRecipientQueue return jobs() .filter(job -> job.state() == RecipientDeliveryState.DISPATCHING) .filter(job -> job.leaseUntil().filter(until -> until.isBefore(threshold)).isPresent()) + // Only what a claim wrote: an owner and a fence. The JPA store skips the rest the same + // way. + .filter(job -> job.leaseOwner().isPresent() && fences.getOrDefault(job.id(), 0L) > 0) .sorted(Comparator.comparing((RecipientDeliveryRecord job) -> job.leaseUntil().get())) .limit(limit) - .map(RecipientDeliveryRecord::id) + .map( + job -> + new RecipientLease( + job.id(), + job.leaseOwner().orElseThrow(), + fences.getOrDefault(job.id(), 0L), + job.leaseUntil().orElseThrow())) .toList(); } } @@ -294,6 +303,35 @@ public final class InMemoryRecipientQueue } } + @Override + public Optional requeueAbandoned( + RecipientLease abandoned, Instant nextDispatchAt) { + Objects.requireNonNull(abandoned, "abandoned"); + Objects.requireNonNull(nextDispatchAt, "nextDispatchAt"); + synchronized (requests) { + return heldBy(abandoned) + .filter(job -> job.state() == RecipientDeliveryState.DISPATCHING) + .filter( + job -> + job.leaseUntil() + .filter(until -> !until.isAfter(abandoned.leaseUntil())) + .isPresent()) + .map( + job -> { + RecipientDeliveryRecord moved = + copy( + job, + RecipientDeliveryState.READY_TO_DISPATCH, + Optional.of(nextDispatchAt), + job.leaseOwner(), + job.leaseUntil(), + clock.instant()); + requests.replace(moved); + return moved; + }); + } + } + // ---- rows ------------------------------------------------------------------------------------- private Stream jobs() { diff --git a/src/application/notification/src/testFixtures/java/dev/caskeleton/application/notification/platform/dispatch/testkit/RecipientDeliveryStoreContract.java b/src/application/notification/src/testFixtures/java/dev/caskeleton/application/notification/platform/dispatch/testkit/RecipientDeliveryStoreContract.java index 7ceefe82..5ec3dead 100644 --- a/src/application/notification/src/testFixtures/java/dev/caskeleton/application/notification/platform/dispatch/testkit/RecipientDeliveryStoreContract.java +++ b/src/application/notification/src/testFixtures/java/dev/caskeleton/application/notification/platform/dispatch/testkit/RecipientDeliveryStoreContract.java @@ -244,6 +244,84 @@ public abstract class RecipientDeliveryStoreContract { List.of(Optional.empty(), Optional.empty()), RecipientDeliveryState.DISPATCHING, 0); } + @Test + void anAbandonedDispatchIsRequeued() { + // given + RecipientDeliveryRecord job = givenAQueuedJob(); + RecipientLease abandoned = claim(); + clock.moveTo(NOW.plus(LEASE).plusSeconds(1)); + + // when + Optional requeued = + inTransaction(() -> deliveries().requeueAbandoned(abandoned, clock.instant())); + + // then + assertThat( + requeued.map(stored -> List.of(stored.id(), stored.state(), stored.nextDispatchAt()))) + .contains( + List.of( + job.id(), + RecipientDeliveryState.READY_TO_DISPATCH, + Optional.of(NOW.plus(LEASE).plusSeconds(1)))); + } + + @Test + void requeueOfAJobItsHolderFinishedWritesNothing() { + // given + RecipientDeliveryRecord job = givenAQueuedJob(); + RecipientLease lease = claim(); + clock.moveTo(NOW.plus(LEASE).plusSeconds(1)); + inTransaction( + () -> + deliveries() + .transitionHeldBy( + job.id(), RecipientDeliveryState.COMPLETED, Optional.empty(), lease)); + + // when + Optional requeued = + inTransaction(() -> deliveries().requeueAbandoned(lease, clock.instant())); + + // then + assertThat(List.of(requeued, deliveries().find(job.id()).orElseThrow().state())) + .containsExactly(Optional.empty(), RecipientDeliveryState.COMPLETED); + } + + @Test + void requeueOfAJobAnotherClaimTookWritesNothing() { + // given + RecipientDeliveryRecord job = givenAQueuedJob(); + RecipientLease abandoned = claim(); + clock.moveTo(NOW.plus(LEASE).plusSeconds(1)); + inTransaction(() -> deliveries().requeueAbandoned(abandoned, clock.instant())); + claim(); + + // when + Optional requeued = + inTransaction(() -> deliveries().requeueAbandoned(abandoned, clock.instant())); + + // then + assertThat(List.of(requeued, deliveries().find(job.id()).orElseThrow().state())) + .as("the second recovery found the same abandoned lease; the new claim raised the fence") + .containsExactly(Optional.empty(), RecipientDeliveryState.DISPATCHING); + } + + @Test + void requeueOfAJobWhoseLeaseWasRenewedWritesNothing() { + // given + RecipientDeliveryRecord job = givenAQueuedJob(); + RecipientLease found = claim(); + clock.moveTo(NOW.plus(LEASE).minusSeconds(1)); + inTransaction(() -> leases().renew(found, LEASE)); + + // when + Optional requeued = + inTransaction(() -> deliveries().requeueAbandoned(found, clock.instant())); + + // then + assertThat(List.of(requeued, deliveries().find(job.id()).orElseThrow().state())) + .containsExactly(Optional.empty(), RecipientDeliveryState.DISPATCHING); + } + private RecipientDeliveryRecord givenAQueuedJob() { NotificationRequestRecord request = NotificationRows.request(TENANT, "idem-" + UUID.randomUUID(), NOW); diff --git a/src/application/notification/src/testFixtures/java/dev/caskeleton/application/notification/platform/dispatch/testkit/RecipientLeaseStoreContract.java b/src/application/notification/src/testFixtures/java/dev/caskeleton/application/notification/platform/dispatch/testkit/RecipientLeaseStoreContract.java index 0a97a7c1..4ddfb077 100644 --- a/src/application/notification/src/testFixtures/java/dev/caskeleton/application/notification/platform/dispatch/testkit/RecipientLeaseStoreContract.java +++ b/src/application/notification/src/testFixtures/java/dev/caskeleton/application/notification/platform/dispatch/testkit/RecipientLeaseStoreContract.java @@ -278,25 +278,77 @@ public abstract class RecipientLeaseStoreContract { } @Test - @DisplayName("lease가 기준보다 오래 전에 끝난 DISPATCHING 작업만 복구 대상으로 나온다") + @DisplayName("lease가 기준보다 오래 전에 끝난 DISPATCHING 작업만, 버려질 때의 lease로 복구 대상에 나온다") void onlyJobsWhoseLeaseEndedLongEnoughAgoAreAbandoned() { // given - RecipientDeliveryId abandoned = given(RecipientDeliveryState.READY_TO_DISPATCH, NOW, null); - claim("worker-a"); + given(RecipientDeliveryState.READY_TO_DISPATCH, NOW, null); + RecipientLease abandoned = claim("worker-a").getFirst(); clock.moveTo(NOW.plus(LEASE).plusSeconds(30)); given(RecipientDeliveryState.READY_TO_DISPATCH, NOW.plus(LEASE), null); claim("worker-b"); clock.moveTo(NOW.plus(LEASE).plus(Duration.ofMinutes(2))); // when - List expired = leases().expiredDispatching(10, Duration.ofMinutes(1)); + List expired = leases().expiredDispatching(10, Duration.ofMinutes(1)); // then assertThat(expired) - .as("a job whose lease is still running belongs to a live worker") + .as( + "a job whose lease is still running belongs to a live worker; the abandoned one comes" + + " back as the lease it was abandoned under") .containsExactly(abandoned); } + @Test + @DisplayName("claim이 쓰지 않은 DISPATCHING 작업(owner는 있고 fence가 없음)은 복구 대상에 나오지 않는다") + void aDispatchingJobNoClaimWroteIsNotOfferedForRecovery() { + // given + // The shape a row left dispatching before the fence existed has: an owner and a lease expiry + // written by hand, and the fence column's default. + NotificationRequestRecord request = + NotificationRows.request(TENANT, "idem-" + UUID.randomUUID(), clock.instant()); + RecipientDeliveryRecord job = + NotificationRows.job( + request, + RecipientDeliveryState.DISPATCHING, + Optional.of(NOW), + Optional.empty(), + clock.instant()); + RecipientDeliveryRecord leasedByHand = + new RecipientDeliveryRecord( + job.id(), + job.notificationId(), + job.tenantId(), + job.recipientRef(), + job.locale(), + job.timeZone(), + job.routingPlan(), + job.routeCursor(), + job.state(), + job.submissionOutcome(), + job.deliveryOutcome(), + job.evidenceLevel(), + job.ambiguousAttemptExists(), + job.duplicateRisk(), + job.nextDispatchAt(), + job.expiresAt(), + Optional.of("legacy-worker"), + Optional.of(NOW.minus(Duration.ofHours(1))), + job.attemptCount(), + job.lastFailureCategory(), + job.createdAt(), + job.updatedAt()); + inTransaction(() -> requests().insert(request, List.of(leasedByHand))); + + // when + List expired = leases().expiredDispatching(10, Duration.ofMinutes(1)); + + // then + assertThat(expired) + .as("a lease the application cannot build would fail every recovery pass after it") + .isEmpty(); + } + private RecipientDeliveryId given( RecipientDeliveryState state, Instant dueAt, Instant expiresAt) { NotificationRequestRecord request = diff --git a/src/bootstrap/notification-platform/src/main/resources/config/module-defaults/notification.yml b/src/bootstrap/notification-platform/src/main/resources/config/module-defaults/notification.yml index 959da41d..1aaac56e 100644 --- a/src/bootstrap/notification-platform/src/main/resources/config/module-defaults/notification.yml +++ b/src/bootstrap/notification-platform/src/main/resources/config/module-defaults/notification.yml @@ -43,8 +43,9 @@ app: default-locale: ${APP_NOTIFICATION_PLATFORM_DEFAULT_LOCALE:ko-KR} dispatch: claim-batch-size: ${APP_NOTIFICATION_PLATFORM_CLAIM_BATCH_SIZE:50} - # The lease must outlast a provider call plus its timeout, or a live worker's delivery is - # claimed by a second one while the first is still waiting on the provider. + # The lease must outlast a provider call, or a live worker's delivery is claimed by a second + # one while the first is still waiting on the provider. Startup refuses a lease that is not + # longer than every enabled provider profile's timeout. lease-duration: ${APP_NOTIFICATION_PLATFORM_LEASE_DURATION:2m} poll-interval: ${APP_NOTIFICATION_PLATFORM_POLL_INTERVAL:1s} max-global-concurrency: ${APP_NOTIFICATION_PLATFORM_MAX_CONCURRENCY:64}