refactor(application)!: 시스템 쓰기와 운영자 연산을 유스케이스 계약 안으로 옮긴다
pr / done (pull_request) Canceled after 0s

시스템이 부르는 쓰기가 유스케이스 계약 밖에 있다. fileserver 정리·맞추기,
알림 claim·발송·lease 회복·맞추기·provider event replay는 capability도
트랜잭션 선언도 없는 서비스를 worker가 바로 부르고, 그 worker와 조립은
outbound adapter 안에 있다. 알림 운영자 연산 넷은 permission 없이 서비스
하나에 모여 있고 재시도 시각은 상수다. web stream bridge는 outbound가
구현한 replay 포트를 바로 읽는다. fileserver 관리 HTTP 기본값은
controller가 정한다.

고른 방법: 시스템 쓰기는 @SystemInvoked 유스케이스로 나누고 composition
worker 설정이 new로 만든다(bean 아님). 알림 worker와 fileserver 정리 조립은
composition으로 옮기고, 맞추기의 재시도·포기 시점은 유스케이스가 정한다.
운영자 연산은 platform.admin의 유스케이스 클래스 넷과 notification-admin:*
permission이다(port.in 인터페이스는 port.in과 admin의 cycle을 만들어 두지
않는다). replay 읽기는 조회 유스케이스 둘을 거치고 포트는
LiveEventReplayQueryPort로 이름을 바꾼다. 아키텍처 규칙 둘: 사용자 전송은
저장·id 포트를 쥐지 못하고, inbound는 application 서비스·coordinator를
interface로도 쥐지 못한다. 재시도 시각은 설정, 관리 기본값은 command
팩터리다. 재시도 시각의 규칙은 policy의 값 RetryTiming 하나가 갖고 설정은
그 값을 만든다. retry·routing 값들은 서버가 만드는 값이라 잘못된 값을
IllegalStateException으로 거절한다(manual 06). 모듈 기본값이 role notification-admin에 운영자 permission
넷을 준다(fileserver의 ROLE_FILE_ADMIN과 같다). 버린 대안: 서비스에
@UseCaseCapability만 달기(사용자와 시스템이 같은 진입점을 나눠 써서 권한
규칙이 막지 못한다), 운영자 서비스 삭제, 설정에서 policy를 만들어 버리며
검증하기(규칙이 바뀐 policy bean과 상관없이 남는다).

위험·한계: 공개 타입·생성자·bean 이름·아키텍처 규칙이 바뀐다. 파생
서비스가 할 일은 docs/upgrade/2026-11-round9-application.md 1~3·5·6행이다.
claim·fence 커밋 위에 쌓았다.

검증: PR 본문의 desktop 결과.

Refs: RR9-05-06, RR9-05-07, RR9-05-08, RR9-05-10, RR9-05-19
BREAKING CHANGE: NotificationAdminService·NotificationAdminApplicationService가 없어지고 운영자 연산은 platform.admin의 RedriveDeliveryUseCase 등이다(permission notification-admin:*). 알림 worker가 composition:notification-platform으로 옮기고 ReconciliationJobWorker·ProviderEventReplayWorker는 유스케이스가 됐다. MessagingReplayBridge 생성자가 조회 유스케이스를 받고 LiveEventReplayPort는 LiveEventReplayQueryPort다. 사용자 전송이 저장·id 포트나 application 서비스 interface를 쥐면 architectureTest가 실패한다. 운영자 요청 record 이름이 *Request로 바뀌고 DefaultNotificationRetryPolicy 생성자가 RetryTiming을 받는다. RetryBackoff·RetryBudget·RetryContext·RetryDecision.RetryAfter·RouteCandidate·RoutingContext·RoutingDecision·EligibilityResult가 잘못된 값에 IllegalStateException을 던진다.
This commit is contained in:
DongHyeonka
2026-11-13 16:37:03 +09:00
parent e60408e931
commit 2d274a6842
143 changed files with 4251 additions and 1383 deletions
@@ -13,8 +13,10 @@
composition:notification-platform
├─ NotificationApplicationCompositionAutoConfiguration (use case 조립, `enabled` 조건)
│ ├─ NotificationSubmissionConfiguration 접수·예약·취소·조회·템플릿 발행 use case
│ ├─ NotificationDeliveryConfiguration 발송·재조정·lease 회수·ledger 투영·운영자 plane
│ └─ NotificationCallbackConfiguration provider 콜백 수집 + ProviderCallbackIngressBridge
│ ├─ NotificationDeliveryConfiguration 발송·재조정·lease 회수·ledger 투영 서비스
│ ├─ NotificationAdminConfiguration 운영자 use case 넷(redrive·reconcile·suppress·provider 상태)
│ ├─ NotificationCallbackConfiguration provider 콜백 수집 + ProviderCallbackIngressBridge
│ └─ NotificationWorkerConfiguration (SERVING만) 발송 scheduler와 회복·replay·맞추기 worker
├─ config/module-defaults/notification.yml (설정 기본값, support 로더가 읽음)
├─ adapter:outbound:notification provider(SMTP·SES), 저장 외 런타임, NotificationPlatformAutoConfiguration
├─ adapter:outbound:persistence-notification (runtimeOnly) 알림 entity·저장소·Flyway stream
@@ -25,6 +27,11 @@ composition:notification-platform
- 조립 순서: 이 모듈의 root는 `NotificationPlatformAutoConfiguration`(outbound leaf) 뒤에 온다
(`afterName`).
- worker: 두 worker(`NotificationSchedulerWorker`, `NotificationBackgroundWorkers`)는 스레드와 일정만 가진다. 하는 일은
`@SystemInvoked` use case(claim·발송·lease 회복·provider event replay·맞추기)와 backlog 측정 query use case이고, worker
설정이 `new`로 만들어 넘긴다.
bean이 아니어서 어떤 전송도 닿지 못한다(manual 07 「시스템이 부르는 쓰기」). `mode=INGEST_ONLY`면 worker 설정 자체가
실리지 않는다.
- 설정: `config/module-defaults/notification.yml`을 support의 `ModuleDefaultsEnvironmentPostProcessor`가 읽는다. 우선순위는 가장 낮다. 그래서
`application*.yml`, 환경 변수, 시스템 속성이 항상 이긴다. app-bootstrap은 이 파일을 모른다.
- 저장소: 요청·발송·ledger 테이블과 migration은 `adapter:outbound:persistence-notification`에 있고,
@@ -27,6 +27,8 @@ dependencies {
implementation 'org.springframework.boot:spring-boot-autoconfigure'
implementation 'org.springframework:spring-context'
// The two worker threads log a failed pass instead of letting it cancel the schedule.
implementation 'org.slf4j:slf4j-api'
testImplementation 'org.springframework.boot:spring-boot-starter-test'
// NotificationModuleDefaultsTest reads the shipped file through the loader that serves it at
@@ -151,7 +151,7 @@ org.reactivestreams:reactive-streams:1.0.4=runtimeClasspath,testRuntimeClasspath
org.reflections:reflections:0.10.2=checkstyle
org.skyscreamer:jsonassert:1.5.3=testCompileClasspath,testRuntimeClasspath
org.slf4j:jul-to-slf4j:2.0.20=runtimeClasspath,testCompileClasspath,testRuntimeClasspath
org.slf4j:slf4j-api:2.0.20=runtimeClasspath,spotbugs,spotbugsSlf4j,testCompileClasspath,testRuntimeClasspath
org.slf4j:slf4j-api:2.0.20=compileClasspath,runtimeClasspath,spotbugs,spotbugsSlf4j,testCompileClasspath,testRuntimeClasspath
org.slf4j:slf4j-simple:2.0.17=spotbugsSlf4j
org.slf4j:slf4j-simple:2.0.20=checkstyle
org.springdoc:springdoc-openapi-starter-common:3.0.0=runtimeClasspath,testRuntimeClasspath
@@ -0,0 +1,92 @@
package dev.caskeleton.composition.notification;
import dev.caskeleton.application.notification.platform.admin.AdminAuthorizationGuard;
import dev.caskeleton.application.notification.platform.admin.AdminOperationStorePort;
import dev.caskeleton.application.notification.platform.admin.ChangeProviderStateUseCase;
import dev.caskeleton.application.notification.platform.admin.ChangeSuppressionUseCase;
import dev.caskeleton.application.notification.platform.admin.DuplicateRiskGuard;
import dev.caskeleton.application.notification.platform.admin.ProviderRuntimeControlPort;
import dev.caskeleton.application.notification.platform.admin.ReconcileDeliveriesUseCase;
import dev.caskeleton.application.notification.platform.admin.RedriveDeliveryUseCase;
import dev.caskeleton.application.notification.platform.dispatch.DeliveryAttemptStorePort;
import dev.caskeleton.application.notification.platform.dispatch.NotificationIdGeneratorPort;
import dev.caskeleton.application.notification.platform.dispatch.RecipientDeliveryStorePort;
import dev.caskeleton.application.notification.platform.dispatch.ReconciliationService;
import dev.caskeleton.application.notification.platform.observation.NotificationAuditPort;
import dev.caskeleton.application.notification.platform.policy.SuppressionStorePort;
import dev.caskeleton.application.transaction.TransactionPort;
import java.time.Clock;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
/**
* The operator plane: redrive, reconcile, suppress and provider control, one use case each.
*
* <p>Beans, so the method-security proxy applies each one's {@code @RequiresPermission}. The
* template ships no transport that calls them; a deployment that exposes the operator plane adds
* one. Which roles hold the {@code notification-admin:*} permissions by default is this module's
* {@code config/module-defaults/notification.yml}.
*/
@Configuration(proxyBeanMethods = false)
class NotificationAdminConfiguration {
@Bean
RedriveDeliveryUseCase notificationRedriveDeliveryUseCase(
DeliveryAttemptStorePort attempts,
RecipientDeliveryStorePort recipients,
AdminOperationStorePort operations,
NotificationAuditPort audit,
TransactionPort transactions,
Clock clock) {
return new RedriveDeliveryUseCase(
new AdminAuthorizationGuard(),
new DuplicateRiskGuard(),
attempts,
recipients,
operations,
audit,
transactions,
clock);
}
@Bean
ReconcileDeliveriesUseCase notificationReconcileDeliveriesUseCase(
DeliveryAttemptStorePort attempts,
ReconciliationService reconciliation,
AdminOperationStorePort operations,
NotificationAuditPort audit,
TransactionPort transactions,
Clock clock) {
return new ReconcileDeliveriesUseCase(
new AdminAuthorizationGuard(),
attempts,
reconciliation,
operations,
audit,
transactions,
clock);
}
@Bean
ChangeSuppressionUseCase notificationChangeSuppressionUseCase(
SuppressionStorePort suppressions,
AdminOperationStorePort operations,
NotificationAuditPort audit,
TransactionPort transactions,
NotificationIdGeneratorPort ids,
Clock clock) {
return new ChangeSuppressionUseCase(
new AdminAuthorizationGuard(), suppressions, operations, audit, transactions, ids, clock);
}
@Bean
ChangeProviderStateUseCase notificationChangeProviderStateUseCase(
ProviderRuntimeControlPort runtimes,
AdminOperationStorePort operations,
NotificationAuditPort audit,
TransactionPort transactions,
Clock clock) {
return new ChangeProviderStateUseCase(
new AdminAuthorizationGuard(), runtimes, operations, audit, transactions, clock);
}
}
@@ -7,9 +7,10 @@ import org.springframework.context.annotation.Import;
/**
* Application-owned Notification orchestration and transaction-bound use-case assembly.
*
* <p>The root holds the capability switch and imports one configuration per concern. It orders
* itself after the outbound leaf's root, whose stores, providers and policies every use case here
* consumes.
* <p>The root holds the capability switch and imports one configuration per concern, the workers
* included: nothing else loads {@link NotificationWorkerConfiguration}, which carries the mode
* condition. It orders itself after the outbound leaf's root, whose stores, providers and policies
* every use case here consumes.
*/
@AutoConfiguration(
afterName =
@@ -22,6 +23,8 @@ import org.springframework.context.annotation.Import;
@Import({
NotificationSubmissionConfiguration.class,
NotificationDeliveryConfiguration.class,
NotificationCallbackConfiguration.class
NotificationAdminConfiguration.class,
NotificationCallbackConfiguration.class,
NotificationWorkerConfiguration.class
})
public class NotificationApplicationCompositionAutoConfiguration {}
@@ -0,0 +1,159 @@
package dev.caskeleton.composition.notification;
import dev.caskeleton.application.notification.platform.callback.ReplayProviderEventsCommand;
import dev.caskeleton.application.notification.platform.callback.ReplayProviderEventsUseCase;
import dev.caskeleton.application.notification.platform.dispatch.ReconcileOutstandingAttemptsCommand;
import dev.caskeleton.application.notification.platform.dispatch.ReconcileOutstandingAttemptsUseCase;
import dev.caskeleton.application.notification.platform.dispatch.RecoverAbandonedDeliveriesCommand;
import dev.caskeleton.application.notification.platform.dispatch.RecoverAbandonedDeliveriesUseCase;
import java.time.Duration;
import java.util.List;
import java.util.Objects;
import java.util.concurrent.CopyOnWriteArrayList;
import java.util.concurrent.Executors;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.ScheduledFuture;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicBoolean;
import java.util.function.IntSupplier;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
/**
* Runs the lease recovery, provider-event replay and reconciliation passes, and stops them on
* shutdown.
*
* <p>The first two existed as classes with no caller. {@code LeaseRecoveryService} was written to
* recover deliveries a dead worker left in flight and was never scheduled; the ledger's replay
* queries were implemented in persistence and never read. The dispatch scheduler's own error path
* assumed recovery existed — it leaves a lease to expire rather than releasing it optimistically,
* on the grounds that recovery will decide — so the absence turned a deliberate design into a leak.
*
* <p>One executor for the three, because they are cheap, periodic, and must stop together. A
* failure in one pass is logged and the schedule continues: an exception escaping a scheduled task
* cancels it silently, which is how a background worker stops running without anybody being told.
*
* <p>Each pass is a {@code @SystemInvoked} use case this composition constructs and hands in, the
* way the outbox relay's scheduler holds its use case. The class owns the thread and the schedule
* and nothing else; it lived in the notification adapter, which owned the reconciliation and replay
* decisions too (round9 review 2026-11-12, RR9-05-06, RR9-05-08).
*/
// Public, unlike the scheduler worker: its bean is @ConditionalOnMissingBean, so a deployment may
// replace it with one of its own, built from the same use cases.
public final class NotificationBackgroundWorkers implements AutoCloseable {
private static final Logger log = LoggerFactory.getLogger(NotificationBackgroundWorkers.class);
private final RecoverAbandonedDeliveriesUseCase recovery;
private final ReplayProviderEventsUseCase replay;
private final ReconcileOutstandingAttemptsUseCase reconciliation;
private final Duration interval;
private final Duration shutdownGrace;
private final ScheduledExecutorService scheduler;
private final AtomicBoolean started = new AtomicBoolean();
private final List<ScheduledFuture<?>> passes = new CopyOnWriteArrayList<>();
/**
* Creates the workers.
*
* @param recovery recovers abandoned deliveries
* @param replay projects stored provider events that were never applied
* @param reconciliation asks providers about attempts whose outcome is unknown
* @param interval how often each pass runs
* @param shutdownGrace how long shutdown waits for a pass in flight
*/
public NotificationBackgroundWorkers(
RecoverAbandonedDeliveriesUseCase recovery,
ReplayProviderEventsUseCase replay,
ReconcileOutstandingAttemptsUseCase reconciliation,
Duration interval,
Duration shutdownGrace) {
this.recovery = Objects.requireNonNull(recovery, "recovery");
this.replay = Objects.requireNonNull(replay, "replay");
this.reconciliation = Objects.requireNonNull(reconciliation, "reconciliation");
this.interval = requirePositive(interval, "interval");
this.shutdownGrace = requirePositive(shutdownGrace, "shutdownGrace");
this.scheduler =
Executors.newScheduledThreadPool(
1,
runnable -> {
Thread thread = new Thread(runnable, "notification-background");
// A daemon thread: this executor must never be the reason a JVM refuses to exit.
thread.setDaemon(true);
return thread;
});
}
/** Starts the three passes. Idempotent. */
public void start() {
if (!started.compareAndSet(false, true)) {
return;
}
schedule("recovery", () -> recovery.handle(new RecoverAbandonedDeliveriesCommand()));
schedule("provider-event-replay", () -> replay.handle(new ReplayProviderEventsCommand()));
schedule(
"reconciliation", () -> reconciliation.handle(new ReconcileOutstandingAttemptsCommand()));
}
private void schedule(String name, IntSupplier pass) {
// The handle is kept so cancellation is possible and so the ignored-future check has an
// answer: the task swallows its own exceptions, so the future never completes exceptionally
// and there is nothing for a caller to observe on it.
ScheduledFuture<?> scheduled =
scheduler.scheduleWithFixedDelay(
() -> {
try {
int handled = pass.getAsInt();
if (handled > 0) {
log.debug("notification {} pass handled {}", name, handled);
}
} catch (RuntimeException failure) {
// Swallowed on purpose: an exception that escapes here cancels the schedule for the
// lifetime of the process, and a recovery worker that stopped silently is worse
// than
// one that fails a pass.
log.warn(
"notification {} pass failed reason={}",
name,
failure.getClass().getSimpleName());
}
},
interval.toMillis(),
interval.toMillis(),
TimeUnit.MILLISECONDS);
passes.add(scheduled);
}
@Override
public void close() {
passes.forEach(pass -> pass.cancel(false));
passes.clear();
scheduler.shutdown();
try {
if (!scheduler.awaitTermination(shutdownGrace.toMillis(), TimeUnit.MILLISECONDS)) {
scheduler.shutdownNow();
}
} catch (InterruptedException interrupted) {
Thread.currentThread().interrupt();
scheduler.shutdownNow();
}
}
/**
* Whether the passes are scheduled.
*
* @return true after start
*/
public boolean started() {
return started.get();
}
private static Duration requirePositive(Duration value, String name) {
Objects.requireNonNull(value, name);
if (value.isNegative() || value.isZero()) {
throw new IllegalStateException(name + " must be positive and finite");
}
return value;
}
}
@@ -1,12 +1,6 @@
package dev.caskeleton.composition.notification;
import dev.caskeleton.adapter.outbound.notification.platform.dispatch.NotificationDispatchProperties;
import dev.caskeleton.application.notification.platform.admin.AdminAuthorizationGuard;
import dev.caskeleton.application.notification.platform.admin.AdminOperationStorePort;
import dev.caskeleton.application.notification.platform.admin.DuplicateRiskGuard;
import dev.caskeleton.application.notification.platform.admin.NotificationAdminApplicationService;
import dev.caskeleton.application.notification.platform.admin.NotificationAdminService;
import dev.caskeleton.application.notification.platform.admin.ProviderRuntimeControlPort;
import dev.caskeleton.application.notification.platform.callback.DeliveryAttemptResolverPort;
import dev.caskeleton.application.notification.platform.callback.DeliveryProjectionStorePort;
import dev.caskeleton.application.notification.platform.callback.NotificationSideEffectPort;
@@ -42,8 +36,9 @@ import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
/**
* Doing the work and correcting it: dispatch, reconciliation, lease recovery, ledger projection and
* the operator plane that redrives and suppresses deliveries.
* Doing the work and correcting it: dispatch, reconciliation, lease recovery and ledger projection.
* The operator plane that redrives and suppresses deliveries is {@link
* NotificationAdminConfiguration}.
*
* <p>Here rather than in the notification adapter because each of these opens its own transactions
* around each write, and the adapter must not decide where a unit of work begins (NTF-020).
@@ -163,34 +158,4 @@ class NotificationDeliveryConfiguration {
return new ProviderEventProjectionService(
ledger, projector, resolver, projections, sideEffects, transactions, metrics);
}
/** The N4 operator plane. */
@Bean
NotificationAdminService notificationAdminService(
AdminAuthorizationGuard authorization,
DuplicateRiskGuard duplicateRiskGuard,
DeliveryAttemptStorePort attempts,
RecipientDeliveryStorePort recipients,
ReconciliationService reconciliation,
SuppressionStorePort suppressions,
ProviderRuntimeControlPort runtimes,
AdminOperationStorePort operations,
NotificationAuditPort audit,
TransactionPort transactions,
NotificationIdGeneratorPort ids,
Clock clock) {
return new NotificationAdminApplicationService(
authorization,
duplicateRiskGuard,
attempts,
recipients,
reconciliation,
suppressions,
runtimes,
operations,
audit,
transactions,
ids,
clock);
}
}
@@ -0,0 +1,208 @@
package dev.caskeleton.composition.notification;
import dev.caskeleton.adapter.outbound.notification.platform.dispatch.NotificationDispatchProperties;
import dev.caskeleton.application.notification.platform.dispatch.ClaimDueDeliveriesCommand;
import dev.caskeleton.application.notification.platform.dispatch.ClaimDueDeliveriesUseCase;
import dev.caskeleton.application.notification.platform.dispatch.DispatchClaimedDeliveryCommand;
import dev.caskeleton.application.notification.platform.dispatch.DispatchClaimedDeliveryUseCase;
import dev.caskeleton.application.notification.platform.dispatch.RecipientLease;
import dev.caskeleton.application.notification.platform.observation.MeasureDispatchBacklogQuery;
import dev.caskeleton.application.notification.platform.observation.MeasureDispatchBacklogUseCase;
import dev.caskeleton.application.notification.platform.observation.NotificationMetricName;
import dev.caskeleton.application.notification.platform.observation.NotificationMetricsPort;
import java.util.List;
import java.util.Map;
import java.util.Objects;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Semaphore;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicBoolean;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
/**
* Claims due deliveries and hands them to the dispatcher.
*
* <p>The worker never calls a provider itself. It claims, submits to a bounded executor, and stops
* claiming the moment shutdown begins — so a rolling restart drains rather than abandoning leases
* that then have to time out.
*
* <p>Claiming is bounded twice over: by the claim batch size and by a global concurrency permit.
* The second bound matters because a slow provider would otherwise let the queue depth become the
* thread count.
*
* <p>What is claimed and how one delivery is sent are two {@code @SystemInvoked} use cases this
* composition constructs and hands in, and the backlog it publishes is a query use case; the worker
* owns the polling thread, the executor and the permits. It lived in the notification adapter
* (round9 review 2026-11-12, RR9-05-06, RR9-05-08).
*/
final class NotificationSchedulerWorker implements AutoCloseable {
private static final Logger log = LoggerFactory.getLogger(NotificationSchedulerWorker.class);
private final ClaimDueDeliveriesUseCase claim;
private final DispatchClaimedDeliveryUseCase dispatch;
private final NotificationMetricsPort metrics;
private final NotificationDispatchProperties properties;
private final MeasureDispatchBacklogUseCase backlog;
private final String workerId;
private final ExecutorService dispatchExecutor;
private final Semaphore globalConcurrency;
private final AtomicBoolean running = new AtomicBoolean();
private final AtomicBoolean shuttingDown = new AtomicBoolean();
/** The polling thread, kept so shutdown can actually stop it. */
private volatile Thread pollingThread;
NotificationSchedulerWorker(
ClaimDueDeliveriesUseCase claim,
DispatchClaimedDeliveryUseCase dispatch,
NotificationMetricsPort metrics,
NotificationDispatchProperties properties,
MeasureDispatchBacklogUseCase backlog,
String workerId) {
this.claim = Objects.requireNonNull(claim, "claim");
this.dispatch = Objects.requireNonNull(dispatch, "dispatch");
this.metrics = Objects.requireNonNull(metrics, "metrics");
this.properties = Objects.requireNonNull(properties, "properties");
this.backlog = Objects.requireNonNull(backlog, "backlog");
this.workerId = Objects.requireNonNull(workerId, "workerId");
this.dispatchExecutor = Executors.newVirtualThreadPerTaskExecutor();
this.globalConcurrency = new Semaphore(properties.maxGlobalConcurrency());
}
/** Claim and dispatch one batch. Returns how many deliveries were claimed. */
public int runOnce() {
if (shuttingDown.get()) {
return 0;
}
// Only as many as can start now. The batch used to be claimed in full and then queued behind
// the semaphore, so a batch larger than the concurrency limit held leases on deliveries nobody
// was working on — and with a short lease those expired before their turn came, letting another
// worker claim a delivery this one still had queued.
int executable = Math.min(properties.claimBatchSize(), globalConcurrency.availablePermits());
if (executable < 1) {
return 0;
}
List<RecipientLease> claimed =
claim.handle(
new ClaimDueDeliveriesCommand(workerId, executable, properties.leaseDuration()));
// The backlog, not the batch. This gauge used to report claimed.size(), which is bounded above
// by the claim batch size — so a queue of ten and a queue of ten million published the same
// number, and the one metric named "queue depth" was the one that could not show a queue
// growing.
metrics.gauge(
NotificationMetricName.QUEUE_DEPTH,
Map.of(),
(double) backlog.handle(new MeasureDispatchBacklogQuery()));
for (RecipientLease lease : claimed) {
globalConcurrency.acquireUninterruptibly();
dispatchExecutor.execute(
() -> {
try {
dispatch.handle(new DispatchClaimedDeliveryCommand(lease));
} catch (RuntimeException failure) {
// The lease is left to expire rather than being released optimistically: a worker
// that
// failed mid-dispatch cannot prove what the provider did.
// The cause chain by type, never by message. "reason=NotificationValidationException"
// alone identified nothing — a dozen checks raise it — and a library's exception text
// can carry a recipient address, so the types are named and the messages are not.
log.warn(
"notification dispatch failed worker={} reason={} causes={}",
workerId,
failure.getClass().getSimpleName(),
causeChain(failure));
} finally {
globalConcurrency.release();
}
});
}
return claimed.size();
}
/**
* The exception's cause chain, as type names only.
*
* @param failure the dispatch failure
* @return the chain, outermost first, bounded so a self-referential cause cannot loop
*/
private static String causeChain(Throwable failure) {
StringBuilder chain = new StringBuilder();
Throwable current = failure.getCause();
for (int depth = 0; current != null && depth < 8; depth++) {
if (chain.length() > 0) {
chain.append('<');
}
chain.append(current.getClass().getSimpleName());
current = current.getCause();
}
return chain.length() == 0 ? "none" : chain.toString();
}
/** Start the polling loop on a dedicated thread. */
public void start() {
if (!running.compareAndSet(false, true)) {
return;
}
pollingThread =
Thread.ofVirtual()
.name("notification-scheduler-" + workerId)
.unstarted(
() -> {
while (running.get() && !shuttingDown.get()) {
try {
if (runOnce() == 0) {
Thread.sleep(properties.pollInterval().toMillis());
}
} catch (InterruptedException interrupted) {
Thread.currentThread().interrupt();
return;
} catch (RuntimeException failure) {
log.warn(
"notification scheduler tick failed worker={} reason={}",
workerId,
failure.getClass().getSimpleName());
}
}
});
pollingThread.start();
}
@Override
public void close() {
shuttingDown.set(true);
running.set(false);
// The polling thread was started and forgotten: close() shut the dispatch executor down and
// returned while the loop was still free to claim another batch, so shutdown could leave leases
// held by a process that was already gone. Interrupting it breaks the poll-interval sleep, and
// joining it means "closed" is a fact rather than a request.
Thread poller = pollingThread;
if (poller != null) {
poller.interrupt();
try {
poller.join(properties.shutdownGrace().toMillis());
} catch (InterruptedException interrupted) {
Thread.currentThread().interrupt();
}
pollingThread = null;
}
dispatchExecutor.shutdown();
try {
if (!dispatchExecutor.awaitTermination(
properties.shutdownGrace().toMillis(), TimeUnit.MILLISECONDS)) {
dispatchExecutor.shutdownNow();
}
} catch (InterruptedException interrupted) {
Thread.currentThread().interrupt();
dispatchExecutor.shutdownNow();
}
}
/** Whether the worker has stopped claiming new work. */
public boolean shuttingDown() {
return shuttingDown.get();
}
}
@@ -0,0 +1,129 @@
package dev.caskeleton.composition.notification;
import dev.caskeleton.adapter.outbound.notification.platform.dispatch.NotificationDispatchProperties;
import dev.caskeleton.application.notification.platform.callback.ProviderEventProjectionService;
import dev.caskeleton.application.notification.platform.callback.ReplayProviderEventsUseCase;
import dev.caskeleton.application.notification.platform.dispatch.ClaimDueDeliveriesUseCase;
import dev.caskeleton.application.notification.platform.dispatch.DispatchClaimedDeliveryUseCase;
import dev.caskeleton.application.notification.platform.dispatch.LeaseRecoveryService;
import dev.caskeleton.application.notification.platform.dispatch.NotificationDispatchService;
import dev.caskeleton.application.notification.platform.dispatch.RecipientLeaseStorePort;
import dev.caskeleton.application.notification.platform.dispatch.ReconcileOutstandingAttemptsUseCase;
import dev.caskeleton.application.notification.platform.dispatch.ReconciliationJobStorePort;
import dev.caskeleton.application.notification.platform.dispatch.ReconciliationService;
import dev.caskeleton.application.notification.platform.dispatch.RecoverAbandonedDeliveriesUseCase;
import dev.caskeleton.application.notification.platform.observation.MeasureDispatchBacklogUseCase;
import dev.caskeleton.application.notification.platform.observation.NotificationMetricsPort;
import dev.caskeleton.application.notification.platform.observation.NotificationServingStatePort;
import dev.caskeleton.application.transaction.TransactionPort;
import java.time.Clock;
import java.util.UUID;
import org.springframework.boot.autoconfigure.condition.ConditionalOnMissingBean;
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
/**
* Everything that runs on a schedule rather than on a request, and therefore everything {@code
* INGEST_ONLY} must not have.
*
* <p>The system's writes are {@code @SystemInvoked} use cases constructed here with {@code new} and
* handed to the two thread-owning workers, the way the outbox relay's composition hands its use
* case to its scheduler (manual 07 「시스템이 부르는 쓰기」). None of them is a bean: a bean is what a
* transport could reach, and these write with no permission check. The workers and this
* configuration lived in the notification adapter, which decided when to reconcile and replay there
* (round9 review 2026-11-12, RR9-05-06, RR9-05-08).
*
* <p>Gated on the mode as one configuration rather than a condition per bean. Both beans call
* {@code start()}, so a deployment in {@code INGEST_ONLY} — the mode whose whole purpose is to
* accept and store without delivering — would otherwise start the dispatch scheduler, lease
* recovery, provider-event replay and reconciliation. A per-bean condition is one a future bean can
* forget; a configuration that is not loaded cannot be. {@code matchIfMissing = true} because
* {@code SERVING} is the shipped default: an absent mode must still deliver.
*/
@Configuration(proxyBeanMethods = false)
@ConditionalOnProperty(
prefix = "app.notification.platform",
name = "mode",
havingValue = "SERVING",
matchIfMissing = true)
class NotificationWorkerConfiguration {
/**
* Schedules lease recovery, provider-event replay and reconciliation, and stops them on shutdown.
*
* @param recovery the lease recovery service
* @param projection the projection the events feed
* @param jobs the outstanding reconciliation questions
* @param reconciliation asks one attempt's provider
* @param transactions opens the use cases' own boundaries
* @param clock the clock
* @param properties the dispatch bounds
* @return the started background workers
*/
@Bean(destroyMethod = "close")
@ConditionalOnMissingBean
NotificationBackgroundWorkers notificationBackgroundWorkers(
LeaseRecoveryService recovery,
ProviderEventProjectionService projection,
ReconciliationJobStorePort jobs,
ReconciliationService reconciliation,
TransactionPort transactions,
Clock clock,
NotificationDispatchProperties properties) {
NotificationBackgroundWorkers workers =
new NotificationBackgroundWorkers(
new RecoverAbandonedDeliveriesUseCase(recovery),
new ReplayProviderEventsUseCase(projection, properties.claimBatchSize()),
new ReconcileOutstandingAttemptsUseCase(
jobs,
reconciliation::reconcile,
transactions,
clock,
properties.leaseDuration(),
properties.claimBatchSize(),
properties.maxAdditionalAttempts()),
properties.leaseDuration(),
properties.shutdownGrace());
workers.start();
return workers;
}
/**
* The dispatch scheduler.
*
* <p>The owner carries a per-process nonce, not just a configured name. Every instance used to
* call itself {@code notification-worker-1}, so a restarted process and the one it replaced were
* the same owner as far as the database was concerned — and a stale process's release could clear
* the lease its successor was working under. The nonce makes each incarnation distinguishable;
* the lease fence handles the rest.
*
* @param leases the recipient lease store
* @param dispatcher the dispatch service
* @param transactions opens the claim's write transaction and the backlog's read
* @param metrics the metrics port
* @param properties the dispatch bounds
* @param servingState the serving-state port
* @return the started scheduler worker
*/
@Bean(destroyMethod = "close")
NotificationSchedulerWorker notificationSchedulerWorker(
RecipientLeaseStorePort leases,
NotificationDispatchService dispatcher,
TransactionPort transactions,
NotificationMetricsPort metrics,
NotificationDispatchProperties properties,
NotificationServingStatePort servingState) {
String owner = "notification-worker-1:" + UUID.randomUUID();
NotificationSchedulerWorker worker =
new NotificationSchedulerWorker(
new ClaimDueDeliveriesUseCase(leases, transactions),
new DispatchClaimedDeliveryUseCase(dispatcher),
metrics,
properties,
new MeasureDispatchBacklogUseCase(servingState, transactions),
owner);
worker.start();
return worker;
}
}
@@ -21,16 +21,20 @@ info:
enabled: ${app.notification.platform.enabled:true}
app:
# The notification platform's own two permissions, granted to two roles rather than one.
# Publishing a template changes what every future submission renders, so a service that may send
# notifications is not thereby a service that may rewrite them; a deployment that wants both
# grants both roles. Here rather than in app-bootstrap's security.yml (07-08): the mapping goes
# when this module's line is removed. Map entries from every source merge, so a deployment's own
# app.authz.role-permissions adds roles beside these and can override either key.
# The notification platform's permissions, granted to three roles. Sending and publishing a
# template are two roles rather than one: publishing changes what every future submission
# renders, so a service that may send notifications is not thereby a service that may rewrite
# them; a deployment that wants both grants both roles. The operator plane's four permissions go
# to one operator role; the operator's authorities and tenant scope are checked again inside each
# use case, on the AdminActor the transport resolves. Here rather than in app-bootstrap's
# security.yml (07-08): the mapping goes when this module's line is removed. Map entries from
# every source merge, so a deployment's own app.authz.role-permissions adds roles beside these
# and can override any of the three keys.
authz:
role-permissions:
notification-submit: notification:submit
notification-template-publish: notification-template:publish
notification-admin: notification-admin:redrive,notification-admin:reconcile,notification-admin:suppress,notification-admin:provider-control
notification:
platform:
# The capability switch. False binds nothing at all.
@@ -57,6 +61,15 @@ app:
# An ambiguous attempt means the request reached the provider and the outcome could not be
# read. Falling back would risk sending the same notification twice, so it is off.
allow-ambiguous-fallback: ${APP_NOTIFICATION_PLATFORM_ALLOW_AMBIGUOUS_FALLBACK:false}
# How long a failed delivery waits before its next attempt (full jitter between base and max; a
# provider's Retry-After counts up to max), and how long after an ambiguous submission
# reconciliation first asks the provider. Tune them against the provider's own rate limits.
# Left unset here: the shipped timing (1s, 5m, 1m) lives in NotificationPlatformSettings.Retry
# alone. Override with
# retry:
# backoff-base: 1s
# backoff-max: 5m
# reconcile-delay: 1m
# Readiness is deployment-specific: throughput and operational SLOs determine what backlog
# is acceptable. These defaults preserve the original behavior but are no longer hard-coded.
serving-thresholds:
@@ -6,6 +6,7 @@ import dev.caskeleton.adapter.inbound.web.notification.platform.callback.Notific
import dev.caskeleton.adapter.outbound.notification.platform.provider.ses.SesCallbackAdapter;
import dev.caskeleton.adapter.outbound.notification.platform.security.AesGcmCallbackPayloadProtection;
import dev.caskeleton.adapter.outbound.support.startup.ModuleDefaultsEnvironmentPostProcessor;
import dev.caskeleton.application.notification.platform.admin.NotificationAdminPermissions;
import java.time.Duration;
import java.util.List;
import java.util.Map;
@@ -113,11 +114,20 @@ class NotificationModuleDefaultsTest {
List<String> permissions =
List.of(
environment.getProperty("app.authz.role-permissions.notification-submit", ""),
environment.getProperty(
"app.authz.role-permissions.notification-template-publish", ""));
environment.getProperty("app.authz.role-permissions.notification-template-publish", ""),
environment.getProperty("app.authz.role-permissions.notification-admin", ""));
// then
assertThat(permissions).containsExactly("notification:submit", "notification-template:publish");
assertThat(permissions)
.containsExactly(
"notification:submit",
"notification-template:publish",
String.join(
",",
NotificationAdminPermissions.REDRIVE,
NotificationAdminPermissions.RECONCILE,
NotificationAdminPermissions.SUPPRESS,
NotificationAdminPermissions.PROVIDER_CONTROL));
}
private static StandardEnvironment processed(Map<String, Object> deployment) {
@@ -0,0 +1,111 @@
package dev.caskeleton.composition.notification;
import static org.assertj.core.api.Assertions.assertThat;
import dev.caskeleton.application.security.SystemInvoked;
import java.lang.reflect.Method;
import java.util.Arrays;
import java.util.List;
import org.junit.jupiter.api.DisplayName;
import org.junit.jupiter.api.Test;
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Import;
/**
* {@code INGEST_ONLY} starts no dispatch, recovery or reconciliation thread (NTF-INT-003).
*
* <p>The worker configuration once carried the master switch and nothing else, and both of its
* worker beans called {@code start()} in the factory method. So a deployment in {@code INGEST_ONLY}
* — the mode whose entire purpose is to accept and store without delivering — started the dispatch
* scheduler, the lease recovery pass, the provider-event replay worker and the reconciliation job.
*
* <p>The gate is structural rather than a condition repeated per bean: the two started beans live
* in one configuration that carries the mode condition. Asserted on the configuration's own
* annotations rather than by booting the platform: a full notification context needs a database,
* provider credentials and a secrets resolver. The live thread count is asserted by the {@code
* local-notification-ingest} Compose lane.
*/
class NotificationWorkerLifecycleTest {
/** The bean methods that start something, and therefore may not exist outside SERVING. */
private static final List<String> THREAD_STARTING_BEANS =
List.of("notificationBackgroundWorkers", "notificationSchedulerWorker");
@Test
@DisplayName("스레드를 시작하는 worker bean은 모두 SERVING 모드 조건이 걸린 configuration에 있다")
void everyThreadStartingBeanIsServingOnly() {
// given
ConditionalOnProperty condition =
NotificationWorkerConfiguration.class.getAnnotation(ConditionalOnProperty.class);
// when
List<String> declared =
Arrays.stream(NotificationWorkerConfiguration.class.getDeclaredMethods())
.filter(method -> method.isAnnotationPresent(Bean.class))
.map(Method::getName)
.sorted()
.toList();
// then
assertThat(
List.of(
declared,
condition.prefix(),
List.of(condition.name()),
condition.havingValue(),
condition.matchIfMissing()))
.as(
"SERVING is the shipped default, so an absent mode must still deliver; a worker bean "
+ "outside this configuration would run in INGEST_ONLY too")
.containsExactly(
THREAD_STARTING_BEANS.stream().sorted().toList(),
"app.notification.platform",
List.of("mode"),
"SERVING",
true);
}
@Test
@DisplayName("worker configuration 밖의 조립은 SERVING 조건이 없어 INGEST_ONLY에서도 받는다")
void theIngestSideStaysAvailableInBothModes() {
// given
Import imports =
NotificationApplicationCompositionAutoConfiguration.class.getAnnotation(Import.class);
// when
List<String> gatedOnTheMode =
Arrays.stream(imports.value())
.filter(type -> type.isAnnotationPresent(ConditionalOnProperty.class))
.filter(
type ->
Arrays.asList(type.getAnnotation(ConditionalOnProperty.class).name())
.contains("mode"))
.map(Class::getSimpleName)
.toList();
// then
assertThat(gatedOnTheMode)
.as(
"only the workers stop in INGEST_ONLY; submission, delivery services, callbacks and "
+ "the operator plane are what an ingest-only deployment still serves")
.containsExactly("NotificationWorkerConfiguration");
}
@Test
@DisplayName("worker가 쥐는 시스템 쓰기 use case는 bean으로 내놓지 않는다")
void noSystemInvokedUseCaseIsABeanOfTheWorkerConfiguration() {
// when
List<String> systemInvokedBeans =
Arrays.stream(NotificationWorkerConfiguration.class.getDeclaredMethods())
.filter(method -> method.isAnnotationPresent(Bean.class))
.filter(method -> method.getReturnType().isAnnotationPresent(SystemInvoked.class))
.map(Method::getName)
.toList();
// then
assertThat(systemInvokedBeans)
.as("a bean is what a transport could reach, and these write with no permission check")
.isEmpty();
}
}