# Operations ## Runtime shape ```text durable queue (PostgreSQL, FOR UPDATE SKIP LOCKED) → expiry check → suppression and eligibility re-check → provider health gate → rate limiter → concurrency limiter → provider adapter ``` Provider calls run outside every database transaction. The attempt row is committed first, so after a crash the row is either absent (nothing was sent) or present in `DISPATCHING` (reconciliation has something to ask about). ## Guards that exist for specific incidents | Guard | The incident it prevents | |---|---| | Credential failure opens the provider route | One expired key multiplied by a queue becomes a self-inflicted outage | | Retry budget per provider profile | A provider outage turning every queued notification into its own retry loop | | Ambiguous attempts block automatic fallback | A push whose response was lost arriving alongside the "just in case" SMS | | Permits released during backoff | A slow provider pinning the whole concurrency budget on work that is only waiting | | Bounded drain on rotation | A provider that never answers holding a credential rotation open forever | | Fail-fast intake on capacity | An unbounded in-memory queue absorbing a burst it cannot survive | ## Scheduling `scheduleAt` activates the job, `notBefore` is the earliest permitted provider submission, and `expiresAt` blocks new attempts, retries and fallbacks. Suppression and expiry are re-checked immediately before dispatch, because a scheduled notification can sit in the queue for hours and the user may have opted out in the meantime. ## Redrive A redrive preserves `NotificationId` and `RecipientDeliveryId`, creates a new `DeliveryAttemptId`, and reuses the pinned template version and rendered digest. Sending different content is a new notification, not a redrive. Redriving an ambiguous attempt requires explicit duplicate-risk approval, because the platform genuinely cannot tell whether the first submission reached the user. ## Actuator surface Provider runtime states and generations, queue depth and age, callback and reconciliation health. Never addresses, never credentials.