--- title: Runbook — IDEMPOTENCY_UNAVAILABLE (멱등성 조정 저장소 사용 불가) category: TRANSIENT_DEPENDENCY error_codes: [IDEMPOTENCY_UNAVAILABLE, IDEMPOTENCY_COMPLETION_UNKNOWN, IDEMPOTENCY_RECOVERY_REQUIRED] severity: P2 owner: oncall last_updated: 2026-11-05 status: active --- # Runbook: IDEMPOTENCY_UNAVAILABLE (`runbook://idempotency/unavailable`) ## Symptoms - HTTP 503 (`error.code=IDEMPOTENCY_UNAVAILABLE`, `retryable=true`) on endpoints that take an `Idempotency-Key`; the same code on the gRPC `error-code` trailer (`UNAVAILABLE`). - `IdempotencyExecutor` refused to run the action because the replay store (JPA or Redis, whichever `app.idempotency` selects) did not answer. Nothing was executed: the refusal happens before the action, so a client retry is safe. - The same 503 when the store refused to start a claim whose processing lease had already run out — the request spent longer than the lease between its claim and the start of its action (a stalled process, a slow store call). The ERROR line, `failure classified as IDEMPOTENCY_UNAVAILABLE (…)`, has the exception attached, and the exception's message reads `the claim's processing lease ran out before the action started; nothing ran: ` — in prod JSON logs that is in `stack_trace`, not `message`. The store may be healthy. Nothing ran here either, and the next request with the key takes the claim over and runs it. - HTTP 503 `IDEMPOTENCY_COMPLETION_UNKNOWN` (`retryable=true`) is the same outage hitting after the action ran: the store did not answer while the result was being recorded. When the use case called the executor inside its write transaction, that transaction's database writes rolled back with the exception; an action that committed a transaction of its own kept its writes, and anything outside a transaction (an outbound call, a message) may have happened. This is not "nothing was executed". Clients resend with the same key. - HTTP 500 `IDEMPOTENCY_RECOVERY_REQUIRED` (`retryable=false`) means the opposite case: the store cannot tell whether an earlier attempt's effect already happened. Two records produce it: - `ABANDONED`: the action threw, or reported its effect unknown. Final until the record expires. - `EXECUTING` past its processing lease (`APP_IDEMPOTENCY_PROCESSING_LEASE`, 30s): the attempt crashed, or is still running longer than the lease. The executor never renews the lease, so a slow but healthy request lands here too. The server never runs it again and does not touch the record. If the attempt is still running and finishes, its result is recorded, the first request gets its normal answer, and later requests with the key get that result replayed; a failure it reports is recorded the same way. So this 500 can stop on its own (owner decision 2026-11-04). This is the Redis store's case. The PostgreSQL store, called inside the use case's write transaction as the template calls it, never shows another request an unfinished record: that request waits for the first transaction to end, and gets the replay if it committed or a fresh run if it rolled back (`docs/web/http-contract.md` §2). ## Diagnosis - Check the health of the idempotency store the service uses (`/actuator/health` component of the JPA datasource or Redis, depending on the selected provider). - Look for connection-pool saturation or timeouts on that store in the same time window; the executor reports saturation as unavailable rather than guessing. - For `IDEMPOTENCY_RECOVERY_REQUIRED`, find the scope in the server's ERROR log of the failed request. Every transport writes the same line, `failure classified as IDEMPOTENCY_RECOVERY_REQUIRED (category=DATA_INTEGRITY, retryable=false)`, with the exception attached (shared-contract `ServerFailureLog`), and adds its own correlation: - web: no suffix; the line carries the request's `X-Request-Id` (MDC `request_id`). - gRPC: ` (method=/)`; MDC `request_id` too when the service closed the call on its callback thread. - GraphQL: ` (executionId=)`, the `executionId` the client got in the error's `extensions`. - STOMP (`@MessageMapping`): no suffix; MDC `request_id` is the frame's. The exception message ends with `::v::`, for example `claim requires reconciliation: CHARGE_CARD::v1::3f9a…` (64 hex characters). Then look at the record in the store the service uses (`APP_IDEMPOTENCY_PROVIDER`). There is no tool for this; these are the commands. - Redis. The key, for example `prod:order:shared:idem:v1:d1:CHARGE_CARD:3f9a…`, is ``` :::idem:v1:d:: ``` ``` HMGET state attempt leaseUntil PTTL ``` `state` `ABANDONED` is final. `state` `EXECUTING` with a `leaseUntil` (lease end, epoch milliseconds, server clock) earlier than the server's `TIME` (first value × 1000) is an attempt past its lease that may still be running. `PTTL` is how long the record still lives, in milliseconds. - PostgreSQL, in the application's schema: ```sql SELECT status, attempt, failure_disposition, last_transition_kind, processing_lease_until, expires_at, updated_at FROM idempotency_record WHERE record_version = 2 AND scope_hash = ''; ``` `status` `ABANDONED` with `last_transition_kind` `FAIL_ABANDONED` means the action threw or reported its effect unknown. `status` `EXECUTING` with `processing_lease_until` before `clock_timestamp()` is an attempt past its lease that may still be running. (`ABANDONED` with `EXPIRED_EXECUTION` is a lease that ran out under a version before 2026-11-04, which abandoned such rows; it goes away when the row expires.) `expires_at` is when the reaper deletes the row. ## Action - `IDEMPOTENCY_UNAVAILABLE`: restore the store; clients retry with the same key and backoff. When the exception message says the claim's processing lease ran out before the action started, the store is not the cause: find what held those requests between claim and start (thread dumps, GC pauses, the store's latency in that window). Nothing to repair; the next same-key request runs. - `IDEMPOTENCY_COMPLETION_UNKNOWN`: restore the store, then check the effects outside the transaction for the failed requests (the ERROR log line names the code). A same-key resend replays a result that was in fact recorded, gets `IDEMPOTENT_IN_FLIGHT` while the attempt's lease lasts, and `IDEMPOTENCY_RECOVERY_REQUIRED` after it — handle that one as below. - `IDEMPOTENCY_RECOVERY_REQUIRED`: do not ask the client to retry blindly. If the record is `EXECUTING` past its lease, first wait for the attempt: look the record up again after the longest time the action can take. It changes by itself when the attempt finishes — `COMPLETED` (resends replay it, nothing to do), `ABANDONED`, or `FAILED_RETRYABLE` (the attempt proved it had no effect, and the next request with the key runs it). A crashed attempt, or one whose request already ended with `IDEMPOTENCY_COMPLETION_UNKNOWN`, never changes it; that record stays `EXECUTING` until it expires. Business data checked while the attempt still runs can say "no effect" a moment before the effect lands. Then confirm from business data (the order, the payment, the downstream system) whether the effect exists. There is no way to complete the record by hand — the stored response is the action's own result. - The effect exists: tell the client it was applied. Leave the record; a resend with the key keeps getting this 500 until the record expires (Redis `PTTL`, PostgreSQL `expires_at`). - The effect does not exist: the client sends the request again under a new key. To let the same key run again instead, delete the record — only while it is `ABANDONED`, because any other state belongs to a request that may still be running or to a stored result. Do not delete an `EXECUTING` record, even past its lease: if its attempt is still alive, its late result finds no record (`ABSENT`) — the result is lost, the first request gets this 500 after its work took effect, and the next request with the key runs the work a second time. Leave it to expire. - Redis: check and delete in one step, as the account the service's registered scripts run as (`APP_REDIS_AUTHENTICATION_ADVANCED_CREDENTIAL_REFERENCE`, or the application account when it is unset). Checking with `HMGET` and then sending `DEL` is two commands: a record that expires between them and is claimed again by a new request is deleted while that request runs, and the next request with the key runs the work a second time (re-review 2026-11-04 RR4-03-02). The program below deletes the key only while its `state` is `ABANDONED`. The application accounts are denied `EVAL` (`infra/redis-sdk/acl/README.md`), so load it and run it by the digest `SCRIPT LOAD` prints, as the service runs its own scripts: ``` SCRIPT LOAD "if redis.call('HGET', KEYS[1], 'state') == 'ABANDONED' then return redis.call('DEL', KEYS[1]) end return 0" EVALSHA 1 ``` `1` deleted the record; the key is new. `0` deleted nothing: the record is no longer `ABANDONED` (or is gone) — look again before doing anything else. `NOSCRIPT` means the node that holds the key (a cluster) has not loaded the program: run `SCRIPT LOAD` on that node and `EVALSHA` again; nothing was deleted. A record that is about to expire needs nothing; once it is gone the key is new. - PostgreSQL: ```sql DELETE FROM idempotency_record WHERE record_version = 2 AND scope_hash = '' AND status = 'ABANDONED'; ``` Without either step the key is released when the record expires. A record whose attempt never finished lives for the processing lease plus `APP_IDEMPOTENCY_TTL` from its claim; one the action marked failed lives for `APP_IDEMPOTENCY_FAILURE_RETENTION` (not shorter than `APP_IDEMPOTENCY_TTL`) from the failure (`docs/web/http-contract.md`, "서버가 기억하는 기간"). ## Escalation - P2 → P1 when the store stays unavailable for more than 10 minutes on a write path. - Page the owner of the idempotency store (database or Redis on-call).