Files
clean-architecture-backend-…/docs/runbooks/idempotency-unavailable.md
T
DongHyeonka 6e992970ca fix!: r9 확인 리뷰가 찾은 결함 25건을 고친다
문제: r9 커밋(45d2a478)을 영역 4개로 다시 리뷰해 결함 25건을 찾았다
(Important 2, Minor 23, round6/01~04). Important 둘: r9가 @Configuration을
사용자 전송에서 통째로 빼서 설정 안의 RouterFunction·DataFetcher가
@SystemCaller를 불러도 잡히지 않는 회귀, 그리고 가이드대로 만든
CREATE INDEX CONCURRENTLY 마이그레이션이 Flyway 기본 트랜잭션 잠금과 서로
기다려 끝나지 않는 것(이전부터). 그 밖에 시작을 거절당한 같은 attempt가
기록 만료까지 503을 받는 것, gRPC 취소 판정과 raw status 설명의 로그 유출,
STOMP의 IAE ERROR 기록, 아무것도 잡지 못하던 shared-contract 패키지 규칙,
workflow 판독기의 거짓 통과 모양(-v·조건부 exit·Windows 기본 셸·요청형
trigger 등).

방법: 영역별 4묶음이 실패 테스트를 먼저 확인하고 고쳤다(재리뷰 README 17절).
- 설정과 그 중첩 클래스는 요청 처리기 타입에 의존하면 사용자 전송으로 센다.
- Flyway PostgreSQL 잠금을 세션 수준으로(transactional-lock=false), 직접
  만드는 Flyway는 FlywaySchemaPolicy 하나로. 버린 대안: CIC를 마이그레이션에서
  금지하고 운영 절차로.
- 같은 attempt로 다시 claim하면 lease가 지난 claim을 넘겨받는다. 옛 revision의
  늦은 시작이 ALREADY를 받아 이중 실행되던 것도 막았다.
- 네 전송이 IAE를 ServerFailureLog의 같은 WARN 한 줄로 남긴다.
- workflow 판독기가 세지 않는 이유를 셋으로 나누고 문서 5곳을 같은 말로.

위험·한계: Flyway 세션 잠금은 transaction-mode PgBouncer 뒤에서 깨진다
(Hikari 직접 연결 전제, 가이드에 적음). 배포 마이그레이션 도구도 같은 값을
써야 한다(업그레이드 안내). 같은 attempt를 두 호출이 동시에 쓰는 것은
저장소가 가려내지 않는다(호출자 계약으로 문서화).

검증: 묶음별 RED·GREEN은 로컬(노트북 빌드 금지 지시 전)에서, 지시 뒤에
바뀐 01 A5 fixture의 RED와 02 늦은 시작 revision 검사의 RED, 03 동시 인덱스
검사기 GREEN은 desktop에서 확인했다. 합친 트리를 desktop에서 돌려 모두 첫
실행 통과: :ci :verifyDependencyLocks --warning-mode=fail, sample check,
두 빌드 verifyStrictLaneStages --dry-run, :mainCheck, :integrationCheck,
-p optional-platforms ci integrationCheck performanceCheck, :performanceCheck,
:releaseCheck, compose 18 lane, -p build-logic check javadoc(README 17.2절).

Refs: docs/reviews/2026-11-03-rereview/README.md 17절
BREAKING CHANGE: Flyway PostgreSQL 잠금이 세션 수준이 되어 배포 도구에도
flyway.postgresql.transactional.lock=false가 필요하다. GraphQL·gRPC가 IAE를
WARN 한 줄로 남긴다. shared-contract 패키지 규칙이 실제로 검사한다.
2026-11-05 21:26:50 +09:00

10 KiB
Raw Blame History

title, category, error_codes, severity, owner, last_updated, status
title category error_codes severity owner last_updated status
Runbook — IDEMPOTENCY_UNAVAILABLE (멱등성 조정 저장소 사용 불가) TRANSIENT_DEPENDENCY
IDEMPOTENCY_UNAVAILABLE
IDEMPOTENCY_COMPLETION_UNKNOWN
IDEMPOTENCY_RECOVERY_REQUIRED
P2 oncall 2026-11-05 active

Runbook: IDEMPOTENCY_UNAVAILABLE (runbook://idempotency/unavailable)

Symptoms

  • HTTP 503 (error.code=IDEMPOTENCY_UNAVAILABLE, retryable=true) on endpoints that take an Idempotency-Key; the same code on the gRPC error-code trailer (UNAVAILABLE).
  • IdempotencyExecutor refused to run the action because the replay store (JPA or Redis, whichever app.idempotency selects) did not answer. Nothing was executed: the refusal happens before the action, so a client retry is safe.
  • The same 503 when the store refused to start a claim whose processing lease had already run out — the request spent longer than the lease between its claim and the start of its action (a stalled process, a slow store call). The ERROR line, failure classified as IDEMPOTENCY_UNAVAILABLE (…), has the exception attached, and the exception's message reads the claim's processing lease ran out before the action started; nothing ran: <scope> — in prod JSON logs that is in stack_trace, not message. The store may be healthy. Nothing ran here either, and the next request with the key takes the claim over and runs it.
  • HTTP 503 IDEMPOTENCY_COMPLETION_UNKNOWN (retryable=true) is the same outage hitting after the action ran: the store did not answer while the result was being recorded. When the use case called the executor inside its write transaction, that transaction's database writes rolled back with the exception; an action that committed a transaction of its own kept its writes, and anything outside a transaction (an outbound call, a message) may have happened. This is not "nothing was executed". Clients resend with the same key.
  • HTTP 500 IDEMPOTENCY_RECOVERY_REQUIRED (retryable=false) means the opposite case: the store cannot tell whether an earlier attempt's effect already happened. Two records produce it:
    • ABANDONED: the action threw, or reported its effect unknown. Final until the record expires.
    • EXECUTING past its processing lease (APP_IDEMPOTENCY_PROCESSING_LEASE, 30s): the attempt crashed, or is still running longer than the lease. The executor never renews the lease, so a slow but healthy request lands here too. The server never runs it again and does not touch the record. If the attempt is still running and finishes, its result is recorded, the first request gets its normal answer, and later requests with the key get that result replayed; a failure it reports is recorded the same way. So this 500 can stop on its own (owner decision 2026-11-04). This is the Redis store's case. The PostgreSQL store, called inside the use case's write transaction as the template calls it, never shows another request an unfinished record: that request waits for the first transaction to end, and gets the replay if it committed or a fresh run if it rolled back (docs/web/http-contract.md §2).

Diagnosis

  • Check the health of the idempotency store the service uses (/actuator/health component of the JPA datasource or Redis, depending on the selected provider).

  • Look for connection-pool saturation or timeouts on that store in the same time window; the executor reports saturation as unavailable rather than guessing.

  • For IDEMPOTENCY_RECOVERY_REQUIRED, find the scope in the server's ERROR log of the failed request. Every transport writes the same line, failure classified as IDEMPOTENCY_RECOVERY_REQUIRED (category=DATA_INTEGRITY, retryable=false), with the exception attached (shared-contract ServerFailureLog), and adds its own correlation:

    • web: no suffix; the line carries the request's X-Request-Id (MDC request_id).
    • gRPC: (method=<service>/<method>); MDC request_id too when the service closed the call on its callback thread.
    • GraphQL: (executionId=<id>), the executionId the client got in the error's extensions.
    • STOMP (@MessageMapping): no suffix; MDC request_id is the frame's.

    The exception message ends with <operation code>::v<digest version>::<scope digest>, for example claim requires reconciliation: CHARGE_CARD::v1::3f9a… (64 hex characters). Then look at the record in the store the service uses (APP_IDEMPOTENCY_PROVIDER). There is no tool for this; these are the commands.

    • Redis. The key, for example prod:order:shared:idem:v1:d1:CHARGE_CARD:3f9a…, is

      <APP_REDIS_NAMESPACE_ENVIRONMENT>:<APP_REDIS_NAMESPACE_SERVICE>:<APP_REDIS_NAMESPACE_DOMAIN>:idem:v1:d<digest version>:<operation code>:<scope digest>
      
      HMGET <key> state attempt leaseUntil
      PTTL <key>
      

      state ABANDONED is final. state EXECUTING with a leaseUntil (lease end, epoch milliseconds, server clock) earlier than the server's TIME (first value × 1000) is an attempt past its lease that may still be running. PTTL is how long the record still lives, in milliseconds.

    • PostgreSQL, in the application's schema:

      SELECT status, attempt, failure_disposition, last_transition_kind,
             processing_lease_until, expires_at, updated_at
        FROM idempotency_record
       WHERE record_version = 2
         AND scope_hash = '<scope digest>';
      

      status ABANDONED with last_transition_kind FAIL_ABANDONED means the action threw or reported its effect unknown. status EXECUTING with processing_lease_until before clock_timestamp() is an attempt past its lease that may still be running. (ABANDONED with EXPIRED_EXECUTION is a lease that ran out under a version before 2026-11-04, which abandoned such rows; it goes away when the row expires.) expires_at is when the reaper deletes the row.

Action

  • IDEMPOTENCY_UNAVAILABLE: restore the store; clients retry with the same key and backoff. When the exception message says the claim's processing lease ran out before the action started, the store is not the cause: find what held those requests between claim and start (thread dumps, GC pauses, the store's latency in that window). Nothing to repair; the next same-key request runs.

  • IDEMPOTENCY_COMPLETION_UNKNOWN: restore the store, then check the effects outside the transaction for the failed requests (the ERROR log line names the code). A same-key resend replays a result that was in fact recorded, gets IDEMPOTENT_IN_FLIGHT while the attempt's lease lasts, and IDEMPOTENCY_RECOVERY_REQUIRED after it — handle that one as below.

  • IDEMPOTENCY_RECOVERY_REQUIRED: do not ask the client to retry blindly. If the record is EXECUTING past its lease, first wait for the attempt: look the record up again after the longest time the action can take. It changes by itself when the attempt finishes — COMPLETED (resends replay it, nothing to do), ABANDONED, or FAILED_RETRYABLE (the attempt proved it had no effect, and the next request with the key runs it). A crashed attempt, or one whose request already ended with IDEMPOTENCY_COMPLETION_UNKNOWN, never changes it; that record stays EXECUTING until it expires. Business data checked while the attempt still runs can say "no effect" a moment before the effect lands. Then confirm from business data (the order, the payment, the downstream system) whether the effect exists. There is no way to complete the record by hand — the stored response is the action's own result.

    • The effect exists: tell the client it was applied. Leave the record; a resend with the key keeps getting this 500 until the record expires (Redis PTTL, PostgreSQL expires_at).
    • The effect does not exist: the client sends the request again under a new key. To let the same key run again instead, delete the record — only while it is ABANDONED, because any other state belongs to a request that may still be running or to a stored result. Do not delete an EXECUTING record, even past its lease: if its attempt is still alive, its late result finds no record (ABSENT) — the result is lost, the first request gets this 500 after its work took effect, and the next request with the key runs the work a second time. Leave it to expire.
      • Redis: check and delete in one step, as the account the service's registered scripts run as (APP_REDIS_AUTHENTICATION_ADVANCED_CREDENTIAL_REFERENCE, or the application account when it is unset). Checking with HMGET and then sending DEL is two commands: a record that expires between them and is claimed again by a new request is deleted while that request runs, and the next request with the key runs the work a second time (re-review 2026-11-04 RR4-03-02). The program below deletes the key only while its state is ABANDONED. The application accounts are denied EVAL (infra/redis-sdk/acl/README.md), so load it and run it by the digest SCRIPT LOAD prints, as the service runs its own scripts:

        SCRIPT LOAD "if redis.call('HGET', KEYS[1], 'state') == 'ABANDONED' then return redis.call('DEL', KEYS[1]) end return 0"
        EVALSHA <digest SCRIPT LOAD printed> 1 <key>
        

        1 deleted the record; the key is new. 0 deleted nothing: the record is no longer ABANDONED (or is gone) — look again before doing anything else. NOSCRIPT means the node that holds the key (a cluster) has not loaded the program: run SCRIPT LOAD on that node and EVALSHA again; nothing was deleted. A record that is about to expire needs nothing; once it is gone the key is new.

      • PostgreSQL:

        DELETE FROM idempotency_record
         WHERE record_version = 2
           AND scope_hash = '<scope digest>'
           AND status = 'ABANDONED';
        

    Without either step the key is released when the record expires. A record whose attempt never finished lives for the processing lease plus APP_IDEMPOTENCY_TTL from its claim; one the action marked failed lives for APP_IDEMPOTENCY_FAILURE_RETENTION (not shorter than APP_IDEMPOTENCY_TTL) from the failure (docs/web/http-contract.md, "서버가 기억하는 기간").

Escalation

  • P2 → P1 when the store stays unavailable for more than 10 minutes on a write path.
  • Page the owner of the idempotency store (database or Redis on-call).