Files
clean-architecture-backend-…/docs/runbooks/idempotency-unavailable.md
T
DongHyeonka 50cd7154d0 refactor(compose)!: compose 파일을 개발 환경 하나와 lane fixture로 나눈다
문제:
- 루트 compose 6개와 infra/ 아래 12개가 개발 환경, 테스트 lane fixture,
  보조 파일을 가리지 않고 섞여 있다. 이름 모양도 셋이다.
- README 빠른 시작 scripts/bootstrap-local.sh는 docker-compose.yml과
  .local.yml만 합친 뒤, 둘 어디에도 없는 db 서비스를 띄우려다 멈춘다.
- compose smoke all-adapters lane은 앱이 mem_limit 512m를 넘어 커널에
  죽고(OOM), restart: unless-stopped로 다시 떠서 죽는 때가 smoke 요청과
  겹칠 때만 실패한다. smoke는 재시작을 보지 않아 나머지 실행은 거짓 통과다.

고른 방법(소유자 결정 2026-11-07):
- 루트에 개발용 compose.yaml 하나. app(local 런타임)과 뒷단 서비스를
  capability profile(jpa, redis, kafka, mongo, objectstorage, auth, mail)로
  고른다. 마운트하는 보조 파일은 compose/(keycloak, minio init, Redis ACL,
  mailpit 인증서 폴더).
- lane fixture는 lane을 가진 leaf로: dev·prod-smoke·tls overlay, fileserver
  overlay, smoke·스키마 적용 서비스(lane-services.yaml)와 스크립트, lane
  계약 JSON → src/app-bootstrap/compose-lanes/. Redis topology →
  redis-sdk/compose/, NFS → fileserver/compose/nfs/. infra/는 없어진다.
- 계약의 composeProfile(문자열)을 composeProfiles(목록)로 바꾼다. lane이
  backing profile과 lane profile을 함께 켠다. 18 lane의 서비스 집합은
  전과 같다(verify-compose-profile-contracts.sh가 정확히 대조).
- smoke: lane이 끝나기 전에 그 project의 모든 컨테이너가 재시작 0,
  OOMKilled false인지 보고 아니면 실패한다. 앱의 memory.peak를 증거로
  남긴다. all-adapters만 appMemoryLimit 1g(계약 칸, smoke가 overlay에 씀).
- bootstrap-local.sh는 compose.yaml로 db와 app을 띄운다(인자 없음).
- 개발 compose와 lane 바탕이 같은 파일이라, app healthcheck는 모든
  런타임에서 답하는 관리 포트 readiness로 둔다.
버린 대안: lane을 Testcontainers로 옮기기(설계 다-2, 별도 판단), lane마다
기본 메모리를 올리기(배포 기본값 512m 계약을 흐림).

위험·한계:
- 개발 compose의 app이 src/.env를 읽으므로, 개발자 src/.env가 dev·prod
  lane에도 들어간다. lane 값은 생성 overlay의 environment가 이기고, CI에는
  src/.env가 없다.
- MinIO 이미지 기본값이 compose.yaml과 lane-services.yaml 두 곳에 있다
  (YAML anchor는 파일을 넘지 못한다). ComposeMinioImageMatchesEvidenceTest가
  둘 다 evidence digest와 대조한다.
- env 키 검사(verifyEnvKeys)는 compose.yaml과 lane overlay를 읽고
  lane-services.yaml(smoke client 변수)은 뺀다.

검증(desktop, 트리 6d1e9f9d → 주석·문서만 고친 d0703901, ~/t1-layout-logs/):
- verify-compose-profile-contracts.sh rc=0 (18 lane 정확히 일치)
- :app-bootstrap:check :app-bootstrap:documentContractTest
  :app-bootstrap:verifyEnvKeys와 objectstorage·cache-redis·redis-sdk·
  redis-runtime·fileserver·composition:fileserver-platform check rc=0
- :app-bootstrap:integrationTest(ComposeMergeCharacterizationTest 포함),
  :adapter:outbound:cache-redis:redisContainerContractTest rc=0
- -p build-logic check rc=0, :verifyStrictLaneStages --dry-run rc=0
- scripts/run-compose-runtime-smoke.sh --matrix(18 lane) rc=0, 794초.
  모든 lane 컨테이너 재시작 0·OOMKilled false. 앱 memory.peak 최고
  all-adapters 532MiB(512m 한도를 넘는 값이라 1g의 근거), 나머지 244~422MiB
- scripts/bootstrap-local.sh rc=0(db·app 기동, /api/healthcheck 200 UP).
  기준 ac03192d의 base+local 조합은 config --services가 app뿐이고
  up --wait db가 "no such service: db"로 rc=1
- 정리할 때 `docker compose down`은 profile 서비스(db)를 남겨
  README·compose.yaml 안내를 `--profile '*' down`으로 고치고 다시 확인
- 주석·문서 수정 뒤 :app-bootstrap:repositoryContractTest
  :app-bootstrap:documentContractTest, verify-compose rc=0

Refs: design-docs repo-structure/README.md 3.1, repo-structure/01-backend-layout.md 6.3
BREAKING CHANGE: 루트 docker-compose*.yml과 infra/가 없어지고 compose.yaml(profile)·compose/·leaf 안 lane fixture로 옮겨졌다. lane 계약은 src/app-bootstrap/compose-lanes/compose-profile-contracts.json, composeProfile → composeProfiles. docs/upgrade/2026-11-maturity-review.md B-51.
2026-11-07 12:52:36 +09:00

10 KiB
Raw Blame History

title, category, error_codes, severity, owner, last_updated, status
title category error_codes severity owner last_updated status
Runbook — IDEMPOTENCY_UNAVAILABLE (멱등성 조정 저장소 사용 불가) TRANSIENT_DEPENDENCY
IDEMPOTENCY_UNAVAILABLE
IDEMPOTENCY_COMPLETION_UNKNOWN
IDEMPOTENCY_RECOVERY_REQUIRED
P2 oncall 2026-11-05 active

Runbook: IDEMPOTENCY_UNAVAILABLE (runbook://idempotency/unavailable)

Symptoms

  • HTTP 503 (error.code=IDEMPOTENCY_UNAVAILABLE, retryable=true) on endpoints that take an Idempotency-Key; the same code on the gRPC error-code trailer (UNAVAILABLE).
  • IdempotencyExecutor refused to run the action because the replay store (JPA or Redis, whichever app.idempotency selects) did not answer. Nothing was executed: the refusal happens before the action, so a client retry is safe.
  • The same 503 when the store refused to start a claim whose processing lease had already run out — the request spent longer than the lease between its claim and the start of its action (a stalled process, a slow store call). The ERROR line, failure classified as IDEMPOTENCY_UNAVAILABLE (…), has the exception attached, and the exception's message reads the claim's processing lease ran out before the action started; nothing ran: <scope> — in prod JSON logs that is in stack_trace, not message. The store may be healthy. Nothing ran here either, and the next request with the key takes the claim over and runs it.
  • HTTP 503 IDEMPOTENCY_COMPLETION_UNKNOWN (retryable=true) is the same outage hitting after the action ran: the store did not answer while the result was being recorded. When the use case called the executor inside its write transaction, that transaction's database writes rolled back with the exception; an action that committed a transaction of its own kept its writes, and anything outside a transaction (an outbound call, a message) may have happened. This is not "nothing was executed". Clients resend with the same key.
  • HTTP 500 IDEMPOTENCY_RECOVERY_REQUIRED (retryable=false) means the opposite case: the store cannot tell whether an earlier attempt's effect already happened. Two records produce it:
    • ABANDONED: the action threw, or reported its effect unknown. Final until the record expires.
    • EXECUTING past its processing lease (APP_IDEMPOTENCY_PROCESSING_LEASE, 30s): the attempt crashed, or is still running longer than the lease. The executor never renews the lease, so a slow but healthy request lands here too. The server never runs it again and does not touch the record. If the attempt is still running and finishes, its result is recorded, the first request gets its normal answer, and later requests with the key get that result replayed; a failure it reports is recorded the same way. So this 500 can stop on its own (owner decision 2026-11-04). This is the Redis store's case. The PostgreSQL store, called inside the use case's write transaction as the template calls it, never shows another request an unfinished record: that request waits for the first transaction to end, and gets the replay if it committed or a fresh run if it rolled back (docs/web/http-contract.md §2).

Diagnosis

  • Check the health of the idempotency store the service uses (/actuator/health component of the JPA datasource or Redis, depending on the selected provider).

  • Look for connection-pool saturation or timeouts on that store in the same time window; the executor reports saturation as unavailable rather than guessing.

  • For IDEMPOTENCY_RECOVERY_REQUIRED, find the scope in the server's ERROR log of the failed request. Every transport writes the same line, failure classified as IDEMPOTENCY_RECOVERY_REQUIRED (category=DATA_INTEGRITY, retryable=false), with the exception attached (shared-contract ServerFailureLog), and adds its own correlation:

    • web: no suffix; the line carries the request's X-Request-Id (MDC request_id).
    • gRPC: (method=<service>/<method>); MDC request_id too when the service closed the call on its callback thread.
    • GraphQL: (executionId=<id>), the executionId the client got in the error's extensions.
    • STOMP (@MessageMapping): no suffix; MDC request_id is the frame's.

    The exception message ends with <operation code>::v<digest version>::<scope digest>, for example claim requires reconciliation: CHARGE_CARD::v1::3f9a… (64 hex characters). Then look at the record in the store the service uses (APP_IDEMPOTENCY_PROVIDER). There is no tool for this; these are the commands.

    • Redis. The key, for example prod:order:shared:idem:v1:d1:CHARGE_CARD:3f9a…, is

      <APP_REDIS_NAMESPACE_ENVIRONMENT>:<APP_REDIS_NAMESPACE_SERVICE>:<APP_REDIS_NAMESPACE_DOMAIN>:idem:v1:d<digest version>:<operation code>:<scope digest>
      
      HMGET <key> state attempt leaseUntil
      PTTL <key>
      

      state ABANDONED is final. state EXECUTING with a leaseUntil (lease end, epoch milliseconds, server clock) earlier than the server's TIME (first value × 1000) is an attempt past its lease that may still be running. PTTL is how long the record still lives, in milliseconds.

    • PostgreSQL, in the application's schema:

      SELECT status, attempt, failure_disposition, last_transition_kind,
             processing_lease_until, expires_at, updated_at
        FROM idempotency_record
       WHERE record_version = 2
         AND scope_hash = '<scope digest>';
      

      status ABANDONED with last_transition_kind FAIL_ABANDONED means the action threw or reported its effect unknown. status EXECUTING with processing_lease_until before clock_timestamp() is an attempt past its lease that may still be running. (ABANDONED with EXPIRED_EXECUTION is a lease that ran out under a version before 2026-11-04, which abandoned such rows; it goes away when the row expires.) expires_at is when the reaper deletes the row.

Action

  • IDEMPOTENCY_UNAVAILABLE: restore the store; clients retry with the same key and backoff. When the exception message says the claim's processing lease ran out before the action started, the store is not the cause: find what held those requests between claim and start (thread dumps, GC pauses, the store's latency in that window). Nothing to repair; the next same-key request runs.

  • IDEMPOTENCY_COMPLETION_UNKNOWN: restore the store, then check the effects outside the transaction for the failed requests (the ERROR log line names the code). A same-key resend replays a result that was in fact recorded, gets IDEMPOTENT_IN_FLIGHT while the attempt's lease lasts, and IDEMPOTENCY_RECOVERY_REQUIRED after it — handle that one as below.

  • IDEMPOTENCY_RECOVERY_REQUIRED: do not ask the client to retry blindly. If the record is EXECUTING past its lease, first wait for the attempt: look the record up again after the longest time the action can take. It changes by itself when the attempt finishes — COMPLETED (resends replay it, nothing to do), ABANDONED, or FAILED_RETRYABLE (the attempt proved it had no effect, and the next request with the key runs it). A crashed attempt, or one whose request already ended with IDEMPOTENCY_COMPLETION_UNKNOWN, never changes it; that record stays EXECUTING until it expires. Business data checked while the attempt still runs can say "no effect" a moment before the effect lands. Then confirm from business data (the order, the payment, the downstream system) whether the effect exists. There is no way to complete the record by hand — the stored response is the action's own result.

    • The effect exists: tell the client it was applied. Leave the record; a resend with the key keeps getting this 500 until the record expires (Redis PTTL, PostgreSQL expires_at).
    • The effect does not exist: the client sends the request again under a new key. To let the same key run again instead, delete the record — only while it is ABANDONED, because any other state belongs to a request that may still be running or to a stored result. Do not delete an EXECUTING record, even past its lease: if its attempt is still alive, its late result finds no record (ABSENT) — the result is lost, the first request gets this 500 after its work took effect, and the next request with the key runs the work a second time. Leave it to expire.
      • Redis: check and delete in one step, as the account the service's registered scripts run as (APP_REDIS_AUTHENTICATION_ADVANCED_CREDENTIAL_REFERENCE, or the application account when it is unset). Checking with HMGET and then sending DEL is two commands: a record that expires between them and is claimed again by a new request is deleted while that request runs, and the next request with the key runs the work a second time (re-review 2026-11-04 RR4-03-02). The program below deletes the key only while its state is ABANDONED. The application accounts are denied EVAL (compose/redis/acl/README.md), so load it and run it by the digest SCRIPT LOAD prints, as the service runs its own scripts:

        SCRIPT LOAD "if redis.call('HGET', KEYS[1], 'state') == 'ABANDONED' then return redis.call('DEL', KEYS[1]) end return 0"
        EVALSHA <digest SCRIPT LOAD printed> 1 <key>
        

        1 deleted the record; the key is new. 0 deleted nothing: the record is no longer ABANDONED (or is gone) — look again before doing anything else. NOSCRIPT means the node that holds the key (a cluster) has not loaded the program: run SCRIPT LOAD on that node and EVALSHA again; nothing was deleted. A record that is about to expire needs nothing; once it is gone the key is new.

      • PostgreSQL:

        DELETE FROM idempotency_record
         WHERE record_version = 2
           AND scope_hash = '<scope digest>'
           AND status = 'ABANDONED';
        

    Without either step the key is released when the record expires. A record whose attempt never finished lives for the processing lease plus APP_IDEMPOTENCY_TTL from its claim; one the action marked failed lives for APP_IDEMPOTENCY_FAILURE_RETENTION (not shorter than APP_IDEMPOTENCY_TTL) from the failure (docs/web/http-contract.md, "서버가 기억하는 기간").

Escalation

  • P2 → P1 when the store stays unavailable for more than 10 minutes on a write path.
  • Page the owner of the idempotency store (database or Redis on-call).