문제:
- 루트 compose 6개와 infra/ 아래 12개가 개발 환경, 테스트 lane fixture,
보조 파일을 가리지 않고 섞여 있다. 이름 모양도 셋이다.
- README 빠른 시작 scripts/bootstrap-local.sh는 docker-compose.yml과
.local.yml만 합친 뒤, 둘 어디에도 없는 db 서비스를 띄우려다 멈춘다.
- compose smoke all-adapters lane은 앱이 mem_limit 512m를 넘어 커널에
죽고(OOM), restart: unless-stopped로 다시 떠서 죽는 때가 smoke 요청과
겹칠 때만 실패한다. smoke는 재시작을 보지 않아 나머지 실행은 거짓 통과다.
고른 방법(소유자 결정 2026-11-07):
- 루트에 개발용 compose.yaml 하나. app(local 런타임)과 뒷단 서비스를
capability profile(jpa, redis, kafka, mongo, objectstorage, auth, mail)로
고른다. 마운트하는 보조 파일은 compose/(keycloak, minio init, Redis ACL,
mailpit 인증서 폴더).
- lane fixture는 lane을 가진 leaf로: dev·prod-smoke·tls overlay, fileserver
overlay, smoke·스키마 적용 서비스(lane-services.yaml)와 스크립트, lane
계약 JSON → src/app-bootstrap/compose-lanes/. Redis topology →
redis-sdk/compose/, NFS → fileserver/compose/nfs/. infra/는 없어진다.
- 계약의 composeProfile(문자열)을 composeProfiles(목록)로 바꾼다. lane이
backing profile과 lane profile을 함께 켠다. 18 lane의 서비스 집합은
전과 같다(verify-compose-profile-contracts.sh가 정확히 대조).
- smoke: lane이 끝나기 전에 그 project의 모든 컨테이너가 재시작 0,
OOMKilled false인지 보고 아니면 실패한다. 앱의 memory.peak를 증거로
남긴다. all-adapters만 appMemoryLimit 1g(계약 칸, smoke가 overlay에 씀).
- bootstrap-local.sh는 compose.yaml로 db와 app을 띄운다(인자 없음).
- 개발 compose와 lane 바탕이 같은 파일이라, app healthcheck는 모든
런타임에서 답하는 관리 포트 readiness로 둔다.
버린 대안: lane을 Testcontainers로 옮기기(설계 다-2, 별도 판단), lane마다
기본 메모리를 올리기(배포 기본값 512m 계약을 흐림).
위험·한계:
- 개발 compose의 app이 src/.env를 읽으므로, 개발자 src/.env가 dev·prod
lane에도 들어간다. lane 값은 생성 overlay의 environment가 이기고, CI에는
src/.env가 없다.
- MinIO 이미지 기본값이 compose.yaml과 lane-services.yaml 두 곳에 있다
(YAML anchor는 파일을 넘지 못한다). ComposeMinioImageMatchesEvidenceTest가
둘 다 evidence digest와 대조한다.
- env 키 검사(verifyEnvKeys)는 compose.yaml과 lane overlay를 읽고
lane-services.yaml(smoke client 변수)은 뺀다.
검증(desktop, 트리 6d1e9f9d → 주석·문서만 고친 d0703901, ~/t1-layout-logs/):
- verify-compose-profile-contracts.sh rc=0 (18 lane 정확히 일치)
- :app-bootstrap:check :app-bootstrap:documentContractTest
:app-bootstrap:verifyEnvKeys와 objectstorage·cache-redis·redis-sdk·
redis-runtime·fileserver·composition:fileserver-platform check rc=0
- :app-bootstrap:integrationTest(ComposeMergeCharacterizationTest 포함),
:adapter:outbound:cache-redis:redisContainerContractTest rc=0
- -p build-logic check rc=0, :verifyStrictLaneStages --dry-run rc=0
- scripts/run-compose-runtime-smoke.sh --matrix(18 lane) rc=0, 794초.
모든 lane 컨테이너 재시작 0·OOMKilled false. 앱 memory.peak 최고
all-adapters 532MiB(512m 한도를 넘는 값이라 1g의 근거), 나머지 244~422MiB
- scripts/bootstrap-local.sh rc=0(db·app 기동, /api/healthcheck 200 UP).
기준 ac03192d의 base+local 조합은 config --services가 app뿐이고
up --wait db가 "no such service: db"로 rc=1
- 정리할 때 `docker compose down`은 profile 서비스(db)를 남겨
README·compose.yaml 안내를 `--profile '*' down`으로 고치고 다시 확인
- 주석·문서 수정 뒤 :app-bootstrap:repositoryContractTest
:app-bootstrap:documentContractTest, verify-compose rc=0
Refs: design-docs repo-structure/README.md 3.1, repo-structure/01-backend-layout.md 6.3
BREAKING CHANGE: 루트 docker-compose*.yml과 infra/가 없어지고 compose.yaml(profile)·compose/·leaf 안 lane fixture로 옮겨졌다. lane 계약은 src/app-bootstrap/compose-lanes/compose-profile-contracts.json, composeProfile → composeProfiles. docs/upgrade/2026-11-maturity-review.md B-51.
10 KiB
title, category, error_codes, severity, owner, last_updated, status
| title | category | error_codes | severity | owner | last_updated | status | |||
|---|---|---|---|---|---|---|---|---|---|
| Runbook — IDEMPOTENCY_UNAVAILABLE (멱등성 조정 저장소 사용 불가) | TRANSIENT_DEPENDENCY |
|
P2 | oncall | 2026-11-05 | active |
Runbook: IDEMPOTENCY_UNAVAILABLE (runbook://idempotency/unavailable)
Symptoms
- HTTP 503 (
error.code=IDEMPOTENCY_UNAVAILABLE,retryable=true) on endpoints that take anIdempotency-Key; the same code on the gRPCerror-codetrailer (UNAVAILABLE). IdempotencyExecutorrefused to run the action because the replay store (JPA or Redis, whicheverapp.idempotencyselects) did not answer. Nothing was executed: the refusal happens before the action, so a client retry is safe.- The same 503 when the store refused to start a claim whose processing lease had already run out
— the request spent longer than the lease between its claim and the start of its action (a
stalled process, a slow store call). The ERROR line,
failure classified as IDEMPOTENCY_UNAVAILABLE (…), has the exception attached, and the exception's message readsthe claim's processing lease ran out before the action started; nothing ran: <scope>— in prod JSON logs that is instack_trace, notmessage. The store may be healthy. Nothing ran here either, and the next request with the key takes the claim over and runs it. - HTTP 503
IDEMPOTENCY_COMPLETION_UNKNOWN(retryable=true) is the same outage hitting after the action ran: the store did not answer while the result was being recorded. When the use case called the executor inside its write transaction, that transaction's database writes rolled back with the exception; an action that committed a transaction of its own kept its writes, and anything outside a transaction (an outbound call, a message) may have happened. This is not "nothing was executed". Clients resend with the same key. - HTTP 500
IDEMPOTENCY_RECOVERY_REQUIRED(retryable=false) means the opposite case: the store cannot tell whether an earlier attempt's effect already happened. Two records produce it:ABANDONED: the action threw, or reported its effect unknown. Final until the record expires.EXECUTINGpast its processing lease (APP_IDEMPOTENCY_PROCESSING_LEASE, 30s): the attempt crashed, or is still running longer than the lease. The executor never renews the lease, so a slow but healthy request lands here too. The server never runs it again and does not touch the record. If the attempt is still running and finishes, its result is recorded, the first request gets its normal answer, and later requests with the key get that result replayed; a failure it reports is recorded the same way. So this 500 can stop on its own (owner decision 2026-11-04). This is the Redis store's case. The PostgreSQL store, called inside the use case's write transaction as the template calls it, never shows another request an unfinished record: that request waits for the first transaction to end, and gets the replay if it committed or a fresh run if it rolled back (docs/web/http-contract.md§2).
Diagnosis
-
Check the health of the idempotency store the service uses (
/actuator/healthcomponent of the JPA datasource or Redis, depending on the selected provider). -
Look for connection-pool saturation or timeouts on that store in the same time window; the executor reports saturation as unavailable rather than guessing.
-
For
IDEMPOTENCY_RECOVERY_REQUIRED, find the scope in the server's ERROR log of the failed request. Every transport writes the same line,failure classified as IDEMPOTENCY_RECOVERY_REQUIRED (category=DATA_INTEGRITY, retryable=false), with the exception attached (shared-contractServerFailureLog), and adds its own correlation:- web: no suffix; the line carries the request's
X-Request-Id(MDCrequest_id). - gRPC:
(method=<service>/<method>); MDCrequest_idtoo when the service closed the call on its callback thread. - GraphQL:
(executionId=<id>), theexecutionIdthe client got in the error'sextensions. - STOMP (
@MessageMapping): no suffix; MDCrequest_idis the frame's.
The exception message ends with
<operation code>::v<digest version>::<scope digest>, for exampleclaim requires reconciliation: CHARGE_CARD::v1::3f9a…(64 hex characters). Then look at the record in the store the service uses (APP_IDEMPOTENCY_PROVIDER). There is no tool for this; these are the commands.-
Redis. The key, for example
prod:order:shared:idem:v1:d1:CHARGE_CARD:3f9a…, is<APP_REDIS_NAMESPACE_ENVIRONMENT>:<APP_REDIS_NAMESPACE_SERVICE>:<APP_REDIS_NAMESPACE_DOMAIN>:idem:v1:d<digest version>:<operation code>:<scope digest>HMGET <key> state attempt leaseUntil PTTL <key>stateABANDONEDis final.stateEXECUTINGwith aleaseUntil(lease end, epoch milliseconds, server clock) earlier than the server'sTIME(first value × 1000) is an attempt past its lease that may still be running.PTTLis how long the record still lives, in milliseconds. -
PostgreSQL, in the application's schema:
SELECT status, attempt, failure_disposition, last_transition_kind, processing_lease_until, expires_at, updated_at FROM idempotency_record WHERE record_version = 2 AND scope_hash = '<scope digest>';statusABANDONEDwithlast_transition_kindFAIL_ABANDONEDmeans the action threw or reported its effect unknown.statusEXECUTINGwithprocessing_lease_untilbeforeclock_timestamp()is an attempt past its lease that may still be running. (ABANDONEDwithEXPIRED_EXECUTIONis a lease that ran out under a version before 2026-11-04, which abandoned such rows; it goes away when the row expires.)expires_atis when the reaper deletes the row.
- web: no suffix; the line carries the request's
Action
-
IDEMPOTENCY_UNAVAILABLE: restore the store; clients retry with the same key and backoff. When the exception message says the claim's processing lease ran out before the action started, the store is not the cause: find what held those requests between claim and start (thread dumps, GC pauses, the store's latency in that window). Nothing to repair; the next same-key request runs. -
IDEMPOTENCY_COMPLETION_UNKNOWN: restore the store, then check the effects outside the transaction for the failed requests (the ERROR log line names the code). A same-key resend replays a result that was in fact recorded, getsIDEMPOTENT_IN_FLIGHTwhile the attempt's lease lasts, andIDEMPOTENCY_RECOVERY_REQUIREDafter it — handle that one as below. -
IDEMPOTENCY_RECOVERY_REQUIRED: do not ask the client to retry blindly. If the record isEXECUTINGpast its lease, first wait for the attempt: look the record up again after the longest time the action can take. It changes by itself when the attempt finishes —COMPLETED(resends replay it, nothing to do),ABANDONED, orFAILED_RETRYABLE(the attempt proved it had no effect, and the next request with the key runs it). A crashed attempt, or one whose request already ended withIDEMPOTENCY_COMPLETION_UNKNOWN, never changes it; that record staysEXECUTINGuntil it expires. Business data checked while the attempt still runs can say "no effect" a moment before the effect lands. Then confirm from business data (the order, the payment, the downstream system) whether the effect exists. There is no way to complete the record by hand — the stored response is the action's own result.- The effect exists: tell the client it was applied. Leave the record; a resend with the key keeps
getting this 500 until the record expires (Redis
PTTL, PostgreSQLexpires_at). - The effect does not exist: the client sends the request again under a new key. To let the same
key run again instead, delete the record — only while it is
ABANDONED, because any other state belongs to a request that may still be running or to a stored result. Do not delete anEXECUTINGrecord, even past its lease: if its attempt is still alive, its late result finds no record (ABSENT) — the result is lost, the first request gets this 500 after its work took effect, and the next request with the key runs the work a second time. Leave it to expire.-
Redis: check and delete in one step, as the account the service's registered scripts run as (
APP_REDIS_AUTHENTICATION_ADVANCED_CREDENTIAL_REFERENCE, or the application account when it is unset). Checking withHMGETand then sendingDELis two commands: a record that expires between them and is claimed again by a new request is deleted while that request runs, and the next request with the key runs the work a second time (re-review 2026-11-04 RR4-03-02). The program below deletes the key only while itsstateisABANDONED. The application accounts are deniedEVAL(compose/redis/acl/README.md), so load it and run it by the digestSCRIPT LOADprints, as the service runs its own scripts:SCRIPT LOAD "if redis.call('HGET', KEYS[1], 'state') == 'ABANDONED' then return redis.call('DEL', KEYS[1]) end return 0" EVALSHA <digest SCRIPT LOAD printed> 1 <key>1deleted the record; the key is new.0deleted nothing: the record is no longerABANDONED(or is gone) — look again before doing anything else.NOSCRIPTmeans the node that holds the key (a cluster) has not loaded the program: runSCRIPT LOADon that node andEVALSHAagain; nothing was deleted. A record that is about to expire needs nothing; once it is gone the key is new. -
PostgreSQL:
DELETE FROM idempotency_record WHERE record_version = 2 AND scope_hash = '<scope digest>' AND status = 'ABANDONED';
-
Without either step the key is released when the record expires. A record whose attempt never finished lives for the processing lease plus
APP_IDEMPOTENCY_TTLfrom its claim; one the action marked failed lives forAPP_IDEMPOTENCY_FAILURE_RETENTION(not shorter thanAPP_IDEMPOTENCY_TTL) from the failure (docs/web/http-contract.md, "서버가 기억하는 기간"). - The effect exists: tell the client it was applied. Leave the record; a resend with the key keeps
getting this 500 until the record expires (Redis
Escalation
- P2 → P1 when the store stays unavailable for more than 10 minutes on a write path.
- Page the owner of the idempotency store (database or Redis on-call).