문제: r9 커밋(45d2a478)을 영역 4개로 다시 리뷰해 결함 25건을 찾았다
(Important 2, Minor 23, round6/01~04). Important 둘: r9가 @Configuration을
사용자 전송에서 통째로 빼서 설정 안의 RouterFunction·DataFetcher가
@SystemCaller를 불러도 잡히지 않는 회귀, 그리고 가이드대로 만든
CREATE INDEX CONCURRENTLY 마이그레이션이 Flyway 기본 트랜잭션 잠금과 서로
기다려 끝나지 않는 것(이전부터). 그 밖에 시작을 거절당한 같은 attempt가
기록 만료까지 503을 받는 것, gRPC 취소 판정과 raw status 설명의 로그 유출,
STOMP의 IAE ERROR 기록, 아무것도 잡지 못하던 shared-contract 패키지 규칙,
workflow 판독기의 거짓 통과 모양(-v·조건부 exit·Windows 기본 셸·요청형
trigger 등).
방법: 영역별 4묶음이 실패 테스트를 먼저 확인하고 고쳤다(재리뷰 README 17절).
- 설정과 그 중첩 클래스는 요청 처리기 타입에 의존하면 사용자 전송으로 센다.
- Flyway PostgreSQL 잠금을 세션 수준으로(transactional-lock=false), 직접
만드는 Flyway는 FlywaySchemaPolicy 하나로. 버린 대안: CIC를 마이그레이션에서
금지하고 운영 절차로.
- 같은 attempt로 다시 claim하면 lease가 지난 claim을 넘겨받는다. 옛 revision의
늦은 시작이 ALREADY를 받아 이중 실행되던 것도 막았다.
- 네 전송이 IAE를 ServerFailureLog의 같은 WARN 한 줄로 남긴다.
- workflow 판독기가 세지 않는 이유를 셋으로 나누고 문서 5곳을 같은 말로.
위험·한계: Flyway 세션 잠금은 transaction-mode PgBouncer 뒤에서 깨진다
(Hikari 직접 연결 전제, 가이드에 적음). 배포 마이그레이션 도구도 같은 값을
써야 한다(업그레이드 안내). 같은 attempt를 두 호출이 동시에 쓰는 것은
저장소가 가려내지 않는다(호출자 계약으로 문서화).
검증: 묶음별 RED·GREEN은 로컬(노트북 빌드 금지 지시 전)에서, 지시 뒤에
바뀐 01 A5 fixture의 RED와 02 늦은 시작 revision 검사의 RED, 03 동시 인덱스
검사기 GREEN은 desktop에서 확인했다. 합친 트리를 desktop에서 돌려 모두 첫
실행 통과: :ci :verifyDependencyLocks --warning-mode=fail, sample check,
두 빌드 verifyStrictLaneStages --dry-run, :mainCheck, :integrationCheck,
-p optional-platforms ci integrationCheck performanceCheck, :performanceCheck,
:releaseCheck, compose 18 lane, -p build-logic check javadoc(README 17.2절).
Refs: docs/reviews/2026-11-03-rereview/README.md 17절
BREAKING CHANGE: Flyway PostgreSQL 잠금이 세션 수준이 되어 배포 도구에도
flyway.postgresql.transactional.lock=false가 필요하다. GraphQL·gRPC가 IAE를
WARN 한 줄로 남긴다. shared-contract 패키지 규칙이 실제로 검사한다.
166 lines
10 KiB
Markdown
166 lines
10 KiB
Markdown
---
|
||
title: Runbook — IDEMPOTENCY_UNAVAILABLE (멱등성 조정 저장소 사용 불가)
|
||
category: TRANSIENT_DEPENDENCY
|
||
error_codes: [IDEMPOTENCY_UNAVAILABLE, IDEMPOTENCY_COMPLETION_UNKNOWN, IDEMPOTENCY_RECOVERY_REQUIRED]
|
||
severity: P2
|
||
owner: oncall
|
||
last_updated: 2026-11-05
|
||
status: active
|
||
---
|
||
|
||
# Runbook: IDEMPOTENCY_UNAVAILABLE (`runbook://idempotency/unavailable`)
|
||
|
||
## Symptoms
|
||
|
||
- HTTP 503 (`error.code=IDEMPOTENCY_UNAVAILABLE`, `retryable=true`) on endpoints that take an
|
||
`Idempotency-Key`; the same code on the gRPC `error-code` trailer (`UNAVAILABLE`).
|
||
- `IdempotencyExecutor` refused to run the action because the replay store (JPA or Redis, whichever
|
||
`app.idempotency` selects) did not answer. Nothing was executed: the refusal happens before the
|
||
action, so a client retry is safe.
|
||
- The same 503 when the store refused to start a claim whose processing lease had already run out
|
||
— the request spent longer than the lease between its claim and the start of its action (a
|
||
stalled process, a slow store call). The ERROR line, `failure classified as
|
||
IDEMPOTENCY_UNAVAILABLE (…)`, has the exception attached, and the exception's message reads `the
|
||
claim's processing lease ran out before the action started; nothing ran: <scope>` — in prod JSON
|
||
logs that is in `stack_trace`, not `message`. The store may be healthy.
|
||
Nothing ran here either, and the next request with the key takes the claim over and runs it.
|
||
- HTTP 503 `IDEMPOTENCY_COMPLETION_UNKNOWN` (`retryable=true`) is the same outage hitting after the
|
||
action ran: the store did not answer while the result was being recorded. When the use case called
|
||
the executor inside its write transaction, that transaction's database writes rolled back with the
|
||
exception; an action that committed a transaction of its own kept its writes, and anything outside
|
||
a transaction (an outbound call, a message) may have happened. This is not "nothing was executed".
|
||
Clients resend with the same key.
|
||
- HTTP 500 `IDEMPOTENCY_RECOVERY_REQUIRED` (`retryable=false`) means the opposite case: the store
|
||
cannot tell whether an earlier attempt's effect already happened. Two records produce it:
|
||
- `ABANDONED`: the action threw, or reported its effect unknown. Final until the record expires.
|
||
- `EXECUTING` past its processing lease (`APP_IDEMPOTENCY_PROCESSING_LEASE`, 30s): the attempt
|
||
crashed, or is still running longer than the lease. The executor never renews the lease, so a
|
||
slow but healthy request lands here too. The server never runs it again and does not touch the
|
||
record. If the attempt is still running and finishes, its result is recorded, the first request
|
||
gets its normal answer, and later requests with the key get that result replayed; a failure it
|
||
reports is recorded the same way. So this 500 can stop on its own (owner decision 2026-11-04).
|
||
This is the Redis store's case. The PostgreSQL store, called inside the use case's write
|
||
transaction as the template calls it, never shows another request an unfinished record: that
|
||
request waits for the first transaction to end, and gets the replay if it committed or a fresh
|
||
run if it rolled back (`docs/web/http-contract.md` §2).
|
||
|
||
## Diagnosis
|
||
|
||
- Check the health of the idempotency store the service uses (`/actuator/health` component of the
|
||
JPA datasource or Redis, depending on the selected provider).
|
||
- Look for connection-pool saturation or timeouts on that store in the same time window; the
|
||
executor reports saturation as unavailable rather than guessing.
|
||
- For `IDEMPOTENCY_RECOVERY_REQUIRED`, find the scope in the server's ERROR log of the failed request.
|
||
Every transport writes the same line, `failure classified as IDEMPOTENCY_RECOVERY_REQUIRED
|
||
(category=DATA_INTEGRITY, retryable=false)`, with the exception attached
|
||
(shared-contract `ServerFailureLog`), and adds its own correlation:
|
||
- web: no suffix; the line carries the request's `X-Request-Id` (MDC `request_id`).
|
||
- gRPC: ` (method=<service>/<method>)`; MDC `request_id` too when the service closed the call on
|
||
its callback thread.
|
||
- GraphQL: ` (executionId=<id>)`, the `executionId` the client got in the error's `extensions`.
|
||
- STOMP (`@MessageMapping`): no suffix; MDC `request_id` is the frame's.
|
||
|
||
The exception message ends with
|
||
`<operation code>::v<digest version>::<scope digest>`, for example
|
||
`claim requires reconciliation: CHARGE_CARD::v1::3f9a…` (64 hex characters). Then look at the
|
||
record in the store the service uses (`APP_IDEMPOTENCY_PROVIDER`). There is no tool for this; these
|
||
are the commands.
|
||
- Redis. The key, for example `prod:order:shared:idem:v1:d1:CHARGE_CARD:3f9a…`, is
|
||
|
||
<!-- runbook-redis: idempotency-record-key -->
|
||
```
|
||
<APP_REDIS_NAMESPACE_ENVIRONMENT>:<APP_REDIS_NAMESPACE_SERVICE>:<APP_REDIS_NAMESPACE_DOMAIN>:idem:v1:d<digest version>:<operation code>:<scope digest>
|
||
```
|
||
|
||
```
|
||
HMGET <key> state attempt leaseUntil
|
||
PTTL <key>
|
||
```
|
||
|
||
`state` `ABANDONED` is final. `state` `EXECUTING` with a `leaseUntil` (lease end, epoch
|
||
milliseconds, server clock) earlier than the server's `TIME` (first value × 1000) is an attempt
|
||
past its lease that may still be running. `PTTL` is how long the record still lives, in
|
||
milliseconds.
|
||
- PostgreSQL, in the application's schema:
|
||
|
||
```sql
|
||
SELECT status, attempt, failure_disposition, last_transition_kind,
|
||
processing_lease_until, expires_at, updated_at
|
||
FROM idempotency_record
|
||
WHERE record_version = 2
|
||
AND scope_hash = '<scope digest>';
|
||
```
|
||
|
||
`status` `ABANDONED` with `last_transition_kind` `FAIL_ABANDONED` means the action threw or
|
||
reported its effect unknown. `status` `EXECUTING` with `processing_lease_until` before
|
||
`clock_timestamp()` is an attempt past its lease that may still be running. (`ABANDONED` with
|
||
`EXPIRED_EXECUTION` is a lease that ran out under a version before 2026-11-04, which abandoned
|
||
such rows; it goes away when the row expires.) `expires_at` is when the reaper deletes the row.
|
||
|
||
## Action
|
||
|
||
- `IDEMPOTENCY_UNAVAILABLE`: restore the store; clients retry with the same key and backoff. When
|
||
the exception message says the claim's processing lease ran out before the action started, the
|
||
store is not the cause: find what held those requests between claim and start (thread dumps, GC
|
||
pauses, the store's latency in that window). Nothing to repair; the next same-key request runs.
|
||
- `IDEMPOTENCY_COMPLETION_UNKNOWN`: restore the store, then check the effects outside the
|
||
transaction for the failed requests (the ERROR log line names the code). A same-key resend replays
|
||
a result that was in fact recorded, gets `IDEMPOTENT_IN_FLIGHT` while the attempt's lease lasts,
|
||
and `IDEMPOTENCY_RECOVERY_REQUIRED` after it — handle that one as below.
|
||
- `IDEMPOTENCY_RECOVERY_REQUIRED`: do not ask the client to retry blindly. If the record is
|
||
`EXECUTING` past its lease, first wait for the attempt: look the record up again after the
|
||
longest time the action can take. It changes by itself when the attempt finishes — `COMPLETED`
|
||
(resends replay it, nothing to do), `ABANDONED`, or `FAILED_RETRYABLE` (the attempt proved it had
|
||
no effect, and the next request with the key runs it). A crashed attempt, or one whose request
|
||
already ended with `IDEMPOTENCY_COMPLETION_UNKNOWN`, never changes it; that record stays
|
||
`EXECUTING` until it expires. Business data checked while the attempt still runs can say "no
|
||
effect" a moment before the effect lands. Then confirm from business data (the order, the
|
||
payment, the downstream system) whether the effect exists. There is no way to complete the record
|
||
by hand — the stored response is the action's own result.
|
||
- The effect exists: tell the client it was applied. Leave the record; a resend with the key keeps
|
||
getting this 500 until the record expires (Redis `PTTL`, PostgreSQL `expires_at`).
|
||
- The effect does not exist: the client sends the request again under a new key. To let the same
|
||
key run again instead, delete the record — only while it is `ABANDONED`, because any other state
|
||
belongs to a request that may still be running or to a stored result. Do not delete an
|
||
`EXECUTING` record, even past its lease: if its attempt is still alive, its late result finds no
|
||
record (`ABSENT`) — the result is lost, the first request gets this 500 after its work took
|
||
effect, and the next request with the key runs the work a second time. Leave it to expire.
|
||
- Redis: check and delete in one step, as the account the service's registered scripts run as
|
||
(`APP_REDIS_AUTHENTICATION_ADVANCED_CREDENTIAL_REFERENCE`, or the application account when it
|
||
is unset). Checking with `HMGET` and then sending `DEL` is two commands: a record that expires
|
||
between them and is claimed again by a new request is deleted while that request runs, and
|
||
the next request with the key runs the work a second time (re-review 2026-11-04 RR4-03-02).
|
||
The program below deletes the key only while its `state` is `ABANDONED`. The application
|
||
accounts are denied `EVAL` (`infra/redis-sdk/acl/README.md`), so load it and run it by the
|
||
digest `SCRIPT LOAD` prints, as the service runs its own scripts:
|
||
|
||
<!-- runbook-redis: idempotency-delete-abandoned -->
|
||
```
|
||
SCRIPT LOAD "if redis.call('HGET', KEYS[1], 'state') == 'ABANDONED' then return redis.call('DEL', KEYS[1]) end return 0"
|
||
EVALSHA <digest SCRIPT LOAD printed> 1 <key>
|
||
```
|
||
|
||
`1` deleted the record; the key is new. `0` deleted nothing: the record is no longer
|
||
`ABANDONED` (or is gone) — look again before doing anything else. `NOSCRIPT` means the node
|
||
that holds the key (a cluster) has not loaded the program: run `SCRIPT LOAD` on that node and
|
||
`EVALSHA` again; nothing was deleted. A record that is about to expire needs nothing; once it
|
||
is gone the key is new.
|
||
- PostgreSQL:
|
||
|
||
```sql
|
||
DELETE FROM idempotency_record
|
||
WHERE record_version = 2
|
||
AND scope_hash = '<scope digest>'
|
||
AND status = 'ABANDONED';
|
||
```
|
||
|
||
Without either step the key is released when the record expires. A record whose attempt never
|
||
finished lives for the processing lease plus `APP_IDEMPOTENCY_TTL` from its claim; one the action
|
||
marked failed lives for `APP_IDEMPOTENCY_FAILURE_RETENTION` (not shorter than `APP_IDEMPOTENCY_TTL`)
|
||
from the failure (`docs/web/http-contract.md`, "서버가 기억하는 기간").
|
||
|
||
## Escalation
|
||
|
||
- P2 → P1 when the store stays unavailable for more than 10 minutes on a write path.
|
||
- Page the owner of the idempotency store (database or Redis on-call).
|