--- title: Runbook — LOCK_ACQUISITION_TIMEOUT (분산 락 획득 타임아웃) category: CONFLICT error_codes: [LOCK_ACQUISITION_TIMEOUT] severity: P2 owner: oncall last_updated: 2026-11-07 status: stub --- # Runbook: LOCK_ACQUISITION_TIMEOUT (`runbook://lock/acquisition-timeout`) ## Symptoms - HTTP 409 with `error.code=LOCK_ACQUISITION_TIMEOUT` and `Retry-After: 1`; GraphQL extension `code=LOCK_ACQUISITION_TIMEOUT`, gRPC trailer `error-code: LOCK_ACQUISITION_TIMEOUT` (`ABORTED`), all `retryable=true`. - No server log line. A 4xx answer is not an incident, so no transport logs it, and the lock adapters do not log a timeout either. The exception's message (lock key, wait time) reaches no log and no client, so which lock timed out is recorded nowhere. Read the metric and the lock table below. - Before 2026-11-07 the template answered this as 500 `INTERNAL_ERROR` with an "unhandled exception" ERROR line (the exception was not an `ApiErrorCarrier`). A service on an older template sees that instead. - Distributed lock wait exceeded configured timeout; high contention on a resource ## Diagnosis 1. Contention rate. With `APP_MULTI_INSTANCE_ENABLED=true` the JDBC lock is metered: compare `lock.acquisition{outcome="timeout"}` with `lock.acquisition{outcome="acquired"}`. The counter has no lock name and no wait time — a key is never a metric tag (`docs/registries/metrics.yaml`). The in-process lock (`APP_MULTI_INSTANCE_ENABLED=false`, the default) has no metric and keeps nothing in the database; the steps below apply only to the JDBC lock. 2. Who holds locks now. `INT_LOCK` has one row per held lock. `CLIENT_ID` is the holding instance's lock repository id, and the times are UTC. A row whose `EXPIRED_AFTER` has passed is a dead lease: the next acquisition of that key takes it over. The oldest live rows are the long holders. ```sql SELECT lock_key, region, client_id, created_date, expired_after FROM int_lock ORDER BY created_date; ``` 3. The holder of one lock whose key you know from the code (for example `orders/42`). `LOCK_KEY` is not the key: Spring Integration stores the key's name-based UUID (`UUID.nameUUIDFromBytes` of the key's UTF-8 bytes), so a key cannot be read back from the table. This computes the UUID from the key and finds its row: ```sql SELECT region, client_id, created_date, expired_after FROM int_lock WHERE lock_key = ( SELECT overlay(overlay(h placing '3' from 13) placing to_hex(8 | (('x' || substr(h, 17, 1))::bit(4)::int & 3)) from 17)::uuid::text FROM md5('') AS h); ``` `:adapter:outbound:persistence-jpa:postgresqlDistributedLockIntegrationTest` runs both blocks, read from this file, against real PostgreSQL while locks are held. ## Action - Client should retry with backoff (retryable=true) - Optimize critical section holding time if lock contention is systemic ## Escalation - Escalate if lock holder appears stuck (potential deadlock in distributed lock) --- > **Stub**: Phase D2 — author body after domain adoption. (feature-operational-runbook-contract D9)