fix(lock): 락 timeout runbook이 있는 정보로 경합과 보유자를 찾게 한다

문제(round8 확인 리뷰 RR8-01-02):
- 락 timeout이 4xx carrier가 된 뒤로 어느 락이 막혔는지 서버 어디에도
  남지 않는다. 그런데 runbook은 lock.acquisition metric에서 락 이름과
  시간을 보라고 하고(metric은 outcome tag 하나뿐인 counter), 없는
  integration_lock 테이블에서 보유자를 찾으라고 한다(테이블은 INT_LOCK,
  LOCK_KEY는 key가 아니라 key의 UUID). 예외 Javadoc과 application-core
  README는 메시지가 "서버 로그 전용"이라 적지만 그 메시지를 남기는
  로그가 없다.

고른 방법(리뷰 권장안):
- runbook Symptoms·Diagnosis를 있는 정보로 다시 쓴다: 로그가 없다는 것,
  경합 비율은 lock.acquisition{outcome} 비교(JDBC 락에만, 이름·시간
  없음), INT_LOCK의 보유자 목록, 코드에서 아는 key의 보유자(key를
  Spring Integration과 같은 name-based UUID로 계산하는 SQL).
- 두 SQL에 runbook-sql 표시를 달고 postgresqlDistributedLockIntegrationTest
  가 runbook 파일에서 읽어 실 PostgreSQL에서 락을 잡은 채 돌린다(lane
  입력에 runbook을 넣어 문서를 고치면 lane이 다시 돈다).
- 예외 Javadoc·README의 "서버 로그 전용"을 "클라이언트에 보내지 않고,
  어느 로그에도 남지 않는다. 필요하면 호출자가 key()를 남긴다"로 고친다.
- 버린 안: 락 timeout에 WARN 한 줄(key digest, 대기 시간)을 남기기 —
  로그 정책을 바꾸는 소유자 결정이고, 템플릿은 tryAcquire를 부르지 않는다.

위험·한계:
- runbook은 status: stub 그대로다. Action·Escalation 본문은 도메인을
  고른 서비스가 쓴다.
- key가 UUID 문자열이면 Spring Integration은 그대로 쓰므로 계산 SQL이
  맞지 않는다. runbook은 "코드에서 아는 key"만 말한다.
- OutboxRunbookSql을 그대로 쓴다. 그 이름을 바꾸는 수정(RR8-03-03)과
  합칠 때 이 테스트의 참조도 함께 바뀌어야 한다.

검증(desktop, flock, --no-build-cache, 로그 ~/r9-g1-inbound-logs/):
- RED(문서의 주장을 단언한 임시 탐침, 기준 3eb3b067 + 탐침, 커밋 안 함,
  트리 a675c244):
  - "integration_lock 테이블에서 보유자를 찾는다": 락을 잡은 채
    SELECT count(*) FROM integration_lock → postgresqlDistributedLock
    IntegrationTest 7건 중 1건 실패, BadSqlGrammarException(relation
    "integration_lock" does not exist). red/R6-jpa-table-claim.log
  - "lock.acquisition metric에서 락 이름과 시간을 본다": timeout 뒤
    outcome 밖의 tag가 있어야 한다 → MeteredDistributedLockTest 11건 중
    1건 실패(tag 없음). red/R5-jpa-metric-claim.log
- GREEN(이 커밋 트리 fc5eae5a):
  - :application-core:check :adapter:outbound:persistence-jpa:check rc=0,
    test 1,089건 실패 0(skip 19는 application-core의 이전부터 것). 같은
    worktree의 앞 실행(green/C1-check.log, 이 두 leaf의 입력이 같은 트리
    acf42e98)으로 UP-TO-DATE. green2/C1-check.log
  - :adapter:outbound:persistence-jpa:postgresqlDistributedLockIntegrationTest
    rc=0, 8건(runbook의 두 SQL 블록을 파일에서 읽어 실행하는 새 테스트 2건
    포함) 실패 0. green2/C2-lock-lane.log
  - :app-bootstrap:documentContractTest rc=0, 67건(RunbookCoverageContract
    Test 포함). green2/C3-doc-contract.log

Refs: docs/reviews/2026-11-03-rereview/README.md 19절 RR8-01-02, round8/01-transport-errors.md
This commit is contained in:
DongHyeonka
2026-11-07 08:22:59 +09:00
parent ee5b4e1b7e
commit ba189869e8
5 changed files with 131 additions and 7 deletions
+37 -3
View File
@@ -14,7 +14,10 @@ status: stub
- HTTP 409 with `error.code=LOCK_ACQUISITION_TIMEOUT` and `Retry-After: 1`; GraphQL extension
`code=LOCK_ACQUISITION_TIMEOUT`, gRPC trailer `error-code: LOCK_ACQUISITION_TIMEOUT` (`ABORTED`), all
`retryable=true`. A 4xx answer, so no server ERROR line — look at the metric.
`retryable=true`.
- No server log line. A 4xx answer is not an incident, so no transport logs it, and the lock adapters
do not log a timeout either. The exception's message (lock key, wait time) reaches no log and no
client, so which lock timed out is recorded nowhere. Read the metric and the lock table below.
- Before 2026-11-07 the template answered this as 500 `INTERNAL_ERROR` with an "unhandled exception"
ERROR line (the exception was not an `ApiErrorCarrier`). A service on an older template sees that
instead.
@@ -22,8 +25,39 @@ status: stub
## Diagnosis
- Check `lock.acquisition` metric for lock name and duration
- Identify lock holders (check DB `integration_lock` table)
1. Contention rate. With `APP_MULTI_INSTANCE_ENABLED=true` the JDBC lock is metered: compare
`lock.acquisition{outcome="timeout"}` with `lock.acquisition{outcome="acquired"}`. The counter has
no lock name and no wait time — a key is never a metric tag (`docs/registries/metrics.yaml`). The
in-process lock (`APP_MULTI_INSTANCE_ENABLED=false`, the default) has no metric and keeps nothing in
the database; the steps below apply only to the JDBC lock.
2. Who holds locks now. `INT_LOCK` has one row per held lock. `CLIENT_ID` is the holding instance's
lock repository id, and the times are UTC. A row whose `EXPIRED_AFTER` has passed is a dead lease:
the next acquisition of that key takes it over. The oldest live rows are the long holders.
<!-- runbook-sql: lock-holders -->
```sql
SELECT lock_key, region, client_id, created_date, expired_after
FROM int_lock
ORDER BY created_date;
```
3. The holder of one lock whose key you know from the code (for example `orders/42`). `LOCK_KEY` is
not the key: Spring Integration stores the key's name-based UUID (`UUID.nameUUIDFromBytes` of the
key's UTF-8 bytes), so a key cannot be read back from the table. This computes the UUID from the
key and finds its row:
<!-- runbook-sql: lock-holder-of-key -->
```sql
SELECT region, client_id, created_date, expired_after
FROM int_lock
WHERE lock_key = (
SELECT overlay(overlay(h placing '3' from 13)
placing to_hex(8 | (('x' || substr(h, 17, 1))::bit(4)::int & 3)) from 17)::uuid::text
FROM md5('<lock_key>') AS h);
```
`:adapter:outbound:persistence-jpa:postgresqlDistributedLockIntegrationTest` runs both blocks, read
from this file, against real PostgreSQL while locks are held.
## Action
@@ -273,6 +273,14 @@ tasks.named('postgresqlConcurrentIndexMigrationIntegrationTest', Test) {
systemProperty 'migration.failed.runbook', migrationFailedRunbook.absolutePath
}
// The lock-timeout runbook gives an operator SQL to find who holds a lock; the lock lane reads it out
// of the runbook the same way, so the query an operator runs is the one that ran against PostgreSQL.
tasks.named('postgresqlDistributedLockIntegrationTest', Test) {
def lockTimeoutRunbook = new File(rootProject.projectDir.parentFile, 'docs/runbooks/lock-acquisition-timeout.md')
inputs.file(lockTimeoutRunbook).withPathSensitivity(PathSensitivity.RELATIVE).withPropertyName('lockTimeoutRunbook')
systemProperty 'lock.acquisition-timeout.runbook', lockTimeoutRunbook.absolutePath
}
tasks.named('postgresqlSecurityBaselineIntegrationTest') {
// Cross-leaf task edge: see the handoff. verifyCleanArchitectureDependencies inspects
// configurations, not the task graph, so this edge is invisible to it.
@@ -8,8 +8,12 @@ import dev.caskeleton.application.lock.DistributedLock;
import dev.caskeleton.application.lock.DistributedLockPort;
import dev.caskeleton.application.lock.LockAcquisitionTimeoutException;
import dev.caskeleton.application.lock.testkit.DistributedLockContract;
import java.nio.charset.StandardCharsets;
import java.time.Duration;
import java.util.ConcurrentModificationException;
import java.util.List;
import java.util.Map;
import java.util.UUID;
import javax.sql.DataSource;
import org.flywaydb.core.Flyway;
import org.junit.jupiter.api.AfterAll;
@@ -19,11 +23,18 @@ import org.junit.jupiter.api.Nested;
import org.junit.jupiter.api.Test;
import org.springframework.integration.jdbc.lock.DefaultLockRepository;
import org.springframework.integration.jdbc.lock.JdbcLockRegistry;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.jdbc.datasource.DataSourceTransactionManager;
/** Real PostgreSQL contract for the Spring Integration JDBC distributed-lock adapter. */
/**
* Real PostgreSQL contract for the Spring Integration JDBC distributed-lock adapter, and the SQL
* the lock-timeout runbook gives an operator to find a lock's holder, read out of the runbook
* itself.
*/
class PostgreSqlDistributedLockIntegrationTest {
private static final String LOCK_TIMEOUT_RUNBOOK = "lock.acquisition-timeout.runbook";
private static PostgreSqlReadinessSupport postgres;
private static DataSource dataSource;
@@ -120,6 +131,65 @@ class PostgreSqlDistributedLockIntegrationTest {
}
}
@Test
void theRunbookListsAHeldLockWithTheInstanceHoldingIt() throws Exception {
// given
OutboxRunbookSql runbook = OutboxRunbookSql.fromSystemProperty(LOCK_TIMEOUT_RUNBOOK);
Duration ttl = Duration.ofSeconds(30);
String holder = UUID.randomUUID().toString();
DefaultLockRepository repository = repository(dataSource, holder);
String key = "orders/runbook-holders-" + System.nanoTime();
DistributedLock held = adapter(repository, ttl).tryAcquire(key, Duration.ofSeconds(1), ttl);
try {
// when
List<Map<String, Object>> rows =
new JdbcTemplate(dataSource).queryForList(runbook.block("lock-holders"));
// then
assertThat(rows)
.filteredOn(row -> holder.equals(row.get("client_id")))
.extracting(row -> row.get("lock_key"))
.containsExactly(nameBasedUuidOf(key));
} finally {
held.close();
repository.stop();
}
}
@Test
void theRunbookFindsTheInstanceHoldingTheLockOfAKeyItIsGiven() throws Exception {
// given
OutboxRunbookSql runbook = OutboxRunbookSql.fromSystemProperty(LOCK_TIMEOUT_RUNBOOK);
Duration ttl = Duration.ofSeconds(30);
String holder = UUID.randomUUID().toString();
DefaultLockRepository holderRepository = repository(dataSource, holder);
DefaultLockRepository neighbourRepository = repository(dataSource);
String key = "orders/runbook-key-" + System.nanoTime();
DistributedLock held =
adapter(holderRepository, ttl).tryAcquire(key, Duration.ofSeconds(1), ttl);
DistributedLock neighbour =
adapter(neighbourRepository, ttl).tryAcquire(key + "/next", Duration.ofSeconds(1), ttl);
try {
// when
List<Map<String, Object>> rows =
new JdbcTemplate(dataSource)
.queryForList(runbook.block("lock-holder-of-key", Map.of("lock_key", key)));
// then
assertThat(rows).extracting(row -> row.get("client_id")).containsExactly(holder);
} finally {
held.close();
neighbour.close();
holderRepository.stop();
neighbourRepository.stop();
}
}
/** What Spring Integration stores as {@code LOCK_KEY} for a key that is not itself a UUID. */
private static String nameBasedUuidOf(String key) {
return UUID.nameUUIDFromBytes(key.getBytes(StandardCharsets.UTF_8)).toString();
}
private static DistributedLock acquireWithin(
LockRegistryDistributedLock adapter, String key, Duration leaseTtl, Duration bound)
throws InterruptedException {
@@ -143,7 +213,11 @@ class PostgreSqlDistributedLockIntegrationTest {
}
private static DefaultLockRepository repository(DataSource dataSource) {
DefaultLockRepository repository = new DefaultLockRepository(dataSource);
return repository(dataSource, UUID.randomUUID().toString());
}
private static DefaultLockRepository repository(DataSource dataSource, String clientId) {
DefaultLockRepository repository = new DefaultLockRepository(dataSource, clientId);
repository.setCheckDatabaseOnStart(false);
repository.setTransactionManager(new DataSourceTransactionManager(dataSource));
repository.afterPropertiesSet();
+4 -1
View File
@@ -582,7 +582,10 @@ application 계층은 락 획득/해제 계약만 알고, 실제 구현은 adapt
- `ApiErrorCarrier` 를 구현한다. 그래서 전송이 따로 처리기를 두지 않아도 web 은 409 +
`Retry-After: 1`(레지스트리 값), GraphQL·gRPC 는 같은 코드로 답하고, 4xx 라 서버 로그를 남기지 않는다. 전에는
carrier 가 아니어서 모든 전송이 500 `INTERNAL_ERROR`(재시도 불가)와 "unhandled" ERROR 로 답했다(재리뷰
RR7-01-04). 진단용 `getMessage()`(key·waitTime 포함)는 서버 로그 전용이고 API 클라이언트에 전달하면 안 된다.
RR7-01-04). `getMessage()`(key·waitTime 포함)는 클라이언트에 보내지 않고, 로그에도 남지 않는다. 4xx carrier 라
어느 전송도 남기지 않고, 락 adapter 도 timeout 을 남기지 않는다. 어느 락이 막혔는지 기록이 필요하면 호출자가
`key()` 를 직접 남긴다(엔티티 id 는 빼고). 운영 절차는 `docs/runbooks/lock-acquisition-timeout.md`
(`lock.acquisition` metric, `INT_LOCK` 보유자 조회)다(재리뷰 RR8-01-02).
---
@@ -11,7 +11,12 @@ import java.time.Duration;
* through {@link ApiErrorCarrier}, the hook every inbound transport reads a code from: the web
* adapter answers 409 with {@code Retry-After}, GraphQL and gRPC the same code, and none of them
* logs it as an unhandled failure. Without the hook it was answered as a 500 {@code INTERNAL_ERROR}
* (review 2026-11-06 RR7-01-04). {@link #getMessage()} is server-log-only. See README.
* (review 2026-11-06 RR7-01-04).
*
* <p>{@link #getMessage()} names the key and the wait time. It is never sent to a client, and no
* transport logs it either: a 4xx carrier is an answer, not an incident, so this timeout leaves no
* server line, and the lock adapters do not log one. A caller that wants a record of which lock
* timed out logs {@link #key()} itself, keeping entity ids out of the line. See README.
*/
public final class LockAcquisitionTimeoutException extends RuntimeException
implements ApiErrorCarrier {