fix(lock): 락 timeout runbook이 있는 정보로 경합과 보유자를 찾게 한다
문제(round8 확인 리뷰 RR8-01-02):
- 락 timeout이 4xx carrier가 된 뒤로 어느 락이 막혔는지 서버 어디에도
남지 않는다. 그런데 runbook은 lock.acquisition metric에서 락 이름과
시간을 보라고 하고(metric은 outcome tag 하나뿐인 counter), 없는
integration_lock 테이블에서 보유자를 찾으라고 한다(테이블은 INT_LOCK,
LOCK_KEY는 key가 아니라 key의 UUID). 예외 Javadoc과 application-core
README는 메시지가 "서버 로그 전용"이라 적지만 그 메시지를 남기는
로그가 없다.
고른 방법(리뷰 권장안):
- runbook Symptoms·Diagnosis를 있는 정보로 다시 쓴다: 로그가 없다는 것,
경합 비율은 lock.acquisition{outcome} 비교(JDBC 락에만, 이름·시간
없음), INT_LOCK의 보유자 목록, 코드에서 아는 key의 보유자(key를
Spring Integration과 같은 name-based UUID로 계산하는 SQL).
- 두 SQL에 runbook-sql 표시를 달고 postgresqlDistributedLockIntegrationTest
가 runbook 파일에서 읽어 실 PostgreSQL에서 락을 잡은 채 돌린다(lane
입력에 runbook을 넣어 문서를 고치면 lane이 다시 돈다).
- 예외 Javadoc·README의 "서버 로그 전용"을 "클라이언트에 보내지 않고,
어느 로그에도 남지 않는다. 필요하면 호출자가 key()를 남긴다"로 고친다.
- 버린 안: 락 timeout에 WARN 한 줄(key digest, 대기 시간)을 남기기 —
로그 정책을 바꾸는 소유자 결정이고, 템플릿은 tryAcquire를 부르지 않는다.
위험·한계:
- runbook은 status: stub 그대로다. Action·Escalation 본문은 도메인을
고른 서비스가 쓴다.
- key가 UUID 문자열이면 Spring Integration은 그대로 쓰므로 계산 SQL이
맞지 않는다. runbook은 "코드에서 아는 key"만 말한다.
- OutboxRunbookSql을 그대로 쓴다. 그 이름을 바꾸는 수정(RR8-03-03)과
합칠 때 이 테스트의 참조도 함께 바뀌어야 한다.
검증(desktop, flock, --no-build-cache, 로그 ~/r9-g1-inbound-logs/):
- RED(문서의 주장을 단언한 임시 탐침, 기준 3eb3b067 + 탐침, 커밋 안 함,
트리 a675c244):
- "integration_lock 테이블에서 보유자를 찾는다": 락을 잡은 채
SELECT count(*) FROM integration_lock → postgresqlDistributedLock
IntegrationTest 7건 중 1건 실패, BadSqlGrammarException(relation
"integration_lock" does not exist). red/R6-jpa-table-claim.log
- "lock.acquisition metric에서 락 이름과 시간을 본다": timeout 뒤
outcome 밖의 tag가 있어야 한다 → MeteredDistributedLockTest 11건 중
1건 실패(tag 없음). red/R5-jpa-metric-claim.log
- GREEN(이 커밋 트리 fc5eae5a):
- :application-core:check :adapter:outbound:persistence-jpa:check rc=0,
test 1,089건 실패 0(skip 19는 application-core의 이전부터 것). 같은
worktree의 앞 실행(green/C1-check.log, 이 두 leaf의 입력이 같은 트리
acf42e98)으로 UP-TO-DATE. green2/C1-check.log
- :adapter:outbound:persistence-jpa:postgresqlDistributedLockIntegrationTest
rc=0, 8건(runbook의 두 SQL 블록을 파일에서 읽어 실행하는 새 테스트 2건
포함) 실패 0. green2/C2-lock-lane.log
- :app-bootstrap:documentContractTest rc=0, 67건(RunbookCoverageContract
Test 포함). green2/C3-doc-contract.log
Refs: docs/reviews/2026-11-03-rereview/README.md 19절 RR8-01-02, round8/01-transport-errors.md
This commit is contained in:
@@ -14,7 +14,10 @@ status: stub
|
||||
|
||||
- HTTP 409 with `error.code=LOCK_ACQUISITION_TIMEOUT` and `Retry-After: 1`; GraphQL extension
|
||||
`code=LOCK_ACQUISITION_TIMEOUT`, gRPC trailer `error-code: LOCK_ACQUISITION_TIMEOUT` (`ABORTED`), all
|
||||
`retryable=true`. A 4xx answer, so no server ERROR line — look at the metric.
|
||||
`retryable=true`.
|
||||
- No server log line. A 4xx answer is not an incident, so no transport logs it, and the lock adapters
|
||||
do not log a timeout either. The exception's message (lock key, wait time) reaches no log and no
|
||||
client, so which lock timed out is recorded nowhere. Read the metric and the lock table below.
|
||||
- Before 2026-11-07 the template answered this as 500 `INTERNAL_ERROR` with an "unhandled exception"
|
||||
ERROR line (the exception was not an `ApiErrorCarrier`). A service on an older template sees that
|
||||
instead.
|
||||
@@ -22,8 +25,39 @@ status: stub
|
||||
|
||||
## Diagnosis
|
||||
|
||||
- Check `lock.acquisition` metric for lock name and duration
|
||||
- Identify lock holders (check DB `integration_lock` table)
|
||||
1. Contention rate. With `APP_MULTI_INSTANCE_ENABLED=true` the JDBC lock is metered: compare
|
||||
`lock.acquisition{outcome="timeout"}` with `lock.acquisition{outcome="acquired"}`. The counter has
|
||||
no lock name and no wait time — a key is never a metric tag (`docs/registries/metrics.yaml`). The
|
||||
in-process lock (`APP_MULTI_INSTANCE_ENABLED=false`, the default) has no metric and keeps nothing in
|
||||
the database; the steps below apply only to the JDBC lock.
|
||||
2. Who holds locks now. `INT_LOCK` has one row per held lock. `CLIENT_ID` is the holding instance's
|
||||
lock repository id, and the times are UTC. A row whose `EXPIRED_AFTER` has passed is a dead lease:
|
||||
the next acquisition of that key takes it over. The oldest live rows are the long holders.
|
||||
|
||||
<!-- runbook-sql: lock-holders -->
|
||||
```sql
|
||||
SELECT lock_key, region, client_id, created_date, expired_after
|
||||
FROM int_lock
|
||||
ORDER BY created_date;
|
||||
```
|
||||
|
||||
3. The holder of one lock whose key you know from the code (for example `orders/42`). `LOCK_KEY` is
|
||||
not the key: Spring Integration stores the key's name-based UUID (`UUID.nameUUIDFromBytes` of the
|
||||
key's UTF-8 bytes), so a key cannot be read back from the table. This computes the UUID from the
|
||||
key and finds its row:
|
||||
|
||||
<!-- runbook-sql: lock-holder-of-key -->
|
||||
```sql
|
||||
SELECT region, client_id, created_date, expired_after
|
||||
FROM int_lock
|
||||
WHERE lock_key = (
|
||||
SELECT overlay(overlay(h placing '3' from 13)
|
||||
placing to_hex(8 | (('x' || substr(h, 17, 1))::bit(4)::int & 3)) from 17)::uuid::text
|
||||
FROM md5('<lock_key>') AS h);
|
||||
```
|
||||
|
||||
`:adapter:outbound:persistence-jpa:postgresqlDistributedLockIntegrationTest` runs both blocks, read
|
||||
from this file, against real PostgreSQL while locks are held.
|
||||
|
||||
## Action
|
||||
|
||||
|
||||
@@ -273,6 +273,14 @@ tasks.named('postgresqlConcurrentIndexMigrationIntegrationTest', Test) {
|
||||
systemProperty 'migration.failed.runbook', migrationFailedRunbook.absolutePath
|
||||
}
|
||||
|
||||
// The lock-timeout runbook gives an operator SQL to find who holds a lock; the lock lane reads it out
|
||||
// of the runbook the same way, so the query an operator runs is the one that ran against PostgreSQL.
|
||||
tasks.named('postgresqlDistributedLockIntegrationTest', Test) {
|
||||
def lockTimeoutRunbook = new File(rootProject.projectDir.parentFile, 'docs/runbooks/lock-acquisition-timeout.md')
|
||||
inputs.file(lockTimeoutRunbook).withPathSensitivity(PathSensitivity.RELATIVE).withPropertyName('lockTimeoutRunbook')
|
||||
systemProperty 'lock.acquisition-timeout.runbook', lockTimeoutRunbook.absolutePath
|
||||
}
|
||||
|
||||
tasks.named('postgresqlSecurityBaselineIntegrationTest') {
|
||||
// Cross-leaf task edge: see the handoff. verifyCleanArchitectureDependencies inspects
|
||||
// configurations, not the task graph, so this edge is invisible to it.
|
||||
|
||||
+76
-2
@@ -8,8 +8,12 @@ import dev.caskeleton.application.lock.DistributedLock;
|
||||
import dev.caskeleton.application.lock.DistributedLockPort;
|
||||
import dev.caskeleton.application.lock.LockAcquisitionTimeoutException;
|
||||
import dev.caskeleton.application.lock.testkit.DistributedLockContract;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.time.Duration;
|
||||
import java.util.ConcurrentModificationException;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.UUID;
|
||||
import javax.sql.DataSource;
|
||||
import org.flywaydb.core.Flyway;
|
||||
import org.junit.jupiter.api.AfterAll;
|
||||
@@ -19,11 +23,18 @@ import org.junit.jupiter.api.Nested;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.springframework.integration.jdbc.lock.DefaultLockRepository;
|
||||
import org.springframework.integration.jdbc.lock.JdbcLockRegistry;
|
||||
import org.springframework.jdbc.core.JdbcTemplate;
|
||||
import org.springframework.jdbc.datasource.DataSourceTransactionManager;
|
||||
|
||||
/** Real PostgreSQL contract for the Spring Integration JDBC distributed-lock adapter. */
|
||||
/**
|
||||
* Real PostgreSQL contract for the Spring Integration JDBC distributed-lock adapter, and the SQL
|
||||
* the lock-timeout runbook gives an operator to find a lock's holder, read out of the runbook
|
||||
* itself.
|
||||
*/
|
||||
class PostgreSqlDistributedLockIntegrationTest {
|
||||
|
||||
private static final String LOCK_TIMEOUT_RUNBOOK = "lock.acquisition-timeout.runbook";
|
||||
|
||||
private static PostgreSqlReadinessSupport postgres;
|
||||
private static DataSource dataSource;
|
||||
|
||||
@@ -120,6 +131,65 @@ class PostgreSqlDistributedLockIntegrationTest {
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
void theRunbookListsAHeldLockWithTheInstanceHoldingIt() throws Exception {
|
||||
// given
|
||||
OutboxRunbookSql runbook = OutboxRunbookSql.fromSystemProperty(LOCK_TIMEOUT_RUNBOOK);
|
||||
Duration ttl = Duration.ofSeconds(30);
|
||||
String holder = UUID.randomUUID().toString();
|
||||
DefaultLockRepository repository = repository(dataSource, holder);
|
||||
String key = "orders/runbook-holders-" + System.nanoTime();
|
||||
DistributedLock held = adapter(repository, ttl).tryAcquire(key, Duration.ofSeconds(1), ttl);
|
||||
try {
|
||||
// when
|
||||
List<Map<String, Object>> rows =
|
||||
new JdbcTemplate(dataSource).queryForList(runbook.block("lock-holders"));
|
||||
|
||||
// then
|
||||
assertThat(rows)
|
||||
.filteredOn(row -> holder.equals(row.get("client_id")))
|
||||
.extracting(row -> row.get("lock_key"))
|
||||
.containsExactly(nameBasedUuidOf(key));
|
||||
} finally {
|
||||
held.close();
|
||||
repository.stop();
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
void theRunbookFindsTheInstanceHoldingTheLockOfAKeyItIsGiven() throws Exception {
|
||||
// given
|
||||
OutboxRunbookSql runbook = OutboxRunbookSql.fromSystemProperty(LOCK_TIMEOUT_RUNBOOK);
|
||||
Duration ttl = Duration.ofSeconds(30);
|
||||
String holder = UUID.randomUUID().toString();
|
||||
DefaultLockRepository holderRepository = repository(dataSource, holder);
|
||||
DefaultLockRepository neighbourRepository = repository(dataSource);
|
||||
String key = "orders/runbook-key-" + System.nanoTime();
|
||||
DistributedLock held =
|
||||
adapter(holderRepository, ttl).tryAcquire(key, Duration.ofSeconds(1), ttl);
|
||||
DistributedLock neighbour =
|
||||
adapter(neighbourRepository, ttl).tryAcquire(key + "/next", Duration.ofSeconds(1), ttl);
|
||||
try {
|
||||
// when
|
||||
List<Map<String, Object>> rows =
|
||||
new JdbcTemplate(dataSource)
|
||||
.queryForList(runbook.block("lock-holder-of-key", Map.of("lock_key", key)));
|
||||
|
||||
// then
|
||||
assertThat(rows).extracting(row -> row.get("client_id")).containsExactly(holder);
|
||||
} finally {
|
||||
held.close();
|
||||
neighbour.close();
|
||||
holderRepository.stop();
|
||||
neighbourRepository.stop();
|
||||
}
|
||||
}
|
||||
|
||||
/** What Spring Integration stores as {@code LOCK_KEY} for a key that is not itself a UUID. */
|
||||
private static String nameBasedUuidOf(String key) {
|
||||
return UUID.nameUUIDFromBytes(key.getBytes(StandardCharsets.UTF_8)).toString();
|
||||
}
|
||||
|
||||
private static DistributedLock acquireWithin(
|
||||
LockRegistryDistributedLock adapter, String key, Duration leaseTtl, Duration bound)
|
||||
throws InterruptedException {
|
||||
@@ -143,7 +213,11 @@ class PostgreSqlDistributedLockIntegrationTest {
|
||||
}
|
||||
|
||||
private static DefaultLockRepository repository(DataSource dataSource) {
|
||||
DefaultLockRepository repository = new DefaultLockRepository(dataSource);
|
||||
return repository(dataSource, UUID.randomUUID().toString());
|
||||
}
|
||||
|
||||
private static DefaultLockRepository repository(DataSource dataSource, String clientId) {
|
||||
DefaultLockRepository repository = new DefaultLockRepository(dataSource, clientId);
|
||||
repository.setCheckDatabaseOnStart(false);
|
||||
repository.setTransactionManager(new DataSourceTransactionManager(dataSource));
|
||||
repository.afterPropertiesSet();
|
||||
|
||||
@@ -582,7 +582,10 @@ application 계층은 락 획득/해제 계약만 알고, 실제 구현은 adapt
|
||||
- `ApiErrorCarrier` 를 구현한다. 그래서 전송이 따로 처리기를 두지 않아도 web 은 409 +
|
||||
`Retry-After: 1`(레지스트리 값), GraphQL·gRPC 는 같은 코드로 답하고, 4xx 라 서버 로그를 남기지 않는다. 전에는
|
||||
carrier 가 아니어서 모든 전송이 500 `INTERNAL_ERROR`(재시도 불가)와 "unhandled" ERROR 로 답했다(재리뷰
|
||||
RR7-01-04). 진단용 `getMessage()`(key·waitTime 포함)는 서버 로그 전용이고 API 클라이언트에 전달하면 안 된다.
|
||||
RR7-01-04). `getMessage()`(key·waitTime 포함)는 클라이언트에 보내지 않고, 로그에도 남지 않는다. 4xx carrier 라
|
||||
어느 전송도 남기지 않고, 락 adapter 도 timeout 을 남기지 않는다. 어느 락이 막혔는지 기록이 필요하면 호출자가
|
||||
`key()` 를 직접 남긴다(엔티티 id 는 빼고). 운영 절차는 `docs/runbooks/lock-acquisition-timeout.md`
|
||||
(`lock.acquisition` metric, `INT_LOCK` 보유자 조회)다(재리뷰 RR8-01-02).
|
||||
|
||||
---
|
||||
|
||||
|
||||
+6
-1
@@ -11,7 +11,12 @@ import java.time.Duration;
|
||||
* through {@link ApiErrorCarrier}, the hook every inbound transport reads a code from: the web
|
||||
* adapter answers 409 with {@code Retry-After}, GraphQL and gRPC the same code, and none of them
|
||||
* logs it as an unhandled failure. Without the hook it was answered as a 500 {@code INTERNAL_ERROR}
|
||||
* (review 2026-11-06 RR7-01-04). {@link #getMessage()} is server-log-only. See README.
|
||||
* (review 2026-11-06 RR7-01-04).
|
||||
*
|
||||
* <p>{@link #getMessage()} names the key and the wait time. It is never sent to a client, and no
|
||||
* transport logs it either: a 4xx carrier is an answer, not an incident, so this timeout leaves no
|
||||
* server line, and the lock adapters do not log one. A caller that wants a record of which lock
|
||||
* timed out logs {@link #key()} itself, keeping entity ids out of the line. See README.
|
||||
*/
|
||||
public final class LockAcquisitionTimeoutException extends RuntimeException
|
||||
implements ApiErrorCarrier {
|
||||
|
||||
Reference in New Issue
Block a user