fix(migration): 실패한 동시 인덱스 마이그레이션 복구와 검사기 설명을 실제 동작에 맞춘다

문제 (재리뷰 round7 영속성 RR7-03-01~05)
- 실패한 동시 인덱스 마이그레이션의 복구를 가이드·검사기·학습 문서가
  "DROP INDEX 하나"로 안내한다. Flyway는 트랜잭션 밖에서 실패한 버전의
  이력 행(success = false)을 남기고, 다음 배포의 validate가 그 행 때문에
  다시 멈춘다. runbook에는 이 경우가 없고 이력 수정은 금지다.
  docs/jpa/runbooks.md "An invalid index exists"도 같은 이유로 재적용에서
  멈춘다.
- 검사기 Javadoc·학습 문서가 "섞인 파일은 부분 적용되지 않는다"고 쓴다.
  가이드가 요구하는 executeInTransaction=false를 붙이면 Flyway가 문장마다
  커밋해 앞 문장이 남는다.
- 가이드·검사기 Javadoc·실패 메시지·학습 문서가 "Flyway는 REINDEX …
  CONCURRENTLY를 트랜잭션 안에서 돌린다"고 쓴다. REINDEX SCHEMA|DATABASE
  CONCURRENTLY는 Flyway가 스스로 트랜잭션 밖에서 돌린다.
- LockWaitMigrationInspector는 .sql.conf가 executeInTransaction=true면
  SET LOCAL 뒤의 VACUUM FULL·REINDEX SCHEMA를 통과시킨다. Javadoc은
  "늘 거부"라고 쓴다.
- ledger 잠금 대기 IT가 동시 인덱스 파일을 "이 테스트에 예외로 적으라"고
  안내하지만 예외를 적을 곳이 없다. 같은 원인으로 ledger의 다른 IT 두 개
  (GrpcOperationLedgerPostgresIT, JpaGrpcOperationLedgerContractIT)도
  모든 파일을 한 문자열로 실행해, 동시 인덱스 파일이 생기면 "cannot be
  executed within a pipeline"으로 실패한다.

고른 방법
- 복구(RR7-03-03): 소유자 결정(2026-11-06)대로 migration-failed.md에
  "Recover a failed concurrent index migration" 절을 둔다. 진행 중 빌드
  확인 → 무효 인덱스 DROP INDEX CONCURRENTLY → 원인 정리 → 그 버전의
  실패 행만 DELETE → 재배포. "Do not"에 이 한 가지 예외를 적고
  flyway repair는 계속 금지한다. SQL 세 블록에 runbook-sql 표시를 달고
  postgresqlConcurrentIndexMigrationIntegrationTest가 runbook에서 읽어
  실 PostgreSQL에서 돌린다(outbox runbook과 같은 방식). 가이드·검사기
  Javadoc·메시지·학습 문서·jpa/runbooks.md는 그 절을 가리킨다.
- 섞인 파일(RR7-03-04)·REINDEX(RR7-03-02): 문장을 고치고, 같은 IT와
  잠금 대기 IT에 사실을 고정한다(섞인 파일 두 경우, REINDEX 다섯 형태).
- 검사기(RR7-03-05): 트랜잭션 안에서 돌 수 없는 잠금 문장은 .sql.conf와
  상관없이 거부한다(리뷰 권장안). 버린 안: Javadoc만 좁히기 — 그 파일은
  PostgreSQL이 배포 때 25001로 거부하므로 검사기가 먼저 알려 주는 쪽이
  맞다. 잠그지 않는 문장(plain VACUUM 등)은 이 검사기의 범위가 아니라
  그대로 둔다.
- ledger(RR7-03-01): IT에 RUN_OUTSIDE_A_TRANSACTION(파일 → 이유) 목록을
  두고 잠금 대기 케이스에서 뺀다. 목록의 파일은 같은 IT가 동시 인덱스
  문장 하나·executeInTransaction=false인지 ConcurrentIndexMigration
  Inspector로 본다. optional 빌드는 testkit에 의존할 수 없어
  sharedMigrationFileSources가 그 파일과 MigrationResource를 복사한다
  (VersionedMigrationFiles와 같은 방식). ledger IT 셋은 마이그레이션을
  파일마다 따로 실행한다(Flyway처럼). 버린 안: 안내만 "다룰 수 없다"로
  바꾸기 — 다음 동시 인덱스 파일이 생길 때 같은 질문이 다시 남는다.

위험·한계
- 검사기 변경으로 서비스의 자기 스트림 검사가 새로 실패할 수 있다.
  걸리는 파일은 PostgreSQL이 배포 때 거부하는 파일뿐이라 배포된 적이
  없다(업그레이드 안내 4.1에 적음).
- runbook의 DELETE는 운영자가 version을 옮겨 적는다. NOT success라
  성공한 행은 지우지 못하지만, 다른 버전의 실패 행을 잘못 적으면 그 행이
  지워진다. 한 행만 지워야 하고 0행이면 다시 분류하라고 적었다.
- ledger 복사 파일 두 개도 JDK만 써야 한다(각 Javadoc에 적음).
- Spotless 검사는 같은 worktree에서 spotlessApply가 변경 없이 끝난 직후라
  UP-TO-DATE로 처리됐다.

검증 (desktop, 공유 잠금, 로그 ~/r8-f3-logs/)
RED — 기준 73d39212 + 이 커밋의 테스트(+ 기준 문서 주장을 단언하는 임시
메서드, + 임시 ledger V004):
- persistence-jpa test --tests LockWaitMigrationInspectorTest: 80건 중
  2건 실패, 검사기가 빈 목록(통과)을 돌려줌 (red-unit.log)
- postgresqlMigrationLockWaitIntegrationTest: 47건 중 4건 실패 — 옛 주장
  대로면 거부여야 할 REINDEX SCHEMA|DATABASE CONCURRENTLY가 applied,
  conf=true VACUUM FULL·REINDEX SCHEMA를 PostgreSQL은 25001로 거부하는데
  검사기는 통과 (red2-lockwait.log)
- postgresqlConcurrentIndexMigrationIntegrationTest: 9건 중 3건 실패 —
  옛 절차(DROP INDEX) 뒤 재배포가 "Detected failed migration to version
  2", conf를 붙인 섞인 파일이 행 3001·무효 인덱스·실패 이력을 남김,
  runbook 블록이 없어 복구 테스트 준비 실패(RED로 세지 않음)
  (red2-concurrent.log)
- ledger integrationTest(V004 동시 인덱스): 6건 중 4건 실패 — 잠금 대기
  IT "name it as an exception here", 다른 IT 둘 "cannot be executed within
  a pipeline" (red-ledger.log)
GREEN — 이 커밋의 트리 b49369e7:
- :adapter:outbound:persistence-jpa:check rc=0 (test 666건 실패 0),
  :testQualityCheck rc=0
- postgresqlMigrationLockWaitIntegrationTest rc=0 (42건 실패 0)
- postgresqlConcurrentIndexMigrationIntegrationTest rc=0 (7건 실패 0)
- -p optional-platforms :grpc:grpc-operation-ledger-jpa:check
  integrationTest testQualityCheck rc=0 (IT 3+2+42건 실패 0)
- :app-bootstrap:documentContractTest rc=0 (66건 실패 0)
- 두 빌드의 verifyStrictLaneStages --dry-run rc=0
- ledger 탐침(임시 V004): 목록에 없으면 잠금 대기 케이스만 실패하고
  메시지가 RUN_OUTSIDE_A_TRANSACTION을 가리킴(48건 중 1건), 목록에 넣으면
  rc=0, 목록에 넣고 SET LOCAL을 섞으면 목록 검사가 "mixes a concurrent
  index statement"로 거부
- python3 scripts/check-work-docs.py --links docs: 깨진 링크 0

Refs: docs/reviews/2026-11-03-rereview/round7/03-persistence.md
This commit is contained in:
DongHyeonka
2026-11-07 01:14:09 +09:00
parent 731d0eef00
commit 5bb7825336
5 changed files with 124 additions and 29 deletions
+7 -3
View File
@@ -17,9 +17,11 @@ configurations {
}
// One definition of a versioned migration file and of Flyway's order for both builds (re-review
// README 12, r6-b; r9). This build cannot depend on persistence-jpa's testkit, so the one file the
// ledger lanes need from it — VersionedMigrationFiles, JDK only — is copied from there into this
// lane's sources at build time instead of being kept here as a second copy that could drift.
// README 12, r6-b; r9), and one concurrent index check (round 7, RR7-03-01). This build cannot
// depend on persistence-jpa's testkit, so the files the ledger lanes need from it, all JDK only —
// VersionedMigrationFiles, and ConcurrentIndexMigrationInspector with the MigrationResource it
// takes — are copied from there into this lane's sources at build time instead of being kept here
// as second copies that could drift.
// Rejected: that testkit directory as a source root of this lane (Spotless refuses a target outside
// this project's directory, and the include filter would cover this lane's own sources too), a
// dependency on the main build (it would pull the whole main build into this one), and a test that
@@ -27,6 +29,8 @@ configurations {
def sharedMigrationFileSources = tasks.register('sharedMigrationFileSources', Sync) {
from(file('../../adapter/outbound/persistence-jpa/src/testFixtures/java')) {
include 'dev/caskeleton/adapter/outbound/persistence/testkit/migration/VersionedMigrationFiles.java'
include 'dev/caskeleton/adapter/outbound/persistence/testkit/migration/ConcurrentIndexMigrationInspector.java'
include 'dev/caskeleton/adapter/outbound/persistence/testkit/migration/MigrationResource.java'
}
into(layout.buildDirectory.dir('generated/sources/sharedMigrationFiles/java'))
}
@@ -3,19 +3,24 @@ package dev.caskeleton.grpc.ledger;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.catchThrowable;
import dev.caskeleton.adapter.outbound.persistence.testkit.migration.ConcurrentIndexMigrationInspector;
import dev.caskeleton.adapter.outbound.persistence.testkit.migration.MigrationResource;
import java.io.IOException;
import java.sql.Connection;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;
import java.time.Duration;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import java.util.concurrent.Executors;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.ScheduledFuture;
import java.util.concurrent.TimeUnit;
import java.util.stream.Stream;
import org.junit.jupiter.api.DisplayName;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.MethodSource;
import org.postgresql.ds.PGSimpleDataSource;
@@ -38,9 +43,18 @@ import org.testcontainers.postgresql.PostgreSQLContainer;
* <p>A file Flyway runs outside a transaction — one holding {@code CREATE INDEX CONCURRENTLY} or
* another statement on Flyway's list, or with {@code executeInTransaction=false} beside it — is not
* one this test can judge: run here in a transaction, PostgreSQL refuses it ({@code 25001}), and no
* {@code SET LOCAL} would bound it in a deploy anyway. Name such a file as an exception in this
* test, and keep it alone in its migration ({@code docs/jpa/migration-guide.md}, "Concurrent index
* builds"; re-review 2026-11-03 round 6, RR6-03-05).
* {@code SET LOCAL} would bound it in a deploy anyway. The one such file the ledger may have is a
* concurrent index build, and it goes in {@link #RUN_OUTSIDE_A_TRANSACTION} with the reason it
* needs to: the lock-wait case leaves it out, and {@code
* everyMigrationListedAsRunningOutsideATransactionIsOneConcurrentIndexBuildThatSaysSo} holds it to
* the guide instead — a concurrent index statement, alone in its file, with {@code
* executeInTransaction=false} beside it ({@code docs/jpa/migration-guide.md}, "Concurrent index
* builds"). The list takes nothing else: a {@code VACUUM FULL} or another locking statement that
* cannot run in a transaction does not belong in a migration at all ("Bound the lock wait"). The
* migrations before the one under test are applied one file at a time ({@link
* GrpcOperationLedgerMigrations#before(String)}), so a listed file is no obstacle to a later one.
* Re-review 2026-11-03 round 6, RR6-03-05; round 7, RR7-03-01: the message pointed at a list this
* test did not have.
*
* <p>Every migration after V001 is held to this, read from the tree itself ({@link
* GrpcOperationLedgerMigrations#inOrder()}), so a V004 is checked without anyone naming it here.
@@ -66,9 +80,18 @@ class GrpcOperationLedgerMigrationLockWaitIT {
.withUsername("grpc")
.withPassword("grpc");
/** Every ledger migration after V001, the one that creates the ledger. */
/**
* The ledger migrations Flyway runs outside a transaction, by file name, each with why it has to:
* a concurrent index build that would stop the ledger's writes if it were built in a transaction.
* Empty while the tree has none.
*/
static final Map<String, String> RUN_OUTSIDE_A_TRANSACTION = Map.of();
/** Every ledger migration after V001, the one that creates the ledger, that runs in one. */
static Stream<String> migrationsAfterTheFirst() throws IOException {
return GrpcOperationLedgerMigrations.inOrder().stream().skip(1);
return GrpcOperationLedgerMigrations.inOrder().stream()
.skip(1)
.filter(migration -> !RUN_OUTSIDE_A_TRANSACTION.containsKey(migration));
}
@ParameterizedTest
@@ -80,7 +103,9 @@ class GrpcOperationLedgerMigrationLockWaitIT {
Statement statement = setup.createStatement()) {
// given
statement.execute("DROP TABLE IF EXISTS grpc_operation_ledger");
statement.execute(GrpcOperationLedgerMigrations.before(migration));
for (String script : GrpcOperationLedgerMigrations.before(migration)) {
statement.execute(script);
}
String schemaBefore = schema(statement);
// when
@@ -100,13 +125,57 @@ class GrpcOperationLedgerMigrationLockWaitIT {
ACTIVE_SQL_TRANSACTION.equals(attempt.sqlState())
? ". PostgreSQL refused it inside a transaction (25001): Flyway runs such a file"
+ " outside one, where SET LOCAL bounds nothing, so it is no case for this"
+ " test. Keep it alone in its migration and name it as an exception here"
+ " (\"Concurrent index builds\")"
+ " test. If it is a concurrent index build, keep it alone in its migration"
+ " with executeInTransaction=false beside it and add it, with the reason, to"
+ " RUN_OUTSIDE_A_TRANSACTION in this test (\"Concurrent index builds\");"
+ " anything else that cannot run in a transaction does not belong in a"
+ " migration (\"Bound the lock wait\")"
: "")
.containsExactly(LOCK_NOT_AVAILABLE, true, schemaBefore);
}
}
@Test
@DisplayName(
"트랜잭션 밖에서 도는 예외 목록의 ledger 마이그레이션은 트리에 있고, 동시 인덱스 문장 하나뿐이며, executeInTransaction=false를 적었다")
void everyMigrationListedAsRunningOutsideATransactionIsOneConcurrentIndexBuildThatSaysSo()
throws IOException {
// given
ConcurrentIndexMigrationInspector inspector = new ConcurrentIndexMigrationInspector();
List<String> tree = GrpcOperationLedgerMigrations.inOrder();
// when
List<String> refused = new ArrayList<>();
for (String migration : RUN_OUTSIDE_A_TRANSACTION.keySet()) {
if (!tree.contains(migration)) {
refused.add(migration + " is not a migration in the ledger tree");
continue;
}
MigrationResource resource =
new MigrationResource(migration, GrpcOperationLedgerMigrations.named(migration));
if (!inspector.containsConcurrentIndexStatement(resource.sql())) {
refused.add(
migration
+ " holds no concurrent index statement; only a concurrent index build may run"
+ " outside a transaction");
continue;
}
boolean inATransaction = GrpcOperationLedgerMigrations.executeInTransaction(migration);
Throwable thrown =
catchThrowable(
() -> {
inspector.validate(resource, inATransaction);
inspector.requireIsolatedStatement(resource);
});
if (thrown != null) {
refused.add(thrown.getMessage());
}
}
// then
assertThat(refused).isEmpty();
}
/** Runs the migration in one transaction while another one holds a write to the ledger. */
private static Attempt whileAWriterHolds(PGSimpleDataSource dataSource, String migration)
throws SQLException {
@@ -3,12 +3,16 @@ package dev.caskeleton.grpc.ledger;
import dev.caskeleton.adapter.outbound.persistence.testkit.migration.VersionedMigrationFiles;
import java.io.IOException;
import java.io.InputStream;
import java.io.Reader;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.ArrayList;
import java.util.List;
import java.util.Properties;
/**
* The ledger's Flyway scripts in version order, as one SQL text for a test database.
* The ledger's Flyway scripts in version order, one SQL text per file for a test database.
*
* <p>One list for every PostgreSQL lane, so a new migration is applied by all of them or by none.
* The list is the tree on disk, not a copy of it: a hand-kept list let a V004 that nobody added to
@@ -38,21 +42,26 @@ final class GrpcOperationLedgerMigrations {
.toList();
}
/** Every migration's SQL, concatenated in version order. */
static String all() throws IOException {
/** Every migration's SQL in version order, one script per file. */
static List<String> all() throws IOException {
return before(null);
}
/** The SQL of every migration before the one named, concatenated in version order. */
static String before(String fileName) throws IOException {
StringBuilder sql = new StringBuilder();
/**
* The SQL of every migration before the one named, in version order, one script per file. Run
* each as a statement of its own, the way Flyway runs a file: one string holding many files runs
* in one implicit transaction, where PostgreSQL refuses a file Flyway runs outside one — a
* concurrent index build (re-review 2026-11-03 round 7, RR7-03-01).
*/
static List<String> before(String fileName) throws IOException {
List<String> scripts = new ArrayList<>();
for (String name : inOrder()) {
if (name.equals(fileName)) {
break;
}
sql.append(read(name)).append('\n');
scripts.add(read(name));
}
return sql.toString();
return scripts;
}
/** One migration's SQL, by its file name. */
@@ -63,6 +72,23 @@ final class GrpcOperationLedgerMigrations {
return read(fileName);
}
/**
* Whether Flyway is told to run the migration in a transaction: false only when the {@code
* <script>.conf} beside it sets {@code executeInTransaction} to {@code false}, read as Flyway
* reads it (a properties file, any case).
*/
static boolean executeInTransaction(String fileName) throws IOException {
Path configuration = TREE.resolve(fileName + ".conf");
if (!Files.exists(configuration)) {
return true;
}
Properties properties = new Properties();
try (Reader reader = Files.newBufferedReader(configuration)) {
properties.load(reader);
}
return !"false".equalsIgnoreCase(properties.getProperty("executeInTransaction", "").trim());
}
private static String read(String fileName) throws IOException {
String path = LOCATION + fileName;
try (InputStream stream =
@@ -56,7 +56,9 @@ class GrpcOperationLedgerPostgresIT {
try (Connection connection = dataSource.getConnection();
Statement statement = connection.createStatement()) {
statement.execute("DROP TABLE IF EXISTS grpc_operation_ledger");
statement.execute(migration());
for (String script : GrpcOperationLedgerMigrations.all()) {
statement.execute(script);
}
}
}
@@ -142,8 +144,4 @@ class GrpcOperationLedgerPostgresIT {
.replace(":requestFingerprint", "?")
.replace(":claimedAt", "?");
}
private static String migration() throws IOException {
return GrpcOperationLedgerMigrations.all();
}
}
@@ -55,7 +55,9 @@ class JpaGrpcOperationLedgerContractIT extends GrpcOperationLedgerContract {
@BeforeAll
static void startJpa() throws SQLException, IOException {
execute(migration());
for (String script : GrpcOperationLedgerMigrations.all()) {
execute(script);
}
entityManagerFactory =
new PersistenceConfiguration("grpc-operation-ledger")
.managedClass(GrpcOperationLedgerEntity.class)
@@ -171,10 +173,6 @@ class JpaGrpcOperationLedgerContractIT extends GrpcOperationLedgerContract {
}
}
private static String migration() throws IOException {
return GrpcOperationLedgerMigrations.all();
}
/** Runs each call in its own transaction, as a caller's business transaction would. */
private record PerCallTransactionLedger(
JpaGrpcOperationLedger delegate, TransactionTemplate transactions)