fix(migration): 실패한 DROP INDEX CONCURRENTLY를 runbook 절차로 복구되게 한다

문제 (재리뷰 round8 RR8-03-01)
- migration-failed.md "Recover a failed concurrent index migration"은
  CREATE·DROP·REINDEX 동시 문장 셋을 대상으로 두고, 2단계에서 "빌드가 남긴
  무효 인덱스를 지운다"고 한다. 실패한 DROP INDEX CONCURRENTLY가 남긴 무효
  인덱스는 그 마이그레이션이 지우려던 인덱스다. 절차대로 지우고 실패 행을
  지운 뒤 다시 배포하면 같은 문장이 42704(index does not exist)로 실패하고
  실패 행이 다시 생긴다. 운영자는 처음과 같은 막힌 상태로 돌아간다.
- 고치다 같은 절에서 하나 더 찾았다. 2단계(DROP INDEX CONCURRENTLY)가
  3단계(원인 정리) 앞이다. 취소·timeout이 원인이면 그 열린 트랜잭션이
  운영자의 DROP INDEX CONCURRENTLY도 붙잡아, 2단계가 3단계를 기다린다.
- runbook을 실행하는 IT는 중복으로 실패한 unique CREATE 한 경우만 돌렸다.

고른 방법
- 리뷰 권장안: 무효 인덱스를 지우는 단계를 문장 종류로 나눈다. CREATE는
  그 인덱스를, REINDEX는 _ccnew(또는 _ccold) 사본을 지운다. DROP INDEX
  CONCURRENTLY면 이 단계를 건너뛴다. 다시 배포한 같은 문장이 그 인덱스를
  지운다(PostgreSQL은 중간에 실패한 DROP INDEX CONCURRENTLY의 재시도를
  받는다). 5단계 문장도 "builds the index"에서 세 경우로 고친다.
- 단계 순서를 "빌드 확인 → 원인 정리 → 무효 인덱스 삭제 → 실패 행 삭제 →
  재배포"로 바꾼다. 실패 행만 지운다는 소유자 결정(2026-11-06)은 그대로다.
- IT에 두 경우를 더한다. 열린 쓰기를 기다리다 취소된 REINDEX와 열린
  읽기를 기다리다 취소된 DROP에 runbook SQL을 runbook에서 읽어 돌리고,
  재배포 뒤 이력 [1 true, 2 true], 무효 인덱스 없음, V1 checksum 그대로를
  단언한다. 대기를 30초 기다리지 않고, 문장이 잠금을 기다리는 것을 본
  즉시 취소하는 도우미(cancelledOnceItWaits)를 둔다.
- 같은 절차를 요약한 곳을 같이 고친다: docs/jpa/runbooks.md "An invalid
  index exists", migration-guide "Concurrent index builds"(DROP이 무엇을
  기다리는지 포함), 학습 문서 8.2절, ConcurrentIndexMigrationInspector의
  Javadoc과 실패 메시지, 업그레이드 안내 5절 행(그 범위가 아직 main에
  없어 행을 최종 동작으로 고치고 머리 목록에 더함).
- 버린 대안: 가이드와 검사기가 DROP INDEX CONCURRENTLY에 IF EXISTS를
  요구한다. 이미 배포된 파일은 고칠 수 없어 새 파일에만 통하고, 서비스의
  기존 마이그레이션 검사를 새로 실패시킨다.

위험·한계
- 운영자가 실패한 문장의 종류를 마이그레이션 파일에서 읽어야 한다.
  runbook이 이력 행만으로는 알 수 없는 것을 요구한다.
- REINDEX가 swap 뒤에 실패해 _ccold가 남는 경우는 문서로만 다뤘다(실행
  재현 없음). PostgreSQL 문서의 권장 복구와 같다.
- 단계 번호가 바뀌었다. 리뷰 기록이 옛 번호(2단계 = drop)로 적은 곳은 그때의
  기록이라 두었다.

RED (desktop, 기준 3eb3b067 + 임시 테스트, 로그 ~/r9-g3-logs/red-concurrent.log)
- postgresqlConcurrentIndexMigrationIntegrationTest rc=1, 9건 중 2건 실패
  - redAFailedConcurrentDropIsRecoveredByTheBaseRunbook: 기준 runbook
    1·2·4단계 뒤 재배포가 "42704 ERROR: index "ix_probe_id" does not
    exist"로 실패, 이력 [1 true, 2 false] 그대로
  - redTheBaseRunbookDropsBeforeItEndsTheTransaction: 기준 순서의 2단계
    (_ccnew drop)가 열린 쓰기 트랜잭션을 기다려 취소됨
  - 나머지 7건(기존) 통과

검증 (desktop, 이 커밋의 트리 7cfaa83c, --no-build-cache, 로그 ~/r9-g3-logs/)
- :adapter:outbound:persistence-jpa:check rc=0, test 666건 실패·skip 0
- postgresqlConcurrentIndexMigrationIntegrationTest rc=0, 9건(새 2건 포함)
- :adapter:outbound:persistence-jpa:testQualityCheck rc=0
- :app-bootstrap:documentContractTest rc=0, 67건(RunbookCoverageContractTest
  포함)
- -p optional-platforms :grpc:grpc-operation-ledger-jpa:integrationTest
  rc=0, 47건(검사기 사본이 바뀌어서)
- FROM-CACHE 0
- 상대 링크(노트북): python3 scripts/check-work-docs.py --links로 바꾼
  문서 5개(migration-failed.md, jpa/runbooks.md, migration-guide.md,
  학습 문서, 업그레이드 안내) rc=0, 깨진 링크 0

Refs: docs/reviews/2026-11-03-rereview/round8/03-migration.md RR8-03-01
Fixes: 6ec0c4f7fa ("fix(migration): 실패한 동시 인덱스 마이그레이션 복구와 검사기 설명을 실제 동작에 맞춘다")
This commit is contained in:
DongHyeonka
2026-11-07 08:07:42 +09:00
parent dae6dcaf51
commit 8ebe98e69a
7 changed files with 262 additions and 59 deletions
@@ -9,8 +9,11 @@ import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;
import java.time.Duration;
import java.util.List;
import java.util.Locale;
@@ -57,13 +60,17 @@ import org.springframework.jdbc.datasource.DelegatingDataSource;
* deploy still waiting after {@link #FINISHES_WITHIN} is cancelled, which leaves the invalid index
* an interrupted build leaves in production too.
*
* <p>Three more cases pin what happens when such a deploy fails (re-review 2026-11-03 round 7,
* RR7-03-03 and RR7-03-04). A concurrent build next to another statement is refused before any of
* it runs when nothing configures the file, and is left half applied when {@code
* executeInTransaction=false} does: Flyway commits each statement on its own. A failed build leaves
* an invalid index and a failed history row, and the deploy after it is refused until the row is
* gone: the recovery case runs the SQL of {@code docs/runbooks/migration-failed.md} ("Recover a
* failed concurrent index migration"), read out of the runbook itself, and deploys again.
* <p>Five more cases pin what happens when such a deploy fails (re-review 2026-11-03 round 7,
* RR7-03-03 and RR7-03-04; round 8, RR8-03-01). A concurrent build next to another statement is
* refused before any of it runs when nothing configures the file, and is left half applied when
* {@code executeInTransaction=false} does: Flyway commits each statement on its own. A concurrent
* statement alone that fails leaves an invalid index and a failed history row, and the deploy after
* it is refused until the row is gone. The recovery cases run the SQL of {@code
* docs/runbooks/migration-failed.md} ("Recover a failed concurrent index migration"), read out of
* the runbook itself, for each statement the runbook covers, and deploy again: a unique build that
* fails on a duplicate, and a rebuild and a drop cancelled while they wait for an open transaction.
* The drop leaves invalid the very index it was dropping, so the runbook leaves that index for the
* deploy again to drop: dropped by hand first, the same statement fails with {@code 42704}.
*/
class PostgreSqlConcurrentIndexMigrationIntegrationTest {
@@ -89,6 +96,12 @@ class PostgreSqlConcurrentIndexMigrationIntegrationTest {
private static final String FLYWAY_REFUSES_A_MIXED_MIGRATION =
"Detected both transactional and non-transactional statements";
/** A read of the table, which a {@code DROP INDEX CONCURRENTLY} waits for while it is open. */
private static final String A_READ = "SELECT count(*) FROM probe";
/** A write to the table, which a {@code REINDEX ... CONCURRENTLY} waits for while it is open. */
private static final String A_WRITE = "UPDATE probe SET note = note WHERE id = 1";
/** The system property naming {@code docs/runbooks/migration-failed.md} for this lane. */
private static final String MIGRATION_FAILED_RUNBOOK = "migration.failed.runbook";
@@ -203,7 +216,7 @@ class PostgreSqlConcurrentIndexMigrationIntegrationTest {
@Test
@DisplayName(
"실패한 동시 인덱스 마이그레이션은 runbook 절차(빌드 확인, 무효 인덱스 삭제, 원인 정리, 그 버전의 실패 이력 행 삭제) 뒤 다시 배포하면 적용된다")
"중복 값으로 실패한 unique 동시 인덱스 빌드는 runbook 절차(빌드 확인, 원인 정리, 무효 인덱스 삭제, 그 버전의 실패 이력 행 삭제) 뒤 다시 배포하면 적용된다")
void aFailedConcurrentIndexMigrationIsRecoveredByTheRunbookProcedure(@TempDir Path stream)
throws Exception {
RunbookSql runbook = RunbookSql.fromSystemProperty(MIGRATION_FAILED_RUNBOOK);
@@ -220,10 +233,10 @@ class PostgreSqlConcurrentIndexMigrationIntegrationTest {
// when
List<Map<String, Object>> buildsRunning =
jdbc.queryForList(runbook.block("failed-concurrent-index-builds"));
jdbc.execute("DELETE FROM probe WHERE id = 2001");
jdbc.execute(
runbook.block(
"failed-concurrent-index-drop", Map.of("invalid_index", "ix_probe_note_u")));
jdbc.execute("DELETE FROM probe WHERE id = 2001");
jdbc.execute(
runbook.block(
"failed-concurrent-index-history-row",
@@ -251,6 +264,113 @@ class PostgreSqlConcurrentIndexMigrationIntegrationTest {
}
}
@Test
@DisplayName(
"열린 트랜잭션을 기다리다 취소된 REINDEX CONCURRENTLY는 runbook 절차(빌드 확인, 원인 정리, 남은 _ccnew 무효 인덱스 삭제, 그 버전의 실패 이력 행 삭제) 뒤 다시 배포하면 적용된다")
void aFailedConcurrentRebuildIsRecoveredByTheRunbookProcedure(@TempDir Path stream)
throws Exception {
RunbookSql runbook = RunbookSql.fromSystemProperty(MIGRATION_FAILED_RUNBOOK);
String name = databaseName();
try (HikariDataSource database = postgres.createDatabase(name)) {
// given
Files.writeString(stream.resolve("V1__probe.sql"), FIRST_RELEASE);
migrateTheWayTheTemplateDoes(database, stream);
String checksumOfV1 = checksumOf(database, "1");
writeConcurrentMigration(stream, "REINDEX INDEX CONCURRENTLY ix_probe_id");
Throwable failed;
String afterFailure;
List<Map<String, Object>> buildsRunning;
JdbcTemplate jdbc = new JdbcTemplate(database);
try (Connection writer = openTransactionOn(database, A_WRITE)) {
failed = cancelledOnceItWaits(name, () -> migrateTheWayTheTemplateDoes(database, stream));
afterFailure = aftermath(database);
// when
buildsRunning = jdbc.queryForList(runbook.block("failed-concurrent-index-builds"));
writer.rollback();
}
jdbc.execute(
runbook.block(
"failed-concurrent-index-drop", Map.of("invalid_index", "ix_probe_id_ccnew")));
jdbc.execute(
runbook.block(
"failed-concurrent-index-history-row",
Map.of("owned_history_table", "flyway_schema_history", "version", "2")));
Throwable again = catchThrowable(() -> migrateTheWayTheTemplateDoes(database, stream));
// then
assertThat(
List.of(
failed != null,
afterFailure,
buildsRunning.size(),
String.valueOf(again),
aftermath(database),
checksumOf(database, "1")))
.containsExactly(
true,
"row 3001 0, indexes [ix_probe_id true, ix_probe_id_ccnew false, probe_pkey true],"
+ " history [1 true, 2 false]",
0,
"null",
"row 3001 0, indexes [ix_probe_id true, probe_pkey true], history [1 true, 2 true]",
checksumOfV1);
}
}
@Test
@DisplayName(
"열린 트랜잭션을 기다리다 취소된 DROP INDEX CONCURRENTLY는 runbook 절차대로 빌드 확인, 원인 정리, 그 버전의 실패 이력 행 삭제만 하고 남은 무효 인덱스는 두면 다시 배포한 같은 문장이 그 인덱스를 지우고 적용된다")
void aFailedConcurrentDropIsRecoveredByTheRunbookProcedure(@TempDir Path stream)
throws Exception {
RunbookSql runbook = RunbookSql.fromSystemProperty(MIGRATION_FAILED_RUNBOOK);
String name = databaseName();
try (HikariDataSource database = postgres.createDatabase(name)) {
// given
Files.writeString(stream.resolve("V1__probe.sql"), FIRST_RELEASE);
migrateTheWayTheTemplateDoes(database, stream);
String checksumOfV1 = checksumOf(database, "1");
writeConcurrentMigration(stream, "DROP INDEX CONCURRENTLY ix_probe_id");
Throwable failed;
String afterFailure;
List<Map<String, Object>> buildsRunning;
JdbcTemplate jdbc = new JdbcTemplate(database);
try (Connection reader = openTransactionOn(database, A_READ)) {
failed = cancelledOnceItWaits(name, () -> migrateTheWayTheTemplateDoes(database, stream));
afterFailure = aftermath(database);
// when
buildsRunning = jdbc.queryForList(runbook.block("failed-concurrent-index-builds"));
reader.rollback();
}
int deleted =
jdbc.update(
runbook.block(
"failed-concurrent-index-history-row",
Map.of("owned_history_table", "flyway_schema_history", "version", "2")));
Throwable again = catchThrowable(() -> migrateTheWayTheTemplateDoes(database, stream));
// then
assertThat(
List.of(
failed != null,
afterFailure,
buildsRunning.size(),
deleted,
String.valueOf(again),
aftermath(database),
checksumOf(database, "1")))
.containsExactly(
true,
"row 3001 0, indexes [ix_probe_id false, probe_pkey true], history [1 true, 2 false]",
0,
1,
"null",
"row 3001 0, indexes [probe_pkey true], history [1 true, 2 true]",
checksumOfV1);
}
}
/** V2: the one concurrent statement, with the companion configuration the guide asks for. */
private static void writeConcurrentMigration(Path stream, String statement) throws IOException {
Files.writeString(stream.resolve("V2__concurrent_index.sql"), statement + ";\n");
@@ -332,6 +452,60 @@ class PostgreSqlConcurrentIndexMigrationIntegrationTest {
}
}
/**
* A connection to the database with a transaction open on the table: {@code statement} has run in
* it and holds its lock until the caller ends the transaction, as a report or an operator's
* {@code BEGIN} would. Opened outside the test's pool, which the deploy needs.
*/
private static Connection openTransactionOn(HikariDataSource database, String statement)
throws SQLException {
Connection holder =
DriverManager.getConnection(
database.getJdbcUrl(), database.getUsername(), database.getPassword());
holder.setAutoCommit(false);
try (Statement holding = holder.createStatement()) {
holding.execute(statement);
}
return holder;
}
/**
* Runs the deploy until its statement waits for a lock in the database, cancels that wait as an
* operator ending a stuck deploy would, and returns how the deploy failed.
*/
private static Throwable cancelledOnceItWaits(String databaseName, Runnable migration)
throws Exception {
ExecutorService deployer = Executors.newSingleThreadExecutor();
try {
Future<?> deploy = deployer.submit(migration);
long deadline = System.nanoTime() + FINISHES_WITHIN.toNanos();
while (!aStatementWaitsForALock(databaseName)) {
if (deploy.isDone() || System.nanoTime() > deadline) {
throw new IllegalStateException("the deploy never waited for a lock");
}
Thread.sleep(50);
}
cancelLockWaits(databaseName);
return catchThrowable(() -> deploy.get(FINISHES_WITHIN.toMillis(), TimeUnit.MILLISECONDS));
} finally {
deployer.shutdownNow();
}
}
private static boolean aStatementWaitsForALock(String databaseName) throws SQLException {
try (Connection admin = postgres.connection();
PreparedStatement waiting =
admin.prepareStatement(
"SELECT count(*) FROM pg_stat_activity"
+ " WHERE datname = ? AND wait_event_type = 'Lock'"
+ " AND pid <> pg_backend_pid()")) {
waiting.setString(1, databaseName);
try (ResultSet count = waiting.executeQuery()) {
return count.next() && count.getInt(1) > 0;
}
}
}
/** The version the history ends on and whether every index on the table is usable. */
private static String state(DataSource database) {
JdbcTemplate jdbc = new JdbcTemplate(database);
@@ -22,11 +22,13 @@ import java.util.regex.Pattern;
* statement it is refused by Flyway before any of the script runs when nothing configures the file
* (Flyway's default {@code mixed=false}); with the {@code executeInTransaction=false} that {@link
* #validate} asks for, Flyway runs and commits each statement on its own, and a failed build leaves
* the statements before it applied (round 7, RR7-03-04). Alone, a failed build leaves an invalid
* index and its version's failed history row, which Flyway refuses the next deploy over until the
* row is gone: {@code docs/runbooks/migration-failed.md} ("Recover a failed concurrent index
* migration") drops the index and deletes that one row ({@code
* postgresqlConcurrentIndexMigrationIntegrationTest} runs it; round 7, RR7-03-03).
* the statements before it applied (round 7, RR7-03-04). Alone, a failed statement leaves an
* invalid index and its version's failed history row, which Flyway refuses the next deploy over
* until the row is gone: {@code docs/runbooks/migration-failed.md} ("Recover a failed concurrent
* index migration") deletes that one row, after dropping the index a failed build or rebuild left.
* A failed {@code DROP INDEX CONCURRENTLY} leaves invalid the index it was dropping, and the deploy
* after the row is gone drops it ({@code postgresqlConcurrentIndexMigrationIntegrationTest} runs
* all three; round 7, RR7-03-03; round 8, RR8-03-01).
*
* <p>Outside a transaction, {@code CREATE INDEX CONCURRENTLY} and {@code REINDEX ... CONCURRENTLY}
* still wait for every transaction older than themselves. They finish because the template's Flyway
@@ -84,12 +86,12 @@ public final class ConcurrentIndexMigrationInspector {
/**
* Fails when a concurrent-index migration also contains other statements.
*
* <p>A concurrent index build can fail and leave an invalid index and its version's failed
* <p>A concurrent index statement can fail and leave an invalid index and its version's failed
* history row behind. When the migration did nothing else, recovering is the runbook's few
* statements: drop the index, delete that one failed row, deploy again ({@code
* docs/runbooks/migration-failed.md}, "Recover a failed concurrent index migration"). When it did
* more, Flyway ran the file outside a transaction and has committed the statements before the
* build, and that partial state has to be reconstructed by hand.
* statements: drop the index a failed build left, delete that one failed row, deploy again
* ({@code docs/runbooks/migration-failed.md}, "Recover a failed concurrent index migration").
* When it did more, Flyway ran the file outside a transaction and has committed the statements
* before the failed one, and that partial state has to be reconstructed by hand.
*/
public void requireIsolatedStatement(MigrationResource migration) {
Objects.requireNonNull(migration, "migration");
@@ -109,9 +111,9 @@ public final class ConcurrentIndexMigrationInspector {
migration.name()
+ " mixes a concurrent index statement with other statements; outside a"
+ " transaction Flyway commits each of them on its own, so a failed build leaves the"
+ " others applied. Alone, a failed build is recovered by dropping its invalid index"
+ " and deleting its version's failed history row (docs/runbooks/migration-failed.md,"
+ " \"Recover a failed concurrent index migration\")");
+ " others applied. Alone, a failed statement is recovered by deleting its version's"
+ " failed history row (docs/runbooks/migration-failed.md, \"Recover a failed"
+ " concurrent index migration\")");
}
}
}