fix(migration): 실패한 DROP INDEX CONCURRENTLY를 runbook 절차로 복구되게 한다
문제 (재리뷰 round8 RR8-03-01) - migration-failed.md "Recover a failed concurrent index migration"은 CREATE·DROP·REINDEX 동시 문장 셋을 대상으로 두고, 2단계에서 "빌드가 남긴 무효 인덱스를 지운다"고 한다. 실패한 DROP INDEX CONCURRENTLY가 남긴 무효 인덱스는 그 마이그레이션이 지우려던 인덱스다. 절차대로 지우고 실패 행을 지운 뒤 다시 배포하면 같은 문장이 42704(index does not exist)로 실패하고 실패 행이 다시 생긴다. 운영자는 처음과 같은 막힌 상태로 돌아간다. - 고치다 같은 절에서 하나 더 찾았다. 2단계(DROP INDEX CONCURRENTLY)가 3단계(원인 정리) 앞이다. 취소·timeout이 원인이면 그 열린 트랜잭션이 운영자의 DROP INDEX CONCURRENTLY도 붙잡아, 2단계가 3단계를 기다린다. - runbook을 실행하는 IT는 중복으로 실패한 unique CREATE 한 경우만 돌렸다. 고른 방법 - 리뷰 권장안: 무효 인덱스를 지우는 단계를 문장 종류로 나눈다. CREATE는 그 인덱스를, REINDEX는 _ccnew(또는 _ccold) 사본을 지운다. DROP INDEX CONCURRENTLY면 이 단계를 건너뛴다. 다시 배포한 같은 문장이 그 인덱스를 지운다(PostgreSQL은 중간에 실패한 DROP INDEX CONCURRENTLY의 재시도를 받는다). 5단계 문장도 "builds the index"에서 세 경우로 고친다. - 단계 순서를 "빌드 확인 → 원인 정리 → 무효 인덱스 삭제 → 실패 행 삭제 → 재배포"로 바꾼다. 실패 행만 지운다는 소유자 결정(2026-11-06)은 그대로다. - IT에 두 경우를 더한다. 열린 쓰기를 기다리다 취소된 REINDEX와 열린 읽기를 기다리다 취소된 DROP에 runbook SQL을 runbook에서 읽어 돌리고, 재배포 뒤 이력 [1 true, 2 true], 무효 인덱스 없음, V1 checksum 그대로를 단언한다. 대기를 30초 기다리지 않고, 문장이 잠금을 기다리는 것을 본 즉시 취소하는 도우미(cancelledOnceItWaits)를 둔다. - 같은 절차를 요약한 곳을 같이 고친다: docs/jpa/runbooks.md "An invalid index exists", migration-guide "Concurrent index builds"(DROP이 무엇을 기다리는지 포함), 학습 문서 8.2절, ConcurrentIndexMigrationInspector의 Javadoc과 실패 메시지, 업그레이드 안내 5절 행(그 범위가 아직 main에 없어 행을 최종 동작으로 고치고 머리 목록에 더함). - 버린 대안: 가이드와 검사기가 DROP INDEX CONCURRENTLY에 IF EXISTS를 요구한다. 이미 배포된 파일은 고칠 수 없어 새 파일에만 통하고, 서비스의 기존 마이그레이션 검사를 새로 실패시킨다. 위험·한계 - 운영자가 실패한 문장의 종류를 마이그레이션 파일에서 읽어야 한다. runbook이 이력 행만으로는 알 수 없는 것을 요구한다. - REINDEX가 swap 뒤에 실패해 _ccold가 남는 경우는 문서로만 다뤘다(실행 재현 없음). PostgreSQL 문서의 권장 복구와 같다. - 단계 번호가 바뀌었다. 리뷰 기록이 옛 번호(2단계 = drop)로 적은 곳은 그때의 기록이라 두었다. RED (desktop, 기준3eb3b067+ 임시 테스트, 로그 ~/r9-g3-logs/red-concurrent.log) - postgresqlConcurrentIndexMigrationIntegrationTest rc=1, 9건 중 2건 실패 - redAFailedConcurrentDropIsRecoveredByTheBaseRunbook: 기준 runbook 1·2·4단계 뒤 재배포가 "42704 ERROR: index "ix_probe_id" does not exist"로 실패, 이력 [1 true, 2 false] 그대로 - redTheBaseRunbookDropsBeforeItEndsTheTransaction: 기준 순서의 2단계 (_ccnew drop)가 열린 쓰기 트랜잭션을 기다려 취소됨 - 나머지 7건(기존) 통과 검증 (desktop, 이 커밋의 트리7cfaa83c, --no-build-cache, 로그 ~/r9-g3-logs/) - :adapter:outbound:persistence-jpa:check rc=0, test 666건 실패·skip 0 - postgresqlConcurrentIndexMigrationIntegrationTest rc=0, 9건(새 2건 포함) - :adapter:outbound:persistence-jpa:testQualityCheck rc=0 - :app-bootstrap:documentContractTest rc=0, 67건(RunbookCoverageContractTest 포함) - -p optional-platforms :grpc:grpc-operation-ledger-jpa:integrationTest rc=0, 47건(검사기 사본이 바뀌어서) - FROM-CACHE 0 - 상대 링크(노트북): python3 scripts/check-work-docs.py --links로 바꾼 문서 5개(migration-failed.md, jpa/runbooks.md, migration-guide.md, 학습 문서, 업그레이드 안내) rc=0, 깨진 링크 0 Refs: docs/reviews/2026-11-03-rereview/round8/03-migration.md RR8-03-01 Fixes:6ec0c4f7fa("fix(migration): 실패한 동시 인덱스 마이그레이션 복구와 검사기 설명을 실제 동작에 맞춘다")
This commit is contained in:
+183
-9
@@ -9,8 +9,11 @@ import java.io.IOException;
|
||||
import java.nio.file.Files;
|
||||
import java.nio.file.Path;
|
||||
import java.sql.Connection;
|
||||
import java.sql.DriverManager;
|
||||
import java.sql.PreparedStatement;
|
||||
import java.sql.ResultSet;
|
||||
import java.sql.SQLException;
|
||||
import java.sql.Statement;
|
||||
import java.time.Duration;
|
||||
import java.util.List;
|
||||
import java.util.Locale;
|
||||
@@ -57,13 +60,17 @@ import org.springframework.jdbc.datasource.DelegatingDataSource;
|
||||
* deploy still waiting after {@link #FINISHES_WITHIN} is cancelled, which leaves the invalid index
|
||||
* an interrupted build leaves in production too.
|
||||
*
|
||||
* <p>Three more cases pin what happens when such a deploy fails (re-review 2026-11-03 round 7,
|
||||
* RR7-03-03 and RR7-03-04). A concurrent build next to another statement is refused before any of
|
||||
* it runs when nothing configures the file, and is left half applied when {@code
|
||||
* executeInTransaction=false} does: Flyway commits each statement on its own. A failed build leaves
|
||||
* an invalid index and a failed history row, and the deploy after it is refused until the row is
|
||||
* gone: the recovery case runs the SQL of {@code docs/runbooks/migration-failed.md} ("Recover a
|
||||
* failed concurrent index migration"), read out of the runbook itself, and deploys again.
|
||||
* <p>Five more cases pin what happens when such a deploy fails (re-review 2026-11-03 round 7,
|
||||
* RR7-03-03 and RR7-03-04; round 8, RR8-03-01). A concurrent build next to another statement is
|
||||
* refused before any of it runs when nothing configures the file, and is left half applied when
|
||||
* {@code executeInTransaction=false} does: Flyway commits each statement on its own. A concurrent
|
||||
* statement alone that fails leaves an invalid index and a failed history row, and the deploy after
|
||||
* it is refused until the row is gone. The recovery cases run the SQL of {@code
|
||||
* docs/runbooks/migration-failed.md} ("Recover a failed concurrent index migration"), read out of
|
||||
* the runbook itself, for each statement the runbook covers, and deploy again: a unique build that
|
||||
* fails on a duplicate, and a rebuild and a drop cancelled while they wait for an open transaction.
|
||||
* The drop leaves invalid the very index it was dropping, so the runbook leaves that index for the
|
||||
* deploy again to drop: dropped by hand first, the same statement fails with {@code 42704}.
|
||||
*/
|
||||
class PostgreSqlConcurrentIndexMigrationIntegrationTest {
|
||||
|
||||
@@ -89,6 +96,12 @@ class PostgreSqlConcurrentIndexMigrationIntegrationTest {
|
||||
private static final String FLYWAY_REFUSES_A_MIXED_MIGRATION =
|
||||
"Detected both transactional and non-transactional statements";
|
||||
|
||||
/** A read of the table, which a {@code DROP INDEX CONCURRENTLY} waits for while it is open. */
|
||||
private static final String A_READ = "SELECT count(*) FROM probe";
|
||||
|
||||
/** A write to the table, which a {@code REINDEX ... CONCURRENTLY} waits for while it is open. */
|
||||
private static final String A_WRITE = "UPDATE probe SET note = note WHERE id = 1";
|
||||
|
||||
/** The system property naming {@code docs/runbooks/migration-failed.md} for this lane. */
|
||||
private static final String MIGRATION_FAILED_RUNBOOK = "migration.failed.runbook";
|
||||
|
||||
@@ -203,7 +216,7 @@ class PostgreSqlConcurrentIndexMigrationIntegrationTest {
|
||||
|
||||
@Test
|
||||
@DisplayName(
|
||||
"실패한 동시 인덱스 마이그레이션은 runbook 절차(빌드 확인, 무효 인덱스 삭제, 원인 정리, 그 버전의 실패 이력 행 삭제) 뒤 다시 배포하면 적용된다")
|
||||
"중복 값으로 실패한 unique 동시 인덱스 빌드는 runbook 절차(빌드 확인, 원인 정리, 무효 인덱스 삭제, 그 버전의 실패 이력 행 삭제) 뒤 다시 배포하면 적용된다")
|
||||
void aFailedConcurrentIndexMigrationIsRecoveredByTheRunbookProcedure(@TempDir Path stream)
|
||||
throws Exception {
|
||||
RunbookSql runbook = RunbookSql.fromSystemProperty(MIGRATION_FAILED_RUNBOOK);
|
||||
@@ -220,10 +233,10 @@ class PostgreSqlConcurrentIndexMigrationIntegrationTest {
|
||||
// when
|
||||
List<Map<String, Object>> buildsRunning =
|
||||
jdbc.queryForList(runbook.block("failed-concurrent-index-builds"));
|
||||
jdbc.execute("DELETE FROM probe WHERE id = 2001");
|
||||
jdbc.execute(
|
||||
runbook.block(
|
||||
"failed-concurrent-index-drop", Map.of("invalid_index", "ix_probe_note_u")));
|
||||
jdbc.execute("DELETE FROM probe WHERE id = 2001");
|
||||
jdbc.execute(
|
||||
runbook.block(
|
||||
"failed-concurrent-index-history-row",
|
||||
@@ -251,6 +264,113 @@ class PostgreSqlConcurrentIndexMigrationIntegrationTest {
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName(
|
||||
"열린 트랜잭션을 기다리다 취소된 REINDEX CONCURRENTLY는 runbook 절차(빌드 확인, 원인 정리, 남은 _ccnew 무효 인덱스 삭제, 그 버전의 실패 이력 행 삭제) 뒤 다시 배포하면 적용된다")
|
||||
void aFailedConcurrentRebuildIsRecoveredByTheRunbookProcedure(@TempDir Path stream)
|
||||
throws Exception {
|
||||
RunbookSql runbook = RunbookSql.fromSystemProperty(MIGRATION_FAILED_RUNBOOK);
|
||||
String name = databaseName();
|
||||
try (HikariDataSource database = postgres.createDatabase(name)) {
|
||||
// given
|
||||
Files.writeString(stream.resolve("V1__probe.sql"), FIRST_RELEASE);
|
||||
migrateTheWayTheTemplateDoes(database, stream);
|
||||
String checksumOfV1 = checksumOf(database, "1");
|
||||
writeConcurrentMigration(stream, "REINDEX INDEX CONCURRENTLY ix_probe_id");
|
||||
Throwable failed;
|
||||
String afterFailure;
|
||||
List<Map<String, Object>> buildsRunning;
|
||||
JdbcTemplate jdbc = new JdbcTemplate(database);
|
||||
try (Connection writer = openTransactionOn(database, A_WRITE)) {
|
||||
failed = cancelledOnceItWaits(name, () -> migrateTheWayTheTemplateDoes(database, stream));
|
||||
afterFailure = aftermath(database);
|
||||
|
||||
// when
|
||||
buildsRunning = jdbc.queryForList(runbook.block("failed-concurrent-index-builds"));
|
||||
writer.rollback();
|
||||
}
|
||||
jdbc.execute(
|
||||
runbook.block(
|
||||
"failed-concurrent-index-drop", Map.of("invalid_index", "ix_probe_id_ccnew")));
|
||||
jdbc.execute(
|
||||
runbook.block(
|
||||
"failed-concurrent-index-history-row",
|
||||
Map.of("owned_history_table", "flyway_schema_history", "version", "2")));
|
||||
Throwable again = catchThrowable(() -> migrateTheWayTheTemplateDoes(database, stream));
|
||||
|
||||
// then
|
||||
assertThat(
|
||||
List.of(
|
||||
failed != null,
|
||||
afterFailure,
|
||||
buildsRunning.size(),
|
||||
String.valueOf(again),
|
||||
aftermath(database),
|
||||
checksumOf(database, "1")))
|
||||
.containsExactly(
|
||||
true,
|
||||
"row 3001 0, indexes [ix_probe_id true, ix_probe_id_ccnew false, probe_pkey true],"
|
||||
+ " history [1 true, 2 false]",
|
||||
0,
|
||||
"null",
|
||||
"row 3001 0, indexes [ix_probe_id true, probe_pkey true], history [1 true, 2 true]",
|
||||
checksumOfV1);
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName(
|
||||
"열린 트랜잭션을 기다리다 취소된 DROP INDEX CONCURRENTLY는 runbook 절차대로 빌드 확인, 원인 정리, 그 버전의 실패 이력 행 삭제만 하고 남은 무효 인덱스는 두면 다시 배포한 같은 문장이 그 인덱스를 지우고 적용된다")
|
||||
void aFailedConcurrentDropIsRecoveredByTheRunbookProcedure(@TempDir Path stream)
|
||||
throws Exception {
|
||||
RunbookSql runbook = RunbookSql.fromSystemProperty(MIGRATION_FAILED_RUNBOOK);
|
||||
String name = databaseName();
|
||||
try (HikariDataSource database = postgres.createDatabase(name)) {
|
||||
// given
|
||||
Files.writeString(stream.resolve("V1__probe.sql"), FIRST_RELEASE);
|
||||
migrateTheWayTheTemplateDoes(database, stream);
|
||||
String checksumOfV1 = checksumOf(database, "1");
|
||||
writeConcurrentMigration(stream, "DROP INDEX CONCURRENTLY ix_probe_id");
|
||||
Throwable failed;
|
||||
String afterFailure;
|
||||
List<Map<String, Object>> buildsRunning;
|
||||
JdbcTemplate jdbc = new JdbcTemplate(database);
|
||||
try (Connection reader = openTransactionOn(database, A_READ)) {
|
||||
failed = cancelledOnceItWaits(name, () -> migrateTheWayTheTemplateDoes(database, stream));
|
||||
afterFailure = aftermath(database);
|
||||
|
||||
// when
|
||||
buildsRunning = jdbc.queryForList(runbook.block("failed-concurrent-index-builds"));
|
||||
reader.rollback();
|
||||
}
|
||||
int deleted =
|
||||
jdbc.update(
|
||||
runbook.block(
|
||||
"failed-concurrent-index-history-row",
|
||||
Map.of("owned_history_table", "flyway_schema_history", "version", "2")));
|
||||
Throwable again = catchThrowable(() -> migrateTheWayTheTemplateDoes(database, stream));
|
||||
|
||||
// then
|
||||
assertThat(
|
||||
List.of(
|
||||
failed != null,
|
||||
afterFailure,
|
||||
buildsRunning.size(),
|
||||
deleted,
|
||||
String.valueOf(again),
|
||||
aftermath(database),
|
||||
checksumOf(database, "1")))
|
||||
.containsExactly(
|
||||
true,
|
||||
"row 3001 0, indexes [ix_probe_id false, probe_pkey true], history [1 true, 2 false]",
|
||||
0,
|
||||
1,
|
||||
"null",
|
||||
"row 3001 0, indexes [probe_pkey true], history [1 true, 2 true]",
|
||||
checksumOfV1);
|
||||
}
|
||||
}
|
||||
|
||||
/** V2: the one concurrent statement, with the companion configuration the guide asks for. */
|
||||
private static void writeConcurrentMigration(Path stream, String statement) throws IOException {
|
||||
Files.writeString(stream.resolve("V2__concurrent_index.sql"), statement + ";\n");
|
||||
@@ -332,6 +452,60 @@ class PostgreSqlConcurrentIndexMigrationIntegrationTest {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* A connection to the database with a transaction open on the table: {@code statement} has run in
|
||||
* it and holds its lock until the caller ends the transaction, as a report or an operator's
|
||||
* {@code BEGIN} would. Opened outside the test's pool, which the deploy needs.
|
||||
*/
|
||||
private static Connection openTransactionOn(HikariDataSource database, String statement)
|
||||
throws SQLException {
|
||||
Connection holder =
|
||||
DriverManager.getConnection(
|
||||
database.getJdbcUrl(), database.getUsername(), database.getPassword());
|
||||
holder.setAutoCommit(false);
|
||||
try (Statement holding = holder.createStatement()) {
|
||||
holding.execute(statement);
|
||||
}
|
||||
return holder;
|
||||
}
|
||||
|
||||
/**
|
||||
* Runs the deploy until its statement waits for a lock in the database, cancels that wait as an
|
||||
* operator ending a stuck deploy would, and returns how the deploy failed.
|
||||
*/
|
||||
private static Throwable cancelledOnceItWaits(String databaseName, Runnable migration)
|
||||
throws Exception {
|
||||
ExecutorService deployer = Executors.newSingleThreadExecutor();
|
||||
try {
|
||||
Future<?> deploy = deployer.submit(migration);
|
||||
long deadline = System.nanoTime() + FINISHES_WITHIN.toNanos();
|
||||
while (!aStatementWaitsForALock(databaseName)) {
|
||||
if (deploy.isDone() || System.nanoTime() > deadline) {
|
||||
throw new IllegalStateException("the deploy never waited for a lock");
|
||||
}
|
||||
Thread.sleep(50);
|
||||
}
|
||||
cancelLockWaits(databaseName);
|
||||
return catchThrowable(() -> deploy.get(FINISHES_WITHIN.toMillis(), TimeUnit.MILLISECONDS));
|
||||
} finally {
|
||||
deployer.shutdownNow();
|
||||
}
|
||||
}
|
||||
|
||||
private static boolean aStatementWaitsForALock(String databaseName) throws SQLException {
|
||||
try (Connection admin = postgres.connection();
|
||||
PreparedStatement waiting =
|
||||
admin.prepareStatement(
|
||||
"SELECT count(*) FROM pg_stat_activity"
|
||||
+ " WHERE datname = ? AND wait_event_type = 'Lock'"
|
||||
+ " AND pid <> pg_backend_pid()")) {
|
||||
waiting.setString(1, databaseName);
|
||||
try (ResultSet count = waiting.executeQuery()) {
|
||||
return count.next() && count.getInt(1) > 0;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/** The version the history ends on and whether every index on the table is usable. */
|
||||
private static String state(DataSource database) {
|
||||
JdbcTemplate jdbc = new JdbcTemplate(database);
|
||||
|
||||
+15
-13
@@ -22,11 +22,13 @@ import java.util.regex.Pattern;
|
||||
* statement it is refused by Flyway before any of the script runs when nothing configures the file
|
||||
* (Flyway's default {@code mixed=false}); with the {@code executeInTransaction=false} that {@link
|
||||
* #validate} asks for, Flyway runs and commits each statement on its own, and a failed build leaves
|
||||
* the statements before it applied (round 7, RR7-03-04). Alone, a failed build leaves an invalid
|
||||
* index and its version's failed history row, which Flyway refuses the next deploy over until the
|
||||
* row is gone: {@code docs/runbooks/migration-failed.md} ("Recover a failed concurrent index
|
||||
* migration") drops the index and deletes that one row ({@code
|
||||
* postgresqlConcurrentIndexMigrationIntegrationTest} runs it; round 7, RR7-03-03).
|
||||
* the statements before it applied (round 7, RR7-03-04). Alone, a failed statement leaves an
|
||||
* invalid index and its version's failed history row, which Flyway refuses the next deploy over
|
||||
* until the row is gone: {@code docs/runbooks/migration-failed.md} ("Recover a failed concurrent
|
||||
* index migration") deletes that one row, after dropping the index a failed build or rebuild left.
|
||||
* A failed {@code DROP INDEX CONCURRENTLY} leaves invalid the index it was dropping, and the deploy
|
||||
* after the row is gone drops it ({@code postgresqlConcurrentIndexMigrationIntegrationTest} runs
|
||||
* all three; round 7, RR7-03-03; round 8, RR8-03-01).
|
||||
*
|
||||
* <p>Outside a transaction, {@code CREATE INDEX CONCURRENTLY} and {@code REINDEX ... CONCURRENTLY}
|
||||
* still wait for every transaction older than themselves. They finish because the template's Flyway
|
||||
@@ -84,12 +86,12 @@ public final class ConcurrentIndexMigrationInspector {
|
||||
/**
|
||||
* Fails when a concurrent-index migration also contains other statements.
|
||||
*
|
||||
* <p>A concurrent index build can fail and leave an invalid index and its version's failed
|
||||
* <p>A concurrent index statement can fail and leave an invalid index and its version's failed
|
||||
* history row behind. When the migration did nothing else, recovering is the runbook's few
|
||||
* statements: drop the index, delete that one failed row, deploy again ({@code
|
||||
* docs/runbooks/migration-failed.md}, "Recover a failed concurrent index migration"). When it did
|
||||
* more, Flyway ran the file outside a transaction and has committed the statements before the
|
||||
* build, and that partial state has to be reconstructed by hand.
|
||||
* statements: drop the index a failed build left, delete that one failed row, deploy again
|
||||
* ({@code docs/runbooks/migration-failed.md}, "Recover a failed concurrent index migration").
|
||||
* When it did more, Flyway ran the file outside a transaction and has committed the statements
|
||||
* before the failed one, and that partial state has to be reconstructed by hand.
|
||||
*/
|
||||
public void requireIsolatedStatement(MigrationResource migration) {
|
||||
Objects.requireNonNull(migration, "migration");
|
||||
@@ -109,9 +111,9 @@ public final class ConcurrentIndexMigrationInspector {
|
||||
migration.name()
|
||||
+ " mixes a concurrent index statement with other statements; outside a"
|
||||
+ " transaction Flyway commits each of them on its own, so a failed build leaves the"
|
||||
+ " others applied. Alone, a failed build is recovered by dropping its invalid index"
|
||||
+ " and deleting its version's failed history row (docs/runbooks/migration-failed.md,"
|
||||
+ " \"Recover a failed concurrent index migration\")");
|
||||
+ " others applied. Alone, a failed statement is recovered by deleting its version's"
|
||||
+ " failed history row (docs/runbooks/migration-failed.md, \"Recover a failed"
|
||||
+ " concurrent index migration\")");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user