Files
clean-architecture-backend-…/docs/jpa/migration-guide.md
T
DongHyeonka 8ebe98e69a fix(migration): 실패한 DROP INDEX CONCURRENTLY를 runbook 절차로 복구되게 한다
문제 (재리뷰 round8 RR8-03-01)
- migration-failed.md "Recover a failed concurrent index migration"은
  CREATE·DROP·REINDEX 동시 문장 셋을 대상으로 두고, 2단계에서 "빌드가 남긴
  무효 인덱스를 지운다"고 한다. 실패한 DROP INDEX CONCURRENTLY가 남긴 무효
  인덱스는 그 마이그레이션이 지우려던 인덱스다. 절차대로 지우고 실패 행을
  지운 뒤 다시 배포하면 같은 문장이 42704(index does not exist)로 실패하고
  실패 행이 다시 생긴다. 운영자는 처음과 같은 막힌 상태로 돌아간다.
- 고치다 같은 절에서 하나 더 찾았다. 2단계(DROP INDEX CONCURRENTLY)가
  3단계(원인 정리) 앞이다. 취소·timeout이 원인이면 그 열린 트랜잭션이
  운영자의 DROP INDEX CONCURRENTLY도 붙잡아, 2단계가 3단계를 기다린다.
- runbook을 실행하는 IT는 중복으로 실패한 unique CREATE 한 경우만 돌렸다.

고른 방법
- 리뷰 권장안: 무효 인덱스를 지우는 단계를 문장 종류로 나눈다. CREATE는
  그 인덱스를, REINDEX는 _ccnew(또는 _ccold) 사본을 지운다. DROP INDEX
  CONCURRENTLY면 이 단계를 건너뛴다. 다시 배포한 같은 문장이 그 인덱스를
  지운다(PostgreSQL은 중간에 실패한 DROP INDEX CONCURRENTLY의 재시도를
  받는다). 5단계 문장도 "builds the index"에서 세 경우로 고친다.
- 단계 순서를 "빌드 확인 → 원인 정리 → 무효 인덱스 삭제 → 실패 행 삭제 →
  재배포"로 바꾼다. 실패 행만 지운다는 소유자 결정(2026-11-06)은 그대로다.
- IT에 두 경우를 더한다. 열린 쓰기를 기다리다 취소된 REINDEX와 열린
  읽기를 기다리다 취소된 DROP에 runbook SQL을 runbook에서 읽어 돌리고,
  재배포 뒤 이력 [1 true, 2 true], 무효 인덱스 없음, V1 checksum 그대로를
  단언한다. 대기를 30초 기다리지 않고, 문장이 잠금을 기다리는 것을 본
  즉시 취소하는 도우미(cancelledOnceItWaits)를 둔다.
- 같은 절차를 요약한 곳을 같이 고친다: docs/jpa/runbooks.md "An invalid
  index exists", migration-guide "Concurrent index builds"(DROP이 무엇을
  기다리는지 포함), 학습 문서 8.2절, ConcurrentIndexMigrationInspector의
  Javadoc과 실패 메시지, 업그레이드 안내 5절 행(그 범위가 아직 main에
  없어 행을 최종 동작으로 고치고 머리 목록에 더함).
- 버린 대안: 가이드와 검사기가 DROP INDEX CONCURRENTLY에 IF EXISTS를
  요구한다. 이미 배포된 파일은 고칠 수 없어 새 파일에만 통하고, 서비스의
  기존 마이그레이션 검사를 새로 실패시킨다.

위험·한계
- 운영자가 실패한 문장의 종류를 마이그레이션 파일에서 읽어야 한다.
  runbook이 이력 행만으로는 알 수 없는 것을 요구한다.
- REINDEX가 swap 뒤에 실패해 _ccold가 남는 경우는 문서로만 다뤘다(실행
  재현 없음). PostgreSQL 문서의 권장 복구와 같다.
- 단계 번호가 바뀌었다. 리뷰 기록이 옛 번호(2단계 = drop)로 적은 곳은 그때의
  기록이라 두었다.

RED (desktop, 기준 3eb3b067 + 임시 테스트, 로그 ~/r9-g3-logs/red-concurrent.log)
- postgresqlConcurrentIndexMigrationIntegrationTest rc=1, 9건 중 2건 실패
  - redAFailedConcurrentDropIsRecoveredByTheBaseRunbook: 기준 runbook
    1·2·4단계 뒤 재배포가 "42704 ERROR: index "ix_probe_id" does not
    exist"로 실패, 이력 [1 true, 2 false] 그대로
  - redTheBaseRunbookDropsBeforeItEndsTheTransaction: 기준 순서의 2단계
    (_ccnew drop)가 열린 쓰기 트랜잭션을 기다려 취소됨
  - 나머지 7건(기존) 통과

검증 (desktop, 이 커밋의 트리 7cfaa83c, --no-build-cache, 로그 ~/r9-g3-logs/)
- :adapter:outbound:persistence-jpa:check rc=0, test 666건 실패·skip 0
- postgresqlConcurrentIndexMigrationIntegrationTest rc=0, 9건(새 2건 포함)
- :adapter:outbound:persistence-jpa:testQualityCheck rc=0
- :app-bootstrap:documentContractTest rc=0, 67건(RunbookCoverageContractTest
  포함)
- -p optional-platforms :grpc:grpc-operation-ledger-jpa:integrationTest
  rc=0, 47건(검사기 사본이 바뀌어서)
- FROM-CACHE 0
- 상대 링크(노트북): python3 scripts/check-work-docs.py --links로 바꾼
  문서 5개(migration-failed.md, jpa/runbooks.md, migration-guide.md,
  학습 문서, 업그레이드 안내) rc=0, 깨진 링크 0

Refs: docs/reviews/2026-11-03-rereview/round8/03-migration.md RR8-03-01
Fixes: 6ec0c4f7fa ("fix(migration): 실패한 동시 인덱스 마이그레이션 복구와 검사기 설명을 실제 동작에 맞춘다")
2026-11-07 08:07:42 +09:00

14 KiB

Migration Guide

Design §31-§32. Flyway owns the schema; Hibernate only validates.

Who may change the schema

Environment Mode
local, test, dev migrate at startup with the migration credential
staging, prod deployment-owned migration; the application validates only

Migrating from inside the application in production means every instance of a rolling deploy races to apply the same script, and the loser's failure is indistinguishable from a real one.

Whoever runs the migration holds Flyway's PostgreSQL migration lock for the session, not in a transaction: flyway.postgresql.transactional.lock=false. Flyway's default (true) keeps a transaction open until the whole run ends, and a concurrent index build waits for every transaction older than itself, Flyway's own included — the deploy stops with no error and no time limit ("Concurrent index builds" below). The template sets it for every Flyway it runs: the module defaults (config/module-defaults/persistence-jpa.yml, spring.flyway.postgresql.transactional-lock: false) for the one Spring Boot builds at startup, and FlywaySchemaPolicy.configure() for every stream it builds itself — a service's own stream starts from it too, and JpaProductionArchitectureTest refuses production code that builds a Flyway any other way: calling Flyway.configure(), or constructing a configuration itself (new FluentConfiguration(), new ClassicConfiguration(), a subclass of either, or FluentConfiguration::new). new Flyway(...) over the policy's configuration is allowed. The deployment-owned migration of staging and prod is outside this repository: give its tool the same value (FLYWAY_POSTGRESQL_TRANSACTIONAL_LOCK=false, or the key above in its configuration file).

A session lock assumes the connection Flyway takes stays its own for the whole run, which the template's direct Hikari connection gives. Behind a pooler in transaction mode (PgBouncer pool_mode = transaction) the unlock can reach another server connection and release nothing, and the next migration waits for a lock that a pooled connection still holds. Run migrations over a direct connection or a session-mode pool.

ddl-auto is validate or none. Never update: it never drops or narrows anything, so it produces a schema that is neither the old one nor the one the migrations describe — silently, on whichever instance started first.

Validation fails closed and never repairs

At startup Flyway's own validation (and Hibernate ddl-auto=validate) refuses a checksum mismatch, a missing migration, or a schema Hibernate disagrees with. Nothing in the application calls repair. In tests, the testkit's FlywayValidationGate turns a failed ValidateResult into SchemaMismatchException; the migration contract tests use it.

Repair rewrites the schema history table to match whatever scripts are on disk. That resolves the symptom by deleting the evidence: a checksum mismatch means the deployed script differs from the applied one, and the interesting question is which change is missing from this database. Repair makes that question unaskable. It exists only as an explicit admin operation with an operator, a reason, and an approval (design §8.4).

The one change to the history an operator makes is narrower than repair: after a concurrent index migration fails, the failed row of that one version is deleted so that the version can run again ("Concurrent index builds" below). Every applied row, and its checksum, stays as it was.

Only Flyway's structured error codes reach the exception. Its messages embed the script path and part of the failing statement.

Bound the lock wait

A migration that locks a table already in use starts with SET LOCAL lock_timeout = '5s';, before the first statement that takes the lock. ALTER TABLE takes the strongest lock there is, which conflicts even with a plain read; a non-concurrent CREATE INDEX takes one that stops every write; UPDATE and DELETE wait on the rows other transactions hold. Unbounded, the migration waits for the oldest open transaction that touched the table — an operator's forgotten BEGIN in psql has no deadline — and every read and write that arrives meanwhile queues behind the migration, holding a pool connection while it waits. With the bound, the migration fails after five seconds, Flyway rolls it back whole, the database stays on the version before it and the new release does not start: a failed deploy instead of an outage. Deploy again when the transaction has ended.

LOCAL, because a plain SET outlives the migration on the pooled connection Flyway borrowed and changes every statement that later runs on it. The value is not zero: lock_timeout = 0 (or DEFAULT, or a later RESET) is no bound at all. A migration with executeInTransaction=false cannot use it — SET LOCAL outside a transaction only warns — so a statement that locks a table goes into a migration that runs in one. A statement on a table the same migration created a moment earlier waits for nothing and needs no bound; CREATE TABLE IF NOT EXISTS does not count, because the table may already be there.

Flyway also takes a migration out of its transaction on its own, without a .sql.conf: on PostgreSQL 12 and later, any statement that starts with VACUUM, CREATE [UNIQUE] INDEX CONCURRENTLY, DROP INDEX CONCURRENTLY, REINDEX [VERBOSE] SCHEMA|DATABASE|SYSTEM, CREATE or DROP DATABASE|TABLESPACE|SUBSCRIPTION, ALTER SYSTEM or DISCARD ALL (Flyway 11, PostgreSQLParser; words inside parentheses are skipped, so VACUUM (FULL) and REINDEX (VERBOSE) SCHEMA count). One such statement puts the whole script outside a transaction. Next to any other statement — a SET LOCAL lock_timeout included — Flyway refuses the migration at deploy under its default mixed=false; a service that sets mixed=true gets it run, with the bound silently gone. Put such a statement in a migration of its own. REINDEX TABLE|INDEX CONCURRENTLY is not on Flyway's list: Flyway runs it in the transaction, PostgreSQL refuses it there, and it needs executeInTransaction=false (below). A locking statement that cannot run in a transaction at all — VACUUM FULL, a non-concurrent REINDEX SCHEMA|DATABASE|SYSTEM — can never be bounded this way and does not belong in a migration: run it by hand as an operations step, outside a deploy, in a psql session that sets SET lock_timeout = '5s'; first (the session ends with the step, so the plain SET is the bound there). A .sql.conf with executeInTransaction=true does not change that: it keeps the statement and its SET LOCAL in the transaction only for PostgreSQL to refuse it at deploy, and the inspector below refuses it as well.

Enforced, not remembered: LockWaitMigrationInspector (persistence-jpa testkit) reads a script — and the .sql.conf beside it, as Flyway reads it (a properties file, false in any case), and otherwise the statements Flyway takes out of the transaction on its own — and names the statements from its list that lock an existing table with no bound in force: ALTER TABLE, LOCK TABLE, TRUNCATE, DROP TABLE; CREATE INDEX, DROP INDEX, REINDEX and REFRESH MATERIALIZED VIEW without CONCURRENTLY, CLUSTER, VACUUM FULL; REFERENCES on the table it references, in a new table as much as in an ALTER TABLE (a foreign key locks its parent against writes while it is added); CREATE TRIGGER and DROP TRIGGER on their table; UPDATE and DELETE. It reads text, so a statement outside the list is not checked. MigrationLockWaitTest runs it over every tree this leaf's database runs, SampleMigrationLockWaitTest over the sample's stream; a service stream copies the latter with its own directory. Both find the migrations the way Flyway does, decimal versions (V22_1__) included (VersionedMigrationFiles, testkit). The migrations released before the rule wait unbounded and stay that way (they are applied, and applied files are never edited); the test lists them and refuses any new one. PostgreSqlMigrationLockWaitIntegrationTest (postgresqlMigrationLockWaitIntegrationTest) shows on real PostgreSQL that a bounded migration gives up within the bound while another transaction holds the table, and leaves the version below it; it also compares the inspector's copy of Flyway's list with Flyway itself for the statements it lists (each is named by the inspector exactly when Flyway refuses it next to a SET LOCAL), and shows a VACUUM FULL run under mixed=true waiting out a reader despite its SET LOCAL. A statement a Flyway upgrade adds to the list is outside that comparison; LockWaitMigrationInspectorTest pins Flyway's own patterns (PostgreSQLParser), so such an upgrade fails the pull request's ci until the copy follows. The gRPC ledger's tree lives in the optional build, which cannot depend on the testkit: its GrpcOperationLedgerMigrationLockWaitIT runs every ledger migration after V001, read from the tree, against a held ledger instead, so its next migration is checked on real PostgreSQL. It finds and orders that tree with the testkit's own VersionedMigrationFiles, which the ledger's build.gradle copies into its integration-test sources at build time (sharedMigrationFileSources), so both builds read one definition. A concurrent index build in that tree runs outside a transaction and is no case for a held ledger: it goes in the test's RUN_OUTSIDE_A_TRANSACTION with its reason, and the same test holds each file listed there to ConcurrentIndexMigrationInspector, copied the same way ("Concurrent index builds"). Every ledger lane applies the tree one file at a time, as Flyway does, so such a file is not refused inside the implicit transaction of one string holding many. The rule came from V21 having the bound and V22, in the same commit, not having it (re-review 2026-11-03 round 2, RR-02-09; round 3, RR3-02-01, RR3-02-03, RR3-02-04; round 4, RR4-02-01; round 6, RR6-03-02, RR6-03-03; round 7, RR7-03-01).

Concurrent index builds

CREATE INDEX CONCURRENTLY, DROP INDEX CONCURRENTLY and REINDEX … CONCURRENTLY cannot run inside a transaction block. Flyway runs a script outside a transaction on its own when it holds CREATE/DROP INDEX CONCURRENTLY or REINDEX SCHEMA|DATABASE, with or without CONCURRENTLY (above). It runs REINDEX INDEX|TABLE … CONCURRENTLY — REINDEX (CONCURRENTLY) TABLE included — in the transaction, where PostgreSQL refuses it. PostgreSqlMigrationLockWaitIntegrationTest shows the five forms on real PostgreSQL. Write the companion configuration for every concurrent statement anyway, so the file says what it needs and no reader has to know Flyway's list:

# V42__order_index.sql.conf
executeInTransaction=false

Outside a transaction, CREATE INDEX CONCURRENTLY and REINDEX … CONCURRENTLY still wait until every transaction older than them has ended. Under Flyway's default lock that includes the transaction Flyway itself keeps open, and Flyway waits for the build in turn: the deploy never ends, and the table's other DDL and autovacuum queue behind the build meanwhile. The template's Flyway holds its lock for the session instead ("Who may change the schema"), so the build waits only for other transactions; a deployment tool left on Flyway's default stops on the first such migration. DROP INDEX CONCURRENTLY does not wait that way: it waits only for the transactions that hold a lock on its table. PostgreSqlConcurrentIndexMigrationIntegrationTest (postgresqlConcurrentIndexMigrationIntegrationTest) applies both builds as a second deploy, with the Flyway Spring Boot builds from the module defaults and with FlywaySchemaPolicy.configure(), on real PostgreSQL. The other transactions still count: a long one an operator or a report keeps open holds the build, and the drop too once it has touched the table, and a file run outside a transaction cannot bound that wait with SET LOCAL. Look for old transactions (pg_stat_activity.xact_start) before deploying one.

The testkit's ConcurrentIndexMigrationInspector checks a script for this in a migration test (nothing checks it at startup), and additionally requires the migration to contain nothing else. Next to another statement, a concurrent one makes Flyway refuse the file before any of it runs when nothing configures it; with executeInTransaction=false Flyway commits each statement on its own, and a failed build leaves the statements before it applied — state to reconstruct by hand.

Alone, a failed statement leaves an invalid index and a history row with success = false for its version, and Flyway refuses every later deploy ("Detected failed migration to version …") until that row is gone: dropping the index is not enough. The recovery is migration-failed.md "Recover a failed concurrent index migration": confirm no build is running, remove the cause, drop the invalid index a failed build or rebuild left, delete that version's failed row and nothing else, deploy again. A failed DROP INDEX CONCURRENTLY leaves invalid the index it was dropping; the runbook leaves that one for the deploy again to drop, since dropped by hand it makes the same statement fail. It is not flyway repair, which would also rewrite the remaining rows' checksums. PostgreSqlConcurrentIndexMigrationIntegrationTest (postgresqlConcurrentIndexMigrationIntegrationTest) runs the runbook's SQL, read from the runbook, after a unique build that failed on a duplicate and after a rebuild and a drop cancelled while they waited for an open transaction, and shows both mixed files above.

An invalid index is not merely useless — the planner ignores it while every write still maintains it. runbooks.md has the query that finds them. Nothing drops them automatically: an invalid index can also mean a build is still running, and the two are indistinguishable from the catalog alone.

Upgrade scenarios

Three, each catching something the others do not:

Scenario Catches
empty an early migration edited to match a later one, no longer applying to a fresh database
previous-release the actual deployment path; the only one exercising this release's migrations
oldest-supported a migration that silently assumes state only recent databases have

Each asserts a data invariant, not just the schema version. A migration that renames a column and loses its contents leaves the version correct and the data gone.