문제 (재리뷰 round8 RR8-03-01) - migration-failed.md "Recover a failed concurrent index migration"은 CREATE·DROP·REINDEX 동시 문장 셋을 대상으로 두고, 2단계에서 "빌드가 남긴 무효 인덱스를 지운다"고 한다. 실패한 DROP INDEX CONCURRENTLY가 남긴 무효 인덱스는 그 마이그레이션이 지우려던 인덱스다. 절차대로 지우고 실패 행을 지운 뒤 다시 배포하면 같은 문장이 42704(index does not exist)로 실패하고 실패 행이 다시 생긴다. 운영자는 처음과 같은 막힌 상태로 돌아간다. - 고치다 같은 절에서 하나 더 찾았다. 2단계(DROP INDEX CONCURRENTLY)가 3단계(원인 정리) 앞이다. 취소·timeout이 원인이면 그 열린 트랜잭션이 운영자의 DROP INDEX CONCURRENTLY도 붙잡아, 2단계가 3단계를 기다린다. - runbook을 실행하는 IT는 중복으로 실패한 unique CREATE 한 경우만 돌렸다. 고른 방법 - 리뷰 권장안: 무효 인덱스를 지우는 단계를 문장 종류로 나눈다. CREATE는 그 인덱스를, REINDEX는 _ccnew(또는 _ccold) 사본을 지운다. DROP INDEX CONCURRENTLY면 이 단계를 건너뛴다. 다시 배포한 같은 문장이 그 인덱스를 지운다(PostgreSQL은 중간에 실패한 DROP INDEX CONCURRENTLY의 재시도를 받는다). 5단계 문장도 "builds the index"에서 세 경우로 고친다. - 단계 순서를 "빌드 확인 → 원인 정리 → 무효 인덱스 삭제 → 실패 행 삭제 → 재배포"로 바꾼다. 실패 행만 지운다는 소유자 결정(2026-11-06)은 그대로다. - IT에 두 경우를 더한다. 열린 쓰기를 기다리다 취소된 REINDEX와 열린 읽기를 기다리다 취소된 DROP에 runbook SQL을 runbook에서 읽어 돌리고, 재배포 뒤 이력 [1 true, 2 true], 무효 인덱스 없음, V1 checksum 그대로를 단언한다. 대기를 30초 기다리지 않고, 문장이 잠금을 기다리는 것을 본 즉시 취소하는 도우미(cancelledOnceItWaits)를 둔다. - 같은 절차를 요약한 곳을 같이 고친다: docs/jpa/runbooks.md "An invalid index exists", migration-guide "Concurrent index builds"(DROP이 무엇을 기다리는지 포함), 학습 문서 8.2절, ConcurrentIndexMigrationInspector의 Javadoc과 실패 메시지, 업그레이드 안내 5절 행(그 범위가 아직 main에 없어 행을 최종 동작으로 고치고 머리 목록에 더함). - 버린 대안: 가이드와 검사기가 DROP INDEX CONCURRENTLY에 IF EXISTS를 요구한다. 이미 배포된 파일은 고칠 수 없어 새 파일에만 통하고, 서비스의 기존 마이그레이션 검사를 새로 실패시킨다. 위험·한계 - 운영자가 실패한 문장의 종류를 마이그레이션 파일에서 읽어야 한다. runbook이 이력 행만으로는 알 수 없는 것을 요구한다. - REINDEX가 swap 뒤에 실패해 _ccold가 남는 경우는 문서로만 다뤘다(실행 재현 없음). PostgreSQL 문서의 권장 복구와 같다. - 단계 번호가 바뀌었다. 리뷰 기록이 옛 번호(2단계 = drop)로 적은 곳은 그때의 기록이라 두었다. RED (desktop, 기준3eb3b067+ 임시 테스트, 로그 ~/r9-g3-logs/red-concurrent.log) - postgresqlConcurrentIndexMigrationIntegrationTest rc=1, 9건 중 2건 실패 - redAFailedConcurrentDropIsRecoveredByTheBaseRunbook: 기준 runbook 1·2·4단계 뒤 재배포가 "42704 ERROR: index "ix_probe_id" does not exist"로 실패, 이력 [1 true, 2 false] 그대로 - redTheBaseRunbookDropsBeforeItEndsTheTransaction: 기준 순서의 2단계 (_ccnew drop)가 열린 쓰기 트랜잭션을 기다려 취소됨 - 나머지 7건(기존) 통과 검증 (desktop, 이 커밋의 트리7cfaa83c, --no-build-cache, 로그 ~/r9-g3-logs/) - :adapter:outbound:persistence-jpa:check rc=0, test 666건 실패·skip 0 - postgresqlConcurrentIndexMigrationIntegrationTest rc=0, 9건(새 2건 포함) - :adapter:outbound:persistence-jpa:testQualityCheck rc=0 - :app-bootstrap:documentContractTest rc=0, 67건(RunbookCoverageContractTest 포함) - -p optional-platforms :grpc:grpc-operation-ledger-jpa:integrationTest rc=0, 47건(검사기 사본이 바뀌어서) - FROM-CACHE 0 - 상대 링크(노트북): python3 scripts/check-work-docs.py --links로 바꾼 문서 5개(migration-failed.md, jpa/runbooks.md, migration-guide.md, 학습 문서, 업그레이드 안내) rc=0, 깨진 링크 0 Refs: docs/reviews/2026-11-03-rereview/round8/03-migration.md RR8-03-01 Fixes:6ec0c4f7fa("fix(migration): 실패한 동시 인덱스 마이그레이션 복구와 검사기 설명을 실제 동작에 맞춘다")
200 lines
14 KiB
Markdown
200 lines
14 KiB
Markdown
# Migration Guide
|
|
|
|
Design §31-§32. Flyway owns the schema; Hibernate only validates.
|
|
|
|
## Who may change the schema
|
|
|
|
| Environment | Mode |
|
|
|---|---|
|
|
| local, test, dev | migrate at startup with the migration credential |
|
|
| staging, prod | deployment-owned migration; the application validates only |
|
|
|
|
Migrating from inside the application in production means every instance of a rolling deploy races
|
|
to apply the same script, and the loser's failure is indistinguishable from a real one.
|
|
|
|
Whoever runs the migration holds Flyway's PostgreSQL migration lock for the session, not in a
|
|
transaction: `flyway.postgresql.transactional.lock=false`. Flyway's default (`true`) keeps a
|
|
transaction open until the whole run ends, and a concurrent index build waits for every transaction
|
|
older than itself, Flyway's own included — the deploy stops with no error and no time limit
|
|
("Concurrent index builds" below). The template sets it for every Flyway it runs: the module
|
|
defaults (`config/module-defaults/persistence-jpa.yml`, `spring.flyway.postgresql.transactional-lock:
|
|
false`) for the one Spring Boot builds at startup, and `FlywaySchemaPolicy.configure()` for every
|
|
stream it builds itself — a service's own stream starts from it too, and `JpaProductionArchitectureTest`
|
|
refuses production code that builds a Flyway any other way: calling `Flyway.configure()`, or
|
|
constructing a configuration itself (`new FluentConfiguration()`, `new ClassicConfiguration()`, a
|
|
subclass of either, or `FluentConfiguration::new`). `new Flyway(...)` over the policy's configuration
|
|
is allowed. The deployment-owned migration of
|
|
staging and prod is outside this repository: give its tool the same value
|
|
(`FLYWAY_POSTGRESQL_TRANSACTIONAL_LOCK=false`, or the key above in its configuration file).
|
|
|
|
A session lock assumes the connection Flyway takes stays its own for the whole run, which the
|
|
template's direct Hikari connection gives. Behind a pooler in transaction mode (PgBouncer
|
|
`pool_mode = transaction`) the unlock can reach another server connection and release nothing, and
|
|
the next migration waits for a lock that a pooled connection still holds. Run migrations over a
|
|
direct connection or a session-mode pool.
|
|
|
|
`ddl-auto` is `validate` or `none`. Never `update`: it never drops or narrows anything, so it
|
|
produces a schema that is neither the old one nor the one the migrations describe — silently, on
|
|
whichever instance started first.
|
|
|
|
## Validation fails closed and never repairs
|
|
|
|
At startup Flyway's own validation (and Hibernate `ddl-auto=validate`) refuses a checksum mismatch,
|
|
a missing migration, or a schema Hibernate disagrees with. Nothing in the application calls
|
|
`repair`. In tests, the testkit's `FlywayValidationGate` turns a failed `ValidateResult` into
|
|
`SchemaMismatchException`; the migration contract tests use it.
|
|
|
|
Repair rewrites the schema history table to match whatever scripts are on disk. That resolves the
|
|
symptom by deleting the evidence: a checksum mismatch means the deployed script differs from the
|
|
applied one, and the interesting question is which change is missing from this database. Repair
|
|
makes that question unaskable. It exists only as an explicit admin operation with an operator, a
|
|
reason, and an approval (design §8.4).
|
|
|
|
The one change to the history an operator makes is narrower than repair: after a concurrent index
|
|
migration fails, the failed row of that one version is deleted so that the version can run again
|
|
("Concurrent index builds" below). Every applied row, and its checksum, stays as it was.
|
|
|
|
Only Flyway's structured error codes reach the exception. Its messages embed the script path and
|
|
part of the failing statement.
|
|
|
|
## Bound the lock wait
|
|
|
|
A migration that locks a table already in use starts with `SET LOCAL lock_timeout = '5s';`, before
|
|
the first statement that takes the lock. `ALTER TABLE` takes the strongest lock there is, which
|
|
conflicts even with a plain read; a non-concurrent `CREATE INDEX` takes one that stops every write;
|
|
`UPDATE` and `DELETE` wait on the rows other transactions hold. Unbounded, the migration waits for
|
|
the oldest open transaction that touched the table — an operator's forgotten `BEGIN` in psql has no
|
|
deadline — and every read and write that arrives meanwhile queues behind the migration, holding a
|
|
pool connection while it waits. With the bound, the migration fails after five seconds, Flyway
|
|
rolls it back whole, the database stays on the version before it and the new release does not
|
|
start: a failed deploy instead of an outage. Deploy again when the transaction has ended.
|
|
|
|
`LOCAL`, because a plain `SET` outlives the migration on the pooled connection Flyway borrowed and
|
|
changes every statement that later runs on it. The value is not zero: `lock_timeout = 0` (or
|
|
`DEFAULT`, or a later `RESET`) is no bound at all. A migration with `executeInTransaction=false`
|
|
cannot use it — `SET LOCAL` outside a transaction only warns — so a statement that locks a table
|
|
goes into a migration that runs in one. A statement on a table the same migration created a moment
|
|
earlier waits for nothing and needs no bound; `CREATE TABLE IF NOT EXISTS` does not count, because
|
|
the table may already be there.
|
|
|
|
Flyway also takes a migration out of its transaction on its own, without a `.sql.conf`: on
|
|
PostgreSQL 12 and later, any statement that starts with `VACUUM`, `CREATE [UNIQUE] INDEX
|
|
CONCURRENTLY`, `DROP INDEX CONCURRENTLY`, `REINDEX [VERBOSE] SCHEMA|DATABASE|SYSTEM`, `CREATE` or
|
|
`DROP DATABASE|TABLESPACE|SUBSCRIPTION`, `ALTER SYSTEM` or `DISCARD ALL` (Flyway 11,
|
|
`PostgreSQLParser`; words inside parentheses are skipped, so `VACUUM (FULL)` and `REINDEX (VERBOSE)
|
|
SCHEMA` count). One such statement puts the whole script outside a transaction. Next to any other
|
|
statement — a `SET LOCAL lock_timeout` included — Flyway refuses the migration at deploy under its
|
|
default `mixed=false`; a service that sets `mixed=true` gets it run, with the bound silently gone.
|
|
Put such a statement in a migration of its own. `REINDEX TABLE|INDEX CONCURRENTLY` is not on
|
|
Flyway's list: Flyway runs it in the transaction, PostgreSQL refuses it there, and it needs
|
|
`executeInTransaction=false` (below). A locking statement that cannot run in a transaction at all —
|
|
`VACUUM FULL`, a non-concurrent `REINDEX SCHEMA|DATABASE|SYSTEM` — can never be bounded this way and
|
|
does not belong in a migration: run it by hand as an operations step, outside a deploy, in a psql
|
|
session that sets `SET lock_timeout = '5s';` first (the session ends with the step, so the plain
|
|
`SET` is the bound there). A `.sql.conf` with `executeInTransaction=true` does not change that: it
|
|
keeps the statement and its `SET LOCAL` in the transaction only for PostgreSQL to refuse it at
|
|
deploy, and the inspector below refuses it as well.
|
|
|
|
Enforced, not remembered: `LockWaitMigrationInspector` (persistence-jpa testkit) reads a script —
|
|
and the `.sql.conf` beside it, as Flyway reads it (a properties file, `false` in any case), and
|
|
otherwise the statements Flyway takes out of the transaction on its own — and names the statements
|
|
from its list that lock an existing table with no bound in force: `ALTER TABLE`, `LOCK TABLE`, `TRUNCATE`, `DROP TABLE`; `CREATE INDEX`,
|
|
`DROP INDEX`, `REINDEX` and `REFRESH MATERIALIZED VIEW` without `CONCURRENTLY`, `CLUSTER`, `VACUUM
|
|
FULL`; `REFERENCES` on the table it references, in a new table as much as in an `ALTER TABLE` (a
|
|
foreign key locks its parent against writes while it is added); `CREATE TRIGGER` and `DROP TRIGGER`
|
|
on their table; `UPDATE` and `DELETE`. It reads text, so a statement outside the list is not
|
|
checked. `MigrationLockWaitTest` runs it over every tree this leaf's database runs,
|
|
`SampleMigrationLockWaitTest` over the sample's stream; a service stream copies the latter with its
|
|
own directory. Both find the migrations the way Flyway does, decimal versions (`V22_1__`) included
|
|
(`VersionedMigrationFiles`, testkit). The migrations released before the rule wait unbounded and
|
|
stay that way (they are applied, and applied files are never edited); the test lists them and
|
|
refuses any new one. `PostgreSqlMigrationLockWaitIntegrationTest`
|
|
(`postgresqlMigrationLockWaitIntegrationTest`) shows on real PostgreSQL that a bounded migration
|
|
gives up within the bound while another transaction holds the table, and leaves the version below
|
|
it; it also compares the inspector's copy of Flyway's list with Flyway itself for the statements it
|
|
lists (each is named by the inspector exactly when Flyway refuses it next to a `SET LOCAL`), and
|
|
shows a `VACUUM FULL` run under `mixed=true` waiting out a reader despite its `SET LOCAL`. A
|
|
statement a Flyway upgrade adds to the list is outside that comparison; `LockWaitMigrationInspectorTest`
|
|
pins Flyway's own patterns (`PostgreSQLParser`), so such an upgrade fails the pull request's `ci`
|
|
until the copy follows. The gRPC ledger's tree lives in the
|
|
optional build, which cannot depend on the testkit: its `GrpcOperationLedgerMigrationLockWaitIT`
|
|
runs every ledger migration after V001, read from the tree, against a held ledger instead, so its
|
|
next migration is checked on real PostgreSQL. It finds and orders that tree with the testkit's own
|
|
`VersionedMigrationFiles`, which the ledger's `build.gradle` copies into its integration-test sources
|
|
at build time (`sharedMigrationFileSources`), so both builds read one definition. A concurrent index
|
|
build in that tree runs outside a transaction and is no case for a held ledger: it goes in the
|
|
test's `RUN_OUTSIDE_A_TRANSACTION` with its reason, and the same test holds each file listed there to
|
|
`ConcurrentIndexMigrationInspector`, copied the same way ("Concurrent index builds"). Every ledger
|
|
lane applies the tree one file at a time, as Flyway does, so such a file is not refused inside the
|
|
implicit transaction of one string holding many. The rule came
|
|
from V21 having the bound and V22, in the same commit, not having it (re-review 2026-11-03 round 2,
|
|
RR-02-09; round 3, RR3-02-01, RR3-02-03, RR3-02-04; round 4, RR4-02-01; round 6, RR6-03-02,
|
|
RR6-03-03; round 7, RR7-03-01).
|
|
|
|
## Concurrent index builds
|
|
|
|
`CREATE INDEX CONCURRENTLY`, `DROP INDEX CONCURRENTLY` and `REINDEX … CONCURRENTLY` cannot run
|
|
inside a transaction block. Flyway runs a script outside a transaction on its own when it holds
|
|
`CREATE`/`DROP INDEX CONCURRENTLY` or `REINDEX SCHEMA|DATABASE`, with or without `CONCURRENTLY`
|
|
(above). It runs `REINDEX INDEX|TABLE … CONCURRENTLY` — `REINDEX (CONCURRENTLY) TABLE` included — in
|
|
the transaction, where PostgreSQL refuses it. `PostgreSqlMigrationLockWaitIntegrationTest` shows the
|
|
five forms on real PostgreSQL. Write the companion configuration for every concurrent statement
|
|
anyway, so the file says what it needs and no reader has to know Flyway's list:
|
|
|
|
```conf
|
|
# V42__order_index.sql.conf
|
|
executeInTransaction=false
|
|
```
|
|
|
|
Outside a transaction, `CREATE INDEX CONCURRENTLY` and `REINDEX … CONCURRENTLY` still wait until
|
|
every transaction older than them has ended. Under Flyway's default lock that includes the
|
|
transaction Flyway itself keeps open, and Flyway waits for the build in turn: the deploy never ends,
|
|
and the table's other DDL and autovacuum queue behind the build meanwhile. The template's Flyway
|
|
holds its lock for the session instead ("Who may change the schema"), so the build waits only for
|
|
other transactions; a deployment tool left on Flyway's default stops on the first such migration.
|
|
`DROP INDEX CONCURRENTLY` does not wait that way: it waits only for the transactions that hold a
|
|
lock on its table. `PostgreSqlConcurrentIndexMigrationIntegrationTest`
|
|
(`postgresqlConcurrentIndexMigrationIntegrationTest`) applies both builds as a second deploy, with
|
|
the Flyway Spring Boot builds from the module defaults and with `FlywaySchemaPolicy.configure()`,
|
|
on real PostgreSQL. The other transactions still count: a long one an operator or a report keeps
|
|
open holds the build, and the drop too once it has touched the table, and a file run outside a
|
|
transaction cannot bound that wait with `SET LOCAL`. Look for old transactions
|
|
(`pg_stat_activity.xact_start`) before deploying one.
|
|
|
|
The testkit's `ConcurrentIndexMigrationInspector` checks a script for this in a migration test
|
|
(nothing checks it at startup), and additionally requires the migration to contain nothing else.
|
|
Next to another statement, a concurrent one makes Flyway refuse the file before any of it runs when
|
|
nothing configures it; with `executeInTransaction=false` Flyway commits each statement on its own,
|
|
and a failed build leaves the statements before it applied — state to reconstruct by hand.
|
|
|
|
Alone, a failed statement leaves an invalid index and a history row with `success = false` for its
|
|
version, and Flyway refuses every later deploy ("Detected failed migration to version …") until that
|
|
row is gone: dropping the index is not enough. The recovery is
|
|
[migration-failed.md](../runbooks/migration-failed.md) "Recover a failed concurrent index migration":
|
|
confirm no build is running, remove the cause, drop the invalid index a failed build or rebuild
|
|
left, delete that version's failed row and nothing else, deploy again. A failed `DROP INDEX
|
|
CONCURRENTLY` leaves invalid the index it was dropping; the runbook leaves that one for the deploy
|
|
again to drop, since dropped by hand it makes the same statement fail. It is not `flyway repair`,
|
|
which would also rewrite the remaining rows' checksums. `PostgreSqlConcurrentIndexMigrationIntegrationTest`
|
|
(`postgresqlConcurrentIndexMigrationIntegrationTest`) runs the runbook's SQL, read from the runbook,
|
|
after a unique build that failed on a duplicate and after a rebuild and a drop cancelled while they
|
|
waited for an open transaction, and shows both mixed files above.
|
|
|
|
An invalid index is not merely useless — the planner ignores it while every write still maintains
|
|
it. [runbooks.md](runbooks.md) has the query that finds them. Nothing drops them automatically: an
|
|
invalid index can also mean a build is still running, and the two are indistinguishable from the
|
|
catalog alone.
|
|
|
|
## Upgrade scenarios
|
|
|
|
Three, each catching something the others do not:
|
|
|
|
| Scenario | Catches |
|
|
|---|---|
|
|
| `empty` | an early migration edited to match a later one, no longer applying to a fresh database |
|
|
| `previous-release` | the actual deployment path; the only one exercising this release's migrations |
|
|
| `oldest-supported` | a migration that silently assumes state only recent databases have |
|
|
|
|
Each asserts a data invariant, not just the schema version. A migration that renames a column and
|
|
loses its contents leaves the version correct and the data gone.
|