문제 (재리뷰 round7 영속성 RR7-03-01~05) - 실패한 동시 인덱스 마이그레이션의 복구를 가이드·검사기·학습 문서가 "DROP INDEX 하나"로 안내한다. Flyway는 트랜잭션 밖에서 실패한 버전의 이력 행(success = false)을 남기고, 다음 배포의 validate가 그 행 때문에 다시 멈춘다. runbook에는 이 경우가 없고 이력 수정은 금지다. docs/jpa/runbooks.md "An invalid index exists"도 같은 이유로 재적용에서 멈춘다. - 검사기 Javadoc·학습 문서가 "섞인 파일은 부분 적용되지 않는다"고 쓴다. 가이드가 요구하는 executeInTransaction=false를 붙이면 Flyway가 문장마다 커밋해 앞 문장이 남는다. - 가이드·검사기 Javadoc·실패 메시지·학습 문서가 "Flyway는 REINDEX … CONCURRENTLY를 트랜잭션 안에서 돌린다"고 쓴다. REINDEX SCHEMA|DATABASE CONCURRENTLY는 Flyway가 스스로 트랜잭션 밖에서 돌린다. - LockWaitMigrationInspector는 .sql.conf가 executeInTransaction=true면 SET LOCAL 뒤의 VACUUM FULL·REINDEX SCHEMA를 통과시킨다. Javadoc은 "늘 거부"라고 쓴다. - ledger 잠금 대기 IT가 동시 인덱스 파일을 "이 테스트에 예외로 적으라"고 안내하지만 예외를 적을 곳이 없다. 같은 원인으로 ledger의 다른 IT 두 개 (GrpcOperationLedgerPostgresIT, JpaGrpcOperationLedgerContractIT)도 모든 파일을 한 문자열로 실행해, 동시 인덱스 파일이 생기면 "cannot be executed within a pipeline"으로 실패한다. 고른 방법 - 복구(RR7-03-03): 소유자 결정(2026-11-06)대로 migration-failed.md에 "Recover a failed concurrent index migration" 절을 둔다. 진행 중 빌드 확인 → 무효 인덱스 DROP INDEX CONCURRENTLY → 원인 정리 → 그 버전의 실패 행만 DELETE → 재배포. "Do not"에 이 한 가지 예외를 적고 flyway repair는 계속 금지한다. SQL 세 블록에 runbook-sql 표시를 달고 postgresqlConcurrentIndexMigrationIntegrationTest가 runbook에서 읽어 실 PostgreSQL에서 돌린다(outbox runbook과 같은 방식). 가이드·검사기 Javadoc·메시지·학습 문서·jpa/runbooks.md는 그 절을 가리킨다. - 섞인 파일(RR7-03-04)·REINDEX(RR7-03-02): 문장을 고치고, 같은 IT와 잠금 대기 IT에 사실을 고정한다(섞인 파일 두 경우, REINDEX 다섯 형태). - 검사기(RR7-03-05): 트랜잭션 안에서 돌 수 없는 잠금 문장은 .sql.conf와 상관없이 거부한다(리뷰 권장안). 버린 안: Javadoc만 좁히기 — 그 파일은 PostgreSQL이 배포 때 25001로 거부하므로 검사기가 먼저 알려 주는 쪽이 맞다. 잠그지 않는 문장(plain VACUUM 등)은 이 검사기의 범위가 아니라 그대로 둔다. - ledger(RR7-03-01): IT에 RUN_OUTSIDE_A_TRANSACTION(파일 → 이유) 목록을 두고 잠금 대기 케이스에서 뺀다. 목록의 파일은 같은 IT가 동시 인덱스 문장 하나·executeInTransaction=false인지 ConcurrentIndexMigration Inspector로 본다. optional 빌드는 testkit에 의존할 수 없어 sharedMigrationFileSources가 그 파일과 MigrationResource를 복사한다 (VersionedMigrationFiles와 같은 방식). ledger IT 셋은 마이그레이션을 파일마다 따로 실행한다(Flyway처럼). 버린 안: 안내만 "다룰 수 없다"로 바꾸기 — 다음 동시 인덱스 파일이 생길 때 같은 질문이 다시 남는다. 위험·한계 - 검사기 변경으로 서비스의 자기 스트림 검사가 새로 실패할 수 있다. 걸리는 파일은 PostgreSQL이 배포 때 거부하는 파일뿐이라 배포된 적이 없다(업그레이드 안내 4.1에 적음). - runbook의 DELETE는 운영자가 version을 옮겨 적는다. NOT success라 성공한 행은 지우지 못하지만, 다른 버전의 실패 행을 잘못 적으면 그 행이 지워진다. 한 행만 지워야 하고 0행이면 다시 분류하라고 적었다. - ledger 복사 파일 두 개도 JDK만 써야 한다(각 Javadoc에 적음). - Spotless 검사는 같은 worktree에서 spotlessApply가 변경 없이 끝난 직후라 UP-TO-DATE로 처리됐다. 검증 (desktop, 공유 잠금, 로그 ~/r8-f3-logs/) RED — 기준73d39212+ 이 커밋의 테스트(+ 기준 문서 주장을 단언하는 임시 메서드, + 임시 ledger V004): - persistence-jpa test --tests LockWaitMigrationInspectorTest: 80건 중 2건 실패, 검사기가 빈 목록(통과)을 돌려줌 (red-unit.log) - postgresqlMigrationLockWaitIntegrationTest: 47건 중 4건 실패 — 옛 주장 대로면 거부여야 할 REINDEX SCHEMA|DATABASE CONCURRENTLY가 applied, conf=true VACUUM FULL·REINDEX SCHEMA를 PostgreSQL은 25001로 거부하는데 검사기는 통과 (red2-lockwait.log) - postgresqlConcurrentIndexMigrationIntegrationTest: 9건 중 3건 실패 — 옛 절차(DROP INDEX) 뒤 재배포가 "Detected failed migration to version 2", conf를 붙인 섞인 파일이 행 3001·무효 인덱스·실패 이력을 남김, runbook 블록이 없어 복구 테스트 준비 실패(RED로 세지 않음) (red2-concurrent.log) - ledger integrationTest(V004 동시 인덱스): 6건 중 4건 실패 — 잠금 대기 IT "name it as an exception here", 다른 IT 둘 "cannot be executed within a pipeline" (red-ledger.log) GREEN — 이 커밋의 트리b49369e7: - :adapter:outbound:persistence-jpa:check rc=0 (test 666건 실패 0), :testQualityCheck rc=0 - postgresqlMigrationLockWaitIntegrationTest rc=0 (42건 실패 0) - postgresqlConcurrentIndexMigrationIntegrationTest rc=0 (7건 실패 0) - -p optional-platforms :grpc:grpc-operation-ledger-jpa:check integrationTest testQualityCheck rc=0 (IT 3+2+42건 실패 0) - :app-bootstrap:documentContractTest rc=0 (66건 실패 0) - 두 빌드의 verifyStrictLaneStages --dry-run rc=0 - ledger 탐침(임시 V004): 목록에 없으면 잠금 대기 케이스만 실패하고 메시지가 RUN_OUTSIDE_A_TRANSACTION을 가리킴(48건 중 1건), 목록에 넣으면 rc=0, 목록에 넣고 SET LOCAL을 섞으면 목록 검사가 "mixes a concurrent index statement"로 거부 - python3 scripts/check-work-docs.py --links docs: 깨진 링크 0 Refs: docs/reviews/2026-11-03-rereview/round7/03-persistence.md
192 lines
14 KiB
Markdown
192 lines
14 KiB
Markdown
# Migration Guide
|
|
|
|
Design §31-§32. Flyway owns the schema; Hibernate only validates.
|
|
|
|
## Who may change the schema
|
|
|
|
| Environment | Mode |
|
|
|---|---|
|
|
| local, test, dev | migrate at startup with the migration credential |
|
|
| staging, prod | deployment-owned migration; the application validates only |
|
|
|
|
Migrating from inside the application in production means every instance of a rolling deploy races
|
|
to apply the same script, and the loser's failure is indistinguishable from a real one.
|
|
|
|
Whoever runs the migration holds Flyway's PostgreSQL migration lock for the session, not in a
|
|
transaction: `flyway.postgresql.transactional.lock=false`. Flyway's default (`true`) keeps a
|
|
transaction open until the whole run ends, and a concurrent index build waits for every transaction
|
|
older than itself, Flyway's own included — the deploy stops with no error and no time limit
|
|
("Concurrent index builds" below). The template sets it for every Flyway it runs: the module
|
|
defaults (`config/module-defaults/persistence-jpa.yml`, `spring.flyway.postgresql.transactional-lock:
|
|
false`) for the one Spring Boot builds at startup, and `FlywaySchemaPolicy.configure()` for every
|
|
stream it builds itself — a service's own stream starts from it too, and `JpaProductionArchitectureTest`
|
|
refuses production code that calls `Flyway.configure()` directly. The deployment-owned migration of
|
|
staging and prod is outside this repository: give its tool the same value
|
|
(`FLYWAY_POSTGRESQL_TRANSACTIONAL_LOCK=false`, or the key above in its configuration file).
|
|
|
|
A session lock assumes the connection Flyway takes stays its own for the whole run, which the
|
|
template's direct Hikari connection gives. Behind a pooler in transaction mode (PgBouncer
|
|
`pool_mode = transaction`) the unlock can reach another server connection and release nothing, and
|
|
the next migration waits for a lock that a pooled connection still holds. Run migrations over a
|
|
direct connection or a session-mode pool.
|
|
|
|
`ddl-auto` is `validate` or `none`. Never `update`: it never drops or narrows anything, so it
|
|
produces a schema that is neither the old one nor the one the migrations describe — silently, on
|
|
whichever instance started first.
|
|
|
|
## Validation fails closed and never repairs
|
|
|
|
At startup Flyway's own validation (and Hibernate `ddl-auto=validate`) refuses a checksum mismatch,
|
|
a missing migration, or a schema Hibernate disagrees with. Nothing in the application calls
|
|
`repair`. In tests, the testkit's `FlywayValidationGate` turns a failed `ValidateResult` into
|
|
`SchemaMismatchException`; the migration contract tests use it.
|
|
|
|
Repair rewrites the schema history table to match whatever scripts are on disk. That resolves the
|
|
symptom by deleting the evidence: a checksum mismatch means the deployed script differs from the
|
|
applied one, and the interesting question is which change is missing from this database. Repair
|
|
makes that question unaskable. It exists only as an explicit admin operation with an operator, a
|
|
reason, and an approval (design §8.4).
|
|
|
|
The one change to the history an operator makes is narrower than repair: after a concurrent index
|
|
migration fails, the failed row of that one version is deleted so that the version can run again
|
|
("Concurrent index builds" below). Every applied row, and its checksum, stays as it was.
|
|
|
|
Only Flyway's structured error codes reach the exception. Its messages embed the script path and
|
|
part of the failing statement.
|
|
|
|
## Bound the lock wait
|
|
|
|
A migration that locks a table already in use starts with `SET LOCAL lock_timeout = '5s';`, before
|
|
the first statement that takes the lock. `ALTER TABLE` takes the strongest lock there is, which
|
|
conflicts even with a plain read; a non-concurrent `CREATE INDEX` takes one that stops every write;
|
|
`UPDATE` and `DELETE` wait on the rows other transactions hold. Unbounded, the migration waits for
|
|
the oldest open transaction that touched the table — an operator's forgotten `BEGIN` in psql has no
|
|
deadline — and every read and write that arrives meanwhile queues behind the migration, holding a
|
|
pool connection while it waits. With the bound, the migration fails after five seconds, Flyway
|
|
rolls it back whole, the database stays on the version before it and the new release does not
|
|
start: a failed deploy instead of an outage. Deploy again when the transaction has ended.
|
|
|
|
`LOCAL`, because a plain `SET` outlives the migration on the pooled connection Flyway borrowed and
|
|
changes every statement that later runs on it. The value is not zero: `lock_timeout = 0` (or
|
|
`DEFAULT`, or a later `RESET`) is no bound at all. A migration with `executeInTransaction=false`
|
|
cannot use it — `SET LOCAL` outside a transaction only warns — so a statement that locks a table
|
|
goes into a migration that runs in one. A statement on a table the same migration created a moment
|
|
earlier waits for nothing and needs no bound; `CREATE TABLE IF NOT EXISTS` does not count, because
|
|
the table may already be there.
|
|
|
|
Flyway also takes a migration out of its transaction on its own, without a `.sql.conf`: on
|
|
PostgreSQL 12 and later, any statement that starts with `VACUUM`, `CREATE [UNIQUE] INDEX
|
|
CONCURRENTLY`, `DROP INDEX CONCURRENTLY`, `REINDEX [VERBOSE] SCHEMA|DATABASE|SYSTEM`, `CREATE` or
|
|
`DROP DATABASE|TABLESPACE|SUBSCRIPTION`, `ALTER SYSTEM` or `DISCARD ALL` (Flyway 11,
|
|
`PostgreSQLParser`; words inside parentheses are skipped, so `VACUUM (FULL)` and `REINDEX (VERBOSE)
|
|
SCHEMA` count). One such statement puts the whole script outside a transaction. Next to any other
|
|
statement — a `SET LOCAL lock_timeout` included — Flyway refuses the migration at deploy under its
|
|
default `mixed=false`; a service that sets `mixed=true` gets it run, with the bound silently gone.
|
|
Put such a statement in a migration of its own. `REINDEX TABLE|INDEX CONCURRENTLY` is not on
|
|
Flyway's list: Flyway runs it in the transaction, PostgreSQL refuses it there, and it needs
|
|
`executeInTransaction=false` (below). A locking statement that cannot run in a transaction at all —
|
|
`VACUUM FULL`, a non-concurrent `REINDEX SCHEMA|DATABASE|SYSTEM` — can never be bounded this way and
|
|
does not belong in a migration: run it by hand as an operations step, outside a deploy, in a psql
|
|
session that sets `SET lock_timeout = '5s';` first (the session ends with the step, so the plain
|
|
`SET` is the bound there). A `.sql.conf` with `executeInTransaction=true` does not change that: it
|
|
keeps the statement and its `SET LOCAL` in the transaction only for PostgreSQL to refuse it at
|
|
deploy, and the inspector below refuses it as well.
|
|
|
|
Enforced, not remembered: `LockWaitMigrationInspector` (persistence-jpa testkit) reads a script —
|
|
and the `.sql.conf` beside it, as Flyway reads it (a properties file, `false` in any case), and
|
|
otherwise the statements Flyway takes out of the transaction on its own — and names the statements
|
|
from its list that lock an existing table with no bound in force: `ALTER TABLE`, `LOCK TABLE`, `TRUNCATE`, `DROP TABLE`; `CREATE INDEX`,
|
|
`DROP INDEX`, `REINDEX` and `REFRESH MATERIALIZED VIEW` without `CONCURRENTLY`, `CLUSTER`, `VACUUM
|
|
FULL`; `REFERENCES` on the table it references, in a new table as much as in an `ALTER TABLE` (a
|
|
foreign key locks its parent against writes while it is added); `CREATE TRIGGER` and `DROP TRIGGER`
|
|
on their table; `UPDATE` and `DELETE`. It reads text, so a statement outside the list is not
|
|
checked. `MigrationLockWaitTest` runs it over every tree this leaf's database runs,
|
|
`SampleMigrationLockWaitTest` over the sample's stream; a service stream copies the latter with its
|
|
own directory. Both find the migrations the way Flyway does, decimal versions (`V22_1__`) included
|
|
(`VersionedMigrationFiles`, testkit). The migrations released before the rule wait unbounded and
|
|
stay that way (they are applied, and applied files are never edited); the test lists them and
|
|
refuses any new one. `PostgreSqlMigrationLockWaitIntegrationTest`
|
|
(`postgresqlMigrationLockWaitIntegrationTest`) shows on real PostgreSQL that a bounded migration
|
|
gives up within the bound while another transaction holds the table, and leaves the version below
|
|
it; it also compares the inspector's copy of Flyway's list with Flyway itself for the statements it
|
|
lists (each is named by the inspector exactly when Flyway refuses it next to a `SET LOCAL`), and
|
|
shows a `VACUUM FULL` run under `mixed=true` waiting out a reader despite its `SET LOCAL`. A
|
|
statement a Flyway upgrade adds to the list is outside that comparison; `LockWaitMigrationInspectorTest`
|
|
pins Flyway's own patterns (`PostgreSQLParser`), so such an upgrade fails the pull request's `ci`
|
|
until the copy follows. The gRPC ledger's tree lives in the
|
|
optional build, which cannot depend on the testkit: its `GrpcOperationLedgerMigrationLockWaitIT`
|
|
runs every ledger migration after V001, read from the tree, against a held ledger instead, so its
|
|
next migration is checked on real PostgreSQL. It finds and orders that tree with the testkit's own
|
|
`VersionedMigrationFiles`, which the ledger's `build.gradle` copies into its integration-test sources
|
|
at build time (`sharedMigrationFileSources`), so both builds read one definition. A concurrent index
|
|
build in that tree runs outside a transaction and is no case for a held ledger: it goes in the
|
|
test's `RUN_OUTSIDE_A_TRANSACTION` with its reason, and the same test holds each file listed there to
|
|
`ConcurrentIndexMigrationInspector`, copied the same way ("Concurrent index builds"). Every ledger
|
|
lane applies the tree one file at a time, as Flyway does, so such a file is not refused inside the
|
|
implicit transaction of one string holding many. The rule came
|
|
from V21 having the bound and V22, in the same commit, not having it (re-review 2026-11-03 round 2,
|
|
RR-02-09; round 3, RR3-02-01, RR3-02-03, RR3-02-04; round 4, RR4-02-01; round 6, RR6-03-02,
|
|
RR6-03-03; round 7, RR7-03-01).
|
|
|
|
## Concurrent index builds
|
|
|
|
`CREATE INDEX CONCURRENTLY`, `DROP INDEX CONCURRENTLY` and `REINDEX … CONCURRENTLY` cannot run
|
|
inside a transaction block. Flyway runs a script outside a transaction on its own when it holds
|
|
`CREATE`/`DROP INDEX CONCURRENTLY` or `REINDEX SCHEMA|DATABASE`, with or without `CONCURRENTLY`
|
|
(above). It runs `REINDEX INDEX|TABLE … CONCURRENTLY` — `REINDEX (CONCURRENTLY) TABLE` included — in
|
|
the transaction, where PostgreSQL refuses it. `PostgreSqlMigrationLockWaitIntegrationTest` shows the
|
|
five forms on real PostgreSQL. Write the companion configuration for every concurrent statement
|
|
anyway, so the file says what it needs and no reader has to know Flyway's list:
|
|
|
|
```conf
|
|
# V42__order_index.sql.conf
|
|
executeInTransaction=false
|
|
```
|
|
|
|
Outside a transaction, `CREATE INDEX CONCURRENTLY` and `REINDEX … CONCURRENTLY` still wait until
|
|
every transaction older than them has ended. Under Flyway's default lock that includes the
|
|
transaction Flyway itself keeps open, and Flyway waits for the build in turn: the deploy never ends,
|
|
and the table's other DDL and autovacuum queue behind the build meanwhile. The template's Flyway
|
|
holds its lock for the session instead ("Who may change the schema"), so the build waits only for
|
|
other transactions; a deployment tool left on Flyway's default stops on the first such migration.
|
|
`DROP INDEX CONCURRENTLY` does not wait that way. `PostgreSqlConcurrentIndexMigrationIntegrationTest`
|
|
(`postgresqlConcurrentIndexMigrationIntegrationTest`) applies both builds as a second deploy, with
|
|
the Flyway Spring Boot builds from the module defaults and with `FlywaySchemaPolicy.configure()`,
|
|
on real PostgreSQL. The other transactions still count: a long one an operator or a report keeps
|
|
open holds the build, and a file run outside a transaction cannot bound that wait with `SET LOCAL`.
|
|
Look for old transactions (`pg_stat_activity.xact_start`) before deploying one.
|
|
|
|
The testkit's `ConcurrentIndexMigrationInspector` checks a script for this in a migration test
|
|
(nothing checks it at startup), and additionally requires the migration to contain nothing else.
|
|
Next to another statement, a concurrent one makes Flyway refuse the file before any of it runs when
|
|
nothing configures it; with `executeInTransaction=false` Flyway commits each statement on its own,
|
|
and a failed build leaves the statements before it applied — state to reconstruct by hand.
|
|
|
|
Alone, a failed build leaves an invalid index and a history row with `success = false` for its
|
|
version, and Flyway refuses every later deploy ("Detected failed migration to version …") until that
|
|
row is gone: dropping the index is not enough. The recovery is
|
|
[migration-failed.md](../runbooks/migration-failed.md) "Recover a failed concurrent index migration":
|
|
confirm no build is running, drop the invalid index, remove the cause, delete that version's failed
|
|
row and nothing else, deploy again. It is not `flyway repair`, which would also rewrite the
|
|
remaining rows' checksums. `PostgreSqlConcurrentIndexMigrationIntegrationTest`
|
|
(`postgresqlConcurrentIndexMigrationIntegrationTest`) runs the runbook's SQL, read from the runbook,
|
|
after a unique build that failed on a duplicate, and shows both mixed files above.
|
|
|
|
An invalid index is not merely useless — the planner ignores it while every write still maintains
|
|
it. [runbooks.md](runbooks.md) has the query that finds them. Nothing drops them automatically: an
|
|
invalid index can also mean a build is still running, and the two are indistinguishable from the
|
|
catalog alone.
|
|
|
|
## Upgrade scenarios
|
|
|
|
Three, each catching something the others do not:
|
|
|
|
| Scenario | Catches |
|
|
|---|---|
|
|
| `empty` | an early migration edited to match a later one, no longer applying to a fresh database |
|
|
| `previous-release` | the actual deployment path; the only one exercising this release's migrations |
|
|
| `oldest-supported` | a migration that silently assumes state only recent databases have |
|
|
|
|
Each asserts a data invariant, not just the schema version. A migration that renames a column and
|
|
loses its contents leaves the version correct and the data gone.
|