문제 (재리뷰 round7 영속성 RR7-03-01~05) - 실패한 동시 인덱스 마이그레이션의 복구를 가이드·검사기·학습 문서가 "DROP INDEX 하나"로 안내한다. Flyway는 트랜잭션 밖에서 실패한 버전의 이력 행(success = false)을 남기고, 다음 배포의 validate가 그 행 때문에 다시 멈춘다. runbook에는 이 경우가 없고 이력 수정은 금지다. docs/jpa/runbooks.md "An invalid index exists"도 같은 이유로 재적용에서 멈춘다. - 검사기 Javadoc·학습 문서가 "섞인 파일은 부분 적용되지 않는다"고 쓴다. 가이드가 요구하는 executeInTransaction=false를 붙이면 Flyway가 문장마다 커밋해 앞 문장이 남는다. - 가이드·검사기 Javadoc·실패 메시지·학습 문서가 "Flyway는 REINDEX … CONCURRENTLY를 트랜잭션 안에서 돌린다"고 쓴다. REINDEX SCHEMA|DATABASE CONCURRENTLY는 Flyway가 스스로 트랜잭션 밖에서 돌린다. - LockWaitMigrationInspector는 .sql.conf가 executeInTransaction=true면 SET LOCAL 뒤의 VACUUM FULL·REINDEX SCHEMA를 통과시킨다. Javadoc은 "늘 거부"라고 쓴다. - ledger 잠금 대기 IT가 동시 인덱스 파일을 "이 테스트에 예외로 적으라"고 안내하지만 예외를 적을 곳이 없다. 같은 원인으로 ledger의 다른 IT 두 개 (GrpcOperationLedgerPostgresIT, JpaGrpcOperationLedgerContractIT)도 모든 파일을 한 문자열로 실행해, 동시 인덱스 파일이 생기면 "cannot be executed within a pipeline"으로 실패한다. 고른 방법 - 복구(RR7-03-03): 소유자 결정(2026-11-06)대로 migration-failed.md에 "Recover a failed concurrent index migration" 절을 둔다. 진행 중 빌드 확인 → 무효 인덱스 DROP INDEX CONCURRENTLY → 원인 정리 → 그 버전의 실패 행만 DELETE → 재배포. "Do not"에 이 한 가지 예외를 적고 flyway repair는 계속 금지한다. SQL 세 블록에 runbook-sql 표시를 달고 postgresqlConcurrentIndexMigrationIntegrationTest가 runbook에서 읽어 실 PostgreSQL에서 돌린다(outbox runbook과 같은 방식). 가이드·검사기 Javadoc·메시지·학습 문서·jpa/runbooks.md는 그 절을 가리킨다. - 섞인 파일(RR7-03-04)·REINDEX(RR7-03-02): 문장을 고치고, 같은 IT와 잠금 대기 IT에 사실을 고정한다(섞인 파일 두 경우, REINDEX 다섯 형태). - 검사기(RR7-03-05): 트랜잭션 안에서 돌 수 없는 잠금 문장은 .sql.conf와 상관없이 거부한다(리뷰 권장안). 버린 안: Javadoc만 좁히기 — 그 파일은 PostgreSQL이 배포 때 25001로 거부하므로 검사기가 먼저 알려 주는 쪽이 맞다. 잠그지 않는 문장(plain VACUUM 등)은 이 검사기의 범위가 아니라 그대로 둔다. - ledger(RR7-03-01): IT에 RUN_OUTSIDE_A_TRANSACTION(파일 → 이유) 목록을 두고 잠금 대기 케이스에서 뺀다. 목록의 파일은 같은 IT가 동시 인덱스 문장 하나·executeInTransaction=false인지 ConcurrentIndexMigration Inspector로 본다. optional 빌드는 testkit에 의존할 수 없어 sharedMigrationFileSources가 그 파일과 MigrationResource를 복사한다 (VersionedMigrationFiles와 같은 방식). ledger IT 셋은 마이그레이션을 파일마다 따로 실행한다(Flyway처럼). 버린 안: 안내만 "다룰 수 없다"로 바꾸기 — 다음 동시 인덱스 파일이 생길 때 같은 질문이 다시 남는다. 위험·한계 - 검사기 변경으로 서비스의 자기 스트림 검사가 새로 실패할 수 있다. 걸리는 파일은 PostgreSQL이 배포 때 거부하는 파일뿐이라 배포된 적이 없다(업그레이드 안내 4.1에 적음). - runbook의 DELETE는 운영자가 version을 옮겨 적는다. NOT success라 성공한 행은 지우지 못하지만, 다른 버전의 실패 행을 잘못 적으면 그 행이 지워진다. 한 행만 지워야 하고 0행이면 다시 분류하라고 적었다. - ledger 복사 파일 두 개도 JDK만 써야 한다(각 Javadoc에 적음). - Spotless 검사는 같은 worktree에서 spotlessApply가 변경 없이 끝난 직후라 UP-TO-DATE로 처리됐다. 검증 (desktop, 공유 잠금, 로그 ~/r8-f3-logs/) RED — 기준73d39212+ 이 커밋의 테스트(+ 기준 문서 주장을 단언하는 임시 메서드, + 임시 ledger V004): - persistence-jpa test --tests LockWaitMigrationInspectorTest: 80건 중 2건 실패, 검사기가 빈 목록(통과)을 돌려줌 (red-unit.log) - postgresqlMigrationLockWaitIntegrationTest: 47건 중 4건 실패 — 옛 주장 대로면 거부여야 할 REINDEX SCHEMA|DATABASE CONCURRENTLY가 applied, conf=true VACUUM FULL·REINDEX SCHEMA를 PostgreSQL은 25001로 거부하는데 검사기는 통과 (red2-lockwait.log) - postgresqlConcurrentIndexMigrationIntegrationTest: 9건 중 3건 실패 — 옛 절차(DROP INDEX) 뒤 재배포가 "Detected failed migration to version 2", conf를 붙인 섞인 파일이 행 3001·무효 인덱스·실패 이력을 남김, runbook 블록이 없어 복구 테스트 준비 실패(RED로 세지 않음) (red2-concurrent.log) - ledger integrationTest(V004 동시 인덱스): 6건 중 4건 실패 — 잠금 대기 IT "name it as an exception here", 다른 IT 둘 "cannot be executed within a pipeline" (red-ledger.log) GREEN — 이 커밋의 트리b49369e7: - :adapter:outbound:persistence-jpa:check rc=0 (test 666건 실패 0), :testQualityCheck rc=0 - postgresqlMigrationLockWaitIntegrationTest rc=0 (42건 실패 0) - postgresqlConcurrentIndexMigrationIntegrationTest rc=0 (7건 실패 0) - -p optional-platforms :grpc:grpc-operation-ledger-jpa:check integrationTest testQualityCheck rc=0 (IT 3+2+42건 실패 0) - :app-bootstrap:documentContractTest rc=0 (66건 실패 0) - 두 빌드의 verifyStrictLaneStages --dry-run rc=0 - ledger 탐침(임시 V004): 목록에 없으면 잠금 대기 케이스만 실패하고 메시지가 RUN_OUTSIDE_A_TRANSACTION을 가리킴(48건 중 1건), 목록에 넣으면 rc=0, 목록에 넣고 SET LOCAL을 섞으면 목록 검사가 "mixes a concurrent index statement"로 거부 - python3 scripts/check-work-docs.py --links docs: 깨진 링크 0 Refs: docs/reviews/2026-11-03-rereview/round7/03-persistence.md
14 KiB
Migration Guide
Design §31-§32. Flyway owns the schema; Hibernate only validates.
Who may change the schema
| Environment | Mode |
|---|---|
| local, test, dev | migrate at startup with the migration credential |
| staging, prod | deployment-owned migration; the application validates only |
Migrating from inside the application in production means every instance of a rolling deploy races to apply the same script, and the loser's failure is indistinguishable from a real one.
Whoever runs the migration holds Flyway's PostgreSQL migration lock for the session, not in a
transaction: flyway.postgresql.transactional.lock=false. Flyway's default (true) keeps a
transaction open until the whole run ends, and a concurrent index build waits for every transaction
older than itself, Flyway's own included — the deploy stops with no error and no time limit
("Concurrent index builds" below). The template sets it for every Flyway it runs: the module
defaults (config/module-defaults/persistence-jpa.yml, spring.flyway.postgresql.transactional-lock: false) for the one Spring Boot builds at startup, and FlywaySchemaPolicy.configure() for every
stream it builds itself — a service's own stream starts from it too, and JpaProductionArchitectureTest
refuses production code that calls Flyway.configure() directly. The deployment-owned migration of
staging and prod is outside this repository: give its tool the same value
(FLYWAY_POSTGRESQL_TRANSACTIONAL_LOCK=false, or the key above in its configuration file).
A session lock assumes the connection Flyway takes stays its own for the whole run, which the
template's direct Hikari connection gives. Behind a pooler in transaction mode (PgBouncer
pool_mode = transaction) the unlock can reach another server connection and release nothing, and
the next migration waits for a lock that a pooled connection still holds. Run migrations over a
direct connection or a session-mode pool.
ddl-auto is validate or none. Never update: it never drops or narrows anything, so it
produces a schema that is neither the old one nor the one the migrations describe — silently, on
whichever instance started first.
Validation fails closed and never repairs
At startup Flyway's own validation (and Hibernate ddl-auto=validate) refuses a checksum mismatch,
a missing migration, or a schema Hibernate disagrees with. Nothing in the application calls
repair. In tests, the testkit's FlywayValidationGate turns a failed ValidateResult into
SchemaMismatchException; the migration contract tests use it.
Repair rewrites the schema history table to match whatever scripts are on disk. That resolves the symptom by deleting the evidence: a checksum mismatch means the deployed script differs from the applied one, and the interesting question is which change is missing from this database. Repair makes that question unaskable. It exists only as an explicit admin operation with an operator, a reason, and an approval (design §8.4).
The one change to the history an operator makes is narrower than repair: after a concurrent index migration fails, the failed row of that one version is deleted so that the version can run again ("Concurrent index builds" below). Every applied row, and its checksum, stays as it was.
Only Flyway's structured error codes reach the exception. Its messages embed the script path and part of the failing statement.
Bound the lock wait
A migration that locks a table already in use starts with SET LOCAL lock_timeout = '5s';, before
the first statement that takes the lock. ALTER TABLE takes the strongest lock there is, which
conflicts even with a plain read; a non-concurrent CREATE INDEX takes one that stops every write;
UPDATE and DELETE wait on the rows other transactions hold. Unbounded, the migration waits for
the oldest open transaction that touched the table — an operator's forgotten BEGIN in psql has no
deadline — and every read and write that arrives meanwhile queues behind the migration, holding a
pool connection while it waits. With the bound, the migration fails after five seconds, Flyway
rolls it back whole, the database stays on the version before it and the new release does not
start: a failed deploy instead of an outage. Deploy again when the transaction has ended.
LOCAL, because a plain SET outlives the migration on the pooled connection Flyway borrowed and
changes every statement that later runs on it. The value is not zero: lock_timeout = 0 (or
DEFAULT, or a later RESET) is no bound at all. A migration with executeInTransaction=false
cannot use it — SET LOCAL outside a transaction only warns — so a statement that locks a table
goes into a migration that runs in one. A statement on a table the same migration created a moment
earlier waits for nothing and needs no bound; CREATE TABLE IF NOT EXISTS does not count, because
the table may already be there.
Flyway also takes a migration out of its transaction on its own, without a .sql.conf: on
PostgreSQL 12 and later, any statement that starts with VACUUM, CREATE [UNIQUE] INDEX CONCURRENTLY, DROP INDEX CONCURRENTLY, REINDEX [VERBOSE] SCHEMA|DATABASE|SYSTEM, CREATE or
DROP DATABASE|TABLESPACE|SUBSCRIPTION, ALTER SYSTEM or DISCARD ALL (Flyway 11,
PostgreSQLParser; words inside parentheses are skipped, so VACUUM (FULL) and REINDEX (VERBOSE) SCHEMA count). One such statement puts the whole script outside a transaction. Next to any other
statement — a SET LOCAL lock_timeout included — Flyway refuses the migration at deploy under its
default mixed=false; a service that sets mixed=true gets it run, with the bound silently gone.
Put such a statement in a migration of its own. REINDEX TABLE|INDEX CONCURRENTLY is not on
Flyway's list: Flyway runs it in the transaction, PostgreSQL refuses it there, and it needs
executeInTransaction=false (below). A locking statement that cannot run in a transaction at all —
VACUUM FULL, a non-concurrent REINDEX SCHEMA|DATABASE|SYSTEM — can never be bounded this way and
does not belong in a migration: run it by hand as an operations step, outside a deploy, in a psql
session that sets SET lock_timeout = '5s'; first (the session ends with the step, so the plain
SET is the bound there). A .sql.conf with executeInTransaction=true does not change that: it
keeps the statement and its SET LOCAL in the transaction only for PostgreSQL to refuse it at
deploy, and the inspector below refuses it as well.
Enforced, not remembered: LockWaitMigrationInspector (persistence-jpa testkit) reads a script —
and the .sql.conf beside it, as Flyway reads it (a properties file, false in any case), and
otherwise the statements Flyway takes out of the transaction on its own — and names the statements
from its list that lock an existing table with no bound in force: ALTER TABLE, LOCK TABLE, TRUNCATE, DROP TABLE; CREATE INDEX,
DROP INDEX, REINDEX and REFRESH MATERIALIZED VIEW without CONCURRENTLY, CLUSTER, VACUUM FULL; REFERENCES on the table it references, in a new table as much as in an ALTER TABLE (a
foreign key locks its parent against writes while it is added); CREATE TRIGGER and DROP TRIGGER
on their table; UPDATE and DELETE. It reads text, so a statement outside the list is not
checked. MigrationLockWaitTest runs it over every tree this leaf's database runs,
SampleMigrationLockWaitTest over the sample's stream; a service stream copies the latter with its
own directory. Both find the migrations the way Flyway does, decimal versions (V22_1__) included
(VersionedMigrationFiles, testkit). The migrations released before the rule wait unbounded and
stay that way (they are applied, and applied files are never edited); the test lists them and
refuses any new one. PostgreSqlMigrationLockWaitIntegrationTest
(postgresqlMigrationLockWaitIntegrationTest) shows on real PostgreSQL that a bounded migration
gives up within the bound while another transaction holds the table, and leaves the version below
it; it also compares the inspector's copy of Flyway's list with Flyway itself for the statements it
lists (each is named by the inspector exactly when Flyway refuses it next to a SET LOCAL), and
shows a VACUUM FULL run under mixed=true waiting out a reader despite its SET LOCAL. A
statement a Flyway upgrade adds to the list is outside that comparison; LockWaitMigrationInspectorTest
pins Flyway's own patterns (PostgreSQLParser), so such an upgrade fails the pull request's ci
until the copy follows. The gRPC ledger's tree lives in the
optional build, which cannot depend on the testkit: its GrpcOperationLedgerMigrationLockWaitIT
runs every ledger migration after V001, read from the tree, against a held ledger instead, so its
next migration is checked on real PostgreSQL. It finds and orders that tree with the testkit's own
VersionedMigrationFiles, which the ledger's build.gradle copies into its integration-test sources
at build time (sharedMigrationFileSources), so both builds read one definition. A concurrent index
build in that tree runs outside a transaction and is no case for a held ledger: it goes in the
test's RUN_OUTSIDE_A_TRANSACTION with its reason, and the same test holds each file listed there to
ConcurrentIndexMigrationInspector, copied the same way ("Concurrent index builds"). Every ledger
lane applies the tree one file at a time, as Flyway does, so such a file is not refused inside the
implicit transaction of one string holding many. The rule came
from V21 having the bound and V22, in the same commit, not having it (re-review 2026-11-03 round 2,
RR-02-09; round 3, RR3-02-01, RR3-02-03, RR3-02-04; round 4, RR4-02-01; round 6, RR6-03-02,
RR6-03-03; round 7, RR7-03-01).
Concurrent index builds
CREATE INDEX CONCURRENTLY, DROP INDEX CONCURRENTLY and REINDEX … CONCURRENTLY cannot run
inside a transaction block. Flyway runs a script outside a transaction on its own when it holds
CREATE/DROP INDEX CONCURRENTLY or REINDEX SCHEMA|DATABASE, with or without CONCURRENTLY
(above). It runs REINDEX INDEX|TABLE … CONCURRENTLY — REINDEX (CONCURRENTLY) TABLE included — in
the transaction, where PostgreSQL refuses it. PostgreSqlMigrationLockWaitIntegrationTest shows the
five forms on real PostgreSQL. Write the companion configuration for every concurrent statement
anyway, so the file says what it needs and no reader has to know Flyway's list:
# V42__order_index.sql.conf
executeInTransaction=false
Outside a transaction, CREATE INDEX CONCURRENTLY and REINDEX … CONCURRENTLY still wait until
every transaction older than them has ended. Under Flyway's default lock that includes the
transaction Flyway itself keeps open, and Flyway waits for the build in turn: the deploy never ends,
and the table's other DDL and autovacuum queue behind the build meanwhile. The template's Flyway
holds its lock for the session instead ("Who may change the schema"), so the build waits only for
other transactions; a deployment tool left on Flyway's default stops on the first such migration.
DROP INDEX CONCURRENTLY does not wait that way. PostgreSqlConcurrentIndexMigrationIntegrationTest
(postgresqlConcurrentIndexMigrationIntegrationTest) applies both builds as a second deploy, with
the Flyway Spring Boot builds from the module defaults and with FlywaySchemaPolicy.configure(),
on real PostgreSQL. The other transactions still count: a long one an operator or a report keeps
open holds the build, and a file run outside a transaction cannot bound that wait with SET LOCAL.
Look for old transactions (pg_stat_activity.xact_start) before deploying one.
The testkit's ConcurrentIndexMigrationInspector checks a script for this in a migration test
(nothing checks it at startup), and additionally requires the migration to contain nothing else.
Next to another statement, a concurrent one makes Flyway refuse the file before any of it runs when
nothing configures it; with executeInTransaction=false Flyway commits each statement on its own,
and a failed build leaves the statements before it applied — state to reconstruct by hand.
Alone, a failed build leaves an invalid index and a history row with success = false for its
version, and Flyway refuses every later deploy ("Detected failed migration to version …") until that
row is gone: dropping the index is not enough. The recovery is
migration-failed.md "Recover a failed concurrent index migration":
confirm no build is running, drop the invalid index, remove the cause, delete that version's failed
row and nothing else, deploy again. It is not flyway repair, which would also rewrite the
remaining rows' checksums. PostgreSqlConcurrentIndexMigrationIntegrationTest
(postgresqlConcurrentIndexMigrationIntegrationTest) runs the runbook's SQL, read from the runbook,
after a unique build that failed on a duplicate, and shows both mixed files above.
An invalid index is not merely useless — the planner ignores it while every write still maintains it. runbooks.md has the query that finds them. Nothing drops them automatically: an invalid index can also mean a build is still running, and the two are indistinguishable from the catalog alone.
Upgrade scenarios
Three, each catching something the others do not:
| Scenario | Catches |
|---|---|
empty |
an early migration edited to match a later one, no longer applying to a fresh database |
previous-release |
the actual deployment path; the only one exercising this release's migrations |
oldest-supported |
a migration that silently assumes state only recent databases have |
Each asserts a data invariant, not just the schema version. A migration that renames a column and loses its contents leaves the version correct and the data gone.