From 6ec0c4f7fa0d806bc03487d75c70e333300a4281 Mon Sep 17 00:00:00 2001 From: DongHyeonka Date: Sat, 7 Nov 2026 01:14:09 +0900 Subject: [PATCH] =?UTF-8?q?fix(migration):=20=EC=8B=A4=ED=8C=A8=ED=95=9C?= =?UTF-8?q?=20=EB=8F=99=EC=8B=9C=20=EC=9D=B8=EB=8D=B1=EC=8A=A4=20=EB=A7=88?= =?UTF-8?q?=EC=9D=B4=EA=B7=B8=EB=A0=88=EC=9D=B4=EC=85=98=20=EB=B3=B5?= =?UTF-8?q?=EA=B5=AC=EC=99=80=20=EA=B2=80=EC=82=AC=EA=B8=B0=20=EC=84=A4?= =?UTF-8?q?=EB=AA=85=EC=9D=84=20=EC=8B=A4=EC=A0=9C=20=EB=8F=99=EC=9E=91?= =?UTF-8?q?=EC=97=90=20=EB=A7=9E=EC=B6=98=EB=8B=A4?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 문제 (재리뷰 round7 영속성 RR7-03-01~05) - 실패한 동시 인덱스 마이그레이션의 복구를 가이드·검사기·학습 문서가 "DROP INDEX 하나"로 안내한다. Flyway는 트랜잭션 밖에서 실패한 버전의 이력 행(success = false)을 남기고, 다음 배포의 validate가 그 행 때문에 다시 멈춘다. runbook에는 이 경우가 없고 이력 수정은 금지다. docs/jpa/runbooks.md "An invalid index exists"도 같은 이유로 재적용에서 멈춘다. - 검사기 Javadoc·학습 문서가 "섞인 파일은 부분 적용되지 않는다"고 쓴다. 가이드가 요구하는 executeInTransaction=false를 붙이면 Flyway가 문장마다 커밋해 앞 문장이 남는다. - 가이드·검사기 Javadoc·실패 메시지·학습 문서가 "Flyway는 REINDEX … CONCURRENTLY를 트랜잭션 안에서 돌린다"고 쓴다. REINDEX SCHEMA|DATABASE CONCURRENTLY는 Flyway가 스스로 트랜잭션 밖에서 돌린다. - LockWaitMigrationInspector는 .sql.conf가 executeInTransaction=true면 SET LOCAL 뒤의 VACUUM FULL·REINDEX SCHEMA를 통과시킨다. Javadoc은 "늘 거부"라고 쓴다. - ledger 잠금 대기 IT가 동시 인덱스 파일을 "이 테스트에 예외로 적으라"고 안내하지만 예외를 적을 곳이 없다. 같은 원인으로 ledger의 다른 IT 두 개 (GrpcOperationLedgerPostgresIT, JpaGrpcOperationLedgerContractIT)도 모든 파일을 한 문자열로 실행해, 동시 인덱스 파일이 생기면 "cannot be executed within a pipeline"으로 실패한다. 고른 방법 - 복구(RR7-03-03): 소유자 결정(2026-11-06)대로 migration-failed.md에 "Recover a failed concurrent index migration" 절을 둔다. 진행 중 빌드 확인 → 무효 인덱스 DROP INDEX CONCURRENTLY → 원인 정리 → 그 버전의 실패 행만 DELETE → 재배포. "Do not"에 이 한 가지 예외를 적고 flyway repair는 계속 금지한다. SQL 세 블록에 runbook-sql 표시를 달고 postgresqlConcurrentIndexMigrationIntegrationTest가 runbook에서 읽어 실 PostgreSQL에서 돌린다(outbox runbook과 같은 방식). 가이드·검사기 Javadoc·메시지·학습 문서·jpa/runbooks.md는 그 절을 가리킨다. - 섞인 파일(RR7-03-04)·REINDEX(RR7-03-02): 문장을 고치고, 같은 IT와 잠금 대기 IT에 사실을 고정한다(섞인 파일 두 경우, REINDEX 다섯 형태). - 검사기(RR7-03-05): 트랜잭션 안에서 돌 수 없는 잠금 문장은 .sql.conf와 상관없이 거부한다(리뷰 권장안). 버린 안: Javadoc만 좁히기 — 그 파일은 PostgreSQL이 배포 때 25001로 거부하므로 검사기가 먼저 알려 주는 쪽이 맞다. 잠그지 않는 문장(plain VACUUM 등)은 이 검사기의 범위가 아니라 그대로 둔다. - ledger(RR7-03-01): IT에 RUN_OUTSIDE_A_TRANSACTION(파일 → 이유) 목록을 두고 잠금 대기 케이스에서 뺀다. 목록의 파일은 같은 IT가 동시 인덱스 문장 하나·executeInTransaction=false인지 ConcurrentIndexMigration Inspector로 본다. optional 빌드는 testkit에 의존할 수 없어 sharedMigrationFileSources가 그 파일과 MigrationResource를 복사한다 (VersionedMigrationFiles와 같은 방식). ledger IT 셋은 마이그레이션을 파일마다 따로 실행한다(Flyway처럼). 버린 안: 안내만 "다룰 수 없다"로 바꾸기 — 다음 동시 인덱스 파일이 생길 때 같은 질문이 다시 남는다. 위험·한계 - 검사기 변경으로 서비스의 자기 스트림 검사가 새로 실패할 수 있다. 걸리는 파일은 PostgreSQL이 배포 때 거부하는 파일뿐이라 배포된 적이 없다(업그레이드 안내 4.1에 적음). - runbook의 DELETE는 운영자가 version을 옮겨 적는다. NOT success라 성공한 행은 지우지 못하지만, 다른 버전의 실패 행을 잘못 적으면 그 행이 지워진다. 한 행만 지워야 하고 0행이면 다시 분류하라고 적었다. - ledger 복사 파일 두 개도 JDK만 써야 한다(각 Javadoc에 적음). - Spotless 검사는 같은 worktree에서 spotlessApply가 변경 없이 끝난 직후라 UP-TO-DATE로 처리됐다. 검증 (desktop, 공유 잠금, 로그 ~/r8-f3-logs/) RED — 기준 73d39212 + 이 커밋의 테스트(+ 기준 문서 주장을 단언하는 임시 메서드, + 임시 ledger V004): - persistence-jpa test --tests LockWaitMigrationInspectorTest: 80건 중 2건 실패, 검사기가 빈 목록(통과)을 돌려줌 (red-unit.log) - postgresqlMigrationLockWaitIntegrationTest: 47건 중 4건 실패 — 옛 주장 대로면 거부여야 할 REINDEX SCHEMA|DATABASE CONCURRENTLY가 applied, conf=true VACUUM FULL·REINDEX SCHEMA를 PostgreSQL은 25001로 거부하는데 검사기는 통과 (red2-lockwait.log) - postgresqlConcurrentIndexMigrationIntegrationTest: 9건 중 3건 실패 — 옛 절차(DROP INDEX) 뒤 재배포가 "Detected failed migration to version 2", conf를 붙인 섞인 파일이 행 3001·무효 인덱스·실패 이력을 남김, runbook 블록이 없어 복구 테스트 준비 실패(RED로 세지 않음) (red2-concurrent.log) - ledger integrationTest(V004 동시 인덱스): 6건 중 4건 실패 — 잠금 대기 IT "name it as an exception here", 다른 IT 둘 "cannot be executed within a pipeline" (red-ledger.log) GREEN — 이 커밋의 트리 b49369e7: - :adapter:outbound:persistence-jpa:check rc=0 (test 666건 실패 0), :testQualityCheck rc=0 - postgresqlMigrationLockWaitIntegrationTest rc=0 (42건 실패 0) - postgresqlConcurrentIndexMigrationIntegrationTest rc=0 (7건 실패 0) - -p optional-platforms :grpc:grpc-operation-ledger-jpa:check integrationTest testQualityCheck rc=0 (IT 3+2+42건 실패 0) - :app-bootstrap:documentContractTest rc=0 (66건 실패 0) - 두 빌드의 verifyStrictLaneStages --dry-run rc=0 - ledger 탐침(임시 V004): 목록에 없으면 잠금 대기 케이스만 실패하고 메시지가 RUN_OUTSIDE_A_TRANSACTION을 가리킴(48건 중 1건), 목록에 넣으면 rc=0, 목록에 넣고 SET LOCAL을 섞으면 목록 검사가 "mixes a concurrent index statement"로 거부 - python3 scripts/check-work-docs.py --links docs: 깨진 링크 0 Refs: docs/reviews/2026-11-03-rereview/round7/03-persistence.md --- docs/jpa/migration-guide.md | 46 ++++-- docs/jpa/runbooks.md | 6 +- docs/runbooks/migration-failed.md | 64 ++++++- docs/study/postgresql-jpa-walkthrough.ko.md | 49 ++++-- docs/upgrade/2026-11-maturity-review.md | 4 + .../outbound/persistence-jpa/build.gradle | 8 + .../readiness/OutboxRunbookSql.java | 5 +- ...ncurrentIndexMigrationIntegrationTest.java | 156 ++++++++++++++++++ ...reSqlMigrationLockWaitIntegrationTest.java | 70 +++++++- .../LockWaitMigrationInspectorTest.java | 23 +++ .../ConcurrentIndexMigrationInspector.java | 51 ++++-- .../migration/LockWaitMigrationInspector.java | 44 +++-- .../testkit/migration/MigrationResource.java | 3 + .../grpc-operation-ledger-jpa/build.gradle | 10 +- ...rpcOperationLedgerMigrationLockWaitIT.java | 85 +++++++++- .../ledger/GrpcOperationLedgerMigrations.java | 42 ++++- .../ledger/GrpcOperationLedgerPostgresIT.java | 8 +- .../JpaGrpcOperationLedgerContractIT.java | 8 +- 18 files changed, 587 insertions(+), 95 deletions(-) diff --git a/docs/jpa/migration-guide.md b/docs/jpa/migration-guide.md index e42b7ed1..9223d687 100644 --- a/docs/jpa/migration-guide.md +++ b/docs/jpa/migration-guide.md @@ -47,6 +47,10 @@ applied one, and the interesting question is which change is missing from this d makes that question unaskable. It exists only as an explicit admin operation with an operator, a reason, and an approval (design §8.4). +The one change to the history an operator makes is narrower than repair: after a concurrent index +migration fails, the failed row of that one version is deleted so that the version can run again +("Concurrent index builds" below). Every applied row, and its checksum, stays as it was. + Only Flyway's structured error codes reach the exception. Its messages embed the script path and part of the failing statement. @@ -84,7 +88,9 @@ Flyway's list: Flyway runs it in the transaction, PostgreSQL refuses it there, a `VACUUM FULL`, a non-concurrent `REINDEX SCHEMA|DATABASE|SYSTEM` — can never be bounded this way and does not belong in a migration: run it by hand as an operations step, outside a deploy, in a psql session that sets `SET lock_timeout = '5s';` first (the session ends with the step, so the plain -`SET` is the bound there). +`SET` is the bound there). A `.sql.conf` with `executeInTransaction=true` does not change that: it +keeps the statement and its `SET LOCAL` in the transaction only for PostgreSQL to refuse it at +deploy, and the inspector below refuses it as well. Enforced, not remembered: `LockWaitMigrationInspector` (persistence-jpa testkit) reads a script — and the `.sql.conf` beside it, as Flyway reads it (a properties file, `false` in any case), and @@ -112,18 +118,25 @@ optional build, which cannot depend on the testkit: its `GrpcOperationLedgerMigr runs every ledger migration after V001, read from the tree, against a held ledger instead, so its next migration is checked on real PostgreSQL. It finds and orders that tree with the testkit's own `VersionedMigrationFiles`, which the ledger's `build.gradle` copies into its integration-test sources -at build time (`sharedMigrationFileSources`), so both builds read one definition. The rule came from -V21 having the bound and V22, in the same commit, not having it (re-review 2026-11-03 round 2, +at build time (`sharedMigrationFileSources`), so both builds read one definition. A concurrent index +build in that tree runs outside a transaction and is no case for a held ledger: it goes in the +test's `RUN_OUTSIDE_A_TRANSACTION` with its reason, and the same test holds each file listed there to +`ConcurrentIndexMigrationInspector`, copied the same way ("Concurrent index builds"). Every ledger +lane applies the tree one file at a time, as Flyway does, so such a file is not refused inside the +implicit transaction of one string holding many. The rule came +from V21 having the bound and V22, in the same commit, not having it (re-review 2026-11-03 round 2, RR-02-09; round 3, RR3-02-01, RR3-02-03, RR3-02-04; round 4, RR4-02-01; round 6, RR6-03-02, -RR6-03-03). +RR6-03-03; round 7, RR7-03-01). ## Concurrent index builds `CREATE INDEX CONCURRENTLY`, `DROP INDEX CONCURRENTLY` and `REINDEX … CONCURRENTLY` cannot run -inside a transaction block. Flyway recognizes `CREATE`/`DROP INDEX CONCURRENTLY` and runs such a -script outside a transaction on its own (above), but not `REINDEX … CONCURRENTLY`, which it runs in -the transaction for PostgreSQL to refuse. Write the companion configuration for every concurrent -statement anyway, so the file says what it needs and no reader has to know Flyway's list: +inside a transaction block. Flyway runs a script outside a transaction on its own when it holds +`CREATE`/`DROP INDEX CONCURRENTLY` or `REINDEX SCHEMA|DATABASE`, with or without `CONCURRENTLY` +(above). It runs `REINDEX INDEX|TABLE … CONCURRENTLY` — `REINDEX (CONCURRENTLY) TABLE` included — in +the transaction, where PostgreSQL refuses it. `PostgreSqlMigrationLockWaitIntegrationTest` shows the +five forms on real PostgreSQL. Write the companion configuration for every concurrent statement +anyway, so the file says what it needs and no reader has to know Flyway's list: ```conf # V42__order_index.sql.conf @@ -144,9 +157,20 @@ open holds the build, and a file run outside a transaction cannot bound that wai Look for old transactions (`pg_stat_activity.xact_start`) before deploying one. The testkit's `ConcurrentIndexMigrationInspector` checks a script for this in a migration test -(nothing checks it at startup), and additionally requires the migration to contain nothing else. A failed concurrent build leaves an invalid index behind; -recovering is a single `DROP INDEX` when the migration did nothing else, and a manual reconstruction -of partial state when it did. +(nothing checks it at startup), and additionally requires the migration to contain nothing else. +Next to another statement, a concurrent one makes Flyway refuse the file before any of it runs when +nothing configures it; with `executeInTransaction=false` Flyway commits each statement on its own, +and a failed build leaves the statements before it applied — state to reconstruct by hand. + +Alone, a failed build leaves an invalid index and a history row with `success = false` for its +version, and Flyway refuses every later deploy ("Detected failed migration to version …") until that +row is gone: dropping the index is not enough. The recovery is +[migration-failed.md](../runbooks/migration-failed.md) "Recover a failed concurrent index migration": +confirm no build is running, drop the invalid index, remove the cause, delete that version's failed +row and nothing else, deploy again. It is not `flyway repair`, which would also rewrite the +remaining rows' checksums. `PostgreSqlConcurrentIndexMigrationIntegrationTest` +(`postgresqlConcurrentIndexMigrationIntegrationTest`) runs the runbook's SQL, read from the runbook, +after a unique build that failed on a duplicate, and shows both mixed files above. An invalid index is not merely useless — the planner ignores it while every write still maintains it. [runbooks.md](runbooks.md) has the query that finds them. Nothing drops them automatically: an diff --git a/docs/jpa/runbooks.md b/docs/jpa/runbooks.md index ad14f251..b77d3086 100644 --- a/docs/jpa/runbooks.md +++ b/docs/jpa/runbooks.md @@ -73,7 +73,11 @@ A concurrent build failed. The index is ignored by the planner and maintained by 1. Confirm no build is currently running. An in-progress build looks identical in the catalog. 2. Run `DROP INDEX CONCURRENTLY IF EXISTS ;` outside a migration. -3. Re-apply the index migration. +3. If a migration built it, Flyway recorded that version as failed (`success = false`) and refuses + every later deploy until that row is gone: dropping the index does not clear it. Follow + [migration-failed.md](../runbooks/migration-failed.md) "Recover a failed concurrent index + migration", which removes the cause and deletes that one row. +4. Re-apply the index migration. The platform does not drop these automatically: on a rolling deploy every instance would race to drop an index another instance was about to finish building. diff --git a/docs/runbooks/migration-failed.md b/docs/runbooks/migration-failed.md index a2988d75..1f40de1b 100644 --- a/docs/runbooks/migration-failed.md +++ b/docs/runbooks/migration-failed.md @@ -4,7 +4,7 @@ category: INTERNAL error_codes: [MIGRATION_FAILED] severity: P1 owner: oncall -last_updated: 2026-11-04 +last_updated: 2026-11-07 status: active --- @@ -63,6 +63,13 @@ status: active case — clean it up with `outbox-dead-letter.md` §8 and rerun, the migration stays as it is; - checksum mismatch: compare the deployed artifact with the already applied script before considering repair; + - a concurrent index migration failed (`CREATE [UNIQUE] INDEX CONCURRENTLY` or `REINDEX … + CONCURRENTLY` that Flyway ran outside a transaction): a duplicate value for a unique + index, a deadlock, `statement_timeout` or a cancelled build. Outside a transaction nothing + rolled back: the migration left an invalid index and a history row with + `success = false` for that version, and every later deploy fails validation ("Detected failed + migration to version …") until the row is gone. Dropping the index alone does not clear it. + Follow "Recover a failed concurrent index migration" below; - connection/TLS failure: fix transport or credentials without changing Flyway history. ## Action @@ -92,7 +99,9 @@ status: active Do not: - edit an already applied migration; -- delete or rewrite Flyway history to make validation green; +- delete or rewrite Flyway history to make validation green. The one exception is the failed row + of a failed concurrent index migration, deleted as "Recover a failed concurrent index migration" + below says — that row and nothing else; - run `flyway repair` before checksum provenance is proven and reviewed; - use `clean`, destructive rollback, or schema-wide restore as the first response; - mark a capability `ACTIVE` before its migration and adapter readiness succeed. @@ -101,6 +110,57 @@ If commit outcome was indeterminate during the failure, reconcile by the applica `OperationId`/idempotency reference before retrying business work. Never blind-retry a commit whose result is unknown. +## Recover a failed concurrent index migration + +The one case where this runbook changes Flyway history (owner decision, 2026-11-06). It applies to a +migration that holds a single concurrent index statement which Flyway ran outside a transaction — +with the `executeInTransaction=false` the guide asks for, or on its own for `CREATE`/`DROP INDEX +CONCURRENTLY` (`docs/jpa/migration-guide.md` "Concurrent index builds") — and whose row for this +version in the stream's history table says `success = false`. A migration that mixed other +statements in has already committed them: reconstruct that state with the database owner instead. + +1. Confirm no build is still running. An index that is still being built looks invalid in the + catalog too, and dropping it races the build. This query returns no rows when none is: + + +```sql +SELECT pid, relid::regclass AS table_name, index_relid::regclass AS index_name, phase +FROM pg_stat_progress_create_index +WHERE datname = current_database(); +``` + +2. Drop the invalid index the build left (`docs/jpa/runbooks.md` "An invalid index exists" lists + them). Run it on its own, not inside `BEGIN`: `CONCURRENTLY` refuses a transaction block. + + +```sql +DROP INDEX CONCURRENTLY IF EXISTS ; +``` + +3. Remove the cause. For a unique index, clean up the duplicate rows with the data owner (a forward + fix, never by editing the migration). For a deadlock or a timeout, end or wait out the + transaction that held the build. +4. Delete that version's failed row from the stream's history table, and nothing else. `` + is the `version` column of the failed row as the Diagnosis query shows it (`22.1` for + `V22_1__…`). It must delete exactly one row; if it deletes none, this is not the failure this + section is for, so classify it again. + + +```sql +DELETE FROM +WHERE version = '' AND NOT success; +``` + +5. Deploy again. Flyway runs the same migration from the start and builds the index. + +The failed row is the record of a version that never applied, so deleting it leaves the order and +the checksum of every applied version as they were. Do not use `flyway repair` for this: it also +rewrites the checksums and descriptions of the remaining rows to match the scripts on disk, which +removes the evidence a checksum mismatch carries. `:adapter:outbound:persistence-jpa:postgresqlConcurrentIndexMigrationIntegrationTest` +runs the three SQL blocks above, read from this file, against real PostgreSQL: a unique build fails +on a duplicate, steps 1–4 run, and the next deploy applies the version with the index valid and the +earlier version's checksum unchanged. + ## Escalation - P1 immediate: the required application revision cannot become ready. diff --git a/docs/study/postgresql-jpa-walkthrough.ko.md b/docs/study/postgresql-jpa-walkthrough.ko.md index 6e74af3d..6523ab89 100644 --- a/docs/study/postgresql-jpa-walkthrough.ko.md +++ b/docs/study/postgresql-jpa-walkthrough.ko.md @@ -1337,15 +1337,22 @@ startup 동작으로는 절대 아니다. ### 8.2 `ConcurrentIndexMigrationInspector` PostgreSQL은 `CREATE INDEX CONCURRENTLY`·`DROP INDEX CONCURRENTLY`·`REINDEX … CONCURRENTLY`를 -트랜잭션 블록 안에서 거부한다. Flyway 11은 `CREATE`/`DROP INDEX CONCURRENTLY`가 든 파일을 **스스로 -트랜잭션 밖에서** 돌린다. `REINDEX … CONCURRENTLY`는 모른다. 그 파일은 트랜잭션 안에서 돌고, 배포 때 -PostgreSQL이 거부한다. 그래서 검사기는 동시 인덱스 문장이 있으면 `.sql.conf`에 -`executeInTransaction=false`를 적으라고 요구한다. Flyway가 알아서 빼 주는 문장에도 적는다. 파일이 -자기에게 필요한 것을 스스로 말하게 하려는 것이다. +트랜잭션 블록 안에서 거부한다. Flyway 11은 `CREATE`/`DROP INDEX CONCURRENTLY`나 +`REINDEX SCHEMA|DATABASE`(`CONCURRENTLY`가 있든 없든)가 든 파일을 **스스로 트랜잭션 밖에서** 돌린다. +`REINDEX INDEX|TABLE … CONCURRENTLY`(`REINDEX (CONCURRENTLY) TABLE` 포함)는 모른다. 그 파일은 +트랜잭션 안에서 돌고, 배포 때 PostgreSQL이 거부한다. 다섯 형태를 실제 PostgreSQL에서 돌린 결과는 +`PostgreSqlMigrationLockWaitIntegrationTest`에 있다. 그래서 검사기는 동시 인덱스 문장이 있으면 +`.sql.conf`에 `executeInTransaction=false`를 적으라고 요구한다. Flyway가 알아서 빼 주는 문장에도 +적는다. 파일이 자기에게 필요한 것을 스스로 말하게 하려는 것이다. -> 다른 문장과 섞인 파일은 Flyway가 기본값(`mixed=false`)에서 **아무것도 실행하기 전에 거부**한다. -> 부분 적용은 일어나지 않는다. 그래도 한 파일에 하나만 두는 이유는 아래 `requireIsolatedStatement`의 -> 복구 때문이다. +> 다른 문장과 섞인 파일은 설정에 따라 결과가 다르다. +> +> - `.sql.conf`가 없으면 Flyway가 기본값(`mixed=false`)에서 **아무것도 실행하기 전에 거부**한다. +> - 검사기가 요구하는 `executeInTransaction=false`를 붙이면 Flyway가 **문장마다 따로 커밋**한다. +> 동시 빌드가 실패해도 그 앞 문장은 **이미 적용돼 남는다**(부분 적용). +> +> 그래서 동시 인덱스 문장은 한 파일에 하나만 둔다. 두 경우 모두 +> `PostgreSqlConcurrentIndexMigrationIntegrationTest`가 실제 PostgreSQL에서 보여 준다. 트랜잭션 밖에서도 동시 빌드는 **자기보다 먼저 시작한 트랜잭션이 모두 끝나기를 기다린다.** @@ -1366,18 +1373,32 @@ Flyway 기본값: 잠금 트랜잭션(열린 채) ──기다림──▶ 마 검사 대상: `CREATE [UNIQUE] INDEX CONCURRENTLY`, `DROP INDEX CONCURRENTLY`, `REINDEX … CONCURRENTLY` (`REINDEX (CONCURRENTLY) TABLE` 같은 괄호 옵션 포함). -`requireIsolatedStatement`도 있다 — concurrent 빌드가 실패하면 invalid index가 남고, 그 -복구는 마이그레이션이 다른 걸 아무것도 안 했으면 `DROP INDEX` 하나지만 뭔가 했으면 부분 -상태를 손으로 재구성해야 한다. +`requireIsolatedStatement`도 있다 — concurrent 빌드가 실패하면 invalid index와 그 버전의 +**실패 이력 행**(`success = false`)이 남는다. Flyway는 그 행이 있는 한 다음 배포를 validate에서 거부한다. +그래서 인덱스만 지우고 다시 배포하면 또 멈춘다. -`FailedConcurrentIndexRecovery`가 `pg_index.indisvalid = false`를 조회해서 리포트한다. -**자동으로 지우지 않는다.** +``` +실패한 동시 빌드 → invalid index + 실패 이력 행 + → 빌드가 아직 도는지 확인 → DROP INDEX CONCURRENTLY → 원인 정리(중복 행 등) + → 그 버전의 실패 이력 행만 DELETE → 다시 배포 +``` + +- 절차와 SQL은 `docs/runbooks/migration-failed.md` "Recover a failed concurrent index migration"에 있다. + `PostgreSqlConcurrentIndexMigrationIntegrationTest`가 그 SQL을 runbook에서 읽어 실제 PostgreSQL에서 돌린다. +- 실패 이력 행을 지우는 것은 "이력을 고치지 않는다"의 하나뿐인 예외다(소유자 결정 2026-11-06). + 실패 행은 적용되지 않은 버전의 기록이라, 지워도 성공한 버전의 순서와 checksum은 그대로다. +- `flyway repair`는 여전히 쓰지 않는다. 남은 행의 checksum·설명까지 디스크의 스크립트에 맞춰 다시 쓰기 때문이다. +- 마이그레이션이 다른 문장도 담고 있었으면 그 문장은 이미 커밋돼 있다. 부분 상태를 손으로 재구성해야 한다. + +invalid index를 찾는 쿼리는 `docs/jpa/runbooks.md` "An invalid index exists"에 있다(예전의 +`FailedConcurrentIndexRecovery` 클래스는 호출자가 없어 지웠고, 이 쿼리가 대신한다). +플랫폼은 invalid index를 **자동으로 지우지 않는다.** > invalid index는 **빌드가 아직 돌고 있다**는 뜻일 수도 있고, 카탈로그만으로는 둘을 구별할 수 > 없다. 자동 drop은 곧 끝나려던 concurrent 빌드와 경주하게 되고, rolling deploy의 모든 > 인스턴스에서 동시에 그렇게 한다. -리포트는 운영자가 돌릴 statement를 준다(그 자체가 non-transactional인 `DROP INDEX CONCURRENTLY`). +지우는 문장은 운영자가 직접 돌린다(그 자체가 트랜잭션 밖에서만 도는 `DROP INDEX CONCURRENTLY`). ### 8.3 실제 마이그레이션 트리 — 8개 독립 스트림 diff --git a/docs/upgrade/2026-11-maturity-review.md b/docs/upgrade/2026-11-maturity-review.md index 7a0f1509..73a3f76d 100644 --- a/docs/upgrade/2026-11-maturity-review.md +++ b/docs/upgrade/2026-11-maturity-review.md @@ -155,6 +155,9 @@ `GRANT … REFERENCES ON`과 `REINDEX (CONCURRENTLY) …`는 더 이상 거부하지 않는다. 자기 스트림 검사가 새로 실패하면 그 파일은 Flyway가 거부하거나(`mixed=false`) 상한 없이 도는(`mixed=true`) 파일이다. 아직 배포 전이면 문장을 나누고, 이미 배포했으면 규칙 이전 목록에 더한다(r9, 재리뷰 RR4-02-01). + 재리뷰 round7(RR7-03-05) 수정부터는 `.sql.conf`가 `executeInTransaction=true`를 정한 파일의 `VACUUM FULL`·비동시 + `REINDEX SCHEMA|DATABASE|SYSTEM`도 앞의 `SET LOCAL`과 상관없이 같은 이유로 거부한다. 그런 파일은 PostgreSQL이 배포 때 + `25001`로 거부하므로 배포된 적이 없다. 새로 걸리면 그 문장을 마이그레이션에서 빼 운영 절차로 옮긴다. - **번호 충돌**: 템플릿이 이제 postgresql V20~V22, notification V11~V13, sample V8_1, grpc V002·V003을 쓴다. 파생 서비스가 같은 위치에 같은 번호를 두었다면 Flyway가 같은 버전 두 개로 실패한다. 자기 마이그레이션은 템플릿과 떨어진 번호대로 옮긴다 (이미 적용된 파일 이름을 바꾸면 checksum·history가 어긋나므로 운영 DB에 적용된 것이면 DB 담당과 정한다). @@ -245,6 +248,7 @@ V21 실패의 분류는 `docs/runbooks/migration-failed.md` "Classify the failur | 서버 로그(GraphQL·gRPC·STOMP) | carrier로 답한 실패를 web과 같은 줄(shared-contract `ServerFailureLog`)로 ERROR에 남긴다: persistence 실패는 분류만, 5xx carrier는 예외와 함께, 4xx carrier는 남기지 않는다. gRPC는 ` (method=…)`, GraphQL은 ` (executionId=…)`를 붙인다. gRPC는 인식 못 한 실패(`INTERNAL_ERROR`)도 `unhandled failure on …`로 남긴다. 클라이언트가 취소한 호출은 남기지 않고, 서비스가 `CANCELLED` status로 끝낸 살아 있는 호출은 남긴다. raw status는 넘겼든 던졌든 설명을 남기지 않는다(원인 예외가 있으면 그것만 붙인다). `IllegalArgumentException`(`BAD_PARAMETER`)은 web과 같은 WARN 한 줄 `request refused as BAD_PARAMETER by <예외 타입>`(메시지 없음)으로 남긴다. STOMP `@MessageMapping` 실패는 Spring의 `Unhandled exception from message handler method`(예외 전체) 대신 같은 줄로 남는다 | ERROR 로그가 늘어난다(전에는 gRPC가 아무것도 남기지 않았다). GraphQL·gRPC·STOMP는 잘못된 입력마다 WARN 한 줄이 생긴다. STOMP에서 Spring 문구로 경보를 걸었다면 고친다. `IDEMPOTENCY_RECOVERY_REQUIRED`의 scope digest를 모든 전송에서 찾을 수 있다(`docs/runbooks/idempotency-unavailable.md`) | r9 · `src/shared-contract/.../error/ServerFailureLog.java`, `GrpcExceptionHandlingInterceptor.java`, `GraphQlWireErrorMapper.java`, `StompHandlerFailureLog.java` | | sample-portfolio | 익명 쓰기 401, 포스터 제목 대소문자 무시 유일(409 `POSTER_ALREADY_EXISTS`), 관리자도 남의 work log 수정 불가, 샘플 이벤트 tenant `global` | sample을 남긴 서비스만 | `34565130`, `1400dc15` | | Flyway 잠금 | Flyway의 PostgreSQL 마이그레이션 잠금을 트랜잭션 대신 세션에 둔다. Spring Boot의 Flyway는 module-defaults `spring.flyway.postgresql.transactional-lock: false`, 템플릿이 직접 만드는 Flyway(멱등성·알림·sample stream, schema-per-tenant, testkit `MigrationContractRunner`)는 `FlywaySchemaPolicy.configure()`, messaging inbox stream은 같은 값을 직접 넣는다. 운영 코드의 `Flyway.configure()` 직접 호출은 `JpaProductionArchitectureTest`가 거부한다 | Flyway 기본값에서는 동시 인덱스 마이그레이션이 Flyway 자신의 열린 트랜잭션을 기다려 끝나지 않았다. 이제 끝난다. 마이그레이션은 잠금을 잡은 연결 하나로 돈다. transaction 모드 PgBouncer 뒤에서는 잠금 해제가 다른 서버 연결로 가 다음 배포가 기다릴 수 있으니 마이그레이션은 직접 연결이나 session 모드 pool로 돌린다. 배포 도구는 이 repo 밖이라 2절 15번으로 맞춘다 | 재리뷰 round6 RR6-03-01 · `persistence-jpa.yml`, `.../migration/FlywaySchemaPolicy.java`, `docs/jpa/migration-guide.md` "Who may change the schema"·"Concurrent index builds" | +| 실패한 동시 인덱스 마이그레이션 복구 | `docs/runbooks/migration-failed.md`에 「Recover a failed concurrent index migration」을 더했다: 진행 중인 빌드가 없는지 보고, 무효 인덱스를 `DROP INDEX CONCURRENTLY`로 지우고, 원인을 정리하고, 그 버전의 실패 이력 행(`success = false`)만 지운 뒤 다시 배포한다. 「Do not」의 이력 수정 금지에 이 한 가지 예외를 적었다. `flyway repair`는 계속 쓰지 않는다 | 전에는 안내가 "`DROP INDEX` 하나"였고, 그대로 하면 남은 실패 이력 때문에 다음 배포가 validate에서 다시 멈췄다. 동시 인덱스 마이그레이션을 쓰는 서비스의 운영자는 이 절차로 복구한다 | 재리뷰 round7 RR7-03-03, 소유자 결정 2026-11-06 · `docs/jpa/migration-guide.md` "Concurrent index builds", `PostgreSqlConcurrentIndexMigrationIntegrationTest` | | 성능 lane(HTTP client·Mongo) | httpclient `LargeBodyResourceTest`의 힙 상한(32 MiB 스트리밍 중 힙 증가 16 MiB 미만)을 매번 단언한다. 전에는 `-Pperformance.assertions.enabled=true`일 때만 단언하고 아니면 건너뛴다는 줄만 찍었다. Mongo `MongoResourceBudgetLaneTest`는 표준 gate 전체(p99 500ms, pool 대기 250ms, 힙 512 MiB, examined/returned 10, spill 금지)를 Gradle 밖 실행에서도 단언한다(B-43) | 손으로 돌린 `performanceCheck`가 상한을 실제로 재므로, 전에는 통과하던 느린 기계에서 실패할 수 있다 | B4(2026-11-06) · `LargeBodyResourceTest`, `MongoResourceBudgetLaneTest` | | 보안 필터 로그 레벨 | 보안 필터(`AuthErrorResponseWriter`)의 `security failure` 줄을 5xx(`AUTH_JWKS_UNAVAILABLE` 503, `INTERNAL_AUTH_MISCONFIGURATION` 500)면 ERROR, 4xx면 WARN으로 남긴다. 전에는 모두 WARN이었다. 다른 경로가 5xx를 ERROR로 남기는 규칙(`ServerFailureLog`)과 같아졌다 | ERROR로 거르는 로그 경보·대시보드에 JWKS 장애와 인증 설정 오류가 새로 잡힌다. 401·403은 그대로 WARN | 설계 메모 2026-11-06 7절 결정 2 · `src/adapter/inbound/web/.../auth/AuthErrorResponseWriter.java`, `docs/superpowers/specs/2026-11-06-error-code-log-level-design.md` | diff --git a/src/adapter/outbound/persistence-jpa/build.gradle b/src/adapter/outbound/persistence-jpa/build.gradle index 2061a83f..5d5abe27 100644 --- a/src/adapter/outbound/persistence-jpa/build.gradle +++ b/src/adapter/outbound/persistence-jpa/build.gradle @@ -265,6 +265,14 @@ strictTestLanes { } } +// The recovery of a failed concurrent index migration (re-review 2026-11-03 round 7, RR7-03-03) is +// SQL the migration runbook gives an operator; the lane reads it out of the runbook the same way. +tasks.named('postgresqlConcurrentIndexMigrationIntegrationTest', Test) { + def migrationFailedRunbook = new File(rootProject.projectDir.parentFile, 'docs/runbooks/migration-failed.md') + inputs.file(migrationFailedRunbook).withPathSensitivity(PathSensitivity.RELATIVE).withPropertyName('migrationFailedRunbook') + systemProperty 'migration.failed.runbook', migrationFailedRunbook.absolutePath +} + tasks.named('postgresqlSecurityBaselineIntegrationTest') { // Cross-leaf task edge: see the handoff. verifyCleanArchitectureDependencies inspects // configurations, not the task graph, so this edge is invisible to it. diff --git a/src/adapter/outbound/persistence-jpa/src/postgresqlIntegrationTest/java/dev/caskeleton/adapter/outbound/persistence/readiness/OutboxRunbookSql.java b/src/adapter/outbound/persistence-jpa/src/postgresqlIntegrationTest/java/dev/caskeleton/adapter/outbound/persistence/readiness/OutboxRunbookSql.java index 395ffcc9..0d2556dc 100644 --- a/src/adapter/outbound/persistence-jpa/src/postgresqlIntegrationTest/java/dev/caskeleton/adapter/outbound/persistence/readiness/OutboxRunbookSql.java +++ b/src/adapter/outbound/persistence-jpa/src/postgresqlIntegrationTest/java/dev/caskeleton/adapter/outbound/persistence/readiness/OutboxRunbookSql.java @@ -7,7 +7,8 @@ import java.nio.file.Path; import java.util.Map; /** - * The SQL an outbox runbook gives an operator, read out of the runbook itself. + * The SQL a runbook gives an operator, read out of the runbook itself: the outbox runbooks, and the + * migration runbook's recovery of a failed concurrent index migration. * *

A runbook block is marked {@code } and followed by a {@code sql} * fence. The lanes that run these blocks against real PostgreSQL read them from the document rather @@ -48,7 +49,7 @@ final class OutboxRunbookSql { String marker = ""; int at = runbook.indexOf(marker); if (at < 0) { - throw new IllegalStateException("the outbox runbook has no SQL block marked " + id); + throw new IllegalStateException("the runbook has no SQL block marked " + id); } int open = runbook.indexOf("```sql\n", at); int close = runbook.indexOf("\n```", open + 1); diff --git a/src/adapter/outbound/persistence-jpa/src/postgresqlIntegrationTest/java/dev/caskeleton/adapter/outbound/persistence/readiness/PostgreSqlConcurrentIndexMigrationIntegrationTest.java b/src/adapter/outbound/persistence-jpa/src/postgresqlIntegrationTest/java/dev/caskeleton/adapter/outbound/persistence/readiness/PostgreSqlConcurrentIndexMigrationIntegrationTest.java index ef691bd0..33b1a78f 100644 --- a/src/adapter/outbound/persistence-jpa/src/postgresqlIntegrationTest/java/dev/caskeleton/adapter/outbound/persistence/readiness/PostgreSqlConcurrentIndexMigrationIntegrationTest.java +++ b/src/adapter/outbound/persistence-jpa/src/postgresqlIntegrationTest/java/dev/caskeleton/adapter/outbound/persistence/readiness/PostgreSqlConcurrentIndexMigrationIntegrationTest.java @@ -14,6 +14,7 @@ import java.sql.SQLException; import java.time.Duration; import java.util.List; import java.util.Locale; +import java.util.Map; import java.util.UUID; import java.util.concurrent.ExecutionException; import java.util.concurrent.ExecutorService; @@ -24,6 +25,8 @@ import java.util.concurrent.TimeoutException; import javax.sql.DataSource; import org.junit.jupiter.api.AfterAll; import org.junit.jupiter.api.BeforeAll; +import org.junit.jupiter.api.DisplayName; +import org.junit.jupiter.api.Test; import org.junit.jupiter.api.io.TempDir; import org.junit.jupiter.params.ParameterizedTest; import org.junit.jupiter.params.provider.ValueSource; @@ -53,6 +56,14 @@ import org.springframework.jdbc.datasource.DelegatingDataSource; * way every Flyway this template constructs itself does ({@link FlywaySchemaPolicy#configure()}). A * deploy still waiting after {@link #FINISHES_WITHIN} is cancelled, which leaves the invalid index * an interrupted build leaves in production too. + * + *

Three more cases pin what happens when such a deploy fails (re-review 2026-11-03 round 7, + * RR7-03-03 and RR7-03-04). A concurrent build next to another statement is refused before any of + * it runs when nothing configures the file, and is left half applied when {@code + * executeInTransaction=false} does: Flyway commits each statement on its own. A failed build leaves + * an invalid index and a failed history row, and the deploy after it is refused until the row is + * gone: the recovery case runs the SQL of {@code docs/runbooks/migration-failed.md} ("A concurrent + * index migration failed"), read out of the runbook itself, and deploys again. */ class PostgreSqlConcurrentIndexMigrationIntegrationTest { @@ -64,6 +75,23 @@ class PostgreSqlConcurrentIndexMigrationIntegrationTest { INSERT INTO probe SELECT g, 'note ' || g FROM generate_series(1, 1000) AS g; """; + /** A second row with row 1's note, so that a unique index on the note cannot be built. */ + private static final String DUPLICATE_NOTE = + "INSERT INTO probe (id, note) VALUES (2001, 'note 1');\n"; + + private static final String UNIQUE_NOTE_INDEX = + "CREATE UNIQUE INDEX CONCURRENTLY ix_probe_note_u ON probe (note)"; + + /** A concurrent build after another statement, the file the guide says not to write. */ + private static final String MIXED = + "INSERT INTO probe (id, note) VALUES (3001, 'partial');\n" + UNIQUE_NOTE_INDEX + ";\n"; + + private static final String FLYWAY_REFUSES_A_MIXED_MIGRATION = + "Detected both transactional and non-transactional statements"; + + /** The system property naming {@code docs/runbooks/migration-failed.md} for this lane. */ + private static final String MIGRATION_FAILED_RUNBOOK = "migration.failed.runbook"; + private static PostgreSqlReadinessSupport postgres; @BeforeAll @@ -126,6 +154,103 @@ class PostgreSqlConcurrentIndexMigrationIntegrationTest { } } + @Test + @DisplayName("설정 파일 없이 동시 인덱스 문장과 다른 문장을 섞은 마이그레이션은 Flyway가 아무것도 실행하기 전에 거부한다") + void aMixedMigrationWithoutConfigurationIsRefusedBeforeAnyOfItRuns(@TempDir Path stream) + throws Exception { + try (HikariDataSource database = postgres.createDatabase(databaseName())) { + // given + Files.writeString(stream.resolve("V1__probe.sql"), FIRST_RELEASE + DUPLICATE_NOTE); + migrateTheWayTheTemplateDoes(database, stream); + Files.writeString(stream.resolve("V2__mixed.sql"), MIXED); + + // when + Throwable thrown = catchThrowable(() -> migrateTheWayTheTemplateDoes(database, stream)); + + // then + assertThat( + List.of( + String.valueOf(thrown).contains(FLYWAY_REFUSES_A_MIXED_MIGRATION), + aftermath(database))) + .containsExactly( + true, "row 3001 0, indexes [ix_probe_id true, probe_pkey true], history [1 true]"); + } + } + + @Test + @DisplayName( + "executeInTransaction=false를 붙여 섞은 마이그레이션은 앞 문장을 커밋한 채 동시 빌드에서 실패해 부분 적용, 무효 인덱스, 실패 이력을 남긴다") + void aMixedMigrationMarkedOutsideATransactionIsLeftHalfApplied(@TempDir Path stream) + throws Exception { + try (HikariDataSource database = postgres.createDatabase(databaseName())) { + // given + Files.writeString(stream.resolve("V1__probe.sql"), FIRST_RELEASE + DUPLICATE_NOTE); + migrateTheWayTheTemplateDoes(database, stream); + Files.writeString(stream.resolve("V2__mixed.sql"), MIXED); + Files.writeString(stream.resolve("V2__mixed.sql.conf"), "executeInTransaction=false\n"); + + // when + Throwable thrown = catchThrowable(() -> migrateTheWayTheTemplateDoes(database, stream)); + + // then + assertThat(List.of(thrown != null, aftermath(database))) + .containsExactly( + true, + "row 3001 1, indexes [ix_probe_id true, ix_probe_note_u false, probe_pkey true]," + + " history [1 true, 2 false]"); + } + } + + @Test + @DisplayName( + "실패한 동시 인덱스 마이그레이션은 runbook 절차(빌드 확인, 무효 인덱스 삭제, 원인 정리, 그 버전의 실패 이력 행 삭제) 뒤 다시 배포하면 적용된다") + void aFailedConcurrentIndexMigrationIsRecoveredByTheRunbookProcedure(@TempDir Path stream) + throws Exception { + OutboxRunbookSql runbook = OutboxRunbookSql.fromSystemProperty(MIGRATION_FAILED_RUNBOOK); + try (HikariDataSource database = postgres.createDatabase(databaseName())) { + // given + Files.writeString(stream.resolve("V1__probe.sql"), FIRST_RELEASE + DUPLICATE_NOTE); + migrateTheWayTheTemplateDoes(database, stream); + String checksumOfV1 = checksumOf(database, "1"); + writeConcurrentMigration(stream, UNIQUE_NOTE_INDEX); + Throwable failed = catchThrowable(() -> migrateTheWayTheTemplateDoes(database, stream)); + String afterFailure = aftermath(database); + JdbcTemplate jdbc = new JdbcTemplate(database); + + // when + List> buildsRunning = + jdbc.queryForList(runbook.block("failed-concurrent-index-builds")); + jdbc.execute( + runbook.block( + "failed-concurrent-index-drop", Map.of("invalid_index", "ix_probe_note_u"))); + jdbc.execute("DELETE FROM probe WHERE id = 2001"); + jdbc.execute( + runbook.block( + "failed-concurrent-index-history-row", + Map.of("owned_history_table", "flyway_schema_history", "version", "2"))); + Throwable again = catchThrowable(() -> migrateTheWayTheTemplateDoes(database, stream)); + + // then + assertThat( + List.of( + failed != null, + afterFailure, + buildsRunning.size(), + String.valueOf(again), + aftermath(database), + checksumOf(database, "1"))) + .containsExactly( + true, + "row 3001 0, indexes [ix_probe_id true, ix_probe_note_u false, probe_pkey true]," + + " history [1 true, 2 false]", + 0, + "null", + "row 3001 0, indexes [ix_probe_id true, ix_probe_note_u true, probe_pkey true]," + + " history [1 true, 2 true]", + checksumOfV1); + } + } + /** V2: the one concurrent statement, with the companion configuration the guide asks for. */ private static void writeConcurrentMigration(Path stream, String statement) throws IOException { Files.writeString(stream.resolve("V2__concurrent_index.sql"), statement + ";\n"); @@ -222,6 +347,37 @@ class PostgreSqlConcurrentIndexMigrationIntegrationTest { return "version " + version + ", indexes valid " + valid; } + /** + * What a deploy left behind: whether the row the mixed V2 inserts first is there, every index on + * the table with whether it is valid, and every history row with whether it succeeded. + */ + private static String aftermath(DataSource database) { + JdbcTemplate jdbc = new JdbcTemplate(database); + Integer row = jdbc.queryForObject("SELECT count(*) FROM probe WHERE id = 3001", Integer.class); + List indexes = + jdbc.queryForList( + "SELECT c.relname || ' ' || i.indisvalid FROM pg_index i" + + " JOIN pg_class c ON c.oid = i.indexrelid" + + " WHERE i.indrelid = 'probe'::regclass ORDER BY c.relname COLLATE \"C\"", + String.class); + List history = + jdbc.queryForList( + "SELECT coalesce(version, '-') || ' ' || success FROM flyway_schema_history" + + " ORDER BY installed_rank", + String.class); + return "row 3001 " + row + ", indexes " + indexes + ", history " + history; + } + + /** The checksum the history holds for a successful version. */ + private static String checksumOf(DataSource database, String version) { + return String.valueOf( + new JdbcTemplate(database) + .queryForObject( + "SELECT checksum FROM flyway_schema_history WHERE version = ? AND success", + Integer.class, + version)); + } + private static String databaseName() { return "concurrent_" + UUID.randomUUID().toString().replace("-", "").toLowerCase(Locale.ROOT); } diff --git a/src/adapter/outbound/persistence-jpa/src/postgresqlIntegrationTest/java/dev/caskeleton/adapter/outbound/persistence/readiness/PostgreSqlMigrationLockWaitIntegrationTest.java b/src/adapter/outbound/persistence-jpa/src/postgresqlIntegrationTest/java/dev/caskeleton/adapter/outbound/persistence/readiness/PostgreSqlMigrationLockWaitIntegrationTest.java index c0a730d7..2f3b0e53 100644 --- a/src/adapter/outbound/persistence-jpa/src/postgresqlIntegrationTest/java/dev/caskeleton/adapter/outbound/persistence/readiness/PostgreSqlMigrationLockWaitIntegrationTest.java +++ b/src/adapter/outbound/persistence-jpa/src/postgresqlIntegrationTest/java/dev/caskeleton/adapter/outbound/persistence/readiness/PostgreSqlMigrationLockWaitIntegrationTest.java @@ -53,6 +53,13 @@ import org.springframework.jdbc.core.JdbcTemplate; * LockWaitMigrationInspectorTest}, which pins Flyway's own patterns in the pull request's {@code * ci} (round 6, RR6-03-03). * + *

Two more cases pin what the documents say about the edges of that list (round 7, RR7-03-02 and + * RR7-03-05): Flyway runs {@code REINDEX INDEX|TABLE ... CONCURRENTLY} in the transaction, where + * PostgreSQL refuses it, but {@code REINDEX SCHEMA|DATABASE ... CONCURRENTLY} outside one on its + * own; and a {@code VACUUM FULL} or {@code REINDEX SCHEMA} that a {@code .sql.conf} keeps in the + * transaction ({@code executeInTransaction=true}) is refused by PostgreSQL at deploy, and by the + * inspector before it. + * *

The migrations run under the template's Flyway settings ({@code FlywaySchemaPolicy}), whose * PostgreSQL lock is held for the session rather than in an open transaction (RR6-03-01): the bound * is shown under the lock the deploy actually takes. @@ -231,15 +238,26 @@ class PostgreSqlMigrationLockWaitIntegrationTest { } } - @Test + @ParameterizedTest + @CsvSource( + delimiter = '|', + value = { + "REINDEX TABLE CONCURRENTLY probe | refused 25001", + "REINDEX INDEX CONCURRENTLY probe_pkey | refused 25001", + "REINDEX (CONCURRENTLY) TABLE probe | refused 25001", + "REINDEX SCHEMA CONCURRENTLY public | applied", + "REINDEX DATABASE CONCURRENTLY | applied" + }) @DisplayName( - "Flyway는 REINDEX TABLE CONCURRENTLY를 트랜잭션 안에서 돌려 PostgreSQL이 거부하므로 executeInTransaction=false가 필요하다") - void flywayRunsReindexConcurrentlyInATransactionWherePostgresqlRefusesIt(@TempDir Path stream) - throws Exception { - try (HikariDataSource database = postgres.createDatabase(databaseName())) { + "Flyway는 REINDEX INDEX·TABLE … CONCURRENTLY를 트랜잭션 안에서 돌려 PostgreSQL이 거부하고, REINDEX SCHEMA·DATABASE … CONCURRENTLY는 스스로 트랜잭션 밖에서 돌린다") + void flywayRunsOnlyReindexIndexOrTableConcurrentlyInATransactionWherePostgresqlRefusesIt( + String statement, String expected, @TempDir Path stream) throws Exception { + String name = databaseName(); + try (HikariDataSource database = postgres.createDatabase(name)) { // given new JdbcTemplate(database).execute("CREATE TABLE probe (id INT PRIMARY KEY)"); - Files.writeString(stream.resolve("V1__reindex.sql"), "REINDEX TABLE CONCURRENTLY probe;\n"); + Files.writeString( + stream.resolve("V1__reindex.sql"), statement.replace("", name) + ";\n"); Flyway flyway = FlywaySchemaPolicy.configure() .dataSource(database) @@ -250,9 +268,47 @@ class PostgreSqlMigrationLockWaitIntegrationTest { // when Throwable thrown = catchThrowable(flyway::migrate); + String outcome = + thrown == null ? "applied" : "refused " + new Attempt(thrown, Duration.ZERO).sqlState(); // then - assertThat(new Attempt(thrown, Duration.ZERO).sqlState()).isEqualTo(ACTIVE_SQL_TRANSACTION); + assertThat(outcome).as("%s (%s)", statement, thrown).isEqualTo(expected); + } + } + + @ParameterizedTest + @ValueSource(strings = {"VACUUM FULL probe", "REINDEX SCHEMA public"}) + @DisplayName( + "executeInTransaction=true로 트랜잭션 안에 둔 VACUUM FULL·REINDEX SCHEMA는 PostgreSQL이 배포 때 거부하고 검사기도 거부한다") + void aLockingStatementAConfigurationKeepsInATransactionIsRefusedByPostgresqlAndTheInspector( + String statement, @TempDir Path stream) throws Exception { + try (HikariDataSource database = postgres.createDatabase(databaseName())) { + // given + new JdbcTemplate(database).execute("CREATE TABLE probe (id INT PRIMARY KEY)"); + Path script = + Files.writeString( + stream.resolve("V1__probe.sql"), + "SET LOCAL lock_timeout = '5s';\n" + statement + ";\n"); + Files.writeString(stream.resolve("V1__probe.sql.conf"), "executeInTransaction=true\n"); + Flyway flyway = + FlywaySchemaPolicy.configure() + .dataSource(database) + .locations("filesystem:" + stream.toAbsolutePath()) + .baselineOnMigrate(true) + .baselineVersion("0") + .load(); + + // when + Throwable thrown = catchThrowable(flyway::migrate); + List named = new LockWaitMigrationInspector().unboundedLockingStatements(script); + + // then + assertThat( + List.of( + String.valueOf(new Attempt(thrown, Duration.ZERO).sqlState()), + named.toString().contains("cannot run in a transaction"))) + .as("%s: Flyway %s, inspector %s", statement, thrown, named) + .containsExactly(ACTIVE_SQL_TRANSACTION, true); } } diff --git a/src/adapter/outbound/persistence-jpa/src/test/java/dev/caskeleton/adapter/outbound/persistence/testkit/migration/LockWaitMigrationInspectorTest.java b/src/adapter/outbound/persistence-jpa/src/test/java/dev/caskeleton/adapter/outbound/persistence/testkit/migration/LockWaitMigrationInspectorTest.java index 4cc49c68..8f2cf4ad 100644 --- a/src/adapter/outbound/persistence-jpa/src/test/java/dev/caskeleton/adapter/outbound/persistence/testkit/migration/LockWaitMigrationInspectorTest.java +++ b/src/adapter/outbound/persistence-jpa/src/test/java/dev/caskeleton/adapter/outbound/persistence/testkit/migration/LockWaitMigrationInspectorTest.java @@ -381,6 +381,29 @@ class LockWaitMigrationInspectorTest { assertThat(unbounded).isEmpty(); } + @ParameterizedTest + @ValueSource( + strings = { + "SET LOCAL lock_timeout = '5s';\nVACUUM FULL outbox_event;", + "SET LOCAL lock_timeout = '5s';\nREINDEX SCHEMA public;" + }) + @DisplayName("executeInTransaction=true 설정이 있어도 트랜잭션 안에서 돌 수 없는 잠금 문장은 마이그레이션에 두지 말라는 이유로 거부된다") + void aLockingStatementThatCannotRunInATransactionIsRefusedWhateverTheConfigurationSays( + String sql, @TempDir Path stream) throws IOException { + // given + Path script = Files.writeString(stream.resolve("V99__change.sql"), sql); + Files.writeString(stream.resolve("V99__change.sql.conf"), "executeInTransaction=true\n"); + + // when + List unbounded = inspector.unboundedLockingStatements(script); + + // then + assertThat(unbounded) + .singleElement(STRING) + .contains("cannot run in a transaction") + .contains("executeInTransaction=true"); + } + /** Every static {@link Pattern} the class declares, by field name. */ private static Map staticPatterns(Class type) throws IllegalAccessException { Map patterns = new TreeMap<>(); diff --git a/src/adapter/outbound/persistence-jpa/src/testFixtures/java/dev/caskeleton/adapter/outbound/persistence/testkit/migration/ConcurrentIndexMigrationInspector.java b/src/adapter/outbound/persistence-jpa/src/testFixtures/java/dev/caskeleton/adapter/outbound/persistence/testkit/migration/ConcurrentIndexMigrationInspector.java index b5615b69..7905f8b9 100644 --- a/src/adapter/outbound/persistence-jpa/src/testFixtures/java/dev/caskeleton/adapter/outbound/persistence/testkit/migration/ConcurrentIndexMigrationInspector.java +++ b/src/adapter/outbound/persistence-jpa/src/testFixtures/java/dev/caskeleton/adapter/outbound/persistence/testkit/migration/ConcurrentIndexMigrationInspector.java @@ -10,19 +10,35 @@ import java.util.regex.Pattern; * *

PostgreSQL refuses {@code CREATE INDEX CONCURRENTLY}, {@code DROP INDEX CONCURRENTLY} and * {@code REINDEX ... CONCURRENTLY} inside a transaction block. Flyway 11 runs a script outside its - * transaction on its own when it holds {@code CREATE} or {@code DROP INDEX CONCURRENTLY}, but not - * {@code REINDEX ... CONCURRENTLY}: that one Flyway runs in the transaction, and PostgreSQL refuses - * it at deploy. So {@link #validate} wants {@code executeInTransaction=false} beside every - * concurrent statement, whether Flyway would detect it or not. Next to any other statement, one - * Flyway runs outside a transaction makes Flyway refuse the whole script before it runs any of it - * (its default {@code mixed=false}); {@link #requireIsolatedStatement} wants the statement alone so - * that a failed build is recovered by dropping one invalid index. + * transaction on its own when it holds {@code CREATE} or {@code DROP INDEX CONCURRENTLY}, or {@code + * REINDEX SCHEMA|DATABASE} with or without {@code CONCURRENTLY}. It runs {@code REINDEX INDEX|TABLE + * ... CONCURRENTLY} — the option-list form {@code REINDEX (CONCURRENTLY) TABLE} included — in the + * transaction, and PostgreSQL refuses it at deploy ({@code + * postgresqlMigrationLockWaitIntegrationTest} shows the five forms; re-review 2026-11-03 round 7, + * RR7-03-02). So {@link #validate} wants {@code executeInTransaction=false} beside every concurrent + * statement, whether Flyway would detect it or not. + * + *

{@link #requireIsolatedStatement} wants the statement alone in its migration. Next to another + * statement it is refused by Flyway before any of the script runs when nothing configures the file + * (Flyway's default {@code mixed=false}); with the {@code executeInTransaction=false} that {@link + * #validate} asks for, Flyway runs and commits each statement on its own, and a failed build leaves + * the statements before it applied (round 7, RR7-03-04). Alone, a failed build leaves an invalid + * index and its version's failed history row, which Flyway refuses the next deploy over until the + * row is gone: {@code docs/runbooks/migration-failed.md} ("Recover a failed concurrent index + * migration") drops the index and deletes that one row ({@code + * postgresqlConcurrentIndexMigrationIntegrationTest} runs it; round 7, RR7-03-03). * *

Outside a transaction, {@code CREATE INDEX CONCURRENTLY} and {@code REINDEX ... CONCURRENTLY} * still wait for every transaction older than themselves. They finish because the template's Flyway * holds its PostgreSQL lock for the session ({@code FlywaySchemaPolicy.configure()}): under * Flyway's default the lock's own open transaction is one of those, and the deploy never ends * (re-review 2026-11-03 round 6, RR6-03-01 and RR6-03-05). + * + *

The gRPC ledger's lane uses it too. It lives in the optional build, which cannot depend on + * this testkit, so {@code grpc-operation-ledger-jpa} compiles this file and {@link + * MigrationResource} from here into its integration-test sources at build time (its {@code + * build.gradle}, {@code sharedMigrationFileSources}; round 7, RR7-03-01). Hence the rule for both + * files: the JDK only, nothing else from this testkit. */ public final class ConcurrentIndexMigrationInspector { @@ -53,8 +69,8 @@ public final class ConcurrentIndexMigrationInspector { throw new IllegalStateException( migration.name() + " must set executeInTransaction=false: PostgreSQL refuses a concurrent index" - + " statement inside a transaction block, and Flyway runs REINDEX ... CONCURRENTLY in" - + " one unless the migration says otherwise"); + + " statement inside a transaction block, and Flyway runs REINDEX INDEX|TABLE ..." + + " CONCURRENTLY in one unless the migration says otherwise"); } /** Whether the script contains a statement PostgreSQL forbids inside a transaction. */ @@ -68,9 +84,12 @@ public final class ConcurrentIndexMigrationInspector { /** * Fails when a concurrent-index migration also contains other statements. * - *

A concurrent index build can fail and leave an invalid index behind. Recovering from that is - * a single {@code DROP INDEX} when the migration did nothing else, and a manual reconstruction of - * partial state when it did. + *

A concurrent index build can fail and leave an invalid index and its version's failed + * history row behind. When the migration did nothing else, recovering is the runbook's few + * statements: drop the index, delete that one failed row, deploy again ({@code + * docs/runbooks/migration-failed.md}, "Recover a failed concurrent index migration"). When it did + * more, Flyway ran the file outside a transaction and has committed the statements before the + * build, and that partial state has to be reconstructed by hand. */ public void requireIsolatedStatement(MigrationResource migration) { Objects.requireNonNull(migration, "migration"); @@ -88,9 +107,11 @@ public final class ConcurrentIndexMigrationInspector { if (statements > 1L) { throw new IllegalStateException( migration.name() - + " mixes a concurrent index statement with other statements; a failed concurrent" - + " build must be recoverable by dropping one invalid index, which requires the" - + " migration to contain nothing else"); + + " mixes a concurrent index statement with other statements; outside a" + + " transaction Flyway commits each of them on its own, so a failed build leaves the" + + " others applied. Alone, a failed build is recovered by dropping its invalid index" + + " and deleting its version's failed history row (docs/runbooks/migration-failed.md," + + " \"Recover a failed concurrent index migration\")"); } } } diff --git a/src/adapter/outbound/persistence-jpa/src/testFixtures/java/dev/caskeleton/adapter/outbound/persistence/testkit/migration/LockWaitMigrationInspector.java b/src/adapter/outbound/persistence-jpa/src/testFixtures/java/dev/caskeleton/adapter/outbound/persistence/testkit/migration/LockWaitMigrationInspector.java index 19a4bca5..3752c1fd 100644 --- a/src/adapter/outbound/persistence-jpa/src/testFixtures/java/dev/caskeleton/adapter/outbound/persistence/testkit/migration/LockWaitMigrationInspector.java +++ b/src/adapter/outbound/persistence-jpa/src/testFixtures/java/dev/caskeleton/adapter/outbound/persistence/testkit/migration/LockWaitMigrationInspector.java @@ -61,7 +61,9 @@ import java.util.regex.Pattern; * that holds any other statement is refused at deploy; with {@code mixed=true} it runs, unbounded. * A locking statement that cannot run in a transaction itself ({@code VACUUM FULL}, a * non-concurrent {@code REINDEX SCHEMA}) can never be bounded and is refused with the advice to - * keep it out of migrations (re-review 2026-11-03 round 4, RR4-02-01). The check assumes Flyway's + * keep it out of migrations (re-review 2026-11-03 round 4, RR4-02-01) — whatever the {@code + * .sql.conf} says: {@code executeInTransaction=true} keeps the {@code SET LOCAL} in force, but only + * for PostgreSQL to refuse the statement at deploy (round 7, RR7-03-05). The check assumes Flyway's * defaults, as the template runs it: {@code executeInTransaction=true} and {@code mixed=false} for * the whole run. */ @@ -147,11 +149,11 @@ public final class LockWaitMigrationInspector { * The statements Flyway runs outside a transaction on PostgreSQL 12 and later, as Flyway 11's * {@code PostgreSQLParser#detectCanExecuteInTransaction} lists them: the statement's leading * words outside parentheses, lower case and one space apart, match one of these in full. {@code - * ALTER TYPE ... ADD VALUE} is on Flyway's list only below PostgreSQL 12. {@code REINDEX TABLE - * CONCURRENTLY} is not on it: Flyway runs it in the transaction and PostgreSQL refuses it there. - * {@code LockWaitMigrationInspectorTest} pins Flyway's own patterns, read from {@code - * PostgreSQLParser}, so a Flyway upgrade that adds a statement or changes one fails the pull - * request's {@code ci} until this copy follows. {@code + * ALTER TYPE ... ADD VALUE} is on Flyway's list only below PostgreSQL 12. {@code REINDEX + * INDEX|TABLE CONCURRENTLY} is not on it: Flyway runs it in the transaction and PostgreSQL + * refuses it there. {@code LockWaitMigrationInspectorTest} pins Flyway's own patterns, read from + * {@code PostgreSQLParser}, so a Flyway upgrade that adds a statement or changes one fails the + * pull request's {@code ci} until this copy follows. {@code * postgresqlMigrationLockWaitIntegrationTest} compares this copy with Flyway on real PostgreSQL * for the statements it lists, and only those (re-review 2026-11-03 round 6, RR6-03-03). */ @@ -254,14 +256,15 @@ public final class LockWaitMigrationInspector { for (Pattern pattern : LOCKING) { Matcher statement = pattern.matcher(text); while (statement.find()) { + boolean cannotRunInATransaction = within(statement.start(), outside); Map.Entry setting = bounds.lowerEntry(statement.start()); - boolean bounded = setting != null && setting.getValue(); + boolean bounded = !cannotRunInATransaction && setting != null && setting.getValue(); if (!bounded && !createdEarlier(statement, createdAt)) { String named = statement.group().trim(); found.add( new Located( statement.start(), - inTransaction + inTransaction && !cannotRunInATransaction ? named : named + " (" + whyUnbounded(statement.start(), outside, configured) + ")")); } @@ -274,14 +277,17 @@ public final class LockWaitMigrationInspector { } /** - * Why a locking statement at {@code at} has no bound in a script that runs outside a transaction. + * Why a locking statement at {@code at} has no bound: it cannot run in a transaction itself, or + * the script runs outside one. */ private static String whyUnbounded(int at, List outside, Configured configured) { - for (Statement statement : outside) { - if (statement.start() <= at && at < statement.end()) { - return "cannot run in a transaction, so no SET LOCAL can bound its wait: keep it out of" - + " migrations and run it by hand as an operations step, outside deploys"; - } + if (within(at, outside)) { + return "cannot run in a transaction, so no SET LOCAL can bound its wait" + + (configured == Configured.IN_A_TRANSACTION + ? " (executeInTransaction=true only makes PostgreSQL refuse it at deploy)" + : "") + + ": keep it out of migrations and run it by hand as an operations step, outside" + + " deploys"; } if (configured == Configured.OUTSIDE_A_TRANSACTION) { return "executeInTransaction=false: SET LOCAL does nothing outside a transaction; move it to" @@ -294,6 +300,16 @@ public final class LockWaitMigrationInspector { + " in a migration of its own"; } + /** Whether the position {@code at} is inside one of the statements Flyway runs outside one. */ + private static boolean within(int at, List outside) { + for (Statement statement : outside) { + if (statement.start() <= at && at < statement.end()) { + return true; + } + } + return false; + } + /** * The statements Flyway runs outside a transaction, found the way Flyway finds them: each * statement's words outside parentheses, up to {@link #FLYWAY_DETECTION_WORDS} words in all, diff --git a/src/adapter/outbound/persistence-jpa/src/testFixtures/java/dev/caskeleton/adapter/outbound/persistence/testkit/migration/MigrationResource.java b/src/adapter/outbound/persistence-jpa/src/testFixtures/java/dev/caskeleton/adapter/outbound/persistence/testkit/migration/MigrationResource.java index 614452af..6251b391 100644 --- a/src/adapter/outbound/persistence-jpa/src/testFixtures/java/dev/caskeleton/adapter/outbound/persistence/testkit/migration/MigrationResource.java +++ b/src/adapter/outbound/persistence-jpa/src/testFixtures/java/dev/caskeleton/adapter/outbound/persistence/testkit/migration/MigrationResource.java @@ -5,6 +5,9 @@ import java.util.Objects; /** * One migration script as the inspector sees it (design §32). * + *

The gRPC ledger's lane in the optional build compiles this file with {@link + * ConcurrentIndexMigrationInspector} (see there), so it keeps to the JDK only. + * * @param name the versioned script name, e.g. {@code V42__order_index.sql} * @param sql the script body */ diff --git a/src/grpc/grpc-operation-ledger-jpa/build.gradle b/src/grpc/grpc-operation-ledger-jpa/build.gradle index 05d90ae0..e4a8fc38 100644 --- a/src/grpc/grpc-operation-ledger-jpa/build.gradle +++ b/src/grpc/grpc-operation-ledger-jpa/build.gradle @@ -17,9 +17,11 @@ configurations { } // One definition of a versioned migration file and of Flyway's order for both builds (re-review -// README 12, r6-b; r9). This build cannot depend on persistence-jpa's testkit, so the one file the -// ledger lanes need from it — VersionedMigrationFiles, JDK only — is copied from there into this -// lane's sources at build time instead of being kept here as a second copy that could drift. +// README 12, r6-b; r9), and one concurrent index check (round 7, RR7-03-01). This build cannot +// depend on persistence-jpa's testkit, so the files the ledger lanes need from it, all JDK only — +// VersionedMigrationFiles, and ConcurrentIndexMigrationInspector with the MigrationResource it +// takes — are copied from there into this lane's sources at build time instead of being kept here +// as second copies that could drift. // Rejected: that testkit directory as a source root of this lane (Spotless refuses a target outside // this project's directory, and the include filter would cover this lane's own sources too), a // dependency on the main build (it would pull the whole main build into this one), and a test that @@ -27,6 +29,8 @@ configurations { def sharedMigrationFileSources = tasks.register('sharedMigrationFileSources', Sync) { from(file('../../adapter/outbound/persistence-jpa/src/testFixtures/java')) { include 'dev/caskeleton/adapter/outbound/persistence/testkit/migration/VersionedMigrationFiles.java' + include 'dev/caskeleton/adapter/outbound/persistence/testkit/migration/ConcurrentIndexMigrationInspector.java' + include 'dev/caskeleton/adapter/outbound/persistence/testkit/migration/MigrationResource.java' } into(layout.buildDirectory.dir('generated/sources/sharedMigrationFiles/java')) } diff --git a/src/grpc/grpc-operation-ledger-jpa/src/integrationTest/java/dev/caskeleton/grpc/ledger/GrpcOperationLedgerMigrationLockWaitIT.java b/src/grpc/grpc-operation-ledger-jpa/src/integrationTest/java/dev/caskeleton/grpc/ledger/GrpcOperationLedgerMigrationLockWaitIT.java index 723fe1d4..7efd2632 100644 --- a/src/grpc/grpc-operation-ledger-jpa/src/integrationTest/java/dev/caskeleton/grpc/ledger/GrpcOperationLedgerMigrationLockWaitIT.java +++ b/src/grpc/grpc-operation-ledger-jpa/src/integrationTest/java/dev/caskeleton/grpc/ledger/GrpcOperationLedgerMigrationLockWaitIT.java @@ -3,19 +3,24 @@ package dev.caskeleton.grpc.ledger; import static org.assertj.core.api.Assertions.assertThat; import static org.assertj.core.api.Assertions.catchThrowable; +import dev.caskeleton.adapter.outbound.persistence.testkit.migration.ConcurrentIndexMigrationInspector; +import dev.caskeleton.adapter.outbound.persistence.testkit.migration.MigrationResource; import java.io.IOException; import java.sql.Connection; import java.sql.ResultSet; import java.sql.SQLException; import java.sql.Statement; import java.time.Duration; +import java.util.ArrayList; import java.util.List; +import java.util.Map; import java.util.concurrent.Executors; import java.util.concurrent.ScheduledExecutorService; import java.util.concurrent.ScheduledFuture; import java.util.concurrent.TimeUnit; import java.util.stream.Stream; import org.junit.jupiter.api.DisplayName; +import org.junit.jupiter.api.Test; import org.junit.jupiter.params.ParameterizedTest; import org.junit.jupiter.params.provider.MethodSource; import org.postgresql.ds.PGSimpleDataSource; @@ -38,9 +43,18 @@ import org.testcontainers.postgresql.PostgreSQLContainer; *

A file Flyway runs outside a transaction — one holding {@code CREATE INDEX CONCURRENTLY} or * another statement on Flyway's list, or with {@code executeInTransaction=false} beside it — is not * one this test can judge: run here in a transaction, PostgreSQL refuses it ({@code 25001}), and no - * {@code SET LOCAL} would bound it in a deploy anyway. Name such a file as an exception in this - * test, and keep it alone in its migration ({@code docs/jpa/migration-guide.md}, "Concurrent index - * builds"; re-review 2026-11-03 round 6, RR6-03-05). + * {@code SET LOCAL} would bound it in a deploy anyway. The one such file the ledger may have is a + * concurrent index build, and it goes in {@link #RUN_OUTSIDE_A_TRANSACTION} with the reason it + * needs to: the lock-wait case leaves it out, and {@code + * everyMigrationListedAsRunningOutsideATransactionIsOneConcurrentIndexBuildThatSaysSo} holds it to + * the guide instead — a concurrent index statement, alone in its file, with {@code + * executeInTransaction=false} beside it ({@code docs/jpa/migration-guide.md}, "Concurrent index + * builds"). The list takes nothing else: a {@code VACUUM FULL} or another locking statement that + * cannot run in a transaction does not belong in a migration at all ("Bound the lock wait"). The + * migrations before the one under test are applied one file at a time ({@link + * GrpcOperationLedgerMigrations#before(String)}), so a listed file is no obstacle to a later one. + * Re-review 2026-11-03 round 6, RR6-03-05; round 7, RR7-03-01: the message pointed at a list this + * test did not have. * *

Every migration after V001 is held to this, read from the tree itself ({@link * GrpcOperationLedgerMigrations#inOrder()}), so a V004 is checked without anyone naming it here. @@ -66,9 +80,18 @@ class GrpcOperationLedgerMigrationLockWaitIT { .withUsername("grpc") .withPassword("grpc"); - /** Every ledger migration after V001, the one that creates the ledger. */ + /** + * The ledger migrations Flyway runs outside a transaction, by file name, each with why it has to: + * a concurrent index build that would stop the ledger's writes if it were built in a transaction. + * Empty while the tree has none. + */ + static final Map RUN_OUTSIDE_A_TRANSACTION = Map.of(); + + /** Every ledger migration after V001, the one that creates the ledger, that runs in one. */ static Stream migrationsAfterTheFirst() throws IOException { - return GrpcOperationLedgerMigrations.inOrder().stream().skip(1); + return GrpcOperationLedgerMigrations.inOrder().stream() + .skip(1) + .filter(migration -> !RUN_OUTSIDE_A_TRANSACTION.containsKey(migration)); } @ParameterizedTest @@ -80,7 +103,9 @@ class GrpcOperationLedgerMigrationLockWaitIT { Statement statement = setup.createStatement()) { // given statement.execute("DROP TABLE IF EXISTS grpc_operation_ledger"); - statement.execute(GrpcOperationLedgerMigrations.before(migration)); + for (String script : GrpcOperationLedgerMigrations.before(migration)) { + statement.execute(script); + } String schemaBefore = schema(statement); // when @@ -100,13 +125,57 @@ class GrpcOperationLedgerMigrationLockWaitIT { ACTIVE_SQL_TRANSACTION.equals(attempt.sqlState()) ? ". PostgreSQL refused it inside a transaction (25001): Flyway runs such a file" + " outside one, where SET LOCAL bounds nothing, so it is no case for this" - + " test. Keep it alone in its migration and name it as an exception here" - + " (\"Concurrent index builds\")" + + " test. If it is a concurrent index build, keep it alone in its migration" + + " with executeInTransaction=false beside it and add it, with the reason, to" + + " RUN_OUTSIDE_A_TRANSACTION in this test (\"Concurrent index builds\");" + + " anything else that cannot run in a transaction does not belong in a" + + " migration (\"Bound the lock wait\")" : "") .containsExactly(LOCK_NOT_AVAILABLE, true, schemaBefore); } } + @Test + @DisplayName( + "트랜잭션 밖에서 도는 예외 목록의 ledger 마이그레이션은 트리에 있고, 동시 인덱스 문장 하나뿐이며, executeInTransaction=false를 적었다") + void everyMigrationListedAsRunningOutsideATransactionIsOneConcurrentIndexBuildThatSaysSo() + throws IOException { + // given + ConcurrentIndexMigrationInspector inspector = new ConcurrentIndexMigrationInspector(); + List tree = GrpcOperationLedgerMigrations.inOrder(); + + // when + List refused = new ArrayList<>(); + for (String migration : RUN_OUTSIDE_A_TRANSACTION.keySet()) { + if (!tree.contains(migration)) { + refused.add(migration + " is not a migration in the ledger tree"); + continue; + } + MigrationResource resource = + new MigrationResource(migration, GrpcOperationLedgerMigrations.named(migration)); + if (!inspector.containsConcurrentIndexStatement(resource.sql())) { + refused.add( + migration + + " holds no concurrent index statement; only a concurrent index build may run" + + " outside a transaction"); + continue; + } + boolean inATransaction = GrpcOperationLedgerMigrations.executeInTransaction(migration); + Throwable thrown = + catchThrowable( + () -> { + inspector.validate(resource, inATransaction); + inspector.requireIsolatedStatement(resource); + }); + if (thrown != null) { + refused.add(thrown.getMessage()); + } + } + + // then + assertThat(refused).isEmpty(); + } + /** Runs the migration in one transaction while another one holds a write to the ledger. */ private static Attempt whileAWriterHolds(PGSimpleDataSource dataSource, String migration) throws SQLException { diff --git a/src/grpc/grpc-operation-ledger-jpa/src/integrationTest/java/dev/caskeleton/grpc/ledger/GrpcOperationLedgerMigrations.java b/src/grpc/grpc-operation-ledger-jpa/src/integrationTest/java/dev/caskeleton/grpc/ledger/GrpcOperationLedgerMigrations.java index f8832fc0..c237e178 100644 --- a/src/grpc/grpc-operation-ledger-jpa/src/integrationTest/java/dev/caskeleton/grpc/ledger/GrpcOperationLedgerMigrations.java +++ b/src/grpc/grpc-operation-ledger-jpa/src/integrationTest/java/dev/caskeleton/grpc/ledger/GrpcOperationLedgerMigrations.java @@ -3,12 +3,16 @@ package dev.caskeleton.grpc.ledger; import dev.caskeleton.adapter.outbound.persistence.testkit.migration.VersionedMigrationFiles; import java.io.IOException; import java.io.InputStream; +import java.io.Reader; import java.nio.charset.StandardCharsets; +import java.nio.file.Files; import java.nio.file.Path; +import java.util.ArrayList; import java.util.List; +import java.util.Properties; /** - * The ledger's Flyway scripts in version order, as one SQL text for a test database. + * The ledger's Flyway scripts in version order, one SQL text per file for a test database. * *

One list for every PostgreSQL lane, so a new migration is applied by all of them or by none. * The list is the tree on disk, not a copy of it: a hand-kept list let a V004 that nobody added to @@ -38,21 +42,26 @@ final class GrpcOperationLedgerMigrations { .toList(); } - /** Every migration's SQL, concatenated in version order. */ - static String all() throws IOException { + /** Every migration's SQL in version order, one script per file. */ + static List all() throws IOException { return before(null); } - /** The SQL of every migration before the one named, concatenated in version order. */ - static String before(String fileName) throws IOException { - StringBuilder sql = new StringBuilder(); + /** + * The SQL of every migration before the one named, in version order, one script per file. Run + * each as a statement of its own, the way Flyway runs a file: one string holding many files runs + * in one implicit transaction, where PostgreSQL refuses a file Flyway runs outside one — a + * concurrent index build (re-review 2026-11-03 round 7, RR7-03-01). + */ + static List before(String fileName) throws IOException { + List scripts = new ArrayList<>(); for (String name : inOrder()) { if (name.equals(fileName)) { break; } - sql.append(read(name)).append('\n'); + scripts.add(read(name)); } - return sql.toString(); + return scripts; } /** One migration's SQL, by its file name. */ @@ -63,6 +72,23 @@ final class GrpcOperationLedgerMigrations { return read(fileName); } + /** + * Whether Flyway is told to run the migration in a transaction: false only when the {@code + *