fix(migration): 실패한 DROP INDEX CONCURRENTLY를 runbook 절차로 복구되게 한다

문제 (재리뷰 round8 RR8-03-01)
- migration-failed.md "Recover a failed concurrent index migration"은
  CREATE·DROP·REINDEX 동시 문장 셋을 대상으로 두고, 2단계에서 "빌드가 남긴
  무효 인덱스를 지운다"고 한다. 실패한 DROP INDEX CONCURRENTLY가 남긴 무효
  인덱스는 그 마이그레이션이 지우려던 인덱스다. 절차대로 지우고 실패 행을
  지운 뒤 다시 배포하면 같은 문장이 42704(index does not exist)로 실패하고
  실패 행이 다시 생긴다. 운영자는 처음과 같은 막힌 상태로 돌아간다.
- 고치다 같은 절에서 하나 더 찾았다. 2단계(DROP INDEX CONCURRENTLY)가
  3단계(원인 정리) 앞이다. 취소·timeout이 원인이면 그 열린 트랜잭션이
  운영자의 DROP INDEX CONCURRENTLY도 붙잡아, 2단계가 3단계를 기다린다.
- runbook을 실행하는 IT는 중복으로 실패한 unique CREATE 한 경우만 돌렸다.

고른 방법
- 리뷰 권장안: 무효 인덱스를 지우는 단계를 문장 종류로 나눈다. CREATE는
  그 인덱스를, REINDEX는 _ccnew(또는 _ccold) 사본을 지운다. DROP INDEX
  CONCURRENTLY면 이 단계를 건너뛴다. 다시 배포한 같은 문장이 그 인덱스를
  지운다(PostgreSQL은 중간에 실패한 DROP INDEX CONCURRENTLY의 재시도를
  받는다). 5단계 문장도 "builds the index"에서 세 경우로 고친다.
- 단계 순서를 "빌드 확인 → 원인 정리 → 무효 인덱스 삭제 → 실패 행 삭제 →
  재배포"로 바꾼다. 실패 행만 지운다는 소유자 결정(2026-11-06)은 그대로다.
- IT에 두 경우를 더한다. 열린 쓰기를 기다리다 취소된 REINDEX와 열린
  읽기를 기다리다 취소된 DROP에 runbook SQL을 runbook에서 읽어 돌리고,
  재배포 뒤 이력 [1 true, 2 true], 무효 인덱스 없음, V1 checksum 그대로를
  단언한다. 대기를 30초 기다리지 않고, 문장이 잠금을 기다리는 것을 본
  즉시 취소하는 도우미(cancelledOnceItWaits)를 둔다.
- 같은 절차를 요약한 곳을 같이 고친다: docs/jpa/runbooks.md "An invalid
  index exists", migration-guide "Concurrent index builds"(DROP이 무엇을
  기다리는지 포함), 학습 문서 8.2절, ConcurrentIndexMigrationInspector의
  Javadoc과 실패 메시지, 업그레이드 안내 5절 행(그 범위가 아직 main에
  없어 행을 최종 동작으로 고치고 머리 목록에 더함).
- 버린 대안: 가이드와 검사기가 DROP INDEX CONCURRENTLY에 IF EXISTS를
  요구한다. 이미 배포된 파일은 고칠 수 없어 새 파일에만 통하고, 서비스의
  기존 마이그레이션 검사를 새로 실패시킨다.

위험·한계
- 운영자가 실패한 문장의 종류를 마이그레이션 파일에서 읽어야 한다.
  runbook이 이력 행만으로는 알 수 없는 것을 요구한다.
- REINDEX가 swap 뒤에 실패해 _ccold가 남는 경우는 문서로만 다뤘다(실행
  재현 없음). PostgreSQL 문서의 권장 복구와 같다.
- 단계 번호가 바뀌었다. 리뷰 기록이 옛 번호(2단계 = drop)로 적은 곳은 그때의
  기록이라 두었다.

RED (desktop, 기준 3eb3b067 + 임시 테스트, 로그 ~/r9-g3-logs/red-concurrent.log)
- postgresqlConcurrentIndexMigrationIntegrationTest rc=1, 9건 중 2건 실패
  - redAFailedConcurrentDropIsRecoveredByTheBaseRunbook: 기준 runbook
    1·2·4단계 뒤 재배포가 "42704 ERROR: index "ix_probe_id" does not
    exist"로 실패, 이력 [1 true, 2 false] 그대로
  - redTheBaseRunbookDropsBeforeItEndsTheTransaction: 기준 순서의 2단계
    (_ccnew drop)가 열린 쓰기 트랜잭션을 기다려 취소됨
  - 나머지 7건(기존) 통과

검증 (desktop, 이 커밋의 트리 7cfaa83c, --no-build-cache, 로그 ~/r9-g3-logs/)
- :adapter:outbound:persistence-jpa:check rc=0, test 666건 실패·skip 0
- postgresqlConcurrentIndexMigrationIntegrationTest rc=0, 9건(새 2건 포함)
- :adapter:outbound:persistence-jpa:testQualityCheck rc=0
- :app-bootstrap:documentContractTest rc=0, 67건(RunbookCoverageContractTest
  포함)
- -p optional-platforms :grpc:grpc-operation-ledger-jpa:integrationTest
  rc=0, 47건(검사기 사본이 바뀌어서)
- FROM-CACHE 0
- 상대 링크(노트북): python3 scripts/check-work-docs.py --links로 바꾼
  문서 5개(migration-failed.md, jpa/runbooks.md, migration-guide.md,
  학습 문서, 업그레이드 안내) rc=0, 깨진 링크 0

Refs: docs/reviews/2026-11-03-rereview/round8/03-migration.md RR8-03-01
Fixes: 6ec0c4f7fa ("fix(migration): 실패한 동시 인덱스 마이그레이션 복구와 검사기 설명을 실제 동작에 맞춘다")
This commit is contained in:
DongHyeonka
2026-11-07 08:07:42 +09:00
parent dae6dcaf51
commit 8ebe98e69a
7 changed files with 262 additions and 59 deletions
+13 -8
View File
@@ -152,12 +152,14 @@ transaction Flyway itself keeps open, and Flyway waits for the build in turn: th
and the table's other DDL and autovacuum queue behind the build meanwhile. The template's Flyway
holds its lock for the session instead ("Who may change the schema"), so the build waits only for
other transactions; a deployment tool left on Flyway's default stops on the first such migration.
`DROP INDEX CONCURRENTLY` does not wait that way. `PostgreSqlConcurrentIndexMigrationIntegrationTest`
`DROP INDEX CONCURRENTLY` does not wait that way: it waits only for the transactions that hold a
lock on its table. `PostgreSqlConcurrentIndexMigrationIntegrationTest`
(`postgresqlConcurrentIndexMigrationIntegrationTest`) applies both builds as a second deploy, with
the Flyway Spring Boot builds from the module defaults and with `FlywaySchemaPolicy.configure()`,
on real PostgreSQL. The other transactions still count: a long one an operator or a report keeps
open holds the build, and a file run outside a transaction cannot bound that wait with `SET LOCAL`.
Look for old transactions (`pg_stat_activity.xact_start`) before deploying one.
open holds the build, and the drop too once it has touched the table, and a file run outside a
transaction cannot bound that wait with `SET LOCAL`. Look for old transactions
(`pg_stat_activity.xact_start`) before deploying one.
The testkit's `ConcurrentIndexMigrationInspector` checks a script for this in a migration test
(nothing checks it at startup), and additionally requires the migration to contain nothing else.
@@ -165,15 +167,18 @@ Next to another statement, a concurrent one makes Flyway refuse the file before
nothing configures it; with `executeInTransaction=false` Flyway commits each statement on its own,
and a failed build leaves the statements before it applied — state to reconstruct by hand.
Alone, a failed build leaves an invalid index and a history row with `success = false` for its
Alone, a failed statement leaves an invalid index and a history row with `success = false` for its
version, and Flyway refuses every later deploy ("Detected failed migration to version …") until that
row is gone: dropping the index is not enough. The recovery is
[migration-failed.md](../runbooks/migration-failed.md) "Recover a failed concurrent index migration":
confirm no build is running, drop the invalid index, remove the cause, delete that version's failed
row and nothing else, deploy again. It is not `flyway repair`, which would also rewrite the
remaining rows' checksums. `PostgreSqlConcurrentIndexMigrationIntegrationTest`
confirm no build is running, remove the cause, drop the invalid index a failed build or rebuild
left, delete that version's failed row and nothing else, deploy again. A failed `DROP INDEX
CONCURRENTLY` leaves invalid the index it was dropping; the runbook leaves that one for the deploy
again to drop, since dropped by hand it makes the same statement fail. It is not `flyway repair`,
which would also rewrite the remaining rows' checksums. `PostgreSqlConcurrentIndexMigrationIntegrationTest`
(`postgresqlConcurrentIndexMigrationIntegrationTest`) runs the runbook's SQL, read from the runbook,
after a unique build that failed on a duplicate, and shows both mixed files above.
after a unique build that failed on a duplicate and after a rebuild and a drop cancelled while they
waited for an open transaction, and shows both mixed files above.
An invalid index is not merely useless — the planner ignores it while every write still maintains
it. [runbooks.md](runbooks.md) has the query that finds them. Nothing drops them automatically: an
+11 -7
View File
@@ -69,15 +69,19 @@ join pg_namespace n on n.oid = i.relnamespace
where not x.indisvalid;
```
A concurrent build failed. The index is ignored by the planner and maintained by every write.
A concurrent build, rebuild or drop failed. The index is ignored by the planner and maintained by
every write.
1. Confirm no build is currently running. An in-progress build looks identical in the catalog.
2. Run `DROP INDEX CONCURRENTLY IF EXISTS <invalid_index>;` outside a migration.
3. If a migration built it, Flyway recorded that version as failed (`success = false`) and refuses
every later deploy until that row is gone: dropping the index does not clear it. Follow
[migration-failed.md](../runbooks/migration-failed.md) "Recover a failed concurrent index
migration", which removes the cause and deletes that one row.
4. Re-apply the index migration.
2. If a migration was building, rebuilding or dropping it, Flyway recorded that version as failed
(`success = false`) and refuses every later deploy until that row is gone: dropping the index
does not clear it. Follow [migration-failed.md](../runbooks/migration-failed.md) "Recover a
failed concurrent index migration" instead of the next step. It removes the cause, drops the
index a failed build or rebuild left, and deletes that one row. It leaves the index a failed
`DROP INDEX CONCURRENTLY` left, for the deploy again to drop: dropped by hand, that statement
finds no index and fails again.
3. Otherwise run `DROP INDEX CONCURRENTLY IF EXISTS <invalid_index>;` outside a migration, and
re-apply the index migration.
The platform does not drop these automatically: on a rolling deploy every instance would race to
drop an index another instance was about to finish building.
+28 -16
View File
@@ -63,13 +63,14 @@ status: active
case — clean it up with `outbox-dead-letter.md` §8 and rerun, the migration stays as it is;
- checksum mismatch: compare the deployed artifact with the already applied script before
considering repair;
- a concurrent index migration failed (`CREATE [UNIQUE] INDEX CONCURRENTLY` or `REINDEX …
CONCURRENTLY` that Flyway ran outside a transaction): a duplicate value for a unique
index, a deadlock, `statement_timeout` or a cancelled build. Outside a transaction nothing
rolled back: the migration left an invalid index and a history row with
`success = false` for that version, and every later deploy fails validation ("Detected failed
migration to version …") until the row is gone. Dropping the index alone does not clear it.
Follow "Recover a failed concurrent index migration" below;
- a concurrent index migration failed (`CREATE [UNIQUE] INDEX CONCURRENTLY`, `DROP INDEX
CONCURRENTLY` or `REINDEX … CONCURRENTLY` that Flyway ran outside a transaction): a duplicate
value for a unique index, a deadlock, `statement_timeout` or a statement cancelled while it
waited for an open transaction. Outside a transaction nothing rolled back: the migration left
an invalid index and a history row with `success = false` for that version, and every later
deploy fails validation ("Detected failed migration to version …") until the row is gone.
Dropping the index alone does not clear it. Follow "Recover a failed concurrent index
migration" below;
- connection/TLS failure: fix transport or credentials without changing Flyway history.
## Action
@@ -129,17 +130,25 @@ FROM pg_stat_progress_create_index
WHERE datname = current_database();
```
2. Drop the invalid index the build left (`docs/jpa/runbooks.md` "An invalid index exists" lists
them). Run it on its own, not inside `BEGIN`: `CONCURRENTLY` refuses a transaction block.
2. Remove the cause. For a unique index, clean up the duplicate rows with the data owner (a forward
fix, never by editing the migration). For a deadlock, a timeout or a cancelled statement, end or
wait out the transaction that held it: a build or a rebuild waits for every transaction older
than itself, a drop for every transaction that holds a lock on its table. Do this before step 3,
whose `DROP INDEX CONCURRENTLY` waits for the same transactions.
3. Drop the invalid index a failed build or rebuild left: the index a `CREATE [UNIQUE] INDEX
CONCURRENTLY` names, or the copy a `REINDEX … CONCURRENTLY` made, named after the index with
`_ccnew` (or `_ccold`) on the end (`docs/jpa/runbooks.md` "An invalid index exists" lists them).
Run it on its own, not inside `BEGIN`: `CONCURRENTLY` refuses a transaction block.
**Skip this step when the failed statement is `DROP INDEX CONCURRENTLY`.** The invalid index it
left is the index the migration drops, and the deploy in step 5 runs the same statement, which
drops it. Dropped by hand first, that statement fails with `42704` ("index … does not exist")
and leaves a new failed row.
<!-- runbook-sql: failed-concurrent-index-drop -->
```sql
DROP INDEX CONCURRENTLY IF EXISTS <invalid_index>;
```
3. Remove the cause. For a unique index, clean up the duplicate rows with the data owner (a forward
fix, never by editing the migration). For a deadlock or a timeout, end or wait out the
transaction that held the build.
4. Delete that version's failed row from the stream's history table, and nothing else. `<version>`
is the `version` column of the failed row as the Diagnosis query shows it (`22.1` for
`V22_1__…`). It must delete exactly one row; if it deletes none, this is not the failure this
@@ -151,15 +160,18 @@ DELETE FROM <owned_history_table>
WHERE version = '<version>' AND NOT success;
```
5. Deploy again. Flyway runs the same migration from the start and builds the index.
5. Deploy again. Flyway runs the same migration from the start: it builds the index, rebuilds it,
or drops the index the failed drop left.
The failed row is the record of a version that never applied, so deleting it leaves the order and
the checksum of every applied version as they were. Do not use `flyway repair` for this: it also
rewrites the checksums and descriptions of the remaining rows to match the scripts on disk, which
removes the evidence a checksum mismatch carries. `:adapter:outbound:persistence-jpa:postgresqlConcurrentIndexMigrationIntegrationTest`
runs the three SQL blocks above, read from this file, against real PostgreSQL: a unique build fails
on a duplicate, steps 1–4 run, and the next deploy applies the version with the index valid and the
earlier version's checksum unchanged.
runs the three SQL blocks above, read from this file, against real PostgreSQL for each statement
this section covers: a unique build that fails on a duplicate, and a rebuild and a drop cancelled
while they wait for an open transaction. Steps 1–4 run in order, step 3 for the build and the
rebuild only, and the next deploy applies the version with no invalid index left and the earlier
version's checksum unchanged.
## Escalation
+10 -4
View File
@@ -1373,18 +1373,24 @@ Flyway 기본값: 잠금 트랜잭션(열린 채) ──기다림──▶ 마
검사 대상: `CREATE [UNIQUE] INDEX CONCURRENTLY`, `DROP INDEX CONCURRENTLY`, `REINDEX … CONCURRENTLY`
(`REINDEX (CONCURRENTLY) TABLE` 같은 괄호 옵션 포함).
`requireIsolatedStatement`도 있다 — concurrent 빌드가 실패하면 invalid index와 그 버전의
`requireIsolatedStatement`도 있다 — concurrent 문장이 실패하면 invalid index와 그 버전의
**실패 이력 행**(`success = false`)이 남는다. Flyway는 그 행이 있는 한 다음 배포를 validate에서 거부한다.
그래서 인덱스만 지우고 다시 배포하면 또 멈춘다.
```
실패한 동시 빌드 → invalid index + 실패 이력 행
→ 빌드가 아직 도는지 확인 → DROP INDEX CONCURRENTLY → 원인 정리(중복 행 등)
실패한 동시 인덱스 문장 → invalid index + 실패 이력 행
→ 빌드가 아직 도는지 확인 → 원인 정리(중복 행, 붙잡던 열린 트랜잭션)
→ 빌드·재빌드가 남긴 invalid index를 DROP INDEX CONCURRENTLY
→ 그 버전의 실패 이력 행만 DELETE → 다시 배포
```
- 원인 정리가 먼저다. 운영자의 `DROP INDEX CONCURRENTLY`도 그 테이블을 잡은 열린 트랜잭션을 기다린다.
- 실패한 문장이 `DROP INDEX CONCURRENTLY`면 남은 invalid index는 그 마이그레이션이 지우려던 인덱스다.
손으로 지우지 않는다. 다시 배포한 같은 문장이 지운다. 먼저 지우면 그 문장이 없는 인덱스(`42704`)로 또 실패한다.
- 절차와 SQL은 `docs/runbooks/migration-failed.md` "Recover a failed concurrent index migration"에 있다.
`PostgreSqlConcurrentIndexMigrationIntegrationTest`가 그 SQL을 runbook에서 읽어 실제 PostgreSQL에서 돌린다.
`PostgreSqlConcurrentIndexMigrationIntegrationTest`가 그 SQL을 runbook에서 읽어 세 문장(중복으로 실패한 unique 빌드,
열린 트랜잭션을 기다리다 취소된 재빌드와 drop)마다 실제 PostgreSQL에서 돌린다.
- 실패 이력 행을 지우는 것은 "이력을 고치지 않는다"의 하나뿐인 예외다(소유자 결정 2026-11-06).
실패 행은 적용되지 않은 버전의 기록이라, 지워도 성공한 버전의 순서와 checksum은 그대로다.
- `flyway repair`는 여전히 쓰지 않는다. 남은 행의 checksum·설명까지 디스크의 스크립트에 맞춰 다시 쓰기 때문이다.
+2 -2
View File
@@ -4,7 +4,7 @@
- 커밋: `34565130` High·P1 19건 · `1400dc15` 남은 발견·매뉴얼·생성 명령 · `5786126c` 남은 항목과 합친 뒤 결함 ·
`7ef8d0b4` 재리뷰 기록 · `bbe944bd` 재리뷰 결함 · `ecc74b7d` 2차 재리뷰 기록·이 안내 · `1885bf8d` 2차 재리뷰 결함 ·
`339cf8fd` 3차 확인 리뷰 기록 · `f40c03c6` 3차 확인 리뷰 Minor · `c2e55510` 마지막 확인 리뷰 기록 ·
`2678955c` 테스트 코드 정적 분석(B-40) · `70fb80bf` 소유자가 다시 정한 기본값 3가지(늦은 멱등 완료 기록, `failure-retention` 기동 검사, workflow `cd`) · B4 성능 lane 단언 상시화(2026-11-06, B-43) · `paths` 필터 trigger를 단계 실행으로 세지 않음(2026-11-06 소유자 결정, B-36) · r8-f1 전송 오류 처리(2026-11-07, B-45~B-48) · 태그 push·`release` workflow는 `releaseCheck`에만 셈(2026-11-07 소유자 결정, B-36) — `docs(...)` 커밋은 문서만 바꾼다
`2678955c` 테스트 코드 정적 분석(B-40) · `70fb80bf` 소유자가 다시 정한 기본값 3가지(늦은 멱등 완료 기록, `failure-retention` 기동 검사, workflow `cd`) · B4 성능 lane 단언 상시화(2026-11-06, B-43) · `paths` 필터 trigger를 단계 실행으로 세지 않음(2026-11-06 소유자 결정, B-36) · r8-f1 전송 오류 처리(2026-11-07, B-45~B-48) · 태그 push·`release` workflow는 `releaseCheck`에만 셈(2026-11-07 소유자 결정, B-36) · 실패한 동시 인덱스 drop의 복구 절차(2026-11-07, 재리뷰 RR8-03-01, 5절) — `docs(...)` 커밋은 문서만 바꾼다
- 근거 문서: `docs/reviews/2026-11-01-maturity-review/README.md` 13~16절, `docs/reviews/2026-11-03-rereview/README.md`
4·6·8~13절, 각 커밋의 `BREAKING CHANGE:` 꼬리말
- 아래 표의 각 줄은 위 범위의 diff와 코드로 확인했다. 확인하지 못한 것은 그 자리에 "확인 못 함"으로 적었고 9절에
@@ -254,7 +254,7 @@ V21 실패의 분류는 `docs/runbooks/migration-failed.md` "Classify the failur
| 서버 로그(STOMP 거부·감싼 carrier, 2026-11-07) | `@MessageMapping` 앞에서 Spring이 거부한 메시지를 WARN 한 줄 `request refused as <code> by <예외 타입>`으로 남긴다: 읽지 못한 payload(`MessageConversionException`)·비었거나 검증에 걸린 payload(`MethodArgumentNotValidException`)는 `VALIDATION_FAILED`, 변환되지 않는 `@DestinationVariable`·`@Header`(`ConversionFailedException`)는 `BAD_PARAMETER`. 전에는 ERROR에 예외 메시지(받은 JSON 조각·값)까지 남겼다. 응답을 쓰지 못한 변환 실패는 그대로 ERROR. 네 전송 모두 감싼 carrier를 그 carrier 줄로 남긴다(감싼 persistence 실패는 분류만, SQL 원인 없음) | ERROR 경보가 클라이언트의 잘못된 메시지에 울리지 않는다. 그 실패를 찾던 로그 검색은 WARN으로 바꾼다 | r8-f1(재리뷰 RR7-01-06·RR7-01-05) · `src/adapter/inbound/websocket/.../stomp/StompHandlerFailureLog.java`, `src/shared-contract/.../error/ServerFailureLog.java` `refused` |
| sample-portfolio | 익명 쓰기 401, 포스터 제목 대소문자 무시 유일(409 `POSTER_ALREADY_EXISTS`), 관리자도 남의 work log 수정 불가, 샘플 이벤트 tenant `global` | sample을 남긴 서비스만 | `34565130`, `1400dc15` |
| Flyway 잠금 | Flyway의 PostgreSQL 마이그레이션 잠금을 트랜잭션 대신 세션에 둔다. Spring Boot의 Flyway는 module-defaults `spring.flyway.postgresql.transactional-lock: false`, 템플릿이 직접 만드는 Flyway(멱등성·알림·sample stream, schema-per-tenant, testkit `MigrationContractRunner`)는 `FlywaySchemaPolicy.configure()`, messaging inbox stream은 같은 값을 직접 넣는다. 운영 코드의 `Flyway.configure()` 직접 호출과 설정 객체 직접 생성(`new FluentConfiguration()`·`new ClassicConfiguration()`·그 하위 클래스, round7 RR7-03-06)은 `JpaProductionArchitectureTest`가 거부한다 | Flyway 기본값에서는 동시 인덱스 마이그레이션이 Flyway 자신의 열린 트랜잭션을 기다려 끝나지 않았다. 이제 끝난다. 마이그레이션은 잠금을 잡은 연결 하나로 돈다. transaction 모드 PgBouncer 뒤에서는 잠금 해제가 다른 서버 연결로 가 다음 배포가 기다릴 수 있으니 마이그레이션은 직접 연결이나 session 모드 pool로 돌린다. 배포 도구는 이 repo 밖이라 2절 15번으로 맞춘다 | 재리뷰 round6 RR6-03-01 · `persistence-jpa.yml`, `.../migration/FlywaySchemaPolicy.java`, `docs/jpa/migration-guide.md` "Who may change the schema"·"Concurrent index builds" |
| 실패한 동시 인덱스 마이그레이션 복구 | `docs/runbooks/migration-failed.md`에 「Recover a failed concurrent index migration」을 더했다: 진행 중인 빌드가 없는지 보고, 무효 인덱스를 `DROP INDEX CONCURRENTLY`로 지우고, 원인을 정리하고, 그 버전의 실패 이력 행(`success = false`)만 지운 뒤 다시 배포한다. 「Do not」의 이력 수정 금지에 이 한 가지 예외를 적었다. `flyway repair`는 계속 쓰지 않는다 | 전에는 안내가 "`DROP INDEX` 하나"였고, 그대로 하면 남은 실패 이력 때문에 다음 배포가 validate에서 다시 멈췄다. 동시 인덱스 마이그레이션을 쓰는 서비스의 운영자는 이 절차로 복구한다 | 재리뷰 round7 RR7-03-03, 소유자 결정 2026-11-06 · `docs/jpa/migration-guide.md` "Concurrent index builds", `PostgreSqlConcurrentIndexMigrationIntegrationTest` |
| 실패한 동시 인덱스 마이그레이션 복구 | `docs/runbooks/migration-failed.md`에 「Recover a failed concurrent index migration」을 더했다: 진행 중인 빌드가 없는지 보고, 원인(중복 행, 문장을 붙잡던 열린 트랜잭션)을 정리하고, 실패한 빌드·재빌드가 남긴 무효 인덱스를 `DROP INDEX CONCURRENTLY`로 지우고, 그 버전의 실패 이력 행(`success = false`)만 지운 뒤 다시 배포한다. 실패한 문장이 `DROP INDEX CONCURRENTLY`면 남은 무효 인덱스는 지우지 않는다. 그 마이그레이션이 지우려던 인덱스라 다시 배포한 같은 문장이 지운다. 「Do not」의 이력 수정 금지에 이 한 가지 예외를 적었다. `flyway repair`는 계속 쓰지 않는다 | 전에는 안내가 "`DROP INDEX` 하나"였고, 그대로 하면 남은 실패 이력 때문에 다음 배포가 validate에서 다시 멈췄다. 실패한 drop의 인덱스를 손으로 지우면 다시 배포한 문장이 `42704`(없는 인덱스)로 또 실패한다. 동시 인덱스 마이그레이션을 쓰는 서비스의 운영자는 이 절차로 복구한다 | 재리뷰 round7 RR7-03-03, round8 RR8-03-01, 소유자 결정 2026-11-06 · `docs/jpa/migration-guide.md` "Concurrent index builds", `PostgreSqlConcurrentIndexMigrationIntegrationTest` |
| 성능 lane(HTTP client·Mongo) | httpclient `LargeBodyResourceTest`의 힙 상한(32 MiB 스트리밍 중 힙 증가 16 MiB 미만)을 매번 단언한다. 전에는 `-Pperformance.assertions.enabled=true`일 때만 단언하고 아니면 건너뛴다는 줄만 찍었다. Mongo `MongoResourceBudgetLaneTest`는 표준 gate 전체(p99 500ms, pool 대기 250ms, 힙 512 MiB, examined/returned 10, spill 금지)를 Gradle 밖 실행에서도 단언한다(B-43) | 손으로 돌린 `performanceCheck`가 상한을 실제로 재므로, 전에는 통과하던 느린 기계에서 실패할 수 있다 | B4(2026-11-06) · `LargeBodyResourceTest`, `MongoResourceBudgetLaneTest` |
| 보안 필터 로그 레벨 | 보안 필터(`AuthErrorResponseWriter`)의 `security failure` 줄을 5xx(`AUTH_JWKS_UNAVAILABLE` 503, `INTERNAL_AUTH_MISCONFIGURATION` 500)면 ERROR, 4xx면 WARN으로 남긴다. 전에는 모두 WARN이었다. 다른 경로가 5xx를 ERROR로 남기는 규칙(`ServerFailureLog`)과 같아졌다 | ERROR로 거르는 로그 경보·대시보드에 JWKS 장애와 인증 설정 오류가 새로 잡힌다. 401·403은 그대로 WARN | 설계 메모 2026-11-06 7절 결정 2 · `src/adapter/inbound/web/.../auth/AuthErrorResponseWriter.java`, `docs/superpowers/specs/2026-11-06-error-code-log-level-design.md` |
@@ -9,8 +9,11 @@ import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;
import java.time.Duration;
import java.util.List;
import java.util.Locale;
@@ -57,13 +60,17 @@ import org.springframework.jdbc.datasource.DelegatingDataSource;
* deploy still waiting after {@link #FINISHES_WITHIN} is cancelled, which leaves the invalid index
* an interrupted build leaves in production too.
*
* <p>Three more cases pin what happens when such a deploy fails (re-review 2026-11-03 round 7,
* RR7-03-03 and RR7-03-04). A concurrent build next to another statement is refused before any of
* it runs when nothing configures the file, and is left half applied when {@code
* executeInTransaction=false} does: Flyway commits each statement on its own. A failed build leaves
* an invalid index and a failed history row, and the deploy after it is refused until the row is
* gone: the recovery case runs the SQL of {@code docs/runbooks/migration-failed.md} ("Recover a
* failed concurrent index migration"), read out of the runbook itself, and deploys again.
* <p>Five more cases pin what happens when such a deploy fails (re-review 2026-11-03 round 7,
* RR7-03-03 and RR7-03-04; round 8, RR8-03-01). A concurrent build next to another statement is
* refused before any of it runs when nothing configures the file, and is left half applied when
* {@code executeInTransaction=false} does: Flyway commits each statement on its own. A concurrent
* statement alone that fails leaves an invalid index and a failed history row, and the deploy after
* it is refused until the row is gone. The recovery cases run the SQL of {@code
* docs/runbooks/migration-failed.md} ("Recover a failed concurrent index migration"), read out of
* the runbook itself, for each statement the runbook covers, and deploy again: a unique build that
* fails on a duplicate, and a rebuild and a drop cancelled while they wait for an open transaction.
* The drop leaves invalid the very index it was dropping, so the runbook leaves that index for the
* deploy again to drop: dropped by hand first, the same statement fails with {@code 42704}.
*/
class PostgreSqlConcurrentIndexMigrationIntegrationTest {
@@ -89,6 +96,12 @@ class PostgreSqlConcurrentIndexMigrationIntegrationTest {
private static final String FLYWAY_REFUSES_A_MIXED_MIGRATION =
"Detected both transactional and non-transactional statements";
/** A read of the table, which a {@code DROP INDEX CONCURRENTLY} waits for while it is open. */
private static final String A_READ = "SELECT count(*) FROM probe";
/** A write to the table, which a {@code REINDEX ... CONCURRENTLY} waits for while it is open. */
private static final String A_WRITE = "UPDATE probe SET note = note WHERE id = 1";
/** The system property naming {@code docs/runbooks/migration-failed.md} for this lane. */
private static final String MIGRATION_FAILED_RUNBOOK = "migration.failed.runbook";
@@ -203,7 +216,7 @@ class PostgreSqlConcurrentIndexMigrationIntegrationTest {
@Test
@DisplayName(
"실패한 동시 인덱스 마이그레이션은 runbook 절차(빌드 확인, 무효 인덱스 삭제, 원인 정리, 그 버전의 실패 이력 행 삭제) 뒤 다시 배포하면 적용된다")
"중복 값으로 실패한 unique 동시 인덱스 빌드는 runbook 절차(빌드 확인, 원인 정리, 무효 인덱스 삭제, 그 버전의 실패 이력 행 삭제) 뒤 다시 배포하면 적용된다")
void aFailedConcurrentIndexMigrationIsRecoveredByTheRunbookProcedure(@TempDir Path stream)
throws Exception {
RunbookSql runbook = RunbookSql.fromSystemProperty(MIGRATION_FAILED_RUNBOOK);
@@ -220,10 +233,10 @@ class PostgreSqlConcurrentIndexMigrationIntegrationTest {
// when
List<Map<String, Object>> buildsRunning =
jdbc.queryForList(runbook.block("failed-concurrent-index-builds"));
jdbc.execute("DELETE FROM probe WHERE id = 2001");
jdbc.execute(
runbook.block(
"failed-concurrent-index-drop", Map.of("invalid_index", "ix_probe_note_u")));
jdbc.execute("DELETE FROM probe WHERE id = 2001");
jdbc.execute(
runbook.block(
"failed-concurrent-index-history-row",
@@ -251,6 +264,113 @@ class PostgreSqlConcurrentIndexMigrationIntegrationTest {
}
}
@Test
@DisplayName(
"열린 트랜잭션을 기다리다 취소된 REINDEX CONCURRENTLY는 runbook 절차(빌드 확인, 원인 정리, 남은 _ccnew 무효 인덱스 삭제, 그 버전의 실패 이력 행 삭제) 뒤 다시 배포하면 적용된다")
void aFailedConcurrentRebuildIsRecoveredByTheRunbookProcedure(@TempDir Path stream)
throws Exception {
RunbookSql runbook = RunbookSql.fromSystemProperty(MIGRATION_FAILED_RUNBOOK);
String name = databaseName();
try (HikariDataSource database = postgres.createDatabase(name)) {
// given
Files.writeString(stream.resolve("V1__probe.sql"), FIRST_RELEASE);
migrateTheWayTheTemplateDoes(database, stream);
String checksumOfV1 = checksumOf(database, "1");
writeConcurrentMigration(stream, "REINDEX INDEX CONCURRENTLY ix_probe_id");
Throwable failed;
String afterFailure;
List<Map<String, Object>> buildsRunning;
JdbcTemplate jdbc = new JdbcTemplate(database);
try (Connection writer = openTransactionOn(database, A_WRITE)) {
failed = cancelledOnceItWaits(name, () -> migrateTheWayTheTemplateDoes(database, stream));
afterFailure = aftermath(database);
// when
buildsRunning = jdbc.queryForList(runbook.block("failed-concurrent-index-builds"));
writer.rollback();
}
jdbc.execute(
runbook.block(
"failed-concurrent-index-drop", Map.of("invalid_index", "ix_probe_id_ccnew")));
jdbc.execute(
runbook.block(
"failed-concurrent-index-history-row",
Map.of("owned_history_table", "flyway_schema_history", "version", "2")));
Throwable again = catchThrowable(() -> migrateTheWayTheTemplateDoes(database, stream));
// then
assertThat(
List.of(
failed != null,
afterFailure,
buildsRunning.size(),
String.valueOf(again),
aftermath(database),
checksumOf(database, "1")))
.containsExactly(
true,
"row 3001 0, indexes [ix_probe_id true, ix_probe_id_ccnew false, probe_pkey true],"
+ " history [1 true, 2 false]",
0,
"null",
"row 3001 0, indexes [ix_probe_id true, probe_pkey true], history [1 true, 2 true]",
checksumOfV1);
}
}
@Test
@DisplayName(
"열린 트랜잭션을 기다리다 취소된 DROP INDEX CONCURRENTLY는 runbook 절차대로 빌드 확인, 원인 정리, 그 버전의 실패 이력 행 삭제만 하고 남은 무효 인덱스는 두면 다시 배포한 같은 문장이 그 인덱스를 지우고 적용된다")
void aFailedConcurrentDropIsRecoveredByTheRunbookProcedure(@TempDir Path stream)
throws Exception {
RunbookSql runbook = RunbookSql.fromSystemProperty(MIGRATION_FAILED_RUNBOOK);
String name = databaseName();
try (HikariDataSource database = postgres.createDatabase(name)) {
// given
Files.writeString(stream.resolve("V1__probe.sql"), FIRST_RELEASE);
migrateTheWayTheTemplateDoes(database, stream);
String checksumOfV1 = checksumOf(database, "1");
writeConcurrentMigration(stream, "DROP INDEX CONCURRENTLY ix_probe_id");
Throwable failed;
String afterFailure;
List<Map<String, Object>> buildsRunning;
JdbcTemplate jdbc = new JdbcTemplate(database);
try (Connection reader = openTransactionOn(database, A_READ)) {
failed = cancelledOnceItWaits(name, () -> migrateTheWayTheTemplateDoes(database, stream));
afterFailure = aftermath(database);
// when
buildsRunning = jdbc.queryForList(runbook.block("failed-concurrent-index-builds"));
reader.rollback();
}
int deleted =
jdbc.update(
runbook.block(
"failed-concurrent-index-history-row",
Map.of("owned_history_table", "flyway_schema_history", "version", "2")));
Throwable again = catchThrowable(() -> migrateTheWayTheTemplateDoes(database, stream));
// then
assertThat(
List.of(
failed != null,
afterFailure,
buildsRunning.size(),
deleted,
String.valueOf(again),
aftermath(database),
checksumOf(database, "1")))
.containsExactly(
true,
"row 3001 0, indexes [ix_probe_id false, probe_pkey true], history [1 true, 2 false]",
0,
1,
"null",
"row 3001 0, indexes [probe_pkey true], history [1 true, 2 true]",
checksumOfV1);
}
}
/** V2: the one concurrent statement, with the companion configuration the guide asks for. */
private static void writeConcurrentMigration(Path stream, String statement) throws IOException {
Files.writeString(stream.resolve("V2__concurrent_index.sql"), statement + ";\n");
@@ -332,6 +452,60 @@ class PostgreSqlConcurrentIndexMigrationIntegrationTest {
}
}
/**
* A connection to the database with a transaction open on the table: {@code statement} has run in
* it and holds its lock until the caller ends the transaction, as a report or an operator's
* {@code BEGIN} would. Opened outside the test's pool, which the deploy needs.
*/
private static Connection openTransactionOn(HikariDataSource database, String statement)
throws SQLException {
Connection holder =
DriverManager.getConnection(
database.getJdbcUrl(), database.getUsername(), database.getPassword());
holder.setAutoCommit(false);
try (Statement holding = holder.createStatement()) {
holding.execute(statement);
}
return holder;
}
/**
* Runs the deploy until its statement waits for a lock in the database, cancels that wait as an
* operator ending a stuck deploy would, and returns how the deploy failed.
*/
private static Throwable cancelledOnceItWaits(String databaseName, Runnable migration)
throws Exception {
ExecutorService deployer = Executors.newSingleThreadExecutor();
try {
Future<?> deploy = deployer.submit(migration);
long deadline = System.nanoTime() + FINISHES_WITHIN.toNanos();
while (!aStatementWaitsForALock(databaseName)) {
if (deploy.isDone() || System.nanoTime() > deadline) {
throw new IllegalStateException("the deploy never waited for a lock");
}
Thread.sleep(50);
}
cancelLockWaits(databaseName);
return catchThrowable(() -> deploy.get(FINISHES_WITHIN.toMillis(), TimeUnit.MILLISECONDS));
} finally {
deployer.shutdownNow();
}
}
private static boolean aStatementWaitsForALock(String databaseName) throws SQLException {
try (Connection admin = postgres.connection();
PreparedStatement waiting =
admin.prepareStatement(
"SELECT count(*) FROM pg_stat_activity"
+ " WHERE datname = ? AND wait_event_type = 'Lock'"
+ " AND pid <> pg_backend_pid()")) {
waiting.setString(1, databaseName);
try (ResultSet count = waiting.executeQuery()) {
return count.next() && count.getInt(1) > 0;
}
}
}
/** The version the history ends on and whether every index on the table is usable. */
private static String state(DataSource database) {
JdbcTemplate jdbc = new JdbcTemplate(database);
@@ -22,11 +22,13 @@ import java.util.regex.Pattern;
* statement it is refused by Flyway before any of the script runs when nothing configures the file
* (Flyway's default {@code mixed=false}); with the {@code executeInTransaction=false} that {@link
* #validate} asks for, Flyway runs and commits each statement on its own, and a failed build leaves
* the statements before it applied (round 7, RR7-03-04). Alone, a failed build leaves an invalid
* index and its version's failed history row, which Flyway refuses the next deploy over until the
* row is gone: {@code docs/runbooks/migration-failed.md} ("Recover a failed concurrent index
* migration") drops the index and deletes that one row ({@code
* postgresqlConcurrentIndexMigrationIntegrationTest} runs it; round 7, RR7-03-03).
* the statements before it applied (round 7, RR7-03-04). Alone, a failed statement leaves an
* invalid index and its version's failed history row, which Flyway refuses the next deploy over
* until the row is gone: {@code docs/runbooks/migration-failed.md} ("Recover a failed concurrent
* index migration") deletes that one row, after dropping the index a failed build or rebuild left.
* A failed {@code DROP INDEX CONCURRENTLY} leaves invalid the index it was dropping, and the deploy
* after the row is gone drops it ({@code postgresqlConcurrentIndexMigrationIntegrationTest} runs
* all three; round 7, RR7-03-03; round 8, RR8-03-01).
*
* <p>Outside a transaction, {@code CREATE INDEX CONCURRENTLY} and {@code REINDEX ... CONCURRENTLY}
* still wait for every transaction older than themselves. They finish because the template's Flyway
@@ -84,12 +86,12 @@ public final class ConcurrentIndexMigrationInspector {
/**
* Fails when a concurrent-index migration also contains other statements.
*
* <p>A concurrent index build can fail and leave an invalid index and its version's failed
* <p>A concurrent index statement can fail and leave an invalid index and its version's failed
* history row behind. When the migration did nothing else, recovering is the runbook's few
* statements: drop the index, delete that one failed row, deploy again ({@code
* docs/runbooks/migration-failed.md}, "Recover a failed concurrent index migration"). When it did
* more, Flyway ran the file outside a transaction and has committed the statements before the
* build, and that partial state has to be reconstructed by hand.
* statements: drop the index a failed build left, delete that one failed row, deploy again
* ({@code docs/runbooks/migration-failed.md}, "Recover a failed concurrent index migration").
* When it did more, Flyway ran the file outside a transaction and has committed the statements
* before the failed one, and that partial state has to be reconstructed by hand.
*/
public void requireIsolatedStatement(MigrationResource migration) {
Objects.requireNonNull(migration, "migration");
@@ -109,9 +111,9 @@ public final class ConcurrentIndexMigrationInspector {
migration.name()
+ " mixes a concurrent index statement with other statements; outside a"
+ " transaction Flyway commits each of them on its own, so a failed build leaves the"
+ " others applied. Alone, a failed build is recovered by dropping its invalid index"
+ " and deleting its version's failed history row (docs/runbooks/migration-failed.md,"
+ " \"Recover a failed concurrent index migration\")");
+ " others applied. Alone, a failed statement is recovered by deleting its version's"
+ " failed history row (docs/runbooks/migration-failed.md, \"Recover a failed"
+ " concurrent index migration\")");
}
}
}