문제 (재리뷰 round7 영속성 RR7-03-01~05) - 실패한 동시 인덱스 마이그레이션의 복구를 가이드·검사기·학습 문서가 "DROP INDEX 하나"로 안내한다. Flyway는 트랜잭션 밖에서 실패한 버전의 이력 행(success = false)을 남기고, 다음 배포의 validate가 그 행 때문에 다시 멈춘다. runbook에는 이 경우가 없고 이력 수정은 금지다. docs/jpa/runbooks.md "An invalid index exists"도 같은 이유로 재적용에서 멈춘다. - 검사기 Javadoc·학습 문서가 "섞인 파일은 부분 적용되지 않는다"고 쓴다. 가이드가 요구하는 executeInTransaction=false를 붙이면 Flyway가 문장마다 커밋해 앞 문장이 남는다. - 가이드·검사기 Javadoc·실패 메시지·학습 문서가 "Flyway는 REINDEX … CONCURRENTLY를 트랜잭션 안에서 돌린다"고 쓴다. REINDEX SCHEMA|DATABASE CONCURRENTLY는 Flyway가 스스로 트랜잭션 밖에서 돌린다. - LockWaitMigrationInspector는 .sql.conf가 executeInTransaction=true면 SET LOCAL 뒤의 VACUUM FULL·REINDEX SCHEMA를 통과시킨다. Javadoc은 "늘 거부"라고 쓴다. - ledger 잠금 대기 IT가 동시 인덱스 파일을 "이 테스트에 예외로 적으라"고 안내하지만 예외를 적을 곳이 없다. 같은 원인으로 ledger의 다른 IT 두 개 (GrpcOperationLedgerPostgresIT, JpaGrpcOperationLedgerContractIT)도 모든 파일을 한 문자열로 실행해, 동시 인덱스 파일이 생기면 "cannot be executed within a pipeline"으로 실패한다. 고른 방법 - 복구(RR7-03-03): 소유자 결정(2026-11-06)대로 migration-failed.md에 "Recover a failed concurrent index migration" 절을 둔다. 진행 중 빌드 확인 → 무효 인덱스 DROP INDEX CONCURRENTLY → 원인 정리 → 그 버전의 실패 행만 DELETE → 재배포. "Do not"에 이 한 가지 예외를 적고 flyway repair는 계속 금지한다. SQL 세 블록에 runbook-sql 표시를 달고 postgresqlConcurrentIndexMigrationIntegrationTest가 runbook에서 읽어 실 PostgreSQL에서 돌린다(outbox runbook과 같은 방식). 가이드·검사기 Javadoc·메시지·학습 문서·jpa/runbooks.md는 그 절을 가리킨다. - 섞인 파일(RR7-03-04)·REINDEX(RR7-03-02): 문장을 고치고, 같은 IT와 잠금 대기 IT에 사실을 고정한다(섞인 파일 두 경우, REINDEX 다섯 형태). - 검사기(RR7-03-05): 트랜잭션 안에서 돌 수 없는 잠금 문장은 .sql.conf와 상관없이 거부한다(리뷰 권장안). 버린 안: Javadoc만 좁히기 — 그 파일은 PostgreSQL이 배포 때 25001로 거부하므로 검사기가 먼저 알려 주는 쪽이 맞다. 잠그지 않는 문장(plain VACUUM 등)은 이 검사기의 범위가 아니라 그대로 둔다. - ledger(RR7-03-01): IT에 RUN_OUTSIDE_A_TRANSACTION(파일 → 이유) 목록을 두고 잠금 대기 케이스에서 뺀다. 목록의 파일은 같은 IT가 동시 인덱스 문장 하나·executeInTransaction=false인지 ConcurrentIndexMigration Inspector로 본다. optional 빌드는 testkit에 의존할 수 없어 sharedMigrationFileSources가 그 파일과 MigrationResource를 복사한다 (VersionedMigrationFiles와 같은 방식). ledger IT 셋은 마이그레이션을 파일마다 따로 실행한다(Flyway처럼). 버린 안: 안내만 "다룰 수 없다"로 바꾸기 — 다음 동시 인덱스 파일이 생길 때 같은 질문이 다시 남는다. 위험·한계 - 검사기 변경으로 서비스의 자기 스트림 검사가 새로 실패할 수 있다. 걸리는 파일은 PostgreSQL이 배포 때 거부하는 파일뿐이라 배포된 적이 없다(업그레이드 안내 4.1에 적음). - runbook의 DELETE는 운영자가 version을 옮겨 적는다. NOT success라 성공한 행은 지우지 못하지만, 다른 버전의 실패 행을 잘못 적으면 그 행이 지워진다. 한 행만 지워야 하고 0행이면 다시 분류하라고 적었다. - ledger 복사 파일 두 개도 JDK만 써야 한다(각 Javadoc에 적음). - Spotless 검사는 같은 worktree에서 spotlessApply가 변경 없이 끝난 직후라 UP-TO-DATE로 처리됐다. 검증 (desktop, 공유 잠금, 로그 ~/r8-f3-logs/) RED — 기준73d39212+ 이 커밋의 테스트(+ 기준 문서 주장을 단언하는 임시 메서드, + 임시 ledger V004): - persistence-jpa test --tests LockWaitMigrationInspectorTest: 80건 중 2건 실패, 검사기가 빈 목록(통과)을 돌려줌 (red-unit.log) - postgresqlMigrationLockWaitIntegrationTest: 47건 중 4건 실패 — 옛 주장 대로면 거부여야 할 REINDEX SCHEMA|DATABASE CONCURRENTLY가 applied, conf=true VACUUM FULL·REINDEX SCHEMA를 PostgreSQL은 25001로 거부하는데 검사기는 통과 (red2-lockwait.log) - postgresqlConcurrentIndexMigrationIntegrationTest: 9건 중 3건 실패 — 옛 절차(DROP INDEX) 뒤 재배포가 "Detected failed migration to version 2", conf를 붙인 섞인 파일이 행 3001·무효 인덱스·실패 이력을 남김, runbook 블록이 없어 복구 테스트 준비 실패(RED로 세지 않음) (red2-concurrent.log) - ledger integrationTest(V004 동시 인덱스): 6건 중 4건 실패 — 잠금 대기 IT "name it as an exception here", 다른 IT 둘 "cannot be executed within a pipeline" (red-ledger.log) GREEN — 이 커밋의 트리b49369e7: - :adapter:outbound:persistence-jpa:check rc=0 (test 666건 실패 0), :testQualityCheck rc=0 - postgresqlMigrationLockWaitIntegrationTest rc=0 (42건 실패 0) - postgresqlConcurrentIndexMigrationIntegrationTest rc=0 (7건 실패 0) - -p optional-platforms :grpc:grpc-operation-ledger-jpa:check integrationTest testQualityCheck rc=0 (IT 3+2+42건 실패 0) - :app-bootstrap:documentContractTest rc=0 (66건 실패 0) - 두 빌드의 verifyStrictLaneStages --dry-run rc=0 - ledger 탐침(임시 V004): 목록에 없으면 잠금 대기 케이스만 실패하고 메시지가 RUN_OUTSIDE_A_TRANSACTION을 가리킴(48건 중 1건), 목록에 넣으면 rc=0, 목록에 넣고 SET LOCAL을 섞으면 목록 검사가 "mixes a concurrent index statement"로 거부 - python3 scripts/check-work-docs.py --links docs: 깨진 링크 0 Refs: docs/reviews/2026-11-03-rereview/round7/03-persistence.md
176 lines
8.8 KiB
Markdown
176 lines
8.8 KiB
Markdown
---
|
||
title: Runbook — MIGRATION_FAILED (DB 마이그레이션 실패)
|
||
category: INTERNAL
|
||
error_codes: [MIGRATION_FAILED]
|
||
severity: P1
|
||
owner: oncall
|
||
last_updated: 2026-11-07
|
||
status: active
|
||
---
|
||
|
||
# Runbook: MIGRATION_FAILED (`runbook://migration/failed`)
|
||
|
||
## Symptoms
|
||
|
||
- Container exits with code 70 (migration failure exit)
|
||
- Structured log with `error.code=MIGRATION_FAILED`, `startup.phase=migration`
|
||
- App refuses to start (fail-fast)
|
||
- JPA capability adapter refuses activation because its
|
||
`capability_schema_registry.lifecycle_state` is not `ACTIVE`
|
||
|
||
## Diagnosis
|
||
|
||
1. Stop rollout and keep the failed revision out of readiness. Do not route traffic to a partially
|
||
migrated instance.
|
||
2. Identify the exact stream from `src/config/jpa/readiness-cards.yaml`. Each stream has an
|
||
independent `location` and `history-table`; do not infer ownership from a broad
|
||
`classpath:db/migration` scan.
|
||
3. From a privileged migration session, capture the stream state before changing anything:
|
||
|
||
```sql
|
||
select installed_rank, version, description, success
|
||
from <owned_history_table>
|
||
order by installed_rank;
|
||
|
||
select capability_id, installation_origin, core_epoch, feature_revision, lifecycle_state
|
||
from capability_schema_registry
|
||
where capability_id = '<card-id>';
|
||
```
|
||
|
||
4. Check whether any owned relation was created without a successful history entry. Compare only
|
||
against the owned tables in the reviewed migration; do not drop unrelated relations.
|
||
5. Classify the failure:
|
||
- lock/statement timeout: remove the blocker or reduce rollout concurrency, then rerun. A
|
||
migration that locks a table already in use gives up after its own `SET LOCAL lock_timeout`
|
||
(5 s; SQLState `55P03` `lock_not_available`) and rolls back whole, so the history ends one
|
||
version below it. The blocker is a transaction that touched the table and is still open —
|
||
often an idle-in-transaction session. Find it by the lock it holds on that table (`<table>`
|
||
is the table the failing statement locks; for a `REFERENCES`, also the table it references):
|
||
|
||
```sql
|
||
SELECT DISTINCT a.pid, a.state, a.xact_start, a.query
|
||
FROM pg_locks l JOIN pg_stat_activity a USING (pid)
|
||
WHERE l.relation = '<table>'::regclass AND l.granted AND a.pid <> pg_backend_pid()
|
||
ORDER BY a.xact_start;
|
||
```
|
||
|
||
`query` is only that session's last statement, not the one that touched the table, so do not
|
||
look for the table name in it. Listing every long transaction instead names unrelated ones
|
||
(reports, batches) beside the blocker. End the blocker, then rerun
|
||
(`docs/jpa/migration-guide.md` "Bound the lock wait");
|
||
- SQL/data precondition: create a new forward migration that makes the precondition explicit;
|
||
a duplicate outbox order key stopping V21 (`ux_outbox_event_aggregate_order`) is the known
|
||
case — clean it up with `outbox-dead-letter.md` §8 and rerun, the migration stays as it is;
|
||
- checksum mismatch: compare the deployed artifact with the already applied script before
|
||
considering repair;
|
||
- a concurrent index migration failed (`CREATE [UNIQUE] INDEX CONCURRENTLY` or `REINDEX …
|
||
CONCURRENTLY` that Flyway ran outside a transaction): a duplicate value for a unique
|
||
index, a deadlock, `statement_timeout` or a cancelled build. Outside a transaction nothing
|
||
rolled back: the migration left an invalid index and a history row with
|
||
`success = false` for that version, and every later deploy fails validation ("Detected failed
|
||
migration to version …") until the row is gone. Dropping the index alone does not clear it.
|
||
Follow "Recover a failed concurrent index migration" below;
|
||
- connection/TLS failure: fix transport or credentials without changing Flyway history.
|
||
|
||
## Action
|
||
|
||
1. Prefer forward recovery. Fix the environmental blocker or add a new immutable migration, then
|
||
rerun the same owned stream with its exact history table.
|
||
2. For an optional stream that never installed successfully, keep the capability marker absent and
|
||
the runtime adapter disabled until migration succeeds.
|
||
3. After a successful migration, validate:
|
||
- the history contains only successful expected versions;
|
||
- `core_epoch` and `feature_revision` match the readiness registry;
|
||
- the marker is `INSTALLED_INACTIVE`;
|
||
- owned objects and constraints exist.
|
||
4. Change the marker to `ACTIVE` only after the compatible application revision is deployed and its
|
||
readiness check succeeds. Disabling or rolling back application code changes the marker to
|
||
`INSTALLED_INACTIVE`; it does not drop history or owned data.
|
||
5. Re-run the readiness lanes before promoting (every card's scenarios, no skips):
|
||
|
||
```bash
|
||
cd src
|
||
./gradlew :adapter:outbound:persistence-jpa:integrationTestLaneCheck --console=plain
|
||
```
|
||
|
||
6. Record the failed revision, stream/history table, root cause, recovery migration, elapsed time
|
||
and verification artifact in the incident.
|
||
|
||
Do not:
|
||
|
||
- edit an already applied migration;
|
||
- delete or rewrite Flyway history to make validation green. The one exception is the failed row
|
||
of a failed concurrent index migration, deleted as "Recover a failed concurrent index migration"
|
||
below says — that row and nothing else;
|
||
- run `flyway repair` before checksum provenance is proven and reviewed;
|
||
- use `clean`, destructive rollback, or schema-wide restore as the first response;
|
||
- mark a capability `ACTIVE` before its migration and adapter readiness succeed.
|
||
|
||
If commit outcome was indeterminate during the failure, reconcile by the application
|
||
`OperationId`/idempotency reference before retrying business work. Never blind-retry a commit whose
|
||
result is unknown.
|
||
|
||
## Recover a failed concurrent index migration
|
||
|
||
The one case where this runbook changes Flyway history (owner decision, 2026-11-06). It applies to a
|
||
migration that holds a single concurrent index statement which Flyway ran outside a transaction —
|
||
with the `executeInTransaction=false` the guide asks for, or on its own for `CREATE`/`DROP INDEX
|
||
CONCURRENTLY` (`docs/jpa/migration-guide.md` "Concurrent index builds") — and whose row for this
|
||
version in the stream's history table says `success = false`. A migration that mixed other
|
||
statements in has already committed them: reconstruct that state with the database owner instead.
|
||
|
||
1. Confirm no build is still running. An index that is still being built looks invalid in the
|
||
catalog too, and dropping it races the build. This query returns no rows when none is:
|
||
|
||
<!-- runbook-sql: failed-concurrent-index-builds -->
|
||
```sql
|
||
SELECT pid, relid::regclass AS table_name, index_relid::regclass AS index_name, phase
|
||
FROM pg_stat_progress_create_index
|
||
WHERE datname = current_database();
|
||
```
|
||
|
||
2. Drop the invalid index the build left (`docs/jpa/runbooks.md` "An invalid index exists" lists
|
||
them). Run it on its own, not inside `BEGIN`: `CONCURRENTLY` refuses a transaction block.
|
||
|
||
<!-- runbook-sql: failed-concurrent-index-drop -->
|
||
```sql
|
||
DROP INDEX CONCURRENTLY IF EXISTS <invalid_index>;
|
||
```
|
||
|
||
3. Remove the cause. For a unique index, clean up the duplicate rows with the data owner (a forward
|
||
fix, never by editing the migration). For a deadlock or a timeout, end or wait out the
|
||
transaction that held the build.
|
||
4. Delete that version's failed row from the stream's history table, and nothing else. `<version>`
|
||
is the `version` column of the failed row as the Diagnosis query shows it (`22.1` for
|
||
`V22_1__…`). It must delete exactly one row; if it deletes none, this is not the failure this
|
||
section is for, so classify it again.
|
||
|
||
<!-- runbook-sql: failed-concurrent-index-history-row -->
|
||
```sql
|
||
DELETE FROM <owned_history_table>
|
||
WHERE version = '<version>' AND NOT success;
|
||
```
|
||
|
||
5. Deploy again. Flyway runs the same migration from the start and builds the index.
|
||
|
||
The failed row is the record of a version that never applied, so deleting it leaves the order and
|
||
the checksum of every applied version as they were. Do not use `flyway repair` for this: it also
|
||
rewrites the checksums and descriptions of the remaining rows to match the scripts on disk, which
|
||
removes the evidence a checksum mismatch carries. `:adapter:outbound:persistence-jpa:postgresqlConcurrentIndexMigrationIntegrationTest`
|
||
runs the three SQL blocks above, read from this file, against real PostgreSQL: a unique build fails
|
||
on a duplicate, steps 1–4 run, and the next deploy applies the version with the index valid and the
|
||
earlier version's checksum unchanged.
|
||
|
||
## Escalation
|
||
|
||
- P1 immediate: the required application revision cannot become ready.
|
||
- Escalate to the database owner before Flyway history repair, destructive DDL, point-in-time
|
||
recovery, or primary failover.
|
||
- R3 restore/PITR and failover rehearsal requires a target-like backup topology; local
|
||
Testcontainers evidence is not a substitute.
|
||
|
||
---
|
||
|
||
This runbook is forward-only. The reviewed migration artifact and the readiness lanes' JUnit
|
||
results from CI are the audit sources.
|