문제 (재리뷰 round8 RR8-03-01) - migration-failed.md "Recover a failed concurrent index migration"은 CREATE·DROP·REINDEX 동시 문장 셋을 대상으로 두고, 2단계에서 "빌드가 남긴 무효 인덱스를 지운다"고 한다. 실패한 DROP INDEX CONCURRENTLY가 남긴 무효 인덱스는 그 마이그레이션이 지우려던 인덱스다. 절차대로 지우고 실패 행을 지운 뒤 다시 배포하면 같은 문장이 42704(index does not exist)로 실패하고 실패 행이 다시 생긴다. 운영자는 처음과 같은 막힌 상태로 돌아간다. - 고치다 같은 절에서 하나 더 찾았다. 2단계(DROP INDEX CONCURRENTLY)가 3단계(원인 정리) 앞이다. 취소·timeout이 원인이면 그 열린 트랜잭션이 운영자의 DROP INDEX CONCURRENTLY도 붙잡아, 2단계가 3단계를 기다린다. - runbook을 실행하는 IT는 중복으로 실패한 unique CREATE 한 경우만 돌렸다. 고른 방법 - 리뷰 권장안: 무효 인덱스를 지우는 단계를 문장 종류로 나눈다. CREATE는 그 인덱스를, REINDEX는 _ccnew(또는 _ccold) 사본을 지운다. DROP INDEX CONCURRENTLY면 이 단계를 건너뛴다. 다시 배포한 같은 문장이 그 인덱스를 지운다(PostgreSQL은 중간에 실패한 DROP INDEX CONCURRENTLY의 재시도를 받는다). 5단계 문장도 "builds the index"에서 세 경우로 고친다. - 단계 순서를 "빌드 확인 → 원인 정리 → 무효 인덱스 삭제 → 실패 행 삭제 → 재배포"로 바꾼다. 실패 행만 지운다는 소유자 결정(2026-11-06)은 그대로다. - IT에 두 경우를 더한다. 열린 쓰기를 기다리다 취소된 REINDEX와 열린 읽기를 기다리다 취소된 DROP에 runbook SQL을 runbook에서 읽어 돌리고, 재배포 뒤 이력 [1 true, 2 true], 무효 인덱스 없음, V1 checksum 그대로를 단언한다. 대기를 30초 기다리지 않고, 문장이 잠금을 기다리는 것을 본 즉시 취소하는 도우미(cancelledOnceItWaits)를 둔다. - 같은 절차를 요약한 곳을 같이 고친다: docs/jpa/runbooks.md "An invalid index exists", migration-guide "Concurrent index builds"(DROP이 무엇을 기다리는지 포함), 학습 문서 8.2절, ConcurrentIndexMigrationInspector의 Javadoc과 실패 메시지, 업그레이드 안내 5절 행(그 범위가 아직 main에 없어 행을 최종 동작으로 고치고 머리 목록에 더함). - 버린 대안: 가이드와 검사기가 DROP INDEX CONCURRENTLY에 IF EXISTS를 요구한다. 이미 배포된 파일은 고칠 수 없어 새 파일에만 통하고, 서비스의 기존 마이그레이션 검사를 새로 실패시킨다. 위험·한계 - 운영자가 실패한 문장의 종류를 마이그레이션 파일에서 읽어야 한다. runbook이 이력 행만으로는 알 수 없는 것을 요구한다. - REINDEX가 swap 뒤에 실패해 _ccold가 남는 경우는 문서로만 다뤘다(실행 재현 없음). PostgreSQL 문서의 권장 복구와 같다. - 단계 번호가 바뀌었다. 리뷰 기록이 옛 번호(2단계 = drop)로 적은 곳은 그때의 기록이라 두었다. RED (desktop, 기준3eb3b067+ 임시 테스트, 로그 ~/r9-g3-logs/red-concurrent.log) - postgresqlConcurrentIndexMigrationIntegrationTest rc=1, 9건 중 2건 실패 - redAFailedConcurrentDropIsRecoveredByTheBaseRunbook: 기준 runbook 1·2·4단계 뒤 재배포가 "42704 ERROR: index "ix_probe_id" does not exist"로 실패, 이력 [1 true, 2 false] 그대로 - redTheBaseRunbookDropsBeforeItEndsTheTransaction: 기준 순서의 2단계 (_ccnew drop)가 열린 쓰기 트랜잭션을 기다려 취소됨 - 나머지 7건(기존) 통과 검증 (desktop, 이 커밋의 트리7cfaa83c, --no-build-cache, 로그 ~/r9-g3-logs/) - :adapter:outbound:persistence-jpa:check rc=0, test 666건 실패·skip 0 - postgresqlConcurrentIndexMigrationIntegrationTest rc=0, 9건(새 2건 포함) - :adapter:outbound:persistence-jpa:testQualityCheck rc=0 - :app-bootstrap:documentContractTest rc=0, 67건(RunbookCoverageContractTest 포함) - -p optional-platforms :grpc:grpc-operation-ledger-jpa:integrationTest rc=0, 47건(검사기 사본이 바뀌어서) - FROM-CACHE 0 - 상대 링크(노트북): python3 scripts/check-work-docs.py --links로 바꾼 문서 5개(migration-failed.md, jpa/runbooks.md, migration-guide.md, 학습 문서, 업그레이드 안내) rc=0, 깨진 링크 0 Refs: docs/reviews/2026-11-03-rereview/round8/03-migration.md RR8-03-01 Fixes:6ec0c4f7fa("fix(migration): 실패한 동시 인덱스 마이그레이션 복구와 검사기 설명을 실제 동작에 맞춘다")
188 lines
9.8 KiB
Markdown
188 lines
9.8 KiB
Markdown
---
|
||
title: Runbook — MIGRATION_FAILED (DB 마이그레이션 실패)
|
||
category: INTERNAL
|
||
error_codes: [MIGRATION_FAILED]
|
||
severity: P1
|
||
owner: oncall
|
||
last_updated: 2026-11-07
|
||
status: active
|
||
---
|
||
|
||
# Runbook: MIGRATION_FAILED (`runbook://migration/failed`)
|
||
|
||
## Symptoms
|
||
|
||
- Container exits with code 70 (migration failure exit)
|
||
- Structured log with `error.code=MIGRATION_FAILED`, `startup.phase=migration`
|
||
- App refuses to start (fail-fast)
|
||
- JPA capability adapter refuses activation because its
|
||
`capability_schema_registry.lifecycle_state` is not `ACTIVE`
|
||
|
||
## Diagnosis
|
||
|
||
1. Stop rollout and keep the failed revision out of readiness. Do not route traffic to a partially
|
||
migrated instance.
|
||
2. Identify the exact stream from `src/config/jpa/readiness-cards.yaml`. Each stream has an
|
||
independent `location` and `history-table`; do not infer ownership from a broad
|
||
`classpath:db/migration` scan.
|
||
3. From a privileged migration session, capture the stream state before changing anything:
|
||
|
||
```sql
|
||
select installed_rank, version, description, success
|
||
from <owned_history_table>
|
||
order by installed_rank;
|
||
|
||
select capability_id, installation_origin, core_epoch, feature_revision, lifecycle_state
|
||
from capability_schema_registry
|
||
where capability_id = '<card-id>';
|
||
```
|
||
|
||
4. Check whether any owned relation was created without a successful history entry. Compare only
|
||
against the owned tables in the reviewed migration; do not drop unrelated relations.
|
||
5. Classify the failure:
|
||
- lock/statement timeout: remove the blocker or reduce rollout concurrency, then rerun. A
|
||
migration that locks a table already in use gives up after its own `SET LOCAL lock_timeout`
|
||
(5 s; SQLState `55P03` `lock_not_available`) and rolls back whole, so the history ends one
|
||
version below it. The blocker is a transaction that touched the table and is still open —
|
||
often an idle-in-transaction session. Find it by the lock it holds on that table (`<table>`
|
||
is the table the failing statement locks; for a `REFERENCES`, also the table it references):
|
||
|
||
```sql
|
||
SELECT DISTINCT a.pid, a.state, a.xact_start, a.query
|
||
FROM pg_locks l JOIN pg_stat_activity a USING (pid)
|
||
WHERE l.relation = '<table>'::regclass AND l.granted AND a.pid <> pg_backend_pid()
|
||
ORDER BY a.xact_start;
|
||
```
|
||
|
||
`query` is only that session's last statement, not the one that touched the table, so do not
|
||
look for the table name in it. Listing every long transaction instead names unrelated ones
|
||
(reports, batches) beside the blocker. End the blocker, then rerun
|
||
(`docs/jpa/migration-guide.md` "Bound the lock wait");
|
||
- SQL/data precondition: create a new forward migration that makes the precondition explicit;
|
||
a duplicate outbox order key stopping V21 (`ux_outbox_event_aggregate_order`) is the known
|
||
case — clean it up with `outbox-dead-letter.md` §8 and rerun, the migration stays as it is;
|
||
- checksum mismatch: compare the deployed artifact with the already applied script before
|
||
considering repair;
|
||
- a concurrent index migration failed (`CREATE [UNIQUE] INDEX CONCURRENTLY`, `DROP INDEX
|
||
CONCURRENTLY` or `REINDEX … CONCURRENTLY` that Flyway ran outside a transaction): a duplicate
|
||
value for a unique index, a deadlock, `statement_timeout` or a statement cancelled while it
|
||
waited for an open transaction. Outside a transaction nothing rolled back: the migration left
|
||
an invalid index and a history row with `success = false` for that version, and every later
|
||
deploy fails validation ("Detected failed migration to version …") until the row is gone.
|
||
Dropping the index alone does not clear it. Follow "Recover a failed concurrent index
|
||
migration" below;
|
||
- connection/TLS failure: fix transport or credentials without changing Flyway history.
|
||
|
||
## Action
|
||
|
||
1. Prefer forward recovery. Fix the environmental blocker or add a new immutable migration, then
|
||
rerun the same owned stream with its exact history table.
|
||
2. For an optional stream that never installed successfully, keep the capability marker absent and
|
||
the runtime adapter disabled until migration succeeds.
|
||
3. After a successful migration, validate:
|
||
- the history contains only successful expected versions;
|
||
- `core_epoch` and `feature_revision` match the readiness registry;
|
||
- the marker is `INSTALLED_INACTIVE`;
|
||
- owned objects and constraints exist.
|
||
4. Change the marker to `ACTIVE` only after the compatible application revision is deployed and its
|
||
readiness check succeeds. Disabling or rolling back application code changes the marker to
|
||
`INSTALLED_INACTIVE`; it does not drop history or owned data.
|
||
5. Re-run the readiness lanes before promoting (every card's scenarios, no skips):
|
||
|
||
```bash
|
||
cd src
|
||
./gradlew :adapter:outbound:persistence-jpa:integrationTestLaneCheck --console=plain
|
||
```
|
||
|
||
6. Record the failed revision, stream/history table, root cause, recovery migration, elapsed time
|
||
and verification artifact in the incident.
|
||
|
||
Do not:
|
||
|
||
- edit an already applied migration;
|
||
- delete or rewrite Flyway history to make validation green. The one exception is the failed row
|
||
of a failed concurrent index migration, deleted as "Recover a failed concurrent index migration"
|
||
below says — that row and nothing else;
|
||
- run `flyway repair` before checksum provenance is proven and reviewed;
|
||
- use `clean`, destructive rollback, or schema-wide restore as the first response;
|
||
- mark a capability `ACTIVE` before its migration and adapter readiness succeed.
|
||
|
||
If commit outcome was indeterminate during the failure, reconcile by the application
|
||
`OperationId`/idempotency reference before retrying business work. Never blind-retry a commit whose
|
||
result is unknown.
|
||
|
||
## Recover a failed concurrent index migration
|
||
|
||
The one case where this runbook changes Flyway history (owner decision, 2026-11-06). It applies to a
|
||
migration that holds a single concurrent index statement which Flyway ran outside a transaction —
|
||
with the `executeInTransaction=false` the guide asks for, or on its own for `CREATE`/`DROP INDEX
|
||
CONCURRENTLY` (`docs/jpa/migration-guide.md` "Concurrent index builds") — and whose row for this
|
||
version in the stream's history table says `success = false`. A migration that mixed other
|
||
statements in has already committed them: reconstruct that state with the database owner instead.
|
||
|
||
1. Confirm no build is still running. An index that is still being built looks invalid in the
|
||
catalog too, and dropping it races the build. This query returns no rows when none is:
|
||
|
||
<!-- runbook-sql: failed-concurrent-index-builds -->
|
||
```sql
|
||
SELECT pid, relid::regclass AS table_name, index_relid::regclass AS index_name, phase
|
||
FROM pg_stat_progress_create_index
|
||
WHERE datname = current_database();
|
||
```
|
||
|
||
2. Remove the cause. For a unique index, clean up the duplicate rows with the data owner (a forward
|
||
fix, never by editing the migration). For a deadlock, a timeout or a cancelled statement, end or
|
||
wait out the transaction that held it: a build or a rebuild waits for every transaction older
|
||
than itself, a drop for every transaction that holds a lock on its table. Do this before step 3,
|
||
whose `DROP INDEX CONCURRENTLY` waits for the same transactions.
|
||
3. Drop the invalid index a failed build or rebuild left: the index a `CREATE [UNIQUE] INDEX
|
||
CONCURRENTLY` names, or the copy a `REINDEX … CONCURRENTLY` made, named after the index with
|
||
`_ccnew` (or `_ccold`) on the end (`docs/jpa/runbooks.md` "An invalid index exists" lists them).
|
||
Run it on its own, not inside `BEGIN`: `CONCURRENTLY` refuses a transaction block.
|
||
**Skip this step when the failed statement is `DROP INDEX CONCURRENTLY`.** The invalid index it
|
||
left is the index the migration drops, and the deploy in step 5 runs the same statement, which
|
||
drops it. Dropped by hand first, that statement fails with `42704` ("index … does not exist")
|
||
and leaves a new failed row.
|
||
|
||
<!-- runbook-sql: failed-concurrent-index-drop -->
|
||
```sql
|
||
DROP INDEX CONCURRENTLY IF EXISTS <invalid_index>;
|
||
```
|
||
|
||
4. Delete that version's failed row from the stream's history table, and nothing else. `<version>`
|
||
is the `version` column of the failed row as the Diagnosis query shows it (`22.1` for
|
||
`V22_1__…`). It must delete exactly one row; if it deletes none, this is not the failure this
|
||
section is for, so classify it again.
|
||
|
||
<!-- runbook-sql: failed-concurrent-index-history-row -->
|
||
```sql
|
||
DELETE FROM <owned_history_table>
|
||
WHERE version = '<version>' AND NOT success;
|
||
```
|
||
|
||
5. Deploy again. Flyway runs the same migration from the start: it builds the index, rebuilds it,
|
||
or drops the index the failed drop left.
|
||
|
||
The failed row is the record of a version that never applied, so deleting it leaves the order and
|
||
the checksum of every applied version as they were. Do not use `flyway repair` for this: it also
|
||
rewrites the checksums and descriptions of the remaining rows to match the scripts on disk, which
|
||
removes the evidence a checksum mismatch carries. `:adapter:outbound:persistence-jpa:postgresqlConcurrentIndexMigrationIntegrationTest`
|
||
runs the three SQL blocks above, read from this file, against real PostgreSQL for each statement
|
||
this section covers: a unique build that fails on a duplicate, and a rebuild and a drop cancelled
|
||
while they wait for an open transaction. Steps 1–4 run in order, step 3 for the build and the
|
||
rebuild only, and the next deploy applies the version with no invalid index left and the earlier
|
||
version's checksum unchanged.
|
||
|
||
## Escalation
|
||
|
||
- P1 immediate: the required application revision cannot become ready.
|
||
- Escalate to the database owner before Flyway history repair, destructive DDL, point-in-time
|
||
recovery, or primary failover.
|
||
- R3 restore/PITR and failover rehearsal requires a target-like backup topology; local
|
||
Testcontainers evidence is not a substitute.
|
||
|
||
---
|
||
|
||
This runbook is forward-only. The reviewed migration artifact and the readiness lanes' JUnit
|
||
results from CI are the audit sources.
|