Files
clean-architecture-backend-…/docs/runbooks/migration-failed.md
T
DongHyeonka 8ebe98e69a fix(migration): 실패한 DROP INDEX CONCURRENTLY를 runbook 절차로 복구되게 한다
문제 (재리뷰 round8 RR8-03-01)
- migration-failed.md "Recover a failed concurrent index migration"은
  CREATE·DROP·REINDEX 동시 문장 셋을 대상으로 두고, 2단계에서 "빌드가 남긴
  무효 인덱스를 지운다"고 한다. 실패한 DROP INDEX CONCURRENTLY가 남긴 무효
  인덱스는 그 마이그레이션이 지우려던 인덱스다. 절차대로 지우고 실패 행을
  지운 뒤 다시 배포하면 같은 문장이 42704(index does not exist)로 실패하고
  실패 행이 다시 생긴다. 운영자는 처음과 같은 막힌 상태로 돌아간다.
- 고치다 같은 절에서 하나 더 찾았다. 2단계(DROP INDEX CONCURRENTLY)가
  3단계(원인 정리) 앞이다. 취소·timeout이 원인이면 그 열린 트랜잭션이
  운영자의 DROP INDEX CONCURRENTLY도 붙잡아, 2단계가 3단계를 기다린다.
- runbook을 실행하는 IT는 중복으로 실패한 unique CREATE 한 경우만 돌렸다.

고른 방법
- 리뷰 권장안: 무효 인덱스를 지우는 단계를 문장 종류로 나눈다. CREATE는
  그 인덱스를, REINDEX는 _ccnew(또는 _ccold) 사본을 지운다. DROP INDEX
  CONCURRENTLY면 이 단계를 건너뛴다. 다시 배포한 같은 문장이 그 인덱스를
  지운다(PostgreSQL은 중간에 실패한 DROP INDEX CONCURRENTLY의 재시도를
  받는다). 5단계 문장도 "builds the index"에서 세 경우로 고친다.
- 단계 순서를 "빌드 확인 → 원인 정리 → 무효 인덱스 삭제 → 실패 행 삭제 →
  재배포"로 바꾼다. 실패 행만 지운다는 소유자 결정(2026-11-06)은 그대로다.
- IT에 두 경우를 더한다. 열린 쓰기를 기다리다 취소된 REINDEX와 열린
  읽기를 기다리다 취소된 DROP에 runbook SQL을 runbook에서 읽어 돌리고,
  재배포 뒤 이력 [1 true, 2 true], 무효 인덱스 없음, V1 checksum 그대로를
  단언한다. 대기를 30초 기다리지 않고, 문장이 잠금을 기다리는 것을 본
  즉시 취소하는 도우미(cancelledOnceItWaits)를 둔다.
- 같은 절차를 요약한 곳을 같이 고친다: docs/jpa/runbooks.md "An invalid
  index exists", migration-guide "Concurrent index builds"(DROP이 무엇을
  기다리는지 포함), 학습 문서 8.2절, ConcurrentIndexMigrationInspector의
  Javadoc과 실패 메시지, 업그레이드 안내 5절 행(그 범위가 아직 main에
  없어 행을 최종 동작으로 고치고 머리 목록에 더함).
- 버린 대안: 가이드와 검사기가 DROP INDEX CONCURRENTLY에 IF EXISTS를
  요구한다. 이미 배포된 파일은 고칠 수 없어 새 파일에만 통하고, 서비스의
  기존 마이그레이션 검사를 새로 실패시킨다.

위험·한계
- 운영자가 실패한 문장의 종류를 마이그레이션 파일에서 읽어야 한다.
  runbook이 이력 행만으로는 알 수 없는 것을 요구한다.
- REINDEX가 swap 뒤에 실패해 _ccold가 남는 경우는 문서로만 다뤘다(실행
  재현 없음). PostgreSQL 문서의 권장 복구와 같다.
- 단계 번호가 바뀌었다. 리뷰 기록이 옛 번호(2단계 = drop)로 적은 곳은 그때의
  기록이라 두었다.

RED (desktop, 기준 3eb3b067 + 임시 테스트, 로그 ~/r9-g3-logs/red-concurrent.log)
- postgresqlConcurrentIndexMigrationIntegrationTest rc=1, 9건 중 2건 실패
  - redAFailedConcurrentDropIsRecoveredByTheBaseRunbook: 기준 runbook
    1·2·4단계 뒤 재배포가 "42704 ERROR: index "ix_probe_id" does not
    exist"로 실패, 이력 [1 true, 2 false] 그대로
  - redTheBaseRunbookDropsBeforeItEndsTheTransaction: 기준 순서의 2단계
    (_ccnew drop)가 열린 쓰기 트랜잭션을 기다려 취소됨
  - 나머지 7건(기존) 통과

검증 (desktop, 이 커밋의 트리 7cfaa83c, --no-build-cache, 로그 ~/r9-g3-logs/)
- :adapter:outbound:persistence-jpa:check rc=0, test 666건 실패·skip 0
- postgresqlConcurrentIndexMigrationIntegrationTest rc=0, 9건(새 2건 포함)
- :adapter:outbound:persistence-jpa:testQualityCheck rc=0
- :app-bootstrap:documentContractTest rc=0, 67건(RunbookCoverageContractTest
  포함)
- -p optional-platforms :grpc:grpc-operation-ledger-jpa:integrationTest
  rc=0, 47건(검사기 사본이 바뀌어서)
- FROM-CACHE 0
- 상대 링크(노트북): python3 scripts/check-work-docs.py --links로 바꾼
  문서 5개(migration-failed.md, jpa/runbooks.md, migration-guide.md,
  학습 문서, 업그레이드 안내) rc=0, 깨진 링크 0

Refs: docs/reviews/2026-11-03-rereview/round8/03-migration.md RR8-03-01
Fixes: 6ec0c4f7fa ("fix(migration): 실패한 동시 인덱스 마이그레이션 복구와 검사기 설명을 실제 동작에 맞춘다")
2026-11-07 08:07:42 +09:00

188 lines
9.8 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
---
title: Runbook — MIGRATION_FAILED (DB 마이그레이션 실패)
category: INTERNAL
error_codes: [MIGRATION_FAILED]
severity: P1
owner: oncall
last_updated: 2026-11-07
status: active
---
# Runbook: MIGRATION_FAILED (`runbook://migration/failed`)
## Symptoms
- Container exits with code 70 (migration failure exit)
- Structured log with `error.code=MIGRATION_FAILED`, `startup.phase=migration`
- App refuses to start (fail-fast)
- JPA capability adapter refuses activation because its
`capability_schema_registry.lifecycle_state` is not `ACTIVE`
## Diagnosis
1. Stop rollout and keep the failed revision out of readiness. Do not route traffic to a partially
migrated instance.
2. Identify the exact stream from `src/config/jpa/readiness-cards.yaml`. Each stream has an
independent `location` and `history-table`; do not infer ownership from a broad
`classpath:db/migration` scan.
3. From a privileged migration session, capture the stream state before changing anything:
```sql
select installed_rank, version, description, success
from <owned_history_table>
order by installed_rank;
select capability_id, installation_origin, core_epoch, feature_revision, lifecycle_state
from capability_schema_registry
where capability_id = '<card-id>';
```
4. Check whether any owned relation was created without a successful history entry. Compare only
against the owned tables in the reviewed migration; do not drop unrelated relations.
5. Classify the failure:
- lock/statement timeout: remove the blocker or reduce rollout concurrency, then rerun. A
migration that locks a table already in use gives up after its own `SET LOCAL lock_timeout`
(5 s; SQLState `55P03` `lock_not_available`) and rolls back whole, so the history ends one
version below it. The blocker is a transaction that touched the table and is still open —
often an idle-in-transaction session. Find it by the lock it holds on that table (`<table>`
is the table the failing statement locks; for a `REFERENCES`, also the table it references):
```sql
SELECT DISTINCT a.pid, a.state, a.xact_start, a.query
FROM pg_locks l JOIN pg_stat_activity a USING (pid)
WHERE l.relation = '<table>'::regclass AND l.granted AND a.pid <> pg_backend_pid()
ORDER BY a.xact_start;
```
`query` is only that session's last statement, not the one that touched the table, so do not
look for the table name in it. Listing every long transaction instead names unrelated ones
(reports, batches) beside the blocker. End the blocker, then rerun
(`docs/jpa/migration-guide.md` "Bound the lock wait");
- SQL/data precondition: create a new forward migration that makes the precondition explicit;
a duplicate outbox order key stopping V21 (`ux_outbox_event_aggregate_order`) is the known
case — clean it up with `outbox-dead-letter.md` §8 and rerun, the migration stays as it is;
- checksum mismatch: compare the deployed artifact with the already applied script before
considering repair;
- a concurrent index migration failed (`CREATE [UNIQUE] INDEX CONCURRENTLY`, `DROP INDEX
CONCURRENTLY` or `REINDEX … CONCURRENTLY` that Flyway ran outside a transaction): a duplicate
value for a unique index, a deadlock, `statement_timeout` or a statement cancelled while it
waited for an open transaction. Outside a transaction nothing rolled back: the migration left
an invalid index and a history row with `success = false` for that version, and every later
deploy fails validation ("Detected failed migration to version …") until the row is gone.
Dropping the index alone does not clear it. Follow "Recover a failed concurrent index
migration" below;
- connection/TLS failure: fix transport or credentials without changing Flyway history.
## Action
1. Prefer forward recovery. Fix the environmental blocker or add a new immutable migration, then
rerun the same owned stream with its exact history table.
2. For an optional stream that never installed successfully, keep the capability marker absent and
the runtime adapter disabled until migration succeeds.
3. After a successful migration, validate:
- the history contains only successful expected versions;
- `core_epoch` and `feature_revision` match the readiness registry;
- the marker is `INSTALLED_INACTIVE`;
- owned objects and constraints exist.
4. Change the marker to `ACTIVE` only after the compatible application revision is deployed and its
readiness check succeeds. Disabling or rolling back application code changes the marker to
`INSTALLED_INACTIVE`; it does not drop history or owned data.
5. Re-run the readiness lanes before promoting (every card's scenarios, no skips):
```bash
cd src
./gradlew :adapter:outbound:persistence-jpa:integrationTestLaneCheck --console=plain
```
6. Record the failed revision, stream/history table, root cause, recovery migration, elapsed time
and verification artifact in the incident.
Do not:
- edit an already applied migration;
- delete or rewrite Flyway history to make validation green. The one exception is the failed row
of a failed concurrent index migration, deleted as "Recover a failed concurrent index migration"
below says — that row and nothing else;
- run `flyway repair` before checksum provenance is proven and reviewed;
- use `clean`, destructive rollback, or schema-wide restore as the first response;
- mark a capability `ACTIVE` before its migration and adapter readiness succeed.
If commit outcome was indeterminate during the failure, reconcile by the application
`OperationId`/idempotency reference before retrying business work. Never blind-retry a commit whose
result is unknown.
## Recover a failed concurrent index migration
The one case where this runbook changes Flyway history (owner decision, 2026-11-06). It applies to a
migration that holds a single concurrent index statement which Flyway ran outside a transaction —
with the `executeInTransaction=false` the guide asks for, or on its own for `CREATE`/`DROP INDEX
CONCURRENTLY` (`docs/jpa/migration-guide.md` "Concurrent index builds") — and whose row for this
version in the stream's history table says `success = false`. A migration that mixed other
statements in has already committed them: reconstruct that state with the database owner instead.
1. Confirm no build is still running. An index that is still being built looks invalid in the
catalog too, and dropping it races the build. This query returns no rows when none is:
<!-- runbook-sql: failed-concurrent-index-builds -->
```sql
SELECT pid, relid::regclass AS table_name, index_relid::regclass AS index_name, phase
FROM pg_stat_progress_create_index
WHERE datname = current_database();
```
2. Remove the cause. For a unique index, clean up the duplicate rows with the data owner (a forward
fix, never by editing the migration). For a deadlock, a timeout or a cancelled statement, end or
wait out the transaction that held it: a build or a rebuild waits for every transaction older
than itself, a drop for every transaction that holds a lock on its table. Do this before step 3,
whose `DROP INDEX CONCURRENTLY` waits for the same transactions.
3. Drop the invalid index a failed build or rebuild left: the index a `CREATE [UNIQUE] INDEX
CONCURRENTLY` names, or the copy a `REINDEX … CONCURRENTLY` made, named after the index with
`_ccnew` (or `_ccold`) on the end (`docs/jpa/runbooks.md` "An invalid index exists" lists them).
Run it on its own, not inside `BEGIN`: `CONCURRENTLY` refuses a transaction block.
**Skip this step when the failed statement is `DROP INDEX CONCURRENTLY`.** The invalid index it
left is the index the migration drops, and the deploy in step 5 runs the same statement, which
drops it. Dropped by hand first, that statement fails with `42704` ("index … does not exist")
and leaves a new failed row.
<!-- runbook-sql: failed-concurrent-index-drop -->
```sql
DROP INDEX CONCURRENTLY IF EXISTS <invalid_index>;
```
4. Delete that version's failed row from the stream's history table, and nothing else. `<version>`
is the `version` column of the failed row as the Diagnosis query shows it (`22.1` for
`V22_1__…`). It must delete exactly one row; if it deletes none, this is not the failure this
section is for, so classify it again.
<!-- runbook-sql: failed-concurrent-index-history-row -->
```sql
DELETE FROM <owned_history_table>
WHERE version = '<version>' AND NOT success;
```
5. Deploy again. Flyway runs the same migration from the start: it builds the index, rebuilds it,
or drops the index the failed drop left.
The failed row is the record of a version that never applied, so deleting it leaves the order and
the checksum of every applied version as they were. Do not use `flyway repair` for this: it also
rewrites the checksums and descriptions of the remaining rows to match the scripts on disk, which
removes the evidence a checksum mismatch carries. `:adapter:outbound:persistence-jpa:postgresqlConcurrentIndexMigrationIntegrationTest`
runs the three SQL blocks above, read from this file, against real PostgreSQL for each statement
this section covers: a unique build that fails on a duplicate, and a rebuild and a drop cancelled
while they wait for an open transaction. Steps 1–4 run in order, step 3 for the build and the
rebuild only, and the next deploy applies the version with no invalid index left and the earlier
version's checksum unchanged.
## Escalation
- P1 immediate: the required application revision cannot become ready.
- Escalate to the database owner before Flyway history repair, destructive DDL, point-in-time
recovery, or primary failover.
- R3 restore/PITR and failover rehearsal requires a target-like backup topology; local
Testcontainers evidence is not a substitute.
---
This runbook is forward-only. The reviewed migration artifact and the readiness lanes' JUnit
results from CI are the audit sources.