--- title: Runbook — MIGRATION_FAILED (DB 마이그레이션 실패) category: INTERNAL error_codes: [MIGRATION_FAILED] severity: P1 owner: oncall last_updated: 2026-11-07 status: active --- # Runbook: MIGRATION_FAILED (`runbook://migration/failed`) ## Symptoms - Container exits with code 70 (migration failure exit) - Structured log with `error.code=MIGRATION_FAILED`, `startup.phase=migration` - App refuses to start (fail-fast) - JPA capability adapter refuses activation because its `capability_schema_registry.lifecycle_state` is not `ACTIVE` ## Diagnosis 1. Stop rollout and keep the failed revision out of readiness. Do not route traffic to a partially migrated instance. 2. Identify the exact stream from `src/config/jpa/readiness-cards.yaml`. Each stream has an independent `location` and `history-table`; do not infer ownership from a broad `classpath:db/migration` scan. 3. From a privileged migration session, capture the stream state before changing anything: ```sql select installed_rank, version, description, success from order by installed_rank; select capability_id, installation_origin, core_epoch, feature_revision, lifecycle_state from capability_schema_registry where capability_id = ''; ``` 4. Check whether any owned relation was created without a successful history entry. Compare only against the owned tables in the reviewed migration; do not drop unrelated relations. 5. Classify the failure: - lock/statement timeout: remove the blocker or reduce rollout concurrency, then rerun. A migration that locks a table already in use gives up after its own `SET LOCAL lock_timeout` (5 s; SQLState `55P03` `lock_not_available`) and rolls back whole, so the history ends one version below it. The blocker is a transaction that touched the table and is still open — often an idle-in-transaction session. Find it by the lock it holds on that table (`` is the table the failing statement locks; for a `REFERENCES`, also the table it references): ```sql SELECT DISTINCT a.pid, a.state, a.xact_start, a.query FROM pg_locks l JOIN pg_stat_activity a USING (pid) WHERE l.relation = '
'::regclass AND l.granted AND a.pid <> pg_backend_pid() ORDER BY a.xact_start; ``` `query` is only that session's last statement, not the one that touched the table, so do not look for the table name in it. Listing every long transaction instead names unrelated ones (reports, batches) beside the blocker. End the blocker, then rerun (`docs/jpa/migration-guide.md` "Bound the lock wait"); - SQL/data precondition: create a new forward migration that makes the precondition explicit; a duplicate outbox order key stopping V21 (`ux_outbox_event_aggregate_order`) is the known case — clean it up with `outbox-dead-letter.md` §8 and rerun, the migration stays as it is; - checksum mismatch: compare the deployed artifact with the already applied script before considering repair; - a concurrent index migration failed (`CREATE [UNIQUE] INDEX CONCURRENTLY` or `REINDEX … CONCURRENTLY` that Flyway ran outside a transaction): a duplicate value for a unique index, a deadlock, `statement_timeout` or a cancelled build. Outside a transaction nothing rolled back: the migration left an invalid index and a history row with `success = false` for that version, and every later deploy fails validation ("Detected failed migration to version …") until the row is gone. Dropping the index alone does not clear it. Follow "Recover a failed concurrent index migration" below; - connection/TLS failure: fix transport or credentials without changing Flyway history. ## Action 1. Prefer forward recovery. Fix the environmental blocker or add a new immutable migration, then rerun the same owned stream with its exact history table. 2. For an optional stream that never installed successfully, keep the capability marker absent and the runtime adapter disabled until migration succeeds. 3. After a successful migration, validate: - the history contains only successful expected versions; - `core_epoch` and `feature_revision` match the readiness registry; - the marker is `INSTALLED_INACTIVE`; - owned objects and constraints exist. 4. Change the marker to `ACTIVE` only after the compatible application revision is deployed and its readiness check succeeds. Disabling or rolling back application code changes the marker to `INSTALLED_INACTIVE`; it does not drop history or owned data. 5. Re-run the readiness lanes before promoting (every card's scenarios, no skips): ```bash cd src ./gradlew :adapter:outbound:persistence-jpa:integrationTestLaneCheck --console=plain ``` 6. Record the failed revision, stream/history table, root cause, recovery migration, elapsed time and verification artifact in the incident. Do not: - edit an already applied migration; - delete or rewrite Flyway history to make validation green. The one exception is the failed row of a failed concurrent index migration, deleted as "Recover a failed concurrent index migration" below says — that row and nothing else; - run `flyway repair` before checksum provenance is proven and reviewed; - use `clean`, destructive rollback, or schema-wide restore as the first response; - mark a capability `ACTIVE` before its migration and adapter readiness succeed. If commit outcome was indeterminate during the failure, reconcile by the application `OperationId`/idempotency reference before retrying business work. Never blind-retry a commit whose result is unknown. ## Recover a failed concurrent index migration The one case where this runbook changes Flyway history (owner decision, 2026-11-06). It applies to a migration that holds a single concurrent index statement which Flyway ran outside a transaction — with the `executeInTransaction=false` the guide asks for, or on its own for `CREATE`/`DROP INDEX CONCURRENTLY` (`docs/jpa/migration-guide.md` "Concurrent index builds") — and whose row for this version in the stream's history table says `success = false`. A migration that mixed other statements in has already committed them: reconstruct that state with the database owner instead. 1. Confirm no build is still running. An index that is still being built looks invalid in the catalog too, and dropping it races the build. This query returns no rows when none is: ```sql SELECT pid, relid::regclass AS table_name, index_relid::regclass AS index_name, phase FROM pg_stat_progress_create_index WHERE datname = current_database(); ``` 2. Drop the invalid index the build left (`docs/jpa/runbooks.md` "An invalid index exists" lists them). Run it on its own, not inside `BEGIN`: `CONCURRENTLY` refuses a transaction block. ```sql DROP INDEX CONCURRENTLY IF EXISTS ; ``` 3. Remove the cause. For a unique index, clean up the duplicate rows with the data owner (a forward fix, never by editing the migration). For a deadlock or a timeout, end or wait out the transaction that held the build. 4. Delete that version's failed row from the stream's history table, and nothing else. `` is the `version` column of the failed row as the Diagnosis query shows it (`22.1` for `V22_1__…`). It must delete exactly one row; if it deletes none, this is not the failure this section is for, so classify it again. ```sql DELETE FROM WHERE version = '' AND NOT success; ``` 5. Deploy again. Flyway runs the same migration from the start and builds the index. The failed row is the record of a version that never applied, so deleting it leaves the order and the checksum of every applied version as they were. Do not use `flyway repair` for this: it also rewrites the checksums and descriptions of the remaining rows to match the scripts on disk, which removes the evidence a checksum mismatch carries. `:adapter:outbound:persistence-jpa:postgresqlConcurrentIndexMigrationIntegrationTest` runs the three SQL blocks above, read from this file, against real PostgreSQL: a unique build fails on a duplicate, steps 1–4 run, and the next deploy applies the version with the index valid and the earlier version's checksum unchanged. ## Escalation - P1 immediate: the required application revision cannot become ready. - Escalate to the database owner before Flyway history repair, destructive DDL, point-in-time recovery, or primary failover. - R3 restore/PITR and failover rehearsal requires a target-like backup topology; local Testcontainers evidence is not a substitute. --- This runbook is forward-only. The reviewed migration artifact and the readiness lanes' JUnit results from CI are the audit sources.