97 lines
4.0 KiB
Markdown
97 lines
4.0 KiB
Markdown
---
|
|
title: Runbook — MIGRATION_FAILED (DB 마이그레이션 실패)
|
|
category: INTERNAL
|
|
error_codes: [MIGRATION_FAILED]
|
|
severity: P1
|
|
owner: oncall
|
|
last_updated: 2026-07-29
|
|
status: active
|
|
---
|
|
|
|
# Runbook: MIGRATION_FAILED (`runbook://migration/failed`)
|
|
|
|
## Symptoms
|
|
|
|
- Container exits with code 70 (migration failure exit)
|
|
- Structured log with `error.code=MIGRATION_FAILED`, `startup.phase=migration`
|
|
- App refuses to start (fail-fast)
|
|
- JPA capability adapter refuses activation because its
|
|
`capability_schema_registry.lifecycle_state` is not `ACTIVE`
|
|
|
|
## Diagnosis
|
|
|
|
1. Stop rollout and keep the failed revision out of readiness. Do not route traffic to a partially
|
|
migrated instance.
|
|
2. Identify the exact stream from `src/config/jpa/readiness-cards.yaml`. Each stream has an
|
|
independent `location` and `history-table`; do not infer ownership from a broad
|
|
`classpath:db/migration` scan.
|
|
3. From a privileged migration session, capture the stream state before changing anything:
|
|
|
|
```sql
|
|
select installed_rank, version, description, success
|
|
from <owned_history_table>
|
|
order by installed_rank;
|
|
|
|
select capability_id, installation_origin, core_epoch, feature_revision, lifecycle_state
|
|
from capability_schema_registry
|
|
where capability_id = '<card-id>';
|
|
```
|
|
|
|
4. Check whether any owned relation was created without a successful history entry. Compare only
|
|
against the owned tables in the reviewed migration; do not drop unrelated relations.
|
|
5. Classify the failure:
|
|
- lock/statement timeout: remove the blocker or reduce rollout concurrency, then rerun;
|
|
- SQL/data precondition: create a new forward migration that makes the precondition explicit;
|
|
- checksum mismatch: compare the deployed artifact with the already applied script before
|
|
considering repair;
|
|
- connection/TLS failure: fix transport or credentials without changing Flyway history.
|
|
|
|
## Action
|
|
|
|
1. Prefer forward recovery. Fix the environmental blocker or add a new immutable migration, then
|
|
rerun the same owned stream with its exact history table.
|
|
2. For an optional stream that never installed successfully, keep the capability marker absent and
|
|
the runtime adapter disabled until migration succeeds.
|
|
3. After a successful migration, validate:
|
|
- the history contains only successful expected versions;
|
|
- `core_epoch` and `feature_revision` match the readiness registry;
|
|
- the marker is `INSTALLED_INACTIVE`;
|
|
- owned objects and constraints exist.
|
|
4. Change the marker to `ACTIVE` only after the compatible application revision is deployed and its
|
|
readiness check succeeds. Disabling or rolling back application code changes the marker to
|
|
`INSTALLED_INACTIVE`; it does not drop history or owned data.
|
|
5. Re-run the candidate evidence task before promoting:
|
|
|
|
```bash
|
|
cd src
|
|
./gradlew :adapter:outbound:persistence-jpa:verifyJpaCandidateEvidence --console=plain
|
|
```
|
|
|
|
6. Record the failed revision, stream/history table, root cause, recovery migration, elapsed time
|
|
and verification artifact in the incident.
|
|
|
|
Do not:
|
|
|
|
- edit an already applied migration;
|
|
- delete or rewrite Flyway history to make validation green;
|
|
- run `flyway repair` before checksum provenance is proven and reviewed;
|
|
- use `clean`, destructive rollback, or schema-wide restore as the first response;
|
|
- mark a capability `ACTIVE` before its migration and adapter readiness succeed.
|
|
|
|
If commit outcome was indeterminate during the failure, reconcile by the application
|
|
`OperationId`/idempotency reference before retrying business work. Never blind-retry a commit whose
|
|
result is unknown.
|
|
|
|
## Escalation
|
|
|
|
- P1 immediate: the required application revision cannot become ready.
|
|
- Escalate to the database owner before Flyway history repair, destructive DDL, point-in-time
|
|
recovery, or primary failover.
|
|
- R3 restore/PITR and failover rehearsal requires a target-like backup topology; local
|
|
Testcontainers evidence is not a substitute.
|
|
|
|
---
|
|
|
|
This runbook is forward-only. The reviewed migration artifact and the per-card evidence manifest
|
|
are the audit sources.
|