clean-architecture-backend-template/docs/runbooks/migration-failed.md

title, category, error_codes, severity, owner, last_updated, status

title	category	error_codes	severity	owner	last_updated	status
Runbook — MIGRATION_FAILED (DB 마이그레이션 실패)	INTERNAL	MIGRATION_FAILED	P1	oncall	2026-07-29	active

Runbook: MIGRATION_FAILED (runbook://migration/failed)

Symptoms

• Container exits with code 70 (migration failure exit)
• Structured log with error.code=MIGRATION_FAILED, startup.phase=migration
• App refuses to start (fail-fast)
• JPA capability adapter refuses activation because its capability_schema_registry.lifecycle_state is not ACTIVE

Diagnosis

1. Stop rollout and keep the failed revision out of readiness. Do not route traffic to a partially migrated instance.

2. Identify the exact stream from src/config/jpa/readiness-cards.yaml. Each stream has an independent location and history-table; do not infer ownership from a broad classpath:db/migration scan.

3. From a privileged migration session, capture the stream state before changing anything:

   select installed_rank, version, description, success
   from <owned_history_table>
   order by installed_rank;
   
   select capability_id, installation_origin, core_epoch, feature_revision, lifecycle_state
   from capability_schema_registry
   where capability_id = '<card-id>';
   

4. Check whether any owned relation was created without a successful history entry. Compare only against the owned tables in the reviewed migration; do not drop unrelated relations.

5. Classify the failure:

   • lock/statement timeout: remove the blocker or reduce rollout concurrency, then rerun;
   • SQL/data precondition: create a new forward migration that makes the precondition explicit;
   • checksum mismatch: compare the deployed artifact with the already applied script before considering repair;
   • connection/TLS failure: fix transport or credentials without changing Flyway history.

Action

1. Prefer forward recovery. Fix the environmental blocker or add a new immutable migration, then rerun the same owned stream with its exact history table.

2. For an optional stream that never installed successfully, keep the capability marker absent and the runtime adapter disabled until migration succeeds.

3. After a successful migration, validate:

   • the history contains only successful expected versions;
   • core_epoch and feature_revision match the readiness registry;
   • the marker is INSTALLED_INACTIVE;
   • owned objects and constraints exist.

4. Change the marker to ACTIVE only after the compatible application revision is deployed and its readiness check succeeds. Disabling or rolling back application code changes the marker to INSTALLED_INACTIVE; it does not drop history or owned data.

5. Re-run the candidate evidence task before promoting:

   cd src
   ./gradlew :adapter:outbound:persistence-jpa:verifyJpaCandidateEvidence --console=plain
   

6. Record the failed revision, stream/history table, root cause, recovery migration, elapsed time and verification artifact in the incident.

Do not:

• edit an already applied migration;
• delete or rewrite Flyway history to make validation green;
• run flyway repair before checksum provenance is proven and reviewed;
• use clean, destructive rollback, or schema-wide restore as the first response;
• mark a capability ACTIVE before its migration and adapter readiness succeed.

If commit outcome was indeterminate during the failure, reconcile by the application OperationId/idempotency reference before retrying business work. Never blind-retry a commit whose result is unknown.

Escalation

• P1 immediate: the required application revision cannot become ready.
• Escalate to the database owner before Flyway history repair, destructive DDL, point-in-time recovery, or primary failover.
• R3 restore/PITR and failover rehearsal requires a target-like backup topology; local Testcontainers evidence is not a substitute.

---

This runbook is forward-only. The reviewed migration artifact and the per-card evidence manifest are the audit sources.