Files
clean-architecture-backend-…/docs/runbooks/migration-failed.md
T
DongHyeonka 5786126cfa fix!: 남은 리뷰 항목과 합친 뒤 드러난 결함을 고친다
문제: 앞 커밋 뒤에도 알림·파일 서버 묶음이 합쳐지지 않았고, 포트 다수에
fake와 실제 구현이 함께 통과하는 공유 계약이 없었고, SpotBugs가 신뢰도
HIGH만 봤고, application-core에 관계없는 클래스 묶음 패키지(outbound)가
있었다. 묶음을 합치자 lane·검사가 새로 드러낸 결함도 있었다(JpaOutbox
Store가 읽기 전용 트랜잭션에서 이벤트를 조용히 잃음, gRPC admission
controller가 public synchronized라 외부 lock에 막힘, 소비자 decode 타입
등록 경로 없음, compose Redis 익명 접근, local-fileserver lane DB 없음).

방법: 리뷰 README 15절에 남은 결정을 모두 적고 그대로 고쳤다. 트랜잭션·
멱등성·발행·notification 포트에 testFixtures 계약 키트를 두고 fake와
PG·Redis 구현이 상속한다(fake 변이로 계약이 잡는지 확인). SpotBugs는
MEDIUM으로 내리고 진짜 결함은 고치고 오탐은 위치·패턴별 이유를 단
제외로 처리했다. 업무 기능 패키지 규칙을 application-core/CLAUDE.md에
정본으로 두고 ArchUnit 2개와 생성 명령이 강제한다. compose Redis는
infra/redis-sdk ACL로 뜬다. 묶음별 결과는 리뷰 README 16절에 있다.

위험·한계: APP_FLYWAY_ENABLED가 dev·prod에도 적용된다. app.security
오타 키가 이제 기동을 막는다. SES 콜백 replay 창 기본값이 1시간이다.
템플릿 패키지 목록은 CLAUDE.md 표와 TemplatePackageOwnership 두 곳에
있고 일치 검사는 없다. 실제 CI runner 실행은 확인하지 못했다.

검증: ci, mainCheck, integrationCheck, -p optional-platforms ci
integrationCheck, performanceCheck, releaseCheck(-PgitRevision),
verify-compose-profile-contracts(18 lane) 통과. compose lane 실제 기동:
local-fileserver, notification ingest·serving·handoff, local-redis,
all-adapters(smoke 3개 포함).

Refs: docs/reviews/2026-11-01-maturity-review/README.md 15·16절
BREAKING CHANGE: APP_FLYWAY_ENABLED가 dev·prod 프로필에도 적용되고,
app.security의 모르는 키는 기동을 거부한다. MessageHandler에
schemaVersions()가 생겨 decode 목록을 handler 선언에서 만든다.
2026-11-03 19:52:06 +09:00

4.2 KiB

title, category, error_codes, severity, owner, last_updated, status
title category error_codes severity owner last_updated status
Runbook — MIGRATION_FAILED (DB 마이그레이션 실패) INTERNAL
MIGRATION_FAILED
P1 oncall 2026-07-29 active

Runbook: MIGRATION_FAILED (runbook://migration/failed)

Symptoms

  • Container exits with code 70 (migration failure exit)
  • Structured log with error.code=MIGRATION_FAILED, startup.phase=migration
  • App refuses to start (fail-fast)
  • JPA capability adapter refuses activation because its capability_schema_registry.lifecycle_state is not ACTIVE

Diagnosis

  1. Stop rollout and keep the failed revision out of readiness. Do not route traffic to a partially migrated instance.

  2. Identify the exact stream from src/config/jpa/readiness-cards.yaml. Each stream has an independent location and history-table; do not infer ownership from a broad classpath:db/migration scan.

  3. From a privileged migration session, capture the stream state before changing anything:

    select installed_rank, version, description, success
    from <owned_history_table>
    order by installed_rank;
    
    select capability_id, installation_origin, core_epoch, feature_revision, lifecycle_state
    from capability_schema_registry
    where capability_id = '<card-id>';
    
  4. Check whether any owned relation was created without a successful history entry. Compare only against the owned tables in the reviewed migration; do not drop unrelated relations.

  5. Classify the failure:

    • lock/statement timeout: remove the blocker or reduce rollout concurrency, then rerun;
    • SQL/data precondition: create a new forward migration that makes the precondition explicit; a duplicate outbox order key stopping V21 (ux_outbox_event_aggregate_order) is the known case — clean it up with outbox-dead-letter.md §8 and rerun, the migration stays as it is;
    • checksum mismatch: compare the deployed artifact with the already applied script before considering repair;
    • connection/TLS failure: fix transport or credentials without changing Flyway history.

Action

  1. Prefer forward recovery. Fix the environmental blocker or add a new immutable migration, then rerun the same owned stream with its exact history table.

  2. For an optional stream that never installed successfully, keep the capability marker absent and the runtime adapter disabled until migration succeeds.

  3. After a successful migration, validate:

    • the history contains only successful expected versions;
    • core_epoch and feature_revision match the readiness registry;
    • the marker is INSTALLED_INACTIVE;
    • owned objects and constraints exist.
  4. Change the marker to ACTIVE only after the compatible application revision is deployed and its readiness check succeeds. Disabling or rolling back application code changes the marker to INSTALLED_INACTIVE; it does not drop history or owned data.

  5. Re-run the readiness lanes before promoting (every card's scenarios, no skips):

    cd src
    ./gradlew :adapter:outbound:persistence-jpa:integrationTestLaneCheck --console=plain
    
  6. Record the failed revision, stream/history table, root cause, recovery migration, elapsed time and verification artifact in the incident.

Do not:

  • edit an already applied migration;
  • delete or rewrite Flyway history to make validation green;
  • run flyway repair before checksum provenance is proven and reviewed;
  • use clean, destructive rollback, or schema-wide restore as the first response;
  • mark a capability ACTIVE before its migration and adapter readiness succeed.

If commit outcome was indeterminate during the failure, reconcile by the application OperationId/idempotency reference before retrying business work. Never blind-retry a commit whose result is unknown.

Escalation

  • P1 immediate: the required application revision cannot become ready.
  • Escalate to the database owner before Flyway history repair, destructive DDL, point-in-time recovery, or primary failover.
  • R3 restore/PITR and failover rehearsal requires a target-like backup topology; local Testcontainers evidence is not a substitute.

This runbook is forward-only. The reviewed migration artifact and the readiness lanes' JUnit results from CI are the audit sources.