Files
clean-architecture-backend-…/docs/runbooks/migration-failed.md
T
DongHyeonka 8d69e599a3 refactor(repo)!: 들여온 플랫폼을 src/platforms 아래로 묶는다
문제: src/ 최상위에 계층 폴더와 들여온 제품 폴더(messaging, grpc,
grpc-advanced)와 optional 빌드 폴더(optional-platforms)가 같은 높이로
놓여, 무엇이 이 애플리케이션의 계층인지 폴더만으로 안 보인다.
src/config/에는 빌드 전체 설정과 기능 하나가 주인인 데이터(jpa,
messaging, mongodb)가 섞여 있다.

고른 방법(소유자 결정 2026-11-07, 설계 가-1):
- git mv로 src/platforms/{messaging,grpc,grpc-advanced,optional-build}.
  registry source_path 28행만 바꾸고 Gradle 경로(:messaging:*, :grpc:*)는
  그대로다. optional 빌드의 이름(optional-platforms)도 그대로다.
- optional 빌드가 한 단계 깊어져, build-logic이 "optional 빌드의 부모 =
  주 빌드"로 가정한 네 곳(registry settings, ArchitecturePlugin, 테스트
  이미지 catalog, checkstyle·spotbugs 설정)을 MainBuild(위로 올라가며
  registry·설정 파일을 찾음)로 바꾼다.
- settings.gradle·build.gradle의 상대 경로, workflow의 -p 값과 경로,
  buf 작업 폴더를 고친다. buf breaking은 이동 전 baseline에서도 옛
  경로로 비교해 이동 PR에서 건너뛰지 않는다.
- config/jpa, config/messaging, config/mongodb의 데이터를 주인 leaf의
  release/로 옮기고 읽는 빌드 줄·테스트·스크립트·문서를 고친다.
- 루트 CLAUDE.md 5절에 최상위 폴더 지도를 한 줄로 적는다.
버린 대안: Gradle 루트를 저장소 루트로(가-3, CI 재편 때 다시 봄),
application-* 묶기(가-2, 파생 서비스 충돌이 큼).

위험·한계: Gradle 문서의 "이름 = 위치" 권고와 한 단계 어긋난다
(:messaging:x가 platforms/messaging/x). 옮긴 폴더를 고쳐 쓰던 파생
서비스는 이름 바뀜 충돌이 날 수 있다(B-52). docs/capability-status.md
「제거된 코드」 표와 docs/work/evidence.md의 옛 경로는 이력이라 두었다.

검증(desktop, 트리 2b3354a4, ~/t1-layout-logs/):
- ./gradlew :ci :verifyDependencyLocks --warning-mode=fail --continue
  첫 실행 rc=1: MessagingAuthorityContractTest가 src/messaging을 찾아 실패
  → 경로를 고침. 두 번째 rc=1: messaging-policy
  MessagingAdmissionControllerTest.aWaitingCallerIsAdmittedOnceAPermitIsReturned
  (2초 대기 경쟁, 부하 7.3에서 한 번 실패, 코드 변경 없는 leaf) → --rerun
  3회 모두 통과. 세 번째 rc=0
- -p platforms/optional-build ci rc=0, 두 빌드의 verifyStrictLaneStages
  --dry-run rc=0, -p build-logic check rc=0
- :app-bootstrap:documentContractTest
  :adapter:outbound:persistence-mongo:mongoDocumentContractTest rc=0
- ./gradlew projects(주 빌드)와 optional 빌드 projects의 프로젝트 목록이
  이동 전 트리와 같다(Gradle 경로 불변)
- check-work-docs.py, 상대 링크 검사 rc=0

Refs: design-docs repo-structure/README.md 3.1, repo-structure/01-backend-layout.md 6.1
BREAKING CHANGE: src/messaging·src/grpc·src/grpc-advanced·src/optional-platforms가 src/platforms/ 아래로, src/config/{jpa,messaging,mongodb}가 주인 leaf의 release/로 옮겨졌다. optional 빌드는 -p platforms/optional-build로 부른다. docs/upgrade/2026-11-maturity-review.md B-52.
2026-11-07 13:04:54 +09:00

9.9 KiB
Raw Blame History

title, category, error_codes, severity, owner, last_updated, status
title category error_codes severity owner last_updated status
Runbook — MIGRATION_FAILED (DB 마이그레이션 실패) INTERNAL
MIGRATION_FAILED
P1 oncall 2026-11-07 active

Runbook: MIGRATION_FAILED (runbook://migration/failed)

Symptoms

  • Container exits with code 70 (migration failure exit)
  • Structured log with error.code=MIGRATION_FAILED, startup.phase=migration
  • App refuses to start (fail-fast)
  • JPA capability adapter refuses activation because its capability_schema_registry.lifecycle_state is not ACTIVE

Diagnosis

  1. Stop rollout and keep the failed revision out of readiness. Do not route traffic to a partially migrated instance.

  2. Identify the exact stream from src/adapter/outbound/persistence-jpa/release/readiness-cards.yaml. Each stream has an independent location and history-table; do not infer ownership from a broad classpath:db/migration scan.

  3. From a privileged migration session, capture the stream state before changing anything:

    select installed_rank, version, description, success
    from <owned_history_table>
    order by installed_rank;
    
    select capability_id, installation_origin, core_epoch, feature_revision, lifecycle_state
    from capability_schema_registry
    where capability_id = '<card-id>';
    
  4. Check whether any owned relation was created without a successful history entry. Compare only against the owned tables in the reviewed migration; do not drop unrelated relations.

  5. Classify the failure:

    • lock/statement timeout: remove the blocker or reduce rollout concurrency, then rerun. A migration that locks a table already in use gives up after its own SET LOCAL lock_timeout (5 s; SQLState 55P03 lock_not_available) and rolls back whole, so the history ends one version below it. The blocker is a transaction that touched the table and is still open — often an idle-in-transaction session. Find it by the lock it holds on that table (<table> is the table the failing statement locks; for a REFERENCES, also the table it references):

      SELECT DISTINCT a.pid, a.state, a.xact_start, a.query
      FROM pg_locks l JOIN pg_stat_activity a USING (pid)
      WHERE l.relation = '<table>'::regclass AND l.granted AND a.pid <> pg_backend_pid()
      ORDER BY a.xact_start;
      

      query is only that session's last statement, not the one that touched the table, so do not look for the table name in it. Listing every long transaction instead names unrelated ones (reports, batches) beside the blocker. End the blocker, then rerun (docs/jpa/migration-guide.md "Bound the lock wait");

    • SQL/data precondition: create a new forward migration that makes the precondition explicit; a duplicate outbox order key stopping V21 (ux_outbox_event_aggregate_order) is the known case — clean it up with outbox-dead-letter.md §8 and rerun, the migration stays as it is;

    • checksum mismatch: compare the deployed artifact with the already applied script before considering repair;

    • a concurrent index migration failed (CREATE [UNIQUE] INDEX CONCURRENTLY, DROP INDEX CONCURRENTLY or REINDEX … CONCURRENTLY that Flyway ran outside a transaction): a duplicate value for a unique index, a deadlock, statement_timeout or a statement cancelled while it waited for an open transaction. Outside a transaction nothing rolled back: the migration left an invalid index and a history row with success = false for that version, and every later deploy fails validation ("Detected failed migration to version …") until the row is gone. Dropping the index alone does not clear it. Follow "Recover a failed concurrent index migration" below;

    • connection/TLS failure: fix transport or credentials without changing Flyway history.

Action

  1. Prefer forward recovery. Fix the environmental blocker or add a new immutable migration, then rerun the same owned stream with its exact history table.

  2. For an optional stream that never installed successfully, keep the capability marker absent and the runtime adapter disabled until migration succeeds.

  3. After a successful migration, validate:

    • the history contains only successful expected versions;
    • core_epoch and feature_revision match the readiness registry;
    • the marker is INSTALLED_INACTIVE;
    • owned objects and constraints exist.
  4. Change the marker to ACTIVE only after the compatible application revision is deployed and its readiness check succeeds. Disabling or rolling back application code changes the marker to INSTALLED_INACTIVE; it does not drop history or owned data.

  5. Re-run the readiness lanes before promoting (every card's scenarios, no skips):

    cd src
    ./gradlew :adapter:outbound:persistence-jpa:integrationTestLaneCheck --console=plain
    
  6. Record the failed revision, stream/history table, root cause, recovery migration, elapsed time and verification artifact in the incident.

Do not:

  • edit an already applied migration;
  • delete or rewrite Flyway history to make validation green. The one exception is the failed row of a failed concurrent index migration, deleted as "Recover a failed concurrent index migration" below says — that row and nothing else;
  • run flyway repair before checksum provenance is proven and reviewed;
  • use clean, destructive rollback, or schema-wide restore as the first response;
  • mark a capability ACTIVE before its migration and adapter readiness succeed.

If commit outcome was indeterminate during the failure, reconcile by the application OperationId/idempotency reference before retrying business work. Never blind-retry a commit whose result is unknown.

Recover a failed concurrent index migration

The one case where this runbook changes Flyway history (owner decision, 2026-11-06). It applies to a migration that holds a single concurrent index statement which Flyway ran outside a transaction — with the executeInTransaction=false the guide asks for, or on its own for CREATE/DROP INDEX CONCURRENTLY (docs/jpa/migration-guide.md "Concurrent index builds") — and whose row for this version in the stream's history table says success = false. A migration that mixed other statements in has already committed them: reconstruct that state with the database owner instead.

  1. Confirm no build is still running. An index that is still being built looks invalid in the catalog too, and dropping it races the build. This query returns no rows when none is:
SELECT pid, relid::regclass AS table_name, index_relid::regclass AS index_name, phase
FROM pg_stat_progress_create_index
WHERE datname = current_database();
  1. Remove the cause. For a unique index, clean up the duplicate rows with the data owner (a forward fix, never by editing the migration). For a deadlock, a timeout or a cancelled statement, end or wait out the transaction that held it: a build or a rebuild waits for every transaction older than itself, a drop for every transaction that holds a lock on its table. Do this before step 3, whose DROP INDEX CONCURRENTLY waits for the same transactions.
  2. Drop the invalid index a failed build or rebuild left: the index a CREATE [UNIQUE] INDEX CONCURRENTLY names, or the copy a REINDEX … CONCURRENTLY made, named after the index with _ccnew (or _ccold) on the end (docs/jpa/runbooks.md "An invalid index exists" lists them). Run it on its own, not inside BEGIN: CONCURRENTLY refuses a transaction block. Skip this step when the failed statement is DROP INDEX CONCURRENTLY. The invalid index it left is the index the migration drops, and the deploy in step 5 runs the same statement, which drops it. Dropped by hand first, that statement fails with 42704 ("index … does not exist") and leaves a new failed row.
DROP INDEX CONCURRENTLY IF EXISTS <invalid_index>;
  1. Delete that version's failed row from the stream's history table, and nothing else. <version> is the version column of the failed row as the Diagnosis query shows it (22.1 for V22_1__…). It must delete exactly one row; if it deletes none, this is not the failure this section is for, so classify it again.
DELETE FROM <owned_history_table>
WHERE version = '<version>' AND NOT success;
  1. Deploy again. Flyway runs the same migration from the start: it builds the index, rebuilds it, or drops the index the failed drop left.

The failed row is the record of a version that never applied, so deleting it leaves the order and the checksum of every applied version as they were. Do not use flyway repair for this: it also rewrites the checksums and descriptions of the remaining rows to match the scripts on disk, which removes the evidence a checksum mismatch carries. :adapter:outbound:persistence-jpa:postgresqlConcurrentIndexMigrationIntegrationTest runs the three SQL blocks above, read from this file, against real PostgreSQL for each statement this section covers: a unique build that fails on a duplicate, and a rebuild and a drop cancelled while they wait for an open transaction. Steps 1–4 run in order, step 3 for the build and the rebuild only, and the next deploy applies the version with no invalid index left and the earlier version's checksum unchanged.

Escalation

  • P1 immediate: the required application revision cannot become ready.
  • Escalate to the database owner before Flyway history repair, destructive DDL, point-in-time recovery, or primary failover.
  • R3 restore/PITR and failover rehearsal requires a target-like backup topology; local Testcontainers evidence is not a substitute.

This runbook is forward-only. The reviewed migration artifact and the readiness lanes' JUnit results from CI are the audit sources.