Files
clean-architecture-backend-…/docs/runbooks/migration-failed.md
T
DongHyeonka 1885bf8da7 fix!: 2차 재리뷰가 찾은 결함 29건을 고친다
문제: 재리뷰 수정 커밋(bbe944bd)만 다시 본 2차 재리뷰가 Important 7건,
Minor 22건을 찾았다. 결과 불명 503 뒤 같은 키를 다시 보내면 Redis
store가 lease 지난 실행을 넘겨받아 작업을 다시 돌렸다. replay TTL이
lease보다 짧으면 lease 도중 key가 사라져 같은 키가 두 번 실행됐다.
@SystemInvoked 규칙이 매뉴얼 09장의 이벤트 소비 경로까지 막았다. V22가
outbox_event에 가장 강한 잠금을 lock_timeout 없이 걸었다. prod의
fixture- 거부가 HOSTNAME 같은 비밀 아닌 값까지 막았다. 데모 허용 목록의
test·demo 프로필로 sample이 뜨지 않았다. 기동 rule bean의 @Value 필드가
조용히 null이 됐다.

방법: 영역별 4묶음이 test-driven-development 절차로 실패 테스트를
먼저 쓰고 고쳤다. 결정은 재리뷰 README 9.1절이다.
- 두 멱등성 store가 같은 계약을 따른다. lease가 지난 실행 기록은 다시
  돌리지 않고 포기로 표시해 500 RECOVERY_REQUIRED로 답한다. 기록 수명은
  lease + replay TTL이고, replay-ttl < processing-lease면 기동이 실패한다.
- 사용자 transport만 @SystemInvoked use case에 닿지 못한다. 메시지
  handler·scheduler는 새 표시 @SystemCaller(reason)를 달고 use case를
  생성자에서 new로 만든다. 매뉴얼 경로를 테스트로 고정했다.
- 아직 어디에도 적용되지 않은 마이그레이션(V20·V22·알림 V11~V13·sample
  V8_1·grpc V002·V003)에 lock_timeout 5s를 넣고, 텍스트 검사기와 실 PG
  lane을 더했다. 이미 배포된 24개는 동결 목록으로 둔다.
- fixture- 검사는 비밀로 분류된 값만 본다. sample은 local·test·demo에서
  object storage를 끈다. 기동 rule bean에 주입 멤버가 있으면 모든
  프로필에서 기동이 실패한다. 리뷰어가 제안한 "static @Bean 선언 강제"는
  문제없는 등록을 막고 @Value 필드는 통과시켜서 버렸다.
- 이 수정 전에 move된 파일의 쿼터 차이를 찾는 쿼리와 재계산 SQL을
  docs/fileserver/operations.md에 적었다.

위험·한계: lease 뒤 늦게 끝난 첫 시도의 결과는 기록되지 않는다(그 요청도
500). @SystemCaller가 없는 메시지 handler, 주입 멤버가 있는 기동 rule,
working-directory 없이 ./gradlew를 부르는 workflow 명령은 파생 서비스에서
새로 실패하거나 세지지 않는다(업그레이드 안내 B-14~16·B-36·B-39).
spotbugsTestFixtures는 게이트 밖이고 기준 커밋에서도 실패하며, 새 검사기의
정규식 경고 8건이 더해졌다. 실제 CI runner 실행은 확인하지 못했다(Gitea
repo의 Actions가 꺼져 있다).

검증: ci, 두 빌드의 verifyStrictLaneStages --dry-run, mainCheck,
:integrationCheck, -p optional-platforms ci integrationCheck
performanceCheck, :performanceCheck, :releaseCheck(-PgitRevision),
verify-compose-profile-contracts(18 lane) 첫 실행 통과. 쿼터 재계산
SQL은 실 PostgreSQL 16에서 차이 2행 → 0행을 확인했다.

Refs: docs/reviews/2026-11-03-rereview/README.md 9·9.1·10절
BREAKING CHANGE: replay-ttl < processing-lease 설정은 기동 실패, ledger
release가 claim token을 받는다, HttpBodyReadException(IOException 하위)이
본문 실패를 싣는다, 기동 rule·비밀 목록 bean의 주입 멤버 금지,
verifyStrictLaneStages가 workflow 명령의 working-directory로 빌드를 정한다.
2026-11-04 01:47:50 +09:00

4.8 KiB

title, category, error_codes, severity, owner, last_updated, status
title category error_codes severity owner last_updated status
Runbook — MIGRATION_FAILED (DB 마이그레이션 실패) INTERNAL
MIGRATION_FAILED
P1 oncall 2026-11-04 active

Runbook: MIGRATION_FAILED (runbook://migration/failed)

Symptoms

  • Container exits with code 70 (migration failure exit)
  • Structured log with error.code=MIGRATION_FAILED, startup.phase=migration
  • App refuses to start (fail-fast)
  • JPA capability adapter refuses activation because its capability_schema_registry.lifecycle_state is not ACTIVE

Diagnosis

  1. Stop rollout and keep the failed revision out of readiness. Do not route traffic to a partially migrated instance.

  2. Identify the exact stream from src/config/jpa/readiness-cards.yaml. Each stream has an independent location and history-table; do not infer ownership from a broad classpath:db/migration scan.

  3. From a privileged migration session, capture the stream state before changing anything:

    select installed_rank, version, description, success
    from <owned_history_table>
    order by installed_rank;
    
    select capability_id, installation_origin, core_epoch, feature_revision, lifecycle_state
    from capability_schema_registry
    where capability_id = '<card-id>';
    
  4. Check whether any owned relation was created without a successful history entry. Compare only against the owned tables in the reviewed migration; do not drop unrelated relations.

  5. Classify the failure:

    • lock/statement timeout: remove the blocker or reduce rollout concurrency, then rerun. A migration that locks a table already in use gives up after its own SET LOCAL lock_timeout (5 s; SQLState 55P03 lock_not_available) and rolls back whole, so the history ends one version below it. The blocker is a transaction that touched the table and is still open — often an idle-in-transaction session: SELECT pid, state, xact_start, query FROM pg_stat_activity WHERE xact_start < now() - interval '5 seconds' ORDER BY xact_start;. End it, then rerun (docs/jpa/migration-guide.md "Bound the lock wait");
    • SQL/data precondition: create a new forward migration that makes the precondition explicit; a duplicate outbox order key stopping V21 (ux_outbox_event_aggregate_order) is the known case — clean it up with outbox-dead-letter.md §8 and rerun, the migration stays as it is;
    • checksum mismatch: compare the deployed artifact with the already applied script before considering repair;
    • connection/TLS failure: fix transport or credentials without changing Flyway history.

Action

  1. Prefer forward recovery. Fix the environmental blocker or add a new immutable migration, then rerun the same owned stream with its exact history table.

  2. For an optional stream that never installed successfully, keep the capability marker absent and the runtime adapter disabled until migration succeeds.

  3. After a successful migration, validate:

    • the history contains only successful expected versions;
    • core_epoch and feature_revision match the readiness registry;
    • the marker is INSTALLED_INACTIVE;
    • owned objects and constraints exist.
  4. Change the marker to ACTIVE only after the compatible application revision is deployed and its readiness check succeeds. Disabling or rolling back application code changes the marker to INSTALLED_INACTIVE; it does not drop history or owned data.

  5. Re-run the readiness lanes before promoting (every card's scenarios, no skips):

    cd src
    ./gradlew :adapter:outbound:persistence-jpa:integrationTestLaneCheck --console=plain
    
  6. Record the failed revision, stream/history table, root cause, recovery migration, elapsed time and verification artifact in the incident.

Do not:

  • edit an already applied migration;
  • delete or rewrite Flyway history to make validation green;
  • run flyway repair before checksum provenance is proven and reviewed;
  • use clean, destructive rollback, or schema-wide restore as the first response;
  • mark a capability ACTIVE before its migration and adapter readiness succeed.

If commit outcome was indeterminate during the failure, reconcile by the application OperationId/idempotency reference before retrying business work. Never blind-retry a commit whose result is unknown.

Escalation

  • P1 immediate: the required application revision cannot become ready.
  • Escalate to the database owner before Flyway history repair, destructive DDL, point-in-time recovery, or primary failover.
  • R3 restore/PITR and failover rehearsal requires a target-like backup topology; local Testcontainers evidence is not a substitute.

This runbook is forward-only. The reviewed migration artifact and the readiness lanes' JUnit results from CI are the audit sources.