Files
clean-architecture-backend-…/docs/runbooks/migration-failed.md
T
DongHyeonka f40c03c6d3 fix: 3차 확인 리뷰의 Minor 19건을 고친다
문제: 3차 수정의 확인 리뷰가 새 Important 없이 Minor 19건을 남겼다.
동작에 닿는 것이 셋이다. JDK read-idle watchdog이 공용 ForkJoinPool에서
돌아 그 풀이 바쁘면 300ms timeout이 6초 가까이 늦었다. 붙여 쓴
-x<task>를 verifyStrictLaneStages가 읽지 않아 lane을 뺀 workflow가
통과했다. 사용자 전송이 중간 클래스를 거쳐 @SystemCaller bean을 부르면
ArchUnit 규칙을 지났다. 잠금 대기 검사기는 FK·trigger·REINDEX와
lock_timeout = 0을 못 봤고, 쿼터 보정 SQL은 코드와 다른 모양의 행을
넣었다.

방법: 영역별 4묶음이 맡은 Minor만 test-driven-development 절차로
고쳤다(재리뷰 README 12절).
- watchdog은 전용 데몬 스레드 하나에서 돌고 스트림을 닫으면 작업을
  취소한다. read당 비용은 그대로다(21–24ns).
- 사용자 전송 → @SystemCaller는 직접·간접 의존 모두 거부한다. 리뷰가
  권한 직접 의존 검사는 중간 클래스 하나로 다시 열려서 버렸다.
- 검사기를 보강하면서 새로 걸린 이미 배포된 마이그레이션 5개는 내용을
  바꾸지 않고 동결 목록에 더했다(main과 blob이 같다).
- 비밀 목록 bean은 규칙보다 먼저 만들어 실제 클래스로 검사한다.

위험·한계: 회복 필요 예외 메시지 끝에 scope digest가 붙고 기동 실패
문구가 바뀌어, 문구 전체에 맞추던 로그 매처는 고쳐야 한다. 코드가 든
Redis 계정 참조의 fixture- 검사는 문서로만 막았다(코드로 막으려면 새
의존 edge가 필요하다). failure-retention ≥ replay-ttl 기동 검사는 넣지
않았다. -mx 같은 짧은 플래그 묶음과 graphql 전송 fixture는 남았다.
spotbugsTestFixtures(게이트 밖)의 정규식 경고가 15건으로 늘었다.

검증: ci, 두 빌드의 verifyStrictLaneStages --dry-run, mainCheck,
:integrationCheck, -p optional-platforms ci integrationCheck
performanceCheck, :performanceCheck, :releaseCheck(-PgitRevision),
verify-compose-profile-contracts(18 lane) 첫 실행 통과. watchdog은
공용 풀을 채운 상태에서 300ms 설정이 300ms에 끝나는 것을 쟀다.

Refs: docs/reviews/2026-11-03-rereview/README.md 11·12절, round3/
2026-11-04 03:00:21 +09:00

116 lines
5.3 KiB
Markdown

---
title: Runbook — MIGRATION_FAILED (DB 마이그레이션 실패)
category: INTERNAL
error_codes: [MIGRATION_FAILED]
severity: P1
owner: oncall
last_updated: 2026-11-04
status: active
---
# Runbook: MIGRATION_FAILED (`runbook://migration/failed`)
## Symptoms
- Container exits with code 70 (migration failure exit)
- Structured log with `error.code=MIGRATION_FAILED`, `startup.phase=migration`
- App refuses to start (fail-fast)
- JPA capability adapter refuses activation because its
`capability_schema_registry.lifecycle_state` is not `ACTIVE`
## Diagnosis
1. Stop rollout and keep the failed revision out of readiness. Do not route traffic to a partially
migrated instance.
2. Identify the exact stream from `src/config/jpa/readiness-cards.yaml`. Each stream has an
independent `location` and `history-table`; do not infer ownership from a broad
`classpath:db/migration` scan.
3. From a privileged migration session, capture the stream state before changing anything:
```sql
select installed_rank, version, description, success
from <owned_history_table>
order by installed_rank;
select capability_id, installation_origin, core_epoch, feature_revision, lifecycle_state
from capability_schema_registry
where capability_id = '<card-id>';
```
4. Check whether any owned relation was created without a successful history entry. Compare only
against the owned tables in the reviewed migration; do not drop unrelated relations.
5. Classify the failure:
- lock/statement timeout: remove the blocker or reduce rollout concurrency, then rerun. A
migration that locks a table already in use gives up after its own `SET LOCAL lock_timeout`
(5 s; SQLState `55P03` `lock_not_available`) and rolls back whole, so the history ends one
version below it. The blocker is a transaction that touched the table and is still open —
often an idle-in-transaction session. Find it by the lock it holds on that table (`<table>`
is the table the failing statement locks; for a `REFERENCES`, also the table it references):
```sql
SELECT DISTINCT a.pid, a.state, a.xact_start, a.query
FROM pg_locks l JOIN pg_stat_activity a USING (pid)
WHERE l.relation = '<table>'::regclass AND l.granted AND a.pid <> pg_backend_pid()
ORDER BY a.xact_start;
```
`query` is only that session's last statement, not the one that touched the table, so do not
look for the table name in it. Listing every long transaction instead names unrelated ones
(reports, batches) beside the blocker. End the blocker, then rerun
(`docs/jpa/migration-guide.md` "Bound the lock wait");
- SQL/data precondition: create a new forward migration that makes the precondition explicit;
a duplicate outbox order key stopping V21 (`ux_outbox_event_aggregate_order`) is the known
case — clean it up with `outbox-dead-letter.md` §8 and rerun, the migration stays as it is;
- checksum mismatch: compare the deployed artifact with the already applied script before
considering repair;
- connection/TLS failure: fix transport or credentials without changing Flyway history.
## Action
1. Prefer forward recovery. Fix the environmental blocker or add a new immutable migration, then
rerun the same owned stream with its exact history table.
2. For an optional stream that never installed successfully, keep the capability marker absent and
the runtime adapter disabled until migration succeeds.
3. After a successful migration, validate:
- the history contains only successful expected versions;
- `core_epoch` and `feature_revision` match the readiness registry;
- the marker is `INSTALLED_INACTIVE`;
- owned objects and constraints exist.
4. Change the marker to `ACTIVE` only after the compatible application revision is deployed and its
readiness check succeeds. Disabling or rolling back application code changes the marker to
`INSTALLED_INACTIVE`; it does not drop history or owned data.
5. Re-run the readiness lanes before promoting (every card's scenarios, no skips):
```bash
cd src
./gradlew :adapter:outbound:persistence-jpa:integrationTestLaneCheck --console=plain
```
6. Record the failed revision, stream/history table, root cause, recovery migration, elapsed time
and verification artifact in the incident.
Do not:
- edit an already applied migration;
- delete or rewrite Flyway history to make validation green;
- run `flyway repair` before checksum provenance is proven and reviewed;
- use `clean`, destructive rollback, or schema-wide restore as the first response;
- mark a capability `ACTIVE` before its migration and adapter readiness succeed.
If commit outcome was indeterminate during the failure, reconcile by the application
`OperationId`/idempotency reference before retrying business work. Never blind-retry a commit whose
result is unknown.
## Escalation
- P1 immediate: the required application revision cannot become ready.
- Escalate to the database owner before Flyway history repair, destructive DDL, point-in-time
recovery, or primary failover.
- R3 restore/PITR and failover rehearsal requires a target-like backup topology; local
Testcontainers evidence is not a substitute.
---
This runbook is forward-only. The reviewed migration artifact and the readiness lanes' JUnit
results from CI are the audit sources.