Files
clean-architecture-backend-…/docs/runbooks/migration-failed.md
T
donghyeon-kaandClaude Opus 5.5 0ed7bc8df7 review-followup Wave 3 라운드 1(P16): 빌드 게이트 정리 — JPA evidence manifest 폐기, build-tools 제거, API surface를 leaf check로
- 결정 4: JPA evidence·qualification manifest 폐기 → persistence-jpa 테스트, build-tools included build 비움(verifyEnvKeys는
  app-bootstrap 태스크, 구현은 build-logic `ca.config-contract`), gRPC release gate 삭제
- 결정 20: API surface 검사를 leaf `check`로, JPA adopter 검사를 `ca.api-surface`로 합침(구현 하나)
- 레인 단계 정리(문서 대조 레인은 `mainStage()`), integrationCheck가 입력 없는 레인을 이름과 함께 제외 → 로컬 통과(NEW-16)
- release workflow 통합, releaseCheck는 tag 전용, build-logic 비핵심 플러그인 정리, sample을 ArchUnit 대상에
- 보류 4행(B3 변경 모듈만 실행, C 이미지 서명, messaging 카드 검증 이동 2행 — 결정 필요), 남음 2행(O3-5 나머지)
- 통합: P17이 옮긴 gRPC release gate 사본 삭제, `grpc-proto-buf`를 release-gate needs에(P17-new-3)

검증: ./gradlew ci verifyDependencyLocks --warning-mode=fail 통과(685 task), mainCheck 통과(268 task),
-p optional-platforms ci 통과, -p build-logic test 78건 통과(손상된 TestKit 캐시 삭제 후)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 16:44:17 +09:00

97 lines
4.0 KiB
Markdown

---
title: Runbook — MIGRATION_FAILED (DB 마이그레이션 실패)
category: INTERNAL
error_codes: [MIGRATION_FAILED]
severity: P1
owner: oncall
last_updated: 2026-07-29
status: active
---
# Runbook: MIGRATION_FAILED (`runbook://migration/failed`)
## Symptoms
- Container exits with code 70 (migration failure exit)
- Structured log with `error.code=MIGRATION_FAILED`, `startup.phase=migration`
- App refuses to start (fail-fast)
- JPA capability adapter refuses activation because its
`capability_schema_registry.lifecycle_state` is not `ACTIVE`
## Diagnosis
1. Stop rollout and keep the failed revision out of readiness. Do not route traffic to a partially
migrated instance.
2. Identify the exact stream from `src/config/jpa/readiness-cards.yaml`. Each stream has an
independent `location` and `history-table`; do not infer ownership from a broad
`classpath:db/migration` scan.
3. From a privileged migration session, capture the stream state before changing anything:
```sql
select installed_rank, version, description, success
from <owned_history_table>
order by installed_rank;
select capability_id, installation_origin, core_epoch, feature_revision, lifecycle_state
from capability_schema_registry
where capability_id = '<card-id>';
```
4. Check whether any owned relation was created without a successful history entry. Compare only
against the owned tables in the reviewed migration; do not drop unrelated relations.
5. Classify the failure:
- lock/statement timeout: remove the blocker or reduce rollout concurrency, then rerun;
- SQL/data precondition: create a new forward migration that makes the precondition explicit;
- checksum mismatch: compare the deployed artifact with the already applied script before
considering repair;
- connection/TLS failure: fix transport or credentials without changing Flyway history.
## Action
1. Prefer forward recovery. Fix the environmental blocker or add a new immutable migration, then
rerun the same owned stream with its exact history table.
2. For an optional stream that never installed successfully, keep the capability marker absent and
the runtime adapter disabled until migration succeeds.
3. After a successful migration, validate:
- the history contains only successful expected versions;
- `core_epoch` and `feature_revision` match the readiness registry;
- the marker is `INSTALLED_INACTIVE`;
- owned objects and constraints exist.
4. Change the marker to `ACTIVE` only after the compatible application revision is deployed and its
readiness check succeeds. Disabling or rolling back application code changes the marker to
`INSTALLED_INACTIVE`; it does not drop history or owned data.
5. Re-run the readiness lanes before promoting (every card's scenarios, no skips):
```bash
cd src
./gradlew :adapter:outbound:persistence-jpa:integrationTestLaneCheck --console=plain
```
6. Record the failed revision, stream/history table, root cause, recovery migration, elapsed time
and verification artifact in the incident.
Do not:
- edit an already applied migration;
- delete or rewrite Flyway history to make validation green;
- run `flyway repair` before checksum provenance is proven and reviewed;
- use `clean`, destructive rollback, or schema-wide restore as the first response;
- mark a capability `ACTIVE` before its migration and adapter readiness succeed.
If commit outcome was indeterminate during the failure, reconcile by the application
`OperationId`/idempotency reference before retrying business work. Never blind-retry a commit whose
result is unknown.
## Escalation
- P1 immediate: the required application revision cannot become ready.
- Escalate to the database owner before Flyway history repair, destructive DDL, point-in-time
recovery, or primary failover.
- R3 restore/PITR and failover rehearsal requires a target-like backup topology; local
Testcontainers evidence is not a substitute.
---
This runbook is forward-only. The reviewed migration artifact and the readiness lanes' JUnit
results from CI are the audit sources.