- 결정 4: JPA evidence·qualification manifest 폐기 → persistence-jpa 테스트, build-tools included build 비움(verifyEnvKeys는 app-bootstrap 태스크, 구현은 build-logic `ca.config-contract`), gRPC release gate 삭제 - 결정 20: API surface 검사를 leaf `check`로, JPA adopter 검사를 `ca.api-surface`로 합침(구현 하나) - 레인 단계 정리(문서 대조 레인은 `mainStage()`), integrationCheck가 입력 없는 레인을 이름과 함께 제외 → 로컬 통과(NEW-16) - release workflow 통합, releaseCheck는 tag 전용, build-logic 비핵심 플러그인 정리, sample을 ArchUnit 대상에 - 보류 4행(B3 변경 모듈만 실행, C 이미지 서명, messaging 카드 검증 이동 2행 — 결정 필요), 남음 2행(O3-5 나머지) - 통합: P17이 옮긴 gRPC release gate 사본 삭제, `grpc-proto-buf`를 release-gate needs에(P17-new-3) 검증: ./gradlew ci verifyDependencyLocks --warning-mode=fail 통과(685 task), mainCheck 통과(268 task), -p optional-platforms ci 통과, -p build-logic test 78건 통과(손상된 TestKit 캐시 삭제 후) Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
97 lines
4.0 KiB
Markdown
97 lines
4.0 KiB
Markdown
---
|
|
title: Runbook — MIGRATION_FAILED (DB 마이그레이션 실패)
|
|
category: INTERNAL
|
|
error_codes: [MIGRATION_FAILED]
|
|
severity: P1
|
|
owner: oncall
|
|
last_updated: 2026-07-29
|
|
status: active
|
|
---
|
|
|
|
# Runbook: MIGRATION_FAILED (`runbook://migration/failed`)
|
|
|
|
## Symptoms
|
|
|
|
- Container exits with code 70 (migration failure exit)
|
|
- Structured log with `error.code=MIGRATION_FAILED`, `startup.phase=migration`
|
|
- App refuses to start (fail-fast)
|
|
- JPA capability adapter refuses activation because its
|
|
`capability_schema_registry.lifecycle_state` is not `ACTIVE`
|
|
|
|
## Diagnosis
|
|
|
|
1. Stop rollout and keep the failed revision out of readiness. Do not route traffic to a partially
|
|
migrated instance.
|
|
2. Identify the exact stream from `src/config/jpa/readiness-cards.yaml`. Each stream has an
|
|
independent `location` and `history-table`; do not infer ownership from a broad
|
|
`classpath:db/migration` scan.
|
|
3. From a privileged migration session, capture the stream state before changing anything:
|
|
|
|
```sql
|
|
select installed_rank, version, description, success
|
|
from <owned_history_table>
|
|
order by installed_rank;
|
|
|
|
select capability_id, installation_origin, core_epoch, feature_revision, lifecycle_state
|
|
from capability_schema_registry
|
|
where capability_id = '<card-id>';
|
|
```
|
|
|
|
4. Check whether any owned relation was created without a successful history entry. Compare only
|
|
against the owned tables in the reviewed migration; do not drop unrelated relations.
|
|
5. Classify the failure:
|
|
- lock/statement timeout: remove the blocker or reduce rollout concurrency, then rerun;
|
|
- SQL/data precondition: create a new forward migration that makes the precondition explicit;
|
|
- checksum mismatch: compare the deployed artifact with the already applied script before
|
|
considering repair;
|
|
- connection/TLS failure: fix transport or credentials without changing Flyway history.
|
|
|
|
## Action
|
|
|
|
1. Prefer forward recovery. Fix the environmental blocker or add a new immutable migration, then
|
|
rerun the same owned stream with its exact history table.
|
|
2. For an optional stream that never installed successfully, keep the capability marker absent and
|
|
the runtime adapter disabled until migration succeeds.
|
|
3. After a successful migration, validate:
|
|
- the history contains only successful expected versions;
|
|
- `core_epoch` and `feature_revision` match the readiness registry;
|
|
- the marker is `INSTALLED_INACTIVE`;
|
|
- owned objects and constraints exist.
|
|
4. Change the marker to `ACTIVE` only after the compatible application revision is deployed and its
|
|
readiness check succeeds. Disabling or rolling back application code changes the marker to
|
|
`INSTALLED_INACTIVE`; it does not drop history or owned data.
|
|
5. Re-run the readiness lanes before promoting (every card's scenarios, no skips):
|
|
|
|
```bash
|
|
cd src
|
|
./gradlew :adapter:outbound:persistence-jpa:integrationTestLaneCheck --console=plain
|
|
```
|
|
|
|
6. Record the failed revision, stream/history table, root cause, recovery migration, elapsed time
|
|
and verification artifact in the incident.
|
|
|
|
Do not:
|
|
|
|
- edit an already applied migration;
|
|
- delete or rewrite Flyway history to make validation green;
|
|
- run `flyway repair` before checksum provenance is proven and reviewed;
|
|
- use `clean`, destructive rollback, or schema-wide restore as the first response;
|
|
- mark a capability `ACTIVE` before its migration and adapter readiness succeed.
|
|
|
|
If commit outcome was indeterminate during the failure, reconcile by the application
|
|
`OperationId`/idempotency reference before retrying business work. Never blind-retry a commit whose
|
|
result is unknown.
|
|
|
|
## Escalation
|
|
|
|
- P1 immediate: the required application revision cannot become ready.
|
|
- Escalate to the database owner before Flyway history repair, destructive DDL, point-in-time
|
|
recovery, or primary failover.
|
|
- R3 restore/PITR and failover rehearsal requires a target-like backup topology; local
|
|
Testcontainers evidence is not a substitute.
|
|
|
|
---
|
|
|
|
This runbook is forward-only. The reviewed migration artifact and the readiness lanes' JUnit
|
|
results from CI are the audit sources.
|