문제: 3차 수정의 확인 리뷰가 새 Important 없이 Minor 19건을 남겼다. 동작에 닿는 것이 셋이다. JDK read-idle watchdog이 공용 ForkJoinPool에서 돌아 그 풀이 바쁘면 300ms timeout이 6초 가까이 늦었다. 붙여 쓴 -x<task>를 verifyStrictLaneStages가 읽지 않아 lane을 뺀 workflow가 통과했다. 사용자 전송이 중간 클래스를 거쳐 @SystemCaller bean을 부르면 ArchUnit 규칙을 지났다. 잠금 대기 검사기는 FK·trigger·REINDEX와 lock_timeout = 0을 못 봤고, 쿼터 보정 SQL은 코드와 다른 모양의 행을 넣었다. 방법: 영역별 4묶음이 맡은 Minor만 test-driven-development 절차로 고쳤다(재리뷰 README 12절). - watchdog은 전용 데몬 스레드 하나에서 돌고 스트림을 닫으면 작업을 취소한다. read당 비용은 그대로다(21–24ns). - 사용자 전송 → @SystemCaller는 직접·간접 의존 모두 거부한다. 리뷰가 권한 직접 의존 검사는 중간 클래스 하나로 다시 열려서 버렸다. - 검사기를 보강하면서 새로 걸린 이미 배포된 마이그레이션 5개는 내용을 바꾸지 않고 동결 목록에 더했다(main과 blob이 같다). - 비밀 목록 bean은 규칙보다 먼저 만들어 실제 클래스로 검사한다. 위험·한계: 회복 필요 예외 메시지 끝에 scope digest가 붙고 기동 실패 문구가 바뀌어, 문구 전체에 맞추던 로그 매처는 고쳐야 한다. 코드가 든 Redis 계정 참조의 fixture- 검사는 문서로만 막았다(코드로 막으려면 새 의존 edge가 필요하다). failure-retention ≥ replay-ttl 기동 검사는 넣지 않았다. -mx 같은 짧은 플래그 묶음과 graphql 전송 fixture는 남았다. spotbugsTestFixtures(게이트 밖)의 정규식 경고가 15건으로 늘었다. 검증: ci, 두 빌드의 verifyStrictLaneStages --dry-run, mainCheck, :integrationCheck, -p optional-platforms ci integrationCheck performanceCheck, :performanceCheck, :releaseCheck(-PgitRevision), verify-compose-profile-contracts(18 lane) 첫 실행 통과. watchdog은 공용 풀을 채운 상태에서 300ms 설정이 300ms에 끝나는 것을 쟀다. Refs: docs/reviews/2026-11-03-rereview/README.md 11·12절, round3/
116 lines
5.3 KiB
Markdown
116 lines
5.3 KiB
Markdown
---
|
|
title: Runbook — MIGRATION_FAILED (DB 마이그레이션 실패)
|
|
category: INTERNAL
|
|
error_codes: [MIGRATION_FAILED]
|
|
severity: P1
|
|
owner: oncall
|
|
last_updated: 2026-11-04
|
|
status: active
|
|
---
|
|
|
|
# Runbook: MIGRATION_FAILED (`runbook://migration/failed`)
|
|
|
|
## Symptoms
|
|
|
|
- Container exits with code 70 (migration failure exit)
|
|
- Structured log with `error.code=MIGRATION_FAILED`, `startup.phase=migration`
|
|
- App refuses to start (fail-fast)
|
|
- JPA capability adapter refuses activation because its
|
|
`capability_schema_registry.lifecycle_state` is not `ACTIVE`
|
|
|
|
## Diagnosis
|
|
|
|
1. Stop rollout and keep the failed revision out of readiness. Do not route traffic to a partially
|
|
migrated instance.
|
|
2. Identify the exact stream from `src/config/jpa/readiness-cards.yaml`. Each stream has an
|
|
independent `location` and `history-table`; do not infer ownership from a broad
|
|
`classpath:db/migration` scan.
|
|
3. From a privileged migration session, capture the stream state before changing anything:
|
|
|
|
```sql
|
|
select installed_rank, version, description, success
|
|
from <owned_history_table>
|
|
order by installed_rank;
|
|
|
|
select capability_id, installation_origin, core_epoch, feature_revision, lifecycle_state
|
|
from capability_schema_registry
|
|
where capability_id = '<card-id>';
|
|
```
|
|
|
|
4. Check whether any owned relation was created without a successful history entry. Compare only
|
|
against the owned tables in the reviewed migration; do not drop unrelated relations.
|
|
5. Classify the failure:
|
|
- lock/statement timeout: remove the blocker or reduce rollout concurrency, then rerun. A
|
|
migration that locks a table already in use gives up after its own `SET LOCAL lock_timeout`
|
|
(5 s; SQLState `55P03` `lock_not_available`) and rolls back whole, so the history ends one
|
|
version below it. The blocker is a transaction that touched the table and is still open —
|
|
often an idle-in-transaction session. Find it by the lock it holds on that table (`<table>`
|
|
is the table the failing statement locks; for a `REFERENCES`, also the table it references):
|
|
|
|
```sql
|
|
SELECT DISTINCT a.pid, a.state, a.xact_start, a.query
|
|
FROM pg_locks l JOIN pg_stat_activity a USING (pid)
|
|
WHERE l.relation = '<table>'::regclass AND l.granted AND a.pid <> pg_backend_pid()
|
|
ORDER BY a.xact_start;
|
|
```
|
|
|
|
`query` is only that session's last statement, not the one that touched the table, so do not
|
|
look for the table name in it. Listing every long transaction instead names unrelated ones
|
|
(reports, batches) beside the blocker. End the blocker, then rerun
|
|
(`docs/jpa/migration-guide.md` "Bound the lock wait");
|
|
- SQL/data precondition: create a new forward migration that makes the precondition explicit;
|
|
a duplicate outbox order key stopping V21 (`ux_outbox_event_aggregate_order`) is the known
|
|
case — clean it up with `outbox-dead-letter.md` §8 and rerun, the migration stays as it is;
|
|
- checksum mismatch: compare the deployed artifact with the already applied script before
|
|
considering repair;
|
|
- connection/TLS failure: fix transport or credentials without changing Flyway history.
|
|
|
|
## Action
|
|
|
|
1. Prefer forward recovery. Fix the environmental blocker or add a new immutable migration, then
|
|
rerun the same owned stream with its exact history table.
|
|
2. For an optional stream that never installed successfully, keep the capability marker absent and
|
|
the runtime adapter disabled until migration succeeds.
|
|
3. After a successful migration, validate:
|
|
- the history contains only successful expected versions;
|
|
- `core_epoch` and `feature_revision` match the readiness registry;
|
|
- the marker is `INSTALLED_INACTIVE`;
|
|
- owned objects and constraints exist.
|
|
4. Change the marker to `ACTIVE` only after the compatible application revision is deployed and its
|
|
readiness check succeeds. Disabling or rolling back application code changes the marker to
|
|
`INSTALLED_INACTIVE`; it does not drop history or owned data.
|
|
5. Re-run the readiness lanes before promoting (every card's scenarios, no skips):
|
|
|
|
```bash
|
|
cd src
|
|
./gradlew :adapter:outbound:persistence-jpa:integrationTestLaneCheck --console=plain
|
|
```
|
|
|
|
6. Record the failed revision, stream/history table, root cause, recovery migration, elapsed time
|
|
and verification artifact in the incident.
|
|
|
|
Do not:
|
|
|
|
- edit an already applied migration;
|
|
- delete or rewrite Flyway history to make validation green;
|
|
- run `flyway repair` before checksum provenance is proven and reviewed;
|
|
- use `clean`, destructive rollback, or schema-wide restore as the first response;
|
|
- mark a capability `ACTIVE` before its migration and adapter readiness succeed.
|
|
|
|
If commit outcome was indeterminate during the failure, reconcile by the application
|
|
`OperationId`/idempotency reference before retrying business work. Never blind-retry a commit whose
|
|
result is unknown.
|
|
|
|
## Escalation
|
|
|
|
- P1 immediate: the required application revision cannot become ready.
|
|
- Escalate to the database owner before Flyway history repair, destructive DDL, point-in-time
|
|
recovery, or primary failover.
|
|
- R3 restore/PITR and failover rehearsal requires a target-like backup topology; local
|
|
Testcontainers evidence is not a substitute.
|
|
|
|
---
|
|
|
|
This runbook is forward-only. The reviewed migration artifact and the readiness lanes' JUnit
|
|
results from CI are the audit sources.
|