문제: 재리뷰 수정 커밋(bbe944bd)만 다시 본 2차 재리뷰가 Important 7건,
Minor 22건을 찾았다. 결과 불명 503 뒤 같은 키를 다시 보내면 Redis
store가 lease 지난 실행을 넘겨받아 작업을 다시 돌렸다. replay TTL이
lease보다 짧으면 lease 도중 key가 사라져 같은 키가 두 번 실행됐다.
@SystemInvoked 규칙이 매뉴얼 09장의 이벤트 소비 경로까지 막았다. V22가
outbox_event에 가장 강한 잠금을 lock_timeout 없이 걸었다. prod의
fixture- 거부가 HOSTNAME 같은 비밀 아닌 값까지 막았다. 데모 허용 목록의
test·demo 프로필로 sample이 뜨지 않았다. 기동 rule bean의 @Value 필드가
조용히 null이 됐다.
방법: 영역별 4묶음이 test-driven-development 절차로 실패 테스트를
먼저 쓰고 고쳤다. 결정은 재리뷰 README 9.1절이다.
- 두 멱등성 store가 같은 계약을 따른다. lease가 지난 실행 기록은 다시
돌리지 않고 포기로 표시해 500 RECOVERY_REQUIRED로 답한다. 기록 수명은
lease + replay TTL이고, replay-ttl < processing-lease면 기동이 실패한다.
- 사용자 transport만 @SystemInvoked use case에 닿지 못한다. 메시지
handler·scheduler는 새 표시 @SystemCaller(reason)를 달고 use case를
생성자에서 new로 만든다. 매뉴얼 경로를 테스트로 고정했다.
- 아직 어디에도 적용되지 않은 마이그레이션(V20·V22·알림 V11~V13·sample
V8_1·grpc V002·V003)에 lock_timeout 5s를 넣고, 텍스트 검사기와 실 PG
lane을 더했다. 이미 배포된 24개는 동결 목록으로 둔다.
- fixture- 검사는 비밀로 분류된 값만 본다. sample은 local·test·demo에서
object storage를 끈다. 기동 rule bean에 주입 멤버가 있으면 모든
프로필에서 기동이 실패한다. 리뷰어가 제안한 "static @Bean 선언 강제"는
문제없는 등록을 막고 @Value 필드는 통과시켜서 버렸다.
- 이 수정 전에 move된 파일의 쿼터 차이를 찾는 쿼리와 재계산 SQL을
docs/fileserver/operations.md에 적었다.
위험·한계: lease 뒤 늦게 끝난 첫 시도의 결과는 기록되지 않는다(그 요청도
500). @SystemCaller가 없는 메시지 handler, 주입 멤버가 있는 기동 rule,
working-directory 없이 ./gradlew를 부르는 workflow 명령은 파생 서비스에서
새로 실패하거나 세지지 않는다(업그레이드 안내 B-14~16·B-36·B-39).
spotbugsTestFixtures는 게이트 밖이고 기준 커밋에서도 실패하며, 새 검사기의
정규식 경고 8건이 더해졌다. 실제 CI runner 실행은 확인하지 못했다(Gitea
repo의 Actions가 꺼져 있다).
검증: ci, 두 빌드의 verifyStrictLaneStages --dry-run, mainCheck,
:integrationCheck, -p optional-platforms ci integrationCheck
performanceCheck, :performanceCheck, :releaseCheck(-PgitRevision),
verify-compose-profile-contracts(18 lane) 첫 실행 통과. 쿼터 재계산
SQL은 실 PostgreSQL 16에서 차이 2행 → 0행을 확인했다.
Refs: docs/reviews/2026-11-03-rereview/README.md 9·9.1·10절
BREAKING CHANGE: replay-ttl < processing-lease 설정은 기동 실패, ledger
release가 claim token을 받는다, HttpBodyReadException(IOException 하위)이
본문 실패를 싣는다, 기동 rule·비밀 목록 bean의 주입 멤버 금지,
verifyStrictLaneStages가 workflow 명령의 working-directory로 빌드를 정한다.
105 lines
4.8 KiB
Markdown
105 lines
4.8 KiB
Markdown
---
|
|
title: Runbook — MIGRATION_FAILED (DB 마이그레이션 실패)
|
|
category: INTERNAL
|
|
error_codes: [MIGRATION_FAILED]
|
|
severity: P1
|
|
owner: oncall
|
|
last_updated: 2026-11-04
|
|
status: active
|
|
---
|
|
|
|
# Runbook: MIGRATION_FAILED (`runbook://migration/failed`)
|
|
|
|
## Symptoms
|
|
|
|
- Container exits with code 70 (migration failure exit)
|
|
- Structured log with `error.code=MIGRATION_FAILED`, `startup.phase=migration`
|
|
- App refuses to start (fail-fast)
|
|
- JPA capability adapter refuses activation because its
|
|
`capability_schema_registry.lifecycle_state` is not `ACTIVE`
|
|
|
|
## Diagnosis
|
|
|
|
1. Stop rollout and keep the failed revision out of readiness. Do not route traffic to a partially
|
|
migrated instance.
|
|
2. Identify the exact stream from `src/config/jpa/readiness-cards.yaml`. Each stream has an
|
|
independent `location` and `history-table`; do not infer ownership from a broad
|
|
`classpath:db/migration` scan.
|
|
3. From a privileged migration session, capture the stream state before changing anything:
|
|
|
|
```sql
|
|
select installed_rank, version, description, success
|
|
from <owned_history_table>
|
|
order by installed_rank;
|
|
|
|
select capability_id, installation_origin, core_epoch, feature_revision, lifecycle_state
|
|
from capability_schema_registry
|
|
where capability_id = '<card-id>';
|
|
```
|
|
|
|
4. Check whether any owned relation was created without a successful history entry. Compare only
|
|
against the owned tables in the reviewed migration; do not drop unrelated relations.
|
|
5. Classify the failure:
|
|
- lock/statement timeout: remove the blocker or reduce rollout concurrency, then rerun. A
|
|
migration that locks a table already in use gives up after its own `SET LOCAL lock_timeout`
|
|
(5 s; SQLState `55P03` `lock_not_available`) and rolls back whole, so the history ends one
|
|
version below it. The blocker is a transaction that touched the table and is still open —
|
|
often an idle-in-transaction session: `SELECT pid, state, xact_start, query FROM
|
|
pg_stat_activity WHERE xact_start < now() - interval '5 seconds' ORDER BY xact_start;`. End
|
|
it, then rerun (`docs/jpa/migration-guide.md` "Bound the lock wait");
|
|
- SQL/data precondition: create a new forward migration that makes the precondition explicit;
|
|
a duplicate outbox order key stopping V21 (`ux_outbox_event_aggregate_order`) is the known
|
|
case — clean it up with `outbox-dead-letter.md` §8 and rerun, the migration stays as it is;
|
|
- checksum mismatch: compare the deployed artifact with the already applied script before
|
|
considering repair;
|
|
- connection/TLS failure: fix transport or credentials without changing Flyway history.
|
|
|
|
## Action
|
|
|
|
1. Prefer forward recovery. Fix the environmental blocker or add a new immutable migration, then
|
|
rerun the same owned stream with its exact history table.
|
|
2. For an optional stream that never installed successfully, keep the capability marker absent and
|
|
the runtime adapter disabled until migration succeeds.
|
|
3. After a successful migration, validate:
|
|
- the history contains only successful expected versions;
|
|
- `core_epoch` and `feature_revision` match the readiness registry;
|
|
- the marker is `INSTALLED_INACTIVE`;
|
|
- owned objects and constraints exist.
|
|
4. Change the marker to `ACTIVE` only after the compatible application revision is deployed and its
|
|
readiness check succeeds. Disabling or rolling back application code changes the marker to
|
|
`INSTALLED_INACTIVE`; it does not drop history or owned data.
|
|
5. Re-run the readiness lanes before promoting (every card's scenarios, no skips):
|
|
|
|
```bash
|
|
cd src
|
|
./gradlew :adapter:outbound:persistence-jpa:integrationTestLaneCheck --console=plain
|
|
```
|
|
|
|
6. Record the failed revision, stream/history table, root cause, recovery migration, elapsed time
|
|
and verification artifact in the incident.
|
|
|
|
Do not:
|
|
|
|
- edit an already applied migration;
|
|
- delete or rewrite Flyway history to make validation green;
|
|
- run `flyway repair` before checksum provenance is proven and reviewed;
|
|
- use `clean`, destructive rollback, or schema-wide restore as the first response;
|
|
- mark a capability `ACTIVE` before its migration and adapter readiness succeed.
|
|
|
|
If commit outcome was indeterminate during the failure, reconcile by the application
|
|
`OperationId`/idempotency reference before retrying business work. Never blind-retry a commit whose
|
|
result is unknown.
|
|
|
|
## Escalation
|
|
|
|
- P1 immediate: the required application revision cannot become ready.
|
|
- Escalate to the database owner before Flyway history repair, destructive DDL, point-in-time
|
|
recovery, or primary failover.
|
|
- R3 restore/PITR and failover rehearsal requires a target-like backup topology; local
|
|
Testcontainers evidence is not a substitute.
|
|
|
|
---
|
|
|
|
This runbook is forward-only. The reviewed migration artifact and the readiness lanes' JUnit
|
|
results from CI are the audit sources.
|