문제: 재리뷰 수정 커밋(bbe944bd)만 다시 본 2차 재리뷰가 Important 7건,
Minor 22건을 찾았다. 결과 불명 503 뒤 같은 키를 다시 보내면 Redis
store가 lease 지난 실행을 넘겨받아 작업을 다시 돌렸다. replay TTL이
lease보다 짧으면 lease 도중 key가 사라져 같은 키가 두 번 실행됐다.
@SystemInvoked 규칙이 매뉴얼 09장의 이벤트 소비 경로까지 막았다. V22가
outbox_event에 가장 강한 잠금을 lock_timeout 없이 걸었다. prod의
fixture- 거부가 HOSTNAME 같은 비밀 아닌 값까지 막았다. 데모 허용 목록의
test·demo 프로필로 sample이 뜨지 않았다. 기동 rule bean의 @Value 필드가
조용히 null이 됐다.
방법: 영역별 4묶음이 test-driven-development 절차로 실패 테스트를
먼저 쓰고 고쳤다. 결정은 재리뷰 README 9.1절이다.
- 두 멱등성 store가 같은 계약을 따른다. lease가 지난 실행 기록은 다시
돌리지 않고 포기로 표시해 500 RECOVERY_REQUIRED로 답한다. 기록 수명은
lease + replay TTL이고, replay-ttl < processing-lease면 기동이 실패한다.
- 사용자 transport만 @SystemInvoked use case에 닿지 못한다. 메시지
handler·scheduler는 새 표시 @SystemCaller(reason)를 달고 use case를
생성자에서 new로 만든다. 매뉴얼 경로를 테스트로 고정했다.
- 아직 어디에도 적용되지 않은 마이그레이션(V20·V22·알림 V11~V13·sample
V8_1·grpc V002·V003)에 lock_timeout 5s를 넣고, 텍스트 검사기와 실 PG
lane을 더했다. 이미 배포된 24개는 동결 목록으로 둔다.
- fixture- 검사는 비밀로 분류된 값만 본다. sample은 local·test·demo에서
object storage를 끈다. 기동 rule bean에 주입 멤버가 있으면 모든
프로필에서 기동이 실패한다. 리뷰어가 제안한 "static @Bean 선언 강제"는
문제없는 등록을 막고 @Value 필드는 통과시켜서 버렸다.
- 이 수정 전에 move된 파일의 쿼터 차이를 찾는 쿼리와 재계산 SQL을
docs/fileserver/operations.md에 적었다.
위험·한계: lease 뒤 늦게 끝난 첫 시도의 결과는 기록되지 않는다(그 요청도
500). @SystemCaller가 없는 메시지 handler, 주입 멤버가 있는 기동 rule,
working-directory 없이 ./gradlew를 부르는 workflow 명령은 파생 서비스에서
새로 실패하거나 세지지 않는다(업그레이드 안내 B-14~16·B-36·B-39).
spotbugsTestFixtures는 게이트 밖이고 기준 커밋에서도 실패하며, 새 검사기의
정규식 경고 8건이 더해졌다. 실제 CI runner 실행은 확인하지 못했다(Gitea
repo의 Actions가 꺼져 있다).
검증: ci, 두 빌드의 verifyStrictLaneStages --dry-run, mainCheck,
:integrationCheck, -p optional-platforms ci integrationCheck
performanceCheck, :performanceCheck, :releaseCheck(-PgitRevision),
verify-compose-profile-contracts(18 lane) 첫 실행 통과. 쿼터 재계산
SQL은 실 PostgreSQL 16에서 차이 2행 → 0행을 확인했다.
Refs: docs/reviews/2026-11-03-rereview/README.md 9·9.1·10절
BREAKING CHANGE: replay-ttl < processing-lease 설정은 기동 실패, ledger
release가 claim token을 받는다, HttpBodyReadException(IOException 하위)이
본문 실패를 싣는다, 기동 rule·비밀 목록 bean의 주입 멤버 금지,
verifyStrictLaneStages가 workflow 명령의 working-directory로 빌드를 정한다.
4.8 KiB
title, category, error_codes, severity, owner, last_updated, status
| title | category | error_codes | severity | owner | last_updated | status | |
|---|---|---|---|---|---|---|---|
| Runbook — MIGRATION_FAILED (DB 마이그레이션 실패) | INTERNAL |
|
P1 | oncall | 2026-11-04 | active |
Runbook: MIGRATION_FAILED (runbook://migration/failed)
Symptoms
- Container exits with code 70 (migration failure exit)
- Structured log with
error.code=MIGRATION_FAILED,startup.phase=migration - App refuses to start (fail-fast)
- JPA capability adapter refuses activation because its
capability_schema_registry.lifecycle_stateis notACTIVE
Diagnosis
-
Stop rollout and keep the failed revision out of readiness. Do not route traffic to a partially migrated instance.
-
Identify the exact stream from
src/config/jpa/readiness-cards.yaml. Each stream has an independentlocationandhistory-table; do not infer ownership from a broadclasspath:db/migrationscan. -
From a privileged migration session, capture the stream state before changing anything:
select installed_rank, version, description, success from <owned_history_table> order by installed_rank; select capability_id, installation_origin, core_epoch, feature_revision, lifecycle_state from capability_schema_registry where capability_id = '<card-id>'; -
Check whether any owned relation was created without a successful history entry. Compare only against the owned tables in the reviewed migration; do not drop unrelated relations.
-
Classify the failure:
- lock/statement timeout: remove the blocker or reduce rollout concurrency, then rerun. A
migration that locks a table already in use gives up after its own
SET LOCAL lock_timeout(5 s; SQLState55P03lock_not_available) and rolls back whole, so the history ends one version below it. The blocker is a transaction that touched the table and is still open — often an idle-in-transaction session:SELECT pid, state, xact_start, query FROM pg_stat_activity WHERE xact_start < now() - interval '5 seconds' ORDER BY xact_start;. End it, then rerun (docs/jpa/migration-guide.md"Bound the lock wait"); - SQL/data precondition: create a new forward migration that makes the precondition explicit;
a duplicate outbox order key stopping V21 (
ux_outbox_event_aggregate_order) is the known case — clean it up withoutbox-dead-letter.md§8 and rerun, the migration stays as it is; - checksum mismatch: compare the deployed artifact with the already applied script before considering repair;
- connection/TLS failure: fix transport or credentials without changing Flyway history.
- lock/statement timeout: remove the blocker or reduce rollout concurrency, then rerun. A
migration that locks a table already in use gives up after its own
Action
-
Prefer forward recovery. Fix the environmental blocker or add a new immutable migration, then rerun the same owned stream with its exact history table.
-
For an optional stream that never installed successfully, keep the capability marker absent and the runtime adapter disabled until migration succeeds.
-
After a successful migration, validate:
- the history contains only successful expected versions;
core_epochandfeature_revisionmatch the readiness registry;- the marker is
INSTALLED_INACTIVE; - owned objects and constraints exist.
-
Change the marker to
ACTIVEonly after the compatible application revision is deployed and its readiness check succeeds. Disabling or rolling back application code changes the marker toINSTALLED_INACTIVE; it does not drop history or owned data. -
Re-run the readiness lanes before promoting (every card's scenarios, no skips):
cd src ./gradlew :adapter:outbound:persistence-jpa:integrationTestLaneCheck --console=plain -
Record the failed revision, stream/history table, root cause, recovery migration, elapsed time and verification artifact in the incident.
Do not:
- edit an already applied migration;
- delete or rewrite Flyway history to make validation green;
- run
flyway repairbefore checksum provenance is proven and reviewed; - use
clean, destructive rollback, or schema-wide restore as the first response; - mark a capability
ACTIVEbefore its migration and adapter readiness succeed.
If commit outcome was indeterminate during the failure, reconcile by the application
OperationId/idempotency reference before retrying business work. Never blind-retry a commit whose
result is unknown.
Escalation
- P1 immediate: the required application revision cannot become ready.
- Escalate to the database owner before Flyway history repair, destructive DDL, point-in-time recovery, or primary failover.
- R3 restore/PITR and failover rehearsal requires a target-like backup topology; local Testcontainers evidence is not a substitute.
This runbook is forward-only. The reviewed migration artifact and the readiness lanes' JUnit results from CI are the audit sources.