문제: src/ 최상위에 계층 폴더와 들여온 제품 폴더(messaging, grpc,
grpc-advanced)와 optional 빌드 폴더(optional-platforms)가 같은 높이로
놓여, 무엇이 이 애플리케이션의 계층인지 폴더만으로 안 보인다.
src/config/에는 빌드 전체 설정과 기능 하나가 주인인 데이터(jpa,
messaging, mongodb)가 섞여 있다.
고른 방법(소유자 결정 2026-11-07, 설계 가-1):
- git mv로 src/platforms/{messaging,grpc,grpc-advanced,optional-build}.
registry source_path 28행만 바꾸고 Gradle 경로(:messaging:*, :grpc:*)는
그대로다. optional 빌드의 이름(optional-platforms)도 그대로다.
- optional 빌드가 한 단계 깊어져, build-logic이 "optional 빌드의 부모 =
주 빌드"로 가정한 네 곳(registry settings, ArchitecturePlugin, 테스트
이미지 catalog, checkstyle·spotbugs 설정)을 MainBuild(위로 올라가며
registry·설정 파일을 찾음)로 바꾼다.
- settings.gradle·build.gradle의 상대 경로, workflow의 -p 값과 경로,
buf 작업 폴더를 고친다. buf breaking은 이동 전 baseline에서도 옛
경로로 비교해 이동 PR에서 건너뛰지 않는다.
- config/jpa, config/messaging, config/mongodb의 데이터를 주인 leaf의
release/로 옮기고 읽는 빌드 줄·테스트·스크립트·문서를 고친다.
- 루트 CLAUDE.md 5절에 최상위 폴더 지도를 한 줄로 적는다.
버린 대안: Gradle 루트를 저장소 루트로(가-3, CI 재편 때 다시 봄),
application-* 묶기(가-2, 파생 서비스 충돌이 큼).
위험·한계: Gradle 문서의 "이름 = 위치" 권고와 한 단계 어긋난다
(:messaging:x가 platforms/messaging/x). 옮긴 폴더를 고쳐 쓰던 파생
서비스는 이름 바뀜 충돌이 날 수 있다(B-52). docs/capability-status.md
「제거된 코드」 표와 docs/work/evidence.md의 옛 경로는 이력이라 두었다.
검증(desktop, 트리 2b3354a4, ~/t1-layout-logs/):
- ./gradlew :ci :verifyDependencyLocks --warning-mode=fail --continue
첫 실행 rc=1: MessagingAuthorityContractTest가 src/messaging을 찾아 실패
→ 경로를 고침. 두 번째 rc=1: messaging-policy
MessagingAdmissionControllerTest.aWaitingCallerIsAdmittedOnceAPermitIsReturned
(2초 대기 경쟁, 부하 7.3에서 한 번 실패, 코드 변경 없는 leaf) → --rerun
3회 모두 통과. 세 번째 rc=0
- -p platforms/optional-build ci rc=0, 두 빌드의 verifyStrictLaneStages
--dry-run rc=0, -p build-logic check rc=0
- :app-bootstrap:documentContractTest
:adapter:outbound:persistence-mongo:mongoDocumentContractTest rc=0
- ./gradlew projects(주 빌드)와 optional 빌드 projects의 프로젝트 목록이
이동 전 트리와 같다(Gradle 경로 불변)
- check-work-docs.py, 상대 링크 검사 rc=0
Refs: design-docs repo-structure/README.md 3.1, repo-structure/01-backend-layout.md 6.1
BREAKING CHANGE: src/messaging·src/grpc·src/grpc-advanced·src/optional-platforms가 src/platforms/ 아래로, src/config/{jpa,messaging,mongodb}가 주인 leaf의 release/로 옮겨졌다. optional 빌드는 -p platforms/optional-build로 부른다. docs/upgrade/2026-11-maturity-review.md B-52.
188 lines
9.9 KiB
Markdown
188 lines
9.9 KiB
Markdown
---
|
||
title: Runbook — MIGRATION_FAILED (DB 마이그레이션 실패)
|
||
category: INTERNAL
|
||
error_codes: [MIGRATION_FAILED]
|
||
severity: P1
|
||
owner: oncall
|
||
last_updated: 2026-11-07
|
||
status: active
|
||
---
|
||
|
||
# Runbook: MIGRATION_FAILED (`runbook://migration/failed`)
|
||
|
||
## Symptoms
|
||
|
||
- Container exits with code 70 (migration failure exit)
|
||
- Structured log with `error.code=MIGRATION_FAILED`, `startup.phase=migration`
|
||
- App refuses to start (fail-fast)
|
||
- JPA capability adapter refuses activation because its
|
||
`capability_schema_registry.lifecycle_state` is not `ACTIVE`
|
||
|
||
## Diagnosis
|
||
|
||
1. Stop rollout and keep the failed revision out of readiness. Do not route traffic to a partially
|
||
migrated instance.
|
||
2. Identify the exact stream from `src/adapter/outbound/persistence-jpa/release/readiness-cards.yaml`. Each stream has an
|
||
independent `location` and `history-table`; do not infer ownership from a broad
|
||
`classpath:db/migration` scan.
|
||
3. From a privileged migration session, capture the stream state before changing anything:
|
||
|
||
```sql
|
||
select installed_rank, version, description, success
|
||
from <owned_history_table>
|
||
order by installed_rank;
|
||
|
||
select capability_id, installation_origin, core_epoch, feature_revision, lifecycle_state
|
||
from capability_schema_registry
|
||
where capability_id = '<card-id>';
|
||
```
|
||
|
||
4. Check whether any owned relation was created without a successful history entry. Compare only
|
||
against the owned tables in the reviewed migration; do not drop unrelated relations.
|
||
5. Classify the failure:
|
||
- lock/statement timeout: remove the blocker or reduce rollout concurrency, then rerun. A
|
||
migration that locks a table already in use gives up after its own `SET LOCAL lock_timeout`
|
||
(5 s; SQLState `55P03` `lock_not_available`) and rolls back whole, so the history ends one
|
||
version below it. The blocker is a transaction that touched the table and is still open —
|
||
often an idle-in-transaction session. Find it by the lock it holds on that table (`<table>`
|
||
is the table the failing statement locks; for a `REFERENCES`, also the table it references):
|
||
|
||
```sql
|
||
SELECT DISTINCT a.pid, a.state, a.xact_start, a.query
|
||
FROM pg_locks l JOIN pg_stat_activity a USING (pid)
|
||
WHERE l.relation = '<table>'::regclass AND l.granted AND a.pid <> pg_backend_pid()
|
||
ORDER BY a.xact_start;
|
||
```
|
||
|
||
`query` is only that session's last statement, not the one that touched the table, so do not
|
||
look for the table name in it. Listing every long transaction instead names unrelated ones
|
||
(reports, batches) beside the blocker. End the blocker, then rerun
|
||
(`docs/jpa/migration-guide.md` "Bound the lock wait");
|
||
- SQL/data precondition: create a new forward migration that makes the precondition explicit;
|
||
a duplicate outbox order key stopping V21 (`ux_outbox_event_aggregate_order`) is the known
|
||
case — clean it up with `outbox-dead-letter.md` §8 and rerun, the migration stays as it is;
|
||
- checksum mismatch: compare the deployed artifact with the already applied script before
|
||
considering repair;
|
||
- a concurrent index migration failed (`CREATE [UNIQUE] INDEX CONCURRENTLY`, `DROP INDEX
|
||
CONCURRENTLY` or `REINDEX … CONCURRENTLY` that Flyway ran outside a transaction): a duplicate
|
||
value for a unique index, a deadlock, `statement_timeout` or a statement cancelled while it
|
||
waited for an open transaction. Outside a transaction nothing rolled back: the migration left
|
||
an invalid index and a history row with `success = false` for that version, and every later
|
||
deploy fails validation ("Detected failed migration to version …") until the row is gone.
|
||
Dropping the index alone does not clear it. Follow "Recover a failed concurrent index
|
||
migration" below;
|
||
- connection/TLS failure: fix transport or credentials without changing Flyway history.
|
||
|
||
## Action
|
||
|
||
1. Prefer forward recovery. Fix the environmental blocker or add a new immutable migration, then
|
||
rerun the same owned stream with its exact history table.
|
||
2. For an optional stream that never installed successfully, keep the capability marker absent and
|
||
the runtime adapter disabled until migration succeeds.
|
||
3. After a successful migration, validate:
|
||
- the history contains only successful expected versions;
|
||
- `core_epoch` and `feature_revision` match the readiness registry;
|
||
- the marker is `INSTALLED_INACTIVE`;
|
||
- owned objects and constraints exist.
|
||
4. Change the marker to `ACTIVE` only after the compatible application revision is deployed and its
|
||
readiness check succeeds. Disabling or rolling back application code changes the marker to
|
||
`INSTALLED_INACTIVE`; it does not drop history or owned data.
|
||
5. Re-run the readiness lanes before promoting (every card's scenarios, no skips):
|
||
|
||
```bash
|
||
cd src
|
||
./gradlew :adapter:outbound:persistence-jpa:integrationTestLaneCheck --console=plain
|
||
```
|
||
|
||
6. Record the failed revision, stream/history table, root cause, recovery migration, elapsed time
|
||
and verification artifact in the incident.
|
||
|
||
Do not:
|
||
|
||
- edit an already applied migration;
|
||
- delete or rewrite Flyway history to make validation green. The one exception is the failed row
|
||
of a failed concurrent index migration, deleted as "Recover a failed concurrent index migration"
|
||
below says — that row and nothing else;
|
||
- run `flyway repair` before checksum provenance is proven and reviewed;
|
||
- use `clean`, destructive rollback, or schema-wide restore as the first response;
|
||
- mark a capability `ACTIVE` before its migration and adapter readiness succeed.
|
||
|
||
If commit outcome was indeterminate during the failure, reconcile by the application
|
||
`OperationId`/idempotency reference before retrying business work. Never blind-retry a commit whose
|
||
result is unknown.
|
||
|
||
## Recover a failed concurrent index migration
|
||
|
||
The one case where this runbook changes Flyway history (owner decision, 2026-11-06). It applies to a
|
||
migration that holds a single concurrent index statement which Flyway ran outside a transaction —
|
||
with the `executeInTransaction=false` the guide asks for, or on its own for `CREATE`/`DROP INDEX
|
||
CONCURRENTLY` (`docs/jpa/migration-guide.md` "Concurrent index builds") — and whose row for this
|
||
version in the stream's history table says `success = false`. A migration that mixed other
|
||
statements in has already committed them: reconstruct that state with the database owner instead.
|
||
|
||
1. Confirm no build is still running. An index that is still being built looks invalid in the
|
||
catalog too, and dropping it races the build. This query returns no rows when none is:
|
||
|
||
<!-- runbook-sql: failed-concurrent-index-builds -->
|
||
```sql
|
||
SELECT pid, relid::regclass AS table_name, index_relid::regclass AS index_name, phase
|
||
FROM pg_stat_progress_create_index
|
||
WHERE datname = current_database();
|
||
```
|
||
|
||
2. Remove the cause. For a unique index, clean up the duplicate rows with the data owner (a forward
|
||
fix, never by editing the migration). For a deadlock, a timeout or a cancelled statement, end or
|
||
wait out the transaction that held it: a build or a rebuild waits for every transaction older
|
||
than itself, a drop for every transaction that holds a lock on its table. Do this before step 3,
|
||
whose `DROP INDEX CONCURRENTLY` waits for the same transactions.
|
||
3. Drop the invalid index a failed build or rebuild left: the index a `CREATE [UNIQUE] INDEX
|
||
CONCURRENTLY` names, or the copy a `REINDEX … CONCURRENTLY` made, named after the index with
|
||
`_ccnew` (or `_ccold`) on the end (`docs/jpa/runbooks.md` "An invalid index exists" lists them).
|
||
Run it on its own, not inside `BEGIN`: `CONCURRENTLY` refuses a transaction block.
|
||
**Skip this step when the failed statement is `DROP INDEX CONCURRENTLY`.** The invalid index it
|
||
left is the index the migration drops, and the deploy in step 5 runs the same statement, which
|
||
drops it. Dropped by hand first, that statement fails with `42704` ("index … does not exist")
|
||
and leaves a new failed row.
|
||
|
||
<!-- runbook-sql: failed-concurrent-index-drop -->
|
||
```sql
|
||
DROP INDEX CONCURRENTLY IF EXISTS <invalid_index>;
|
||
```
|
||
|
||
4. Delete that version's failed row from the stream's history table, and nothing else. `<version>`
|
||
is the `version` column of the failed row as the Diagnosis query shows it (`22.1` for
|
||
`V22_1__…`). It must delete exactly one row; if it deletes none, this is not the failure this
|
||
section is for, so classify it again.
|
||
|
||
<!-- runbook-sql: failed-concurrent-index-history-row -->
|
||
```sql
|
||
DELETE FROM <owned_history_table>
|
||
WHERE version = '<version>' AND NOT success;
|
||
```
|
||
|
||
5. Deploy again. Flyway runs the same migration from the start: it builds the index, rebuilds it,
|
||
or drops the index the failed drop left.
|
||
|
||
The failed row is the record of a version that never applied, so deleting it leaves the order and
|
||
the checksum of every applied version as they were. Do not use `flyway repair` for this: it also
|
||
rewrites the checksums and descriptions of the remaining rows to match the scripts on disk, which
|
||
removes the evidence a checksum mismatch carries. `:adapter:outbound:persistence-jpa:postgresqlConcurrentIndexMigrationIntegrationTest`
|
||
runs the three SQL blocks above, read from this file, against real PostgreSQL for each statement
|
||
this section covers: a unique build that fails on a duplicate, and a rebuild and a drop cancelled
|
||
while they wait for an open transaction. Steps 1–4 run in order, step 3 for the build and the
|
||
rebuild only, and the next deploy applies the version with no invalid index left and the earlier
|
||
version's checksum unchanged.
|
||
|
||
## Escalation
|
||
|
||
- P1 immediate: the required application revision cannot become ready.
|
||
- Escalate to the database owner before Flyway history repair, destructive DDL, point-in-time
|
||
recovery, or primary failover.
|
||
- R3 restore/PITR and failover rehearsal requires a target-like backup topology; local
|
||
Testcontainers evidence is not a substitute.
|
||
|
||
---
|
||
|
||
This runbook is forward-only. The reviewed migration artifact and the readiness lanes' JUnit
|
||
results from CI are the audit sources.
|