문제 (재리뷰 round7 영속성 RR7-03-01~05) - 실패한 동시 인덱스 마이그레이션의 복구를 가이드·검사기·학습 문서가 "DROP INDEX 하나"로 안내한다. Flyway는 트랜잭션 밖에서 실패한 버전의 이력 행(success = false)을 남기고, 다음 배포의 validate가 그 행 때문에 다시 멈춘다. runbook에는 이 경우가 없고 이력 수정은 금지다. docs/jpa/runbooks.md "An invalid index exists"도 같은 이유로 재적용에서 멈춘다. - 검사기 Javadoc·학습 문서가 "섞인 파일은 부분 적용되지 않는다"고 쓴다. 가이드가 요구하는 executeInTransaction=false를 붙이면 Flyway가 문장마다 커밋해 앞 문장이 남는다. - 가이드·검사기 Javadoc·실패 메시지·학습 문서가 "Flyway는 REINDEX … CONCURRENTLY를 트랜잭션 안에서 돌린다"고 쓴다. REINDEX SCHEMA|DATABASE CONCURRENTLY는 Flyway가 스스로 트랜잭션 밖에서 돌린다. - LockWaitMigrationInspector는 .sql.conf가 executeInTransaction=true면 SET LOCAL 뒤의 VACUUM FULL·REINDEX SCHEMA를 통과시킨다. Javadoc은 "늘 거부"라고 쓴다. - ledger 잠금 대기 IT가 동시 인덱스 파일을 "이 테스트에 예외로 적으라"고 안내하지만 예외를 적을 곳이 없다. 같은 원인으로 ledger의 다른 IT 두 개 (GrpcOperationLedgerPostgresIT, JpaGrpcOperationLedgerContractIT)도 모든 파일을 한 문자열로 실행해, 동시 인덱스 파일이 생기면 "cannot be executed within a pipeline"으로 실패한다. 고른 방법 - 복구(RR7-03-03): 소유자 결정(2026-11-06)대로 migration-failed.md에 "Recover a failed concurrent index migration" 절을 둔다. 진행 중 빌드 확인 → 무효 인덱스 DROP INDEX CONCURRENTLY → 원인 정리 → 그 버전의 실패 행만 DELETE → 재배포. "Do not"에 이 한 가지 예외를 적고 flyway repair는 계속 금지한다. SQL 세 블록에 runbook-sql 표시를 달고 postgresqlConcurrentIndexMigrationIntegrationTest가 runbook에서 읽어 실 PostgreSQL에서 돌린다(outbox runbook과 같은 방식). 가이드·검사기 Javadoc·메시지·학습 문서·jpa/runbooks.md는 그 절을 가리킨다. - 섞인 파일(RR7-03-04)·REINDEX(RR7-03-02): 문장을 고치고, 같은 IT와 잠금 대기 IT에 사실을 고정한다(섞인 파일 두 경우, REINDEX 다섯 형태). - 검사기(RR7-03-05): 트랜잭션 안에서 돌 수 없는 잠금 문장은 .sql.conf와 상관없이 거부한다(리뷰 권장안). 버린 안: Javadoc만 좁히기 — 그 파일은 PostgreSQL이 배포 때 25001로 거부하므로 검사기가 먼저 알려 주는 쪽이 맞다. 잠그지 않는 문장(plain VACUUM 등)은 이 검사기의 범위가 아니라 그대로 둔다. - ledger(RR7-03-01): IT에 RUN_OUTSIDE_A_TRANSACTION(파일 → 이유) 목록을 두고 잠금 대기 케이스에서 뺀다. 목록의 파일은 같은 IT가 동시 인덱스 문장 하나·executeInTransaction=false인지 ConcurrentIndexMigration Inspector로 본다. optional 빌드는 testkit에 의존할 수 없어 sharedMigrationFileSources가 그 파일과 MigrationResource를 복사한다 (VersionedMigrationFiles와 같은 방식). ledger IT 셋은 마이그레이션을 파일마다 따로 실행한다(Flyway처럼). 버린 안: 안내만 "다룰 수 없다"로 바꾸기 — 다음 동시 인덱스 파일이 생길 때 같은 질문이 다시 남는다. 위험·한계 - 검사기 변경으로 서비스의 자기 스트림 검사가 새로 실패할 수 있다. 걸리는 파일은 PostgreSQL이 배포 때 거부하는 파일뿐이라 배포된 적이 없다(업그레이드 안내 4.1에 적음). - runbook의 DELETE는 운영자가 version을 옮겨 적는다. NOT success라 성공한 행은 지우지 못하지만, 다른 버전의 실패 행을 잘못 적으면 그 행이 지워진다. 한 행만 지워야 하고 0행이면 다시 분류하라고 적었다. - ledger 복사 파일 두 개도 JDK만 써야 한다(각 Javadoc에 적음). - Spotless 검사는 같은 worktree에서 spotlessApply가 변경 없이 끝난 직후라 UP-TO-DATE로 처리됐다. 검증 (desktop, 공유 잠금, 로그 ~/r8-f3-logs/) RED — 기준73d39212+ 이 커밋의 테스트(+ 기준 문서 주장을 단언하는 임시 메서드, + 임시 ledger V004): - persistence-jpa test --tests LockWaitMigrationInspectorTest: 80건 중 2건 실패, 검사기가 빈 목록(통과)을 돌려줌 (red-unit.log) - postgresqlMigrationLockWaitIntegrationTest: 47건 중 4건 실패 — 옛 주장 대로면 거부여야 할 REINDEX SCHEMA|DATABASE CONCURRENTLY가 applied, conf=true VACUUM FULL·REINDEX SCHEMA를 PostgreSQL은 25001로 거부하는데 검사기는 통과 (red2-lockwait.log) - postgresqlConcurrentIndexMigrationIntegrationTest: 9건 중 3건 실패 — 옛 절차(DROP INDEX) 뒤 재배포가 "Detected failed migration to version 2", conf를 붙인 섞인 파일이 행 3001·무효 인덱스·실패 이력을 남김, runbook 블록이 없어 복구 테스트 준비 실패(RED로 세지 않음) (red2-concurrent.log) - ledger integrationTest(V004 동시 인덱스): 6건 중 4건 실패 — 잠금 대기 IT "name it as an exception here", 다른 IT 둘 "cannot be executed within a pipeline" (red-ledger.log) GREEN — 이 커밋의 트리b49369e7: - :adapter:outbound:persistence-jpa:check rc=0 (test 666건 실패 0), :testQualityCheck rc=0 - postgresqlMigrationLockWaitIntegrationTest rc=0 (42건 실패 0) - postgresqlConcurrentIndexMigrationIntegrationTest rc=0 (7건 실패 0) - -p optional-platforms :grpc:grpc-operation-ledger-jpa:check integrationTest testQualityCheck rc=0 (IT 3+2+42건 실패 0) - :app-bootstrap:documentContractTest rc=0 (66건 실패 0) - 두 빌드의 verifyStrictLaneStages --dry-run rc=0 - ledger 탐침(임시 V004): 목록에 없으면 잠금 대기 케이스만 실패하고 메시지가 RUN_OUTSIDE_A_TRANSACTION을 가리킴(48건 중 1건), 목록에 넣으면 rc=0, 목록에 넣고 SET LOCAL을 섞으면 목록 검사가 "mixes a concurrent index statement"로 거부 - python3 scripts/check-work-docs.py --links docs: 깨진 링크 0 Refs: docs/reviews/2026-11-03-rereview/round7/03-persistence.md
8.8 KiB
title, category, error_codes, severity, owner, last_updated, status
| title | category | error_codes | severity | owner | last_updated | status | |
|---|---|---|---|---|---|---|---|
| Runbook — MIGRATION_FAILED (DB 마이그레이션 실패) | INTERNAL |
|
P1 | oncall | 2026-11-07 | active |
Runbook: MIGRATION_FAILED (runbook://migration/failed)
Symptoms
- Container exits with code 70 (migration failure exit)
- Structured log with
error.code=MIGRATION_FAILED,startup.phase=migration - App refuses to start (fail-fast)
- JPA capability adapter refuses activation because its
capability_schema_registry.lifecycle_stateis notACTIVE
Diagnosis
-
Stop rollout and keep the failed revision out of readiness. Do not route traffic to a partially migrated instance.
-
Identify the exact stream from
src/config/jpa/readiness-cards.yaml. Each stream has an independentlocationandhistory-table; do not infer ownership from a broadclasspath:db/migrationscan. -
From a privileged migration session, capture the stream state before changing anything:
select installed_rank, version, description, success from <owned_history_table> order by installed_rank; select capability_id, installation_origin, core_epoch, feature_revision, lifecycle_state from capability_schema_registry where capability_id = '<card-id>'; -
Check whether any owned relation was created without a successful history entry. Compare only against the owned tables in the reviewed migration; do not drop unrelated relations.
-
Classify the failure:
-
lock/statement timeout: remove the blocker or reduce rollout concurrency, then rerun. A migration that locks a table already in use gives up after its own
SET LOCAL lock_timeout(5 s; SQLState55P03lock_not_available) and rolls back whole, so the history ends one version below it. The blocker is a transaction that touched the table and is still open — often an idle-in-transaction session. Find it by the lock it holds on that table (<table>is the table the failing statement locks; for aREFERENCES, also the table it references):SELECT DISTINCT a.pid, a.state, a.xact_start, a.query FROM pg_locks l JOIN pg_stat_activity a USING (pid) WHERE l.relation = '<table>'::regclass AND l.granted AND a.pid <> pg_backend_pid() ORDER BY a.xact_start;queryis only that session's last statement, not the one that touched the table, so do not look for the table name in it. Listing every long transaction instead names unrelated ones (reports, batches) beside the blocker. End the blocker, then rerun (docs/jpa/migration-guide.md"Bound the lock wait"); -
SQL/data precondition: create a new forward migration that makes the precondition explicit; a duplicate outbox order key stopping V21 (
ux_outbox_event_aggregate_order) is the known case — clean it up withoutbox-dead-letter.md§8 and rerun, the migration stays as it is; -
checksum mismatch: compare the deployed artifact with the already applied script before considering repair;
-
a concurrent index migration failed (
CREATE [UNIQUE] INDEX CONCURRENTLYorREINDEX … CONCURRENTLYthat Flyway ran outside a transaction): a duplicate value for a unique index, a deadlock,statement_timeoutor a cancelled build. Outside a transaction nothing rolled back: the migration left an invalid index and a history row withsuccess = falsefor that version, and every later deploy fails validation ("Detected failed migration to version …") until the row is gone. Dropping the index alone does not clear it. Follow "Recover a failed concurrent index migration" below; -
connection/TLS failure: fix transport or credentials without changing Flyway history.
-
Action
-
Prefer forward recovery. Fix the environmental blocker or add a new immutable migration, then rerun the same owned stream with its exact history table.
-
For an optional stream that never installed successfully, keep the capability marker absent and the runtime adapter disabled until migration succeeds.
-
After a successful migration, validate:
- the history contains only successful expected versions;
core_epochandfeature_revisionmatch the readiness registry;- the marker is
INSTALLED_INACTIVE; - owned objects and constraints exist.
-
Change the marker to
ACTIVEonly after the compatible application revision is deployed and its readiness check succeeds. Disabling or rolling back application code changes the marker toINSTALLED_INACTIVE; it does not drop history or owned data. -
Re-run the readiness lanes before promoting (every card's scenarios, no skips):
cd src ./gradlew :adapter:outbound:persistence-jpa:integrationTestLaneCheck --console=plain -
Record the failed revision, stream/history table, root cause, recovery migration, elapsed time and verification artifact in the incident.
Do not:
- edit an already applied migration;
- delete or rewrite Flyway history to make validation green. The one exception is the failed row of a failed concurrent index migration, deleted as "Recover a failed concurrent index migration" below says — that row and nothing else;
- run
flyway repairbefore checksum provenance is proven and reviewed; - use
clean, destructive rollback, or schema-wide restore as the first response; - mark a capability
ACTIVEbefore its migration and adapter readiness succeed.
If commit outcome was indeterminate during the failure, reconcile by the application
OperationId/idempotency reference before retrying business work. Never blind-retry a commit whose
result is unknown.
Recover a failed concurrent index migration
The one case where this runbook changes Flyway history (owner decision, 2026-11-06). It applies to a
migration that holds a single concurrent index statement which Flyway ran outside a transaction —
with the executeInTransaction=false the guide asks for, or on its own for CREATE/DROP INDEX CONCURRENTLY (docs/jpa/migration-guide.md "Concurrent index builds") — and whose row for this
version in the stream's history table says success = false. A migration that mixed other
statements in has already committed them: reconstruct that state with the database owner instead.
- Confirm no build is still running. An index that is still being built looks invalid in the catalog too, and dropping it races the build. This query returns no rows when none is:
SELECT pid, relid::regclass AS table_name, index_relid::regclass AS index_name, phase
FROM pg_stat_progress_create_index
WHERE datname = current_database();
- Drop the invalid index the build left (
docs/jpa/runbooks.md"An invalid index exists" lists them). Run it on its own, not insideBEGIN:CONCURRENTLYrefuses a transaction block.
DROP INDEX CONCURRENTLY IF EXISTS <invalid_index>;
- Remove the cause. For a unique index, clean up the duplicate rows with the data owner (a forward fix, never by editing the migration). For a deadlock or a timeout, end or wait out the transaction that held the build.
- Delete that version's failed row from the stream's history table, and nothing else.
<version>is theversioncolumn of the failed row as the Diagnosis query shows it (22.1forV22_1__…). It must delete exactly one row; if it deletes none, this is not the failure this section is for, so classify it again.
DELETE FROM <owned_history_table>
WHERE version = '<version>' AND NOT success;
- Deploy again. Flyway runs the same migration from the start and builds the index.
The failed row is the record of a version that never applied, so deleting it leaves the order and
the checksum of every applied version as they were. Do not use flyway repair for this: it also
rewrites the checksums and descriptions of the remaining rows to match the scripts on disk, which
removes the evidence a checksum mismatch carries. :adapter:outbound:persistence-jpa:postgresqlConcurrentIndexMigrationIntegrationTest
runs the three SQL blocks above, read from this file, against real PostgreSQL: a unique build fails
on a duplicate, steps 1–4 run, and the next deploy applies the version with the index valid and the
earlier version's checksum unchanged.
Escalation
- P1 immediate: the required application revision cannot become ready.
- Escalate to the database owner before Flyway history repair, destructive DDL, point-in-time recovery, or primary failover.
- R3 restore/PITR and failover rehearsal requires a target-like backup topology; local Testcontainers evidence is not a substitute.
This runbook is forward-only. The reviewed migration artifact and the readiness lanes' JUnit results from CI are the audit sources.