문제: 재리뷰 수정 커밋(bbe944bd)만 다시 본 2차 재리뷰가 Important 7건,
Minor 22건을 찾았다. 결과 불명 503 뒤 같은 키를 다시 보내면 Redis
store가 lease 지난 실행을 넘겨받아 작업을 다시 돌렸다. replay TTL이
lease보다 짧으면 lease 도중 key가 사라져 같은 키가 두 번 실행됐다.
@SystemInvoked 규칙이 매뉴얼 09장의 이벤트 소비 경로까지 막았다. V22가
outbox_event에 가장 강한 잠금을 lock_timeout 없이 걸었다. prod의
fixture- 거부가 HOSTNAME 같은 비밀 아닌 값까지 막았다. 데모 허용 목록의
test·demo 프로필로 sample이 뜨지 않았다. 기동 rule bean의 @Value 필드가
조용히 null이 됐다.
방법: 영역별 4묶음이 test-driven-development 절차로 실패 테스트를
먼저 쓰고 고쳤다. 결정은 재리뷰 README 9.1절이다.
- 두 멱등성 store가 같은 계약을 따른다. lease가 지난 실행 기록은 다시
돌리지 않고 포기로 표시해 500 RECOVERY_REQUIRED로 답한다. 기록 수명은
lease + replay TTL이고, replay-ttl < processing-lease면 기동이 실패한다.
- 사용자 transport만 @SystemInvoked use case에 닿지 못한다. 메시지
handler·scheduler는 새 표시 @SystemCaller(reason)를 달고 use case를
생성자에서 new로 만든다. 매뉴얼 경로를 테스트로 고정했다.
- 아직 어디에도 적용되지 않은 마이그레이션(V20·V22·알림 V11~V13·sample
V8_1·grpc V002·V003)에 lock_timeout 5s를 넣고, 텍스트 검사기와 실 PG
lane을 더했다. 이미 배포된 24개는 동결 목록으로 둔다.
- fixture- 검사는 비밀로 분류된 값만 본다. sample은 local·test·demo에서
object storage를 끈다. 기동 rule bean에 주입 멤버가 있으면 모든
프로필에서 기동이 실패한다. 리뷰어가 제안한 "static @Bean 선언 강제"는
문제없는 등록을 막고 @Value 필드는 통과시켜서 버렸다.
- 이 수정 전에 move된 파일의 쿼터 차이를 찾는 쿼리와 재계산 SQL을
docs/fileserver/operations.md에 적었다.
위험·한계: lease 뒤 늦게 끝난 첫 시도의 결과는 기록되지 않는다(그 요청도
500). @SystemCaller가 없는 메시지 handler, 주입 멤버가 있는 기동 rule,
working-directory 없이 ./gradlew를 부르는 workflow 명령은 파생 서비스에서
새로 실패하거나 세지지 않는다(업그레이드 안내 B-14~16·B-36·B-39).
spotbugsTestFixtures는 게이트 밖이고 기준 커밋에서도 실패하며, 새 검사기의
정규식 경고 8건이 더해졌다. 실제 CI runner 실행은 확인하지 못했다(Gitea
repo의 Actions가 꺼져 있다).
검증: ci, 두 빌드의 verifyStrictLaneStages --dry-run, mainCheck,
:integrationCheck, -p optional-platforms ci integrationCheck
performanceCheck, :performanceCheck, :releaseCheck(-PgitRevision),
verify-compose-profile-contracts(18 lane) 첫 실행 통과. 쿼터 재계산
SQL은 실 PostgreSQL 16에서 차이 2행 → 0행을 확인했다.
Refs: docs/reviews/2026-11-03-rereview/README.md 9·9.1·10절
BREAKING CHANGE: replay-ttl < processing-lease 설정은 기동 실패, ledger
release가 claim token을 받는다, HttpBodyReadException(IOException 하위)이
본문 실패를 싣는다, 기동 rule·비밀 목록 bean의 주입 멤버 금지,
verifyStrictLaneStages가 workflow 명령의 working-directory로 빌드를 정한다.
95 lines
5.2 KiB
Markdown
95 lines
5.2 KiB
Markdown
# Migration Guide
|
|
|
|
Design §31-§32. Flyway owns the schema; Hibernate only validates.
|
|
|
|
## Who may change the schema
|
|
|
|
| Environment | Mode |
|
|
|---|---|
|
|
| local, test, dev | migrate at startup with the migration credential |
|
|
| staging, prod | deployment-owned migration; the application validates only |
|
|
|
|
Migrating from inside the application in production means every instance of a rolling deploy races
|
|
to apply the same script, and the loser's failure is indistinguishable from a real one.
|
|
|
|
`ddl-auto` is `validate` or `none`. Never `update`: it never drops or narrows anything, so it
|
|
produces a schema that is neither the old one nor the one the migrations describe — silently, on
|
|
whichever instance started first.
|
|
|
|
## Validation fails closed and never repairs
|
|
|
|
At startup Flyway's own validation (and Hibernate `ddl-auto=validate`) refuses a checksum mismatch,
|
|
a missing migration, or a schema Hibernate disagrees with. Nothing in the application calls
|
|
`repair`. In tests, the testkit's `FlywayValidationGate` turns a failed `ValidateResult` into
|
|
`SchemaMismatchException`; the migration contract tests use it.
|
|
|
|
Repair rewrites the schema history table to match whatever scripts are on disk. That resolves the
|
|
symptom by deleting the evidence: a checksum mismatch means the deployed script differs from the
|
|
applied one, and the interesting question is which change is missing from this database. Repair
|
|
makes that question unaskable. It exists only as an explicit admin operation with an operator, a
|
|
reason, and an approval (design §8.4).
|
|
|
|
Only Flyway's structured error codes reach the exception. Its messages embed the script path and
|
|
part of the failing statement.
|
|
|
|
## Bound the lock wait
|
|
|
|
A migration that locks a table already in use starts with `SET LOCAL lock_timeout = '5s';`, before
|
|
the first statement that takes the lock. `ALTER TABLE` takes the strongest lock there is, which
|
|
conflicts even with a plain read; a non-concurrent `CREATE INDEX` takes one that stops every write;
|
|
`UPDATE` and `DELETE` wait on the rows other transactions hold. Unbounded, the migration waits for
|
|
the oldest open transaction that touched the table — an operator's forgotten `BEGIN` in psql has no
|
|
deadline — and every read and write that arrives meanwhile queues behind the migration, holding a
|
|
pool connection while it waits. With the bound, the migration fails after five seconds, Flyway
|
|
rolls it back whole, the database stays on the version before it and the new release does not
|
|
start: a failed deploy instead of an outage. Deploy again when the transaction has ended.
|
|
|
|
`LOCAL`, because a plain `SET` outlives the migration on the pooled connection Flyway borrowed and
|
|
changes every statement that later runs on it. A statement on a table the same migration created a
|
|
moment earlier waits for nothing and needs no bound.
|
|
|
|
Enforced, not remembered: `LockWaitMigrationInspector` (persistence-jpa testkit) reads a script
|
|
and names every statement that locks an existing table before the bound. `MigrationLockWaitTest`
|
|
runs it over every tree this leaf's database runs, `SampleMigrationLockWaitTest` over the sample's
|
|
stream; a service stream copies the latter with its own directory. The migrations released before
|
|
the rule wait unbounded and stay that way (they are applied, and applied files are never edited);
|
|
the test lists them and refuses any new one. `PostgreSqlMigrationLockWaitIntegrationTest`
|
|
(`postgresqlMigrationLockWaitIntegrationTest`) and the gRPC ledger's
|
|
`GrpcOperationLedgerMigrationLockWaitIT` show on real PostgreSQL that a bounded migration gives up
|
|
within the bound while another transaction holds the table, and leaves the version below it. The
|
|
rule came from V21 having the bound and V22, in the same commit, not having it (re-review
|
|
2026-11-03 round 2, RR-02-09).
|
|
|
|
## Concurrent index builds
|
|
|
|
`CREATE INDEX CONCURRENTLY` cannot run inside a transaction block, and Flyway wraps migrations in
|
|
one by default. The migration therefore needs a companion configuration:
|
|
|
|
```conf
|
|
# V42__order_index.sql.conf
|
|
executeInTransaction=false
|
|
```
|
|
|
|
The testkit's `ConcurrentIndexMigrationInspector` checks a script for this in a migration test
|
|
(nothing checks it at startup), and additionally requires the migration to contain nothing else. A failed concurrent build leaves an invalid index behind;
|
|
recovering is a single `DROP INDEX` when the migration did nothing else, and a manual reconstruction
|
|
of partial state when it did.
|
|
|
|
An invalid index is not merely useless — the planner ignores it while every write still maintains
|
|
it. [runbooks.md](runbooks.md) has the query that finds them. Nothing drops them automatically: an
|
|
invalid index can also mean a build is still running, and the two are indistinguishable from the
|
|
catalog alone.
|
|
|
|
## Upgrade scenarios
|
|
|
|
Three, each catching something the others do not:
|
|
|
|
| Scenario | Catches |
|
|
|---|---|
|
|
| `empty` | an early migration edited to match a later one, no longer applying to a fresh database |
|
|
| `previous-release` | the actual deployment path; the only one exercising this release's migrations |
|
|
| `oldest-supported` | a migration that silently assumes state only recent databases have |
|
|
|
|
Each asserts a data invariant, not just the schema version. A migration that renames a column and
|
|
loses its contents leaves the version correct and the data gone.
|