DongHyeonkaandClaude Opus 5
0e61f86eb5
Implements the Stable and Experimental JPA persistence platform designs against real PostgreSQL, adapted to this repository's fail-closed 19-leaf registry. The design models the platform as 25 Gradle projects. `src/settings.gradle` throws unless the registry holds exactly 19 leaves, so the plan's modules become packages inside `:adapter:outbound:persistence-jpa` (starter in `:app-bootstrap`, testkit in its own source set). The full mapping, the renames this repository's naming gate required, and every deliberate substitution are recorded in `docs/jpa/repository-adaptation.md`. Seven Docker-backed lanes replace the plan's seven JVM test suites. Each fails closed: a lane that discovers nothing, or a container that cannot start, is an error rather than a skip. Three defects the contracts found against a real server: - `CommitFailureClassifier` treated only SQLSTATE 40003, class 08, and transport breaks as completion-unknown. A backend terminated mid-commit reports 57P01, and the commit record may already be in the WAL — so a possibly-committed transaction could be re-run. 57P01/57P02/57P03 now classify as completion-unknown. - `SchemaTenantMigrationOrchestrator` recorded `MigrateResult`'s target version, which is empty for a tenant already current, reporting migrated tenants as unmigrated during a partial rollout. It now reads the applied version back from the tenant's schema history. - `JpaStreamExecutor` checked only the declared return type for reactive publishers, and `RegisteredPostgreSqlCopyLoader` passed the COPY timeout to `SET`, which is parsed before parameter binding. `JpaModuleBoundaryTest` enforces the plan's module map as package rules; `verifyCleanArchitectureDependencies` governs edges between leaves and cannot see these. Its first assertion is that the import is non-empty, because every rule under it is a `noClasses()` rule and would pass vacuously on an empty import. Verified: 128 container tests across all seven lanes, 1183 unit tests, `:adapter:outbound:persistence-jpa:check`, `:app-bootstrap:check`, `verifyCleanArchitectureDependencies`, `verifyOneTypePerFile`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
86 lines
3.7 KiB
Markdown
86 lines
3.7 KiB
Markdown
# JPA Platform Runbooks
|
|
|
|
Operator procedures for the failures this platform is designed to surface rather than hide.
|
|
|
|
## A transaction reported completion unknown
|
|
|
|
**Signal:** `jpa.transaction.completion.unknown` incremented; a `CompletionUnknownRecord` in the
|
|
reconciliation channel.
|
|
|
|
**What it means:** the commit may or may not have happened. It is not a rollback.
|
|
|
|
**Do not** re-run the use case. That is what the platform refused to do automatically, for the same
|
|
reason.
|
|
|
|
**Procedure:**
|
|
|
|
1. Take the `transactionKey` from the record.
|
|
2. Check the idempotency record for that key.
|
|
3. Check the business row the use case would have written.
|
|
4. Check the outbox for a corresponding event.
|
|
5. If all three agree the write happened, mark the record `COMMITTED` and stop.
|
|
6. If all three agree it did not, the use case may be re-run.
|
|
7. If they disagree or are inconclusive, leave it `STILL_UNKNOWN` and escalate. An inconclusive
|
|
answer is a legitimate outcome; guessing is not.
|
|
|
|
A record with no `transactionKey` cannot be resolved automatically — use the operation name and
|
|
timestamp.
|
|
|
|
## Deadlock or serialization rate rising
|
|
|
|
**Signal:** `jpa.retry.attempt` rising; `jpa.retry.exhausted` non-zero.
|
|
|
|
Retries are expected. Exhaustion is not.
|
|
|
|
1. Group `jpa.retry.attempt` by operation. A single operation dominating means a hot row or an
|
|
inconsistent lock order.
|
|
2. For deadlocks, check whether two operations take the same rows in opposite orders — that is a
|
|
code fix, not a tuning one.
|
|
3. For serialization failures under `SERIALIZABLE`, confirm the isolation is actually required.
|
|
4. Only then consider raising `maxAttempts`. A larger budget on a hot row converts a fast failure
|
|
into a slow one.
|
|
|
|
## Pool exhaustion
|
|
|
|
**Signal:** connection acquisition timeouts; `PoolMeasurement.pending` non-zero.
|
|
|
|
1. Check `REQUIRES_NEW` usage. It takes a second connection while pinning the first, so the pool
|
|
must satisfy `(threads x (1 + depth)) + 1`.
|
|
2. Check for streaming outside a bounded scope — a `Stream` returned past the transaction holds its
|
|
connection until the pool notices.
|
|
3. Check for external calls inside a DB transaction. The design forbids them precisely because an
|
|
HTTP timeout then holds a connection for its whole duration.
|
|
|
|
## Flyway validation failed at startup
|
|
|
|
The deployment is running against a schema it was not built for. It failed closed, which is correct.
|
|
|
|
1. Read the reported error codes (the messages are deliberately not propagated).
|
|
2. `CHECKSUM_MISMATCH` — an applied migration was edited afterwards. Find which change is missing
|
|
from this database. **Do not run `repair`**: it rewrites history to match the scripts, which
|
|
resolves the symptom by deleting the evidence.
|
|
3. `MISSING_SCRIPT` — a migration applied here is not in this build. Usually a rollback to an older
|
|
artifact.
|
|
|
|
## An invalid index exists
|
|
|
|
**Signal:** `FailedConcurrentIndexRecovery.invalidIndexes()` is non-empty.
|
|
|
|
A concurrent build failed. The index is ignored by the planner and maintained by every write.
|
|
|
|
1. Confirm no build is currently running. An in-progress build looks identical in the catalog.
|
|
2. Run the reported `DROP INDEX CONCURRENTLY` outside a migration.
|
|
3. Re-apply the index migration.
|
|
|
|
The platform does not drop these automatically: on a rolling deploy every instance would race to
|
|
drop an index another instance was about to finish building.
|
|
|
|
## The runtime role failed verification
|
|
|
|
Startup refused because the runtime credential holds `CREATE`, or `search_path` contains an
|
|
unapproved schema.
|
|
|
|
This is not a false positive to be worked around. Re-provision from
|
|
`infra/jpa/roles/runtime-roles.sql`; the application's credential having DDL is the condition that
|
|
makes every other schema guarantee unenforceable.
|