Implements the Stable and Experimental JPA persistence platform designs against real PostgreSQL, adapted to this repository's fail-closed 19-leaf registry. The design models the platform as 25 Gradle projects. `src/settings.gradle` throws unless the registry holds exactly 19 leaves, so the plan's modules become packages inside `:adapter:outbound:persistence-jpa` (starter in `:app-bootstrap`, testkit in its own source set). The full mapping, the renames this repository's naming gate required, and every deliberate substitution are recorded in `docs/jpa/repository-adaptation.md`. Seven Docker-backed lanes replace the plan's seven JVM test suites. Each fails closed: a lane that discovers nothing, or a container that cannot start, is an error rather than a skip. Three defects the contracts found against a real server: - `CommitFailureClassifier` treated only SQLSTATE 40003, class 08, and transport breaks as completion-unknown. A backend terminated mid-commit reports 57P01, and the commit record may already be in the WAL — so a possibly-committed transaction could be re-run. 57P01/57P02/57P03 now classify as completion-unknown. - `SchemaTenantMigrationOrchestrator` recorded `MigrateResult`'s target version, which is empty for a tenant already current, reporting migrated tenants as unmigrated during a partial rollout. It now reads the applied version back from the tenant's schema history. - `JpaStreamExecutor` checked only the declared return type for reactive publishers, and `RegisteredPostgreSqlCopyLoader` passed the COPY timeout to `SET`, which is parsed before parameter binding. `JpaModuleBoundaryTest` enforces the plan's module map as package rules; `verifyCleanArchitectureDependencies` governs edges between leaves and cannot see these. Its first assertion is that the import is non-empty, because every rule under it is a `noClasses()` rule and would pass vacuously on an empty import. Verified: 128 container tests across all seven lanes, 1183 unit tests, `:adapter:outbound:persistence-jpa:check`, `:app-bootstrap:check`, `verifyCleanArchitectureDependencies`, `verifyOneTypePerFile`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
3.7 KiB
JPA Platform Runbooks
Operator procedures for the failures this platform is designed to surface rather than hide.
A transaction reported completion unknown
Signal: jpa.transaction.completion.unknown incremented; a CompletionUnknownRecord in the
reconciliation channel.
What it means: the commit may or may not have happened. It is not a rollback.
Do not re-run the use case. That is what the platform refused to do automatically, for the same reason.
Procedure:
- Take the
transactionKeyfrom the record. - Check the idempotency record for that key.
- Check the business row the use case would have written.
- Check the outbox for a corresponding event.
- If all three agree the write happened, mark the record
COMMITTEDand stop. - If all three agree it did not, the use case may be re-run.
- If they disagree or are inconclusive, leave it
STILL_UNKNOWNand escalate. An inconclusive answer is a legitimate outcome; guessing is not.
A record with no transactionKey cannot be resolved automatically — use the operation name and
timestamp.
Deadlock or serialization rate rising
Signal: jpa.retry.attempt rising; jpa.retry.exhausted non-zero.
Retries are expected. Exhaustion is not.
- Group
jpa.retry.attemptby operation. A single operation dominating means a hot row or an inconsistent lock order. - For deadlocks, check whether two operations take the same rows in opposite orders — that is a code fix, not a tuning one.
- For serialization failures under
SERIALIZABLE, confirm the isolation is actually required. - Only then consider raising
maxAttempts. A larger budget on a hot row converts a fast failure into a slow one.
Pool exhaustion
Signal: connection acquisition timeouts; PoolMeasurement.pending non-zero.
- Check
REQUIRES_NEWusage. It takes a second connection while pinning the first, so the pool must satisfy(threads x (1 + depth)) + 1. - Check for streaming outside a bounded scope — a
Streamreturned past the transaction holds its connection until the pool notices. - Check for external calls inside a DB transaction. The design forbids them precisely because an HTTP timeout then holds a connection for its whole duration.
Flyway validation failed at startup
The deployment is running against a schema it was not built for. It failed closed, which is correct.
- Read the reported error codes (the messages are deliberately not propagated).
CHECKSUM_MISMATCH— an applied migration was edited afterwards. Find which change is missing from this database. Do not runrepair: it rewrites history to match the scripts, which resolves the symptom by deleting the evidence.MISSING_SCRIPT— a migration applied here is not in this build. Usually a rollback to an older artifact.
An invalid index exists
Signal: FailedConcurrentIndexRecovery.invalidIndexes() is non-empty.
A concurrent build failed. The index is ignored by the planner and maintained by every write.
- Confirm no build is currently running. An in-progress build looks identical in the catalog.
- Run the reported
DROP INDEX CONCURRENTLYoutside a migration. - Re-apply the index migration.
The platform does not drop these automatically: on a rolling deploy every instance would race to drop an index another instance was about to finish building.
The runtime role failed verification
Startup refused because the runtime credential holds CREATE, or search_path contains an
unapproved schema.
This is not a false positive to be worked around. Re-provision from
infra/jpa/roles/runtime-roles.sql; the application's credential having DDL is the condition that
makes every other schema guarantee unenforceable.