Files
clean-architecture-backend-…/docs/jpa/runbooks.md
T
DongHyeonkaandClaude Opus 5 0e61f86eb5 feat(jpa): implement the JPA relational persistence platform
Implements the Stable and Experimental JPA persistence platform designs
against real PostgreSQL, adapted to this repository's fail-closed 19-leaf
registry.

The design models the platform as 25 Gradle projects. `src/settings.gradle`
throws unless the registry holds exactly 19 leaves, so the plan's modules
become packages inside `:adapter:outbound:persistence-jpa` (starter in
`:app-bootstrap`, testkit in its own source set). The full mapping, the
renames this repository's naming gate required, and every deliberate
substitution are recorded in `docs/jpa/repository-adaptation.md`.

Seven Docker-backed lanes replace the plan's seven JVM test suites. Each
fails closed: a lane that discovers nothing, or a container that cannot
start, is an error rather than a skip.

Three defects the contracts found against a real server:

- `CommitFailureClassifier` treated only SQLSTATE 40003, class 08, and
  transport breaks as completion-unknown. A backend terminated mid-commit
  reports 57P01, and the commit record may already be in the WAL — so a
  possibly-committed transaction could be re-run. 57P01/57P02/57P03 now
  classify as completion-unknown.
- `SchemaTenantMigrationOrchestrator` recorded `MigrateResult`'s target
  version, which is empty for a tenant already current, reporting migrated
  tenants as unmigrated during a partial rollout. It now reads the applied
  version back from the tenant's schema history.
- `JpaStreamExecutor` checked only the declared return type for reactive
  publishers, and `RegisteredPostgreSqlCopyLoader` passed the COPY timeout
  to `SET`, which is parsed before parameter binding.

`JpaModuleBoundaryTest` enforces the plan's module map as package rules;
`verifyCleanArchitectureDependencies` governs edges between leaves and
cannot see these. Its first assertion is that the import is non-empty,
because every rule under it is a `noClasses()` rule and would pass
vacuously on an empty import.

Verified: 128 container tests across all seven lanes, 1183 unit tests,
`:adapter:outbound:persistence-jpa:check`, `:app-bootstrap:check`,
`verifyCleanArchitectureDependencies`, `verifyOneTypePerFile`.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-14 14:06:18 +09:00

3.7 KiB

JPA Platform Runbooks

Operator procedures for the failures this platform is designed to surface rather than hide.

A transaction reported completion unknown

Signal: jpa.transaction.completion.unknown incremented; a CompletionUnknownRecord in the reconciliation channel.

What it means: the commit may or may not have happened. It is not a rollback.

Do not re-run the use case. That is what the platform refused to do automatically, for the same reason.

Procedure:

  1. Take the transactionKey from the record.
  2. Check the idempotency record for that key.
  3. Check the business row the use case would have written.
  4. Check the outbox for a corresponding event.
  5. If all three agree the write happened, mark the record COMMITTED and stop.
  6. If all three agree it did not, the use case may be re-run.
  7. If they disagree or are inconclusive, leave it STILL_UNKNOWN and escalate. An inconclusive answer is a legitimate outcome; guessing is not.

A record with no transactionKey cannot be resolved automatically — use the operation name and timestamp.

Deadlock or serialization rate rising

Signal: jpa.retry.attempt rising; jpa.retry.exhausted non-zero.

Retries are expected. Exhaustion is not.

  1. Group jpa.retry.attempt by operation. A single operation dominating means a hot row or an inconsistent lock order.
  2. For deadlocks, check whether two operations take the same rows in opposite orders — that is a code fix, not a tuning one.
  3. For serialization failures under SERIALIZABLE, confirm the isolation is actually required.
  4. Only then consider raising maxAttempts. A larger budget on a hot row converts a fast failure into a slow one.

Pool exhaustion

Signal: connection acquisition timeouts; PoolMeasurement.pending non-zero.

  1. Check REQUIRES_NEW usage. It takes a second connection while pinning the first, so the pool must satisfy (threads x (1 + depth)) + 1.
  2. Check for streaming outside a bounded scope — a Stream returned past the transaction holds its connection until the pool notices.
  3. Check for external calls inside a DB transaction. The design forbids them precisely because an HTTP timeout then holds a connection for its whole duration.

Flyway validation failed at startup

The deployment is running against a schema it was not built for. It failed closed, which is correct.

  1. Read the reported error codes (the messages are deliberately not propagated).
  2. CHECKSUM_MISMATCH — an applied migration was edited afterwards. Find which change is missing from this database. Do not run repair: it rewrites history to match the scripts, which resolves the symptom by deleting the evidence.
  3. MISSING_SCRIPT — a migration applied here is not in this build. Usually a rollback to an older artifact.

An invalid index exists

Signal: FailedConcurrentIndexRecovery.invalidIndexes() is non-empty.

A concurrent build failed. The index is ignored by the planner and maintained by every write.

  1. Confirm no build is currently running. An in-progress build looks identical in the catalog.
  2. Run the reported DROP INDEX CONCURRENTLY outside a migration.
  3. Re-apply the index migration.

The platform does not drop these automatically: on a rolling deploy every instance would race to drop an index another instance was about to finish building.

The runtime role failed verification

Startup refused because the runtime credential holds CREATE, or search_path contains an unapproved schema.

This is not a false positive to be worked around. Re-provision from infra/jpa/roles/runtime-roles.sql; the application's credential having DDL is the condition that makes every other schema guarantee unenforceable.