# JPA Platform Runbooks

Operator procedures for the failures this platform is designed to surface rather than hide.

## A transaction reported completion unknown

**Signal:** `jpa.transaction.completion.unknown` incremented; a `CompletionUnknownRecord` in the
reconciliation channel.

**What it means:** the commit may or may not have happened. It is not a rollback.

**Do not** re-run the use case. That is what the platform refused to do automatically, for the same
reason.

**Procedure:**

1. Take the `transactionKey` from the record.
2. Check the idempotency record for that key.
3. Check the business row the use case would have written.
4. Check the outbox for a corresponding event.
5. If all three agree the write happened, mark the record `COMMITTED` and stop.
6. If all three agree it did not, the use case may be re-run.
7. If they disagree or are inconclusive, leave it `STILL_UNKNOWN` and escalate. An inconclusive
   answer is a legitimate outcome; guessing is not.

A record with no `transactionKey` cannot be resolved automatically — use the operation name and
timestamp.

## Deadlock or serialization rate rising

**Signal:** `jpa.retry.attempt` rising; `jpa.retry.exhausted` non-zero.

Retries are expected. Exhaustion is not.

1. Group `jpa.retry.attempt` by operation. A single operation dominating means a hot row or an
   inconsistent lock order.
2. For deadlocks, check whether two operations take the same rows in opposite orders — that is a
   code fix, not a tuning one.
3. For serialization failures under `SERIALIZABLE`, confirm the isolation is actually required.
4. Only then consider raising `maxAttempts`. A larger budget on a hot row converts a fast failure
   into a slow one.

## Pool exhaustion

**Signal:** connection acquisition timeouts; `PoolMeasurement.pending` non-zero.

1. Check `REQUIRES_NEW` usage. It takes a second connection while pinning the first, so the pool
   must satisfy `(threads x (1 + depth)) + 1`.
2. Check for streaming outside a bounded scope — a `Stream` returned past the transaction holds its
   connection until the pool notices.
3. Check for external calls inside a DB transaction. The design forbids them precisely because an
   HTTP timeout then holds a connection for its whole duration.

## Flyway validation failed at startup

The deployment is running against a schema it was not built for. It failed closed, which is correct.

1. Read the reported error codes (the messages are deliberately not propagated).
2. `CHECKSUM_MISMATCH` — an applied migration was edited afterwards. Find which change is missing
   from this database. **Do not run `repair`**: it rewrites history to match the scripts, which
   resolves the symptom by deleting the evidence.
3. `MISSING_SCRIPT` — a migration applied here is not in this build. Usually a rollback to an older
   artifact.

## An invalid index exists

**Signal:** `FailedConcurrentIndexRecovery.invalidIndexes()` is non-empty.

A concurrent build failed. The index is ignored by the planner and maintained by every write.

1. Confirm no build is currently running. An in-progress build looks identical in the catalog.
2. Run the reported `DROP INDEX CONCURRENTLY` outside a migration.
3. Re-apply the index migration.

The platform does not drop these automatically: on a rolling deploy every instance would race to
drop an index another instance was about to finish building.

## The runtime role failed verification

Startup refused because the runtime credential holds `CREATE`, or `search_path` contains an
unapproved schema.

This is not a false positive to be worked around. Re-provision from
`infra/jpa/roles/runtime-roles.sql`; the application's credential having DDL is the condition that
makes every other schema guarantee unenforceable.