# JPA Platform Runbooks Operator procedures for the failures this platform is designed to surface rather than hide. ## A transaction reported completion unknown **Signal:** a write use case failed with a commit-phase error (connection lost, `57P01`, class `08`) from `TransactionPort`. **What it means:** the commit may or may not have happened. It is not a rollback. **Do not** re-run the use case. That is what the platform refused to do automatically, for the same reason. **Procedure:** 1. Take the idempotency key or operation id of the failed request. 2. Check the idempotency record for that key. 3. Check the business row the use case would have written. 4. Check the outbox for a corresponding event. 5. If all three agree the write happened, stop — the request succeeded. 6. If all three agree it did not, the use case may be re-run. 7. If they disagree or are inconclusive, escalate. An inconclusive answer is a legitimate outcome; guessing is not. ## Deadlock or serialization rate rising **Signal:** `DB_DEADLOCK` / serialization failures rising in the persistence error codes. `SpringTransaction` does not retry, so each one reaches the caller. 1. Group the failures by operation. A single operation dominating means a hot row or an inconsistent lock order. 2. For deadlocks, check whether two operations take the same rows in opposite orders — that is a code fix, not a tuning one. 3. For serialization failures under `SERIALIZABLE`, confirm the isolation is actually required. ## Pool exhaustion **Signal:** connection acquisition timeouts; `PoolMeasurement.pending` non-zero. 1. Check `REQUIRES_NEW` usage. It takes a second connection while pinning the first, so the pool must satisfy `(threads x (1 + depth)) + 1`. 2. Check for streaming outside a bounded scope — a `Stream` returned past the transaction holds its connection until the pool notices. 3. Check for external calls inside a DB transaction. The design forbids them precisely because an HTTP timeout then holds a connection for its whole duration. ## Flyway validation failed at startup The deployment is running against a schema it was not built for. It failed closed, which is correct. 1. Read the reported error codes (the messages are deliberately not propagated). 2. `CHECKSUM_MISMATCH` — an applied migration was edited afterwards. Find which change is missing from this database. **Do not run `repair`**: it rewrites history to match the scripts, which resolves the symptom by deleting the evidence. 3. `MISSING_SCRIPT` — a migration applied here is not in this build. Usually a rollback to an older artifact. ## An invalid index exists **Signal:** this query returns rows. ```sql select n.nspname || '.' || i.relname as invalid_index from pg_index x join pg_class i on i.oid = x.indexrelid join pg_namespace n on n.oid = i.relnamespace where not x.indisvalid; ``` A concurrent build failed. The index is ignored by the planner and maintained by every write. 1. Confirm no build is currently running. An in-progress build looks identical in the catalog. 2. Run `DROP INDEX CONCURRENTLY IF EXISTS ;` outside a migration. 3. If a migration built it, Flyway recorded that version as failed (`success = false`) and refuses every later deploy until that row is gone: dropping the index does not clear it. Follow [migration-failed.md](../runbooks/migration-failed.md) "Recover a failed concurrent index migration", which removes the cause and deletes that one row. 4. Re-apply the index migration. The platform does not drop these automatically: on a rolling deploy every instance would race to drop an index another instance was about to finish building. ## The runtime role failed verification Startup refused because the runtime credential holds `CREATE`, or `search_path` contains an unapproved schema. This is not a false positive to be worked around. Re-provision from `infra/jpa/roles/runtime-roles.sql`; the application's credential having DDL is the condition that makes every other schema guarantee unenforceable.