1103 lines
66 KiB
Markdown
1103 lines
66 KiB
Markdown
# Wave 2 — decisions and evidence
|
|
|
|
Required by the Wave 2 exit criteria. One entry per task that made a choice a reader could
|
|
reasonably have made differently, with what was run rather than what was expected.
|
|
|
|
---
|
|
|
|
## E1 — GraphQL: the two safety axes collapsed into one deployment mode (GQL-INT-002)
|
|
|
|
### What the defect actually was
|
|
|
|
`GraphQlPlatformSettings` declared `@DefaultValue("false") boolean production` and
|
|
`@DefaultValue("PRODUCTION_PUBLIC") GraphQlPlatformEnvironment environment` **in the same record**.
|
|
The shipped default therefore described an internet-facing production endpoint whose protections
|
|
behaved as if it were a laptop, because the two axes were read by different code:
|
|
|
|
| Behaviour | Read from |
|
|
| --- | --- |
|
|
| GraphiQL refusal | both |
|
|
| cursor signing key required | `production` |
|
|
| introspection refusal | `environment` |
|
|
| allow-by-default authorization | `production` |
|
|
| anonymous principal handling in the web interceptor | `production` |
|
|
|
|
So `environment=PRODUCTION_PUBLIC` with the boolean left alone gave a deployment that refused
|
|
introspection while handing out an allow-by-default authorization policy — production by one axis,
|
|
development by the other.
|
|
|
|
### The decision
|
|
|
|
`backend.graphql.deployment-mode` is now the only axis. `GraphQlPlatformEnvironment` became
|
|
`GraphQlDeploymentMode`; `production()` is a derived accessor on the settings, not a component.
|
|
|
|
**No default, deliberately.** The mode is the one setting in this record with none. A guess that
|
|
lands on development is an unauthorized endpoint; a guess that lands on production is an outage an
|
|
operator cannot explain. Absence is refused by name, and `production()` reads fail-closed (`true`)
|
|
for the window between binding and that refusal.
|
|
|
|
**Four modes, not six.** `TEST` and `STAGING` were removed. `CapabilityDependencyValidator` permits
|
|
`local→LOCAL`, `dev→DEV`, `prod→{PRODUCTION_INTERNAL, PRODUCTION_PUBLIC}`, so neither constant was
|
|
selectable by any shipped runtime — a posture nobody can deploy and nobody notices is dead.
|
|
`GraphQlDeploymentModeRegistryParityTest` derives its cases from `GraphQlDeploymentMode.values()` and
|
|
asserts that the enum, `docs/registries/env-keys.yaml`, and the validator agree, so a fifth mode
|
|
cannot be added without a registry row and a runtime that accepts it.
|
|
|
|
**The retired keys fail rather than being ignored.** Spring's binder ignores unknown keys, so an
|
|
operator who set `backend.graphql.production` would have got a clean startup and a silently different
|
|
safety posture — worse than the split-brain, because the old configuration at least did something.
|
|
`GraphQlActivationEnvironmentPostProcessor` refuses either retired key while the master switch is on,
|
|
naming `APP_GRAPHQL_DEPLOYMENT_MODE`, including when the replacement is also set: two keys that can
|
|
disagree is the defect, and the new one winning silently is the same bug.
|
|
|
|
### Two defects the `local-graphql` lane found that the spec did not predict
|
|
|
|
Neither was reachable by reading; both took a real composition.
|
|
|
|
1. **Boot answers introspection by default and this platform does not.** With the switch on and
|
|
nothing else configured, `spring.graphql.schema.introspection.enabled` was `true` while
|
|
`backend.graphql.console.introspection-enabled` was `false`, and the runtime validator correctly
|
|
refused a deployment with two answers to one question — a contradiction nobody had configured.
|
|
The same post-processor now contributes the platform's console values as the framework's defaults
|
|
at the **lowest** precedence, so an operator who sets either key still wins and is still
|
|
validated; what is removed is the disagreement that existed with nothing set at all.
|
|
|
|
2. **The Keycloak realm artifact could never have imported.** Keycloak deserializes the realm into
|
|
`RealmRepresentation` with unknown fields **rejected**, not ignored, so the `"_comment"` key
|
|
documenting why the client secret is a `${...}` reference failed the entire import and exited the
|
|
container 1. Fixing it revealed a second one — `"_flowComment"` on a client — which had been
|
|
invisible behind the first. Both are gone, the rationale moved to `infra/keycloak/README.md`, and
|
|
`verify-compose-profile-contracts.sh` now fails on any key in that artifact starting with `_`.
|
|
That check runs in the ordinary suite through `ComposeMergeCharacterizationTest`.
|
|
|
|
A third, in the same lane: the wrapper writes the client secret at mode 0600 as the host user, the
|
|
Keycloak image happens to run as the same uid, and `curlimages/curl` runs as uid 100 — so
|
|
`auth-smoke` read `Permission denied` and the lane failed on the smoke client rather than on
|
|
anything it was checking. Compose ignores a secret's `uid`/`gid`/`mode` outside swarm, so the
|
|
container reads it as root. The two alternatives are worse: a world-readable host file leaves a
|
|
credential readable by every process on the machine, and an environment variable puts the value in
|
|
`docker compose config` output and in `ps`.
|
|
|
|
### Verification
|
|
|
|
```
|
|
$ cd src && ./gradlew :adapter:inbound:graphql:test :app-bootstrap:test BUILD SUCCESSFUL
|
|
$ ./gradlew :adapter:inbound:graphql:graphqlStableTest BUILD SUCCESSFUL
|
|
$ ./gradlew :adapter:inbound:graphql:verifyGraphQlApiSurface OK
|
|
$ ./scripts/verify-compose-profile-contracts.sh all 15 lanes match
|
|
$ ./scripts/run-compose-runtime-smoke.sh --lane local-graphql == local-graphql: passed
|
|
```
|
|
|
|
`GraphQlDeploymentModeContractTest` — 22 cases, 0 skipped. The lane's reported activation:
|
|
|
|
```json
|
|
{"activeProfile":"local",
|
|
"switches":{"backend.graphql.enabled":true,
|
|
"ca-skeleton.persistence-jpa.enabled":false,
|
|
"ca-skeleton.notification.platform.enabled":false,
|
|
"ca-skeleton.persistence-mongo.enabled":false,
|
|
"app.messaging.enabled":false},
|
|
"dataSourceRequiredBy":[],"persistenceVendor":"none"}
|
|
```
|
|
|
|
GraphQL is the first inbound transport proven on against a real identity provider with every other
|
|
switch off and no database connection required. `local-graphql` is now `blocking: true` in
|
|
`src/config/runtime/compose-profile-contracts.json`; its `notBlockingReason` is gone because the
|
|
reason is gone.
|
|
|
|
### Recorded, not fixed
|
|
|
|
- **`verifyGraphQlApiSurface` was already failing at HEAD.** The committed snapshot predates a rename
|
|
sweep: `GraphQlPlatformProperties`→`GraphQlPlatformSettings`, `GraphQlReleaseFailure`→
|
|
`GraphQlReleaseException`, `GraphQlStructuralLimitViolation`→`GraphQlStructuralLimitException`,
|
|
`GraphQlAdvancedReleaseFailure`→`GraphQlAdvancedReleaseException`,
|
|
`GraphQlWebSocketProtocolError`→`GraphQlWebSocketProtocolException`, plus the Wave 1 root and import
|
|
filter. The approved update absorbed those alongside this wave's four entries. Noted rather than
|
|
buried: a snapshot that has been red for a while stops being a gate.
|
|
- **`BoundedPreparsedDocumentProviderTest.concurrentMissesOnOneKeyParseOnce` is flaky.** It failed
|
|
once under load (two modules compiling in parallel) with two parses where the single-flight
|
|
contract requires one, and passed 3/3 when re-run alone. Untouched by this task and not
|
|
investigated here — a genuine single-flight race under contention is a Wave 4 candidate, and a
|
|
cache test that only holds on an idle machine is not holding.
|
|
|
|
---
|
|
|
|
## Section A — the JPA-on lanes, and the five defects between them and green
|
|
|
|
Eight lanes were blocked on one recorded finding. Fixing it uncovered four more, each hidden behind
|
|
the one before it, and each invisible to the suite that existed. They are listed in the order they
|
|
surfaced, because that order is the point: no amount of reading found any of them.
|
|
|
|
### 1. The fileserver entity scan (the recorded blocker)
|
|
|
|
Six entities in `...persistence.fileserver` map six `fs_*` tables that live only in
|
|
`db/migration/jpa/fileserver`, a stream applied only when that capability is on. The primary Flyway
|
|
location creates none of them. The scan was unconditional, so `ddl-auto=validate` against real
|
|
PostgreSQL failed on `fs_cleanup_item` in every deployment that had switched the capability off.
|
|
|
|
Notification had already been given a gated scan for exactly this reason; fileserver had not.
|
|
`FileserverJpaPersistenceConfig` now carries the capability's own condition — the same one its
|
|
adapter beans already carried, so "disabled" stopped meaning two different things one annotation
|
|
apart.
|
|
|
|
**The half-fix that came with it.** Moving a scan out of the unconditional list registers it
|
|
nowhere: `dev.caskeleton.adapter.outbound.persistence..*` is excluded from the composition root's
|
|
component scan by design, and the leaf's `config` package may depend on `api` alone. So the
|
|
composition root is the only thing that can register it, and `NotificationJpaPersistenceConfig` —
|
|
which nothing imports — has been in that state since it was created. The notification capability has
|
|
no JPA persistence at all. `CapabilityEntityScanRegistrationTest` asserts the fileserver import and
|
|
**asserts the notification gap**, with instructions to invert rather than delete the assertion when
|
|
Section D wires it. A silently unwired capability is what that file exists to make loud.
|
|
|
|
`FileserverRoundTripContractTest` declared its own `@EntityScan`/`@EnableJpaRepositories` over the
|
|
same packages, which both produced a duplicate bean definition and would have kept passing if the
|
|
shipped scan were registered nowhere. It uses the shipped one now.
|
|
|
|
### 2. `request_hash` — `char(64)` in the migration, `varchar(64)` in the entity
|
|
|
|
```
|
|
Schema-validation: wrong column type encountered in column [request_hash] in table
|
|
[idempotency_record]; found [bpchar (Types#CHAR)], but expecting [varchar(64) (Types#VARCHAR)]
|
|
```
|
|
|
|
Always-installed, so every JPA-on deployment failed. Invisible under H2, whose `create-drop` builds
|
|
the schema from the entities and therefore cannot disagree with them — a vendor that generates the
|
|
schema can never report a mismatch with it.
|
|
|
|
Forward migrations in both streams rather than an edit to `V1`: an applied migration's checksum is a
|
|
promise to every deployment that already ran it. Both are guarded on the current column type, because
|
|
the two streams keep separate histories and their relative order is not fixed.
|
|
|
|
`PostgreSqlDefaultPersistenceUnitIntegrationTest` now does for the default persistence unit what the
|
|
notification and fileserver suites already did for theirs, deriving its packages from
|
|
`PersistenceJpaConfig` so it covers the unit as it grows. Seconds, against a four-minute lane.
|
|
|
|
### 3. `connection-timeout: 5s` — a default only its overriders could survive
|
|
|
|
`spring.datasource.hikari.connection-timeout` binds onto `HikariConfig#setConnectionTimeout(long)`.
|
|
The shipped default was `5s`; `application-local.yml` overrides it with `30000`. So **every `prod`
|
|
and `dev` deployment failed to start** and every `local` one worked, which is why only a prod lane
|
|
could find it. The env registry declared `type: duration, default: 5s`, and `application.yml` copied
|
|
that — the registry documented a value the property cannot accept.
|
|
|
|
`HikariPoolConstraintValidator` reads these keys with `DurationStyle` and accepts `5s` happily. That
|
|
tolerance is what made the wrong default look supported. It is a **deliberate, tested contract**
|
|
(`durationStringsParticipateInCrossPropertyValidation`), so it is recorded as an open Section A
|
|
finding rather than quietly changed: a validator that passes values the binder rejects is answering a
|
|
different question than the one it appears to answer.
|
|
|
|
`ShippedDefaultBindabilityTest` is the cheap half — a scan, not a boot, over the millisecond-typed
|
|
pool knobs. Its first version matched on the key's leaf name and reported
|
|
`server.tomcat.connection-timeout: 20s`, a genuine `Duration`, as a defect; it flattens through
|
|
Boot's own `YamlPropertySourceLoader` now. **Verified by reverting the default and watching it fail.**
|
|
|
|
### 4. The dev stack put the application on a different network from its database
|
|
|
|
`UnknownHostException: db`, from a container running and healthy a metre away. The local overlay
|
|
joins `caskeleton-infra` and the dev overlay declared no `networks:` at all, so Compose put it on
|
|
`default` — a network of its own making. The omission reads as a working stack until something has to
|
|
resolve a name across it.
|
|
|
|
### 5. Generated credentials the database could never see
|
|
|
|
The wrapper generates a per-run password into `src/.env.lane`, which the application reads as an
|
|
`env_file`. The `db` service takes `POSTGRES_PASSWORD` from Compose **interpolation**
|
|
(`${APP_DATASOURCE_PASSWORD:-ca_skeleton}`), and interpolation reads the process environment and the
|
|
project `.env` file — never a service's `env_file`. The application got the generated password, the
|
|
database got the literal default, and they could not agree.
|
|
|
|
`shared-infra-local` passed only because the local overlay restates the value, which made this look
|
|
like a dev-specific problem rather than the general one it is. The wrapper exports the credentials
|
|
now, so one value serves both mechanisms.
|
|
|
|
### 6. The public health endpoint was not public anywhere except local
|
|
|
|
`presentation.api-base-path` defaults to `/v1`. `security.public-paths` defaulted to the literal
|
|
`/api/healthcheck`. Two shipped defaults describing one address, disagreeing — so health was
|
|
published at `/v1/healthcheck` while the allowlist opened a path no handler serves, and a load
|
|
balancer polling it would get a 401 and take the instance out of rotation. `local` pins both to
|
|
`/api` and could never reveal it.
|
|
|
|
The allowlist derives from the base path now, and `PublicHealthPathAgreementTest` holds the two
|
|
together in every profile. The smoke client had the same literal baked in; it takes the path from the
|
|
lane, which supplies the one matching the runtime.
|
|
|
|
The `/api` (local) versus `/v1` (everywhere else) split is left as found — changing it is a
|
|
user-facing decision — but it is a live trap for anyone following a local README against a dev host.
|
|
|
|
### Verification
|
|
|
|
```
|
|
$ cd src && ./gradlew test verifyCleanArchitectureDependencies verifyRuntimeModuleMembership \
|
|
verifyEnvKeys verifyPublicPathSnapshot BUILD SUCCESSFUL
|
|
$ ./gradlew :adapter:outbound:persistence-jpa:jpaPlatformMigrationTest BUILD SUCCESSFUL
|
|
$ ./scripts/verify-compose-profile-contracts.sh all 15 lanes match
|
|
```
|
|
|
|
`local-jpa` reports `persistenceVendor: postgresql` — the assertion that previously caught it passing
|
|
against H2 while its PostgreSQL container sat untouched.
|
|
|
|
### Lanes promoted to blocking
|
|
|
|
`local-jpa`, `shared-infra-local`, `shared-infra-dev`, joining `off-local`, `off-dev`, `off-prod`,
|
|
`local-mongo` and `local-graphql`. Eight of fifteen.
|
|
|
|
The seven that remain each carry a reason that is now **specific to what is actually left**, not the
|
|
JPA finding they inherited: `prod-smoke` on transport security (closed below), the two messaging
|
|
lanes on MSG-INT-003, the three notification lanes on the unwired persistence above plus NTF-INT-001
|
|
and NTF-INT-006, and `all-adapters` on the union.
|
|
|
|
---
|
|
|
|
## D1 — the notification mode had a name nobody could bind
|
|
|
|
`NotificationPlatformMode` is `SERVING | INGEST_ONLY`. `docs/registries/env-keys.yaml` declared
|
|
`SERVING | ACCEPT_ONLY`, and so did the comment in `application.yml` and the row in
|
|
`docs/notification/configuration-reference.md`. `ACCEPT_ONLY` is a name the enum has never had.
|
|
|
|
The worst shape a drift can take: an operator follows the registry, sets
|
|
`APP_NOTIFICATION_PLATFORM_MODE=ACCEPT_ONLY`, and gets a binding failure naming a constant that none
|
|
of the three documents they can reach mentions. Every source they consulted agreed with every other
|
|
one, and all of them were wrong.
|
|
|
|
`NotificationModeSsotTest` derives its expectation from `NotificationPlatformMode.values()`, so a
|
|
third mode cannot be added without its registry row and a rename cannot land in one place only.
|
|
|
|
One correction while writing it: the first version scanned the raw file text for the retired name and
|
|
therefore failed on the comment that *records the retirement* — the opposite of the defect, since the
|
|
defect was a value nobody could tell had never existed. It strips YAML comments now and checks what
|
|
an operator would paste.
|
|
|
|
---
|
|
|
|
## The `prod-smoke` lane, and why it got a real certificate
|
|
|
|
The lane's remaining blocker was transport security: the prod runtime connects with
|
|
`sslmode=verify-full&sslrootcert=/run/secrets/postgres-ca`, and the lane's PostgreSQL was a stock
|
|
image with no TLS — `The server does not support SSL.`
|
|
|
|
**The rejected option was relaxing the lane.** `sslmode=disable` would have turned it green in one
|
|
line and made a prod smoke test a smoke test of a configuration production never runs. The one
|
|
failure mode it exists to catch — a chain or a host name that does not check out — is precisely the
|
|
one that appears nowhere else.
|
|
|
|
So the lane brings a certificate. `docker-compose.tls.yml` is a separate overlay, in the lane's file
|
|
stack rather than in the shared infra file, so no other lane pays for it. The wrapper generates a CA
|
|
and a server certificate **for the host name `db`**, valid one day, and removes both halves on
|
|
teardown — the realm-secret pattern applied to a keypair. `verify-full` rather than `verify-ca` is
|
|
deliberate: `verify-ca` proves who issued the certificate and says nothing about who presented it, so
|
|
it does not detect a redirected connection, which is most of what this is for.
|
|
|
|
**The uid problem, for the third time.** PostgreSQL refuses to start on a key that is group- or
|
|
world-readable and reads it as uid 70; the host generates it as uid 1000; a bind mount preserves
|
|
ownership. The same collision as the Keycloak client secret (uid 1000 vs the curl image's uid 100)
|
|
and it has the same shape: bind-mounted credentials and per-image uids do not compose. The
|
|
entrypoint wrapper copies the key at the only moment the container is still root, before the official
|
|
entrypoint drops privileges. The mount stays read-only and the CA certificate — public, so no mode
|
|
problem — is the only thing the application container sees.
|
|
|
|
### Verification
|
|
|
|
```
|
|
$ ./scripts/run-compose-runtime-smoke.sh --lane prod-smoke == prod-smoke: passed
|
|
```
|
|
|
|
The application's own log records the connection it made:
|
|
|
|
```
|
|
Database: jdbc:postgresql://db:5432/ca_skeleton?sslmode=verify-full&sslrootcert=/run/secrets/postgres-ca
|
|
(PostgreSQL 16.14)
|
|
```
|
|
|
|
Zero occurrences of `does not support SSL`, no certificate left on disk, no surviving container.
|
|
A pass *is* the proof here: `verify-full` fails closed, so a lane that silently lost TLS could not
|
|
have started.
|
|
|
|
`prod-smoke` is blocking. Nine of fifteen.
|
|
|
|
---
|
|
|
|
## C4 — one master-gated starter root, and a provider selection that can actually select
|
|
|
|
### The condition that could never select
|
|
|
|
The spec records MSG-INT-004 as "Kafka and Rabbit must never assemble together merely because both
|
|
client libraries are on the classpath". Reading the build files makes it sharper than that: they
|
|
always are.
|
|
|
|
```
|
|
messaging-kafka/build.gradle:16 api 'org.apache.kafka:kafka-clients'
|
|
messaging-rabbit/build.gradle:14 api 'org.springframework.amqp:spring-rabbit'
|
|
messaging-spring-boot-starter implementation project(':messaging:messaging-kafka')
|
|
implementation project(':messaging:messaging-rabbit')
|
|
```
|
|
|
|
Both are `api` dependencies of leaves the starter depends on, so **both client classes are on every
|
|
adopter's runtime classpath, always**. `@ConditionalOnClass(Producer)` and
|
|
`@ConditionalOnClass(Channel)` were therefore both true for everybody: selection by classpath could
|
|
not select. It assembled both providers and let a `@ConditionalOnMissingBean` race decide where a
|
|
message went. Nothing failed; the message simply went somewhere nobody chose.
|
|
|
|
`MessagingStarterOffContractTest.bothClientLibrariesArePresent` asserts that classpath fact directly,
|
|
so the argument for property-based selection stays checkable rather than becoming folklore.
|
|
|
|
### What replaced it
|
|
|
|
`MessagingPlatformRootAutoConfiguration` is the single `.imports` entry — five became one — and owns
|
|
`@ConditionalOnProperty(app.messaging.enabled=true)`. The five former auto-configurations are plain
|
|
`@Configuration` children reached only through it, so a bean added to any of them next month is gated
|
|
without anyone remembering to repeat a condition.
|
|
|
|
`MessagingProviderSelection` resolves `app.messaging.broker` against a closed registry through an
|
|
`ImportSelector`, and turns three silences into startup errors: an unregistered broker id, a
|
|
registered broker whose client library is absent, and a blank broker while the master switch is on.
|
|
Each message names the property and lists what is available.
|
|
|
|
### What the off-contract test found
|
|
|
|
The two "selecting X assembles X" cases failed on first run — not on selection, which worked, but on
|
|
`MessagingCoreAutoConfiguration.deadLetterOrchestrator` having no `MessagePublisher` to depend on.
|
|
**Neither provider configuration contributes one.** That is MSG-INT-003, reproduced at the starter
|
|
boundary at unit speed, where the Compose lane had found it as a missing `KafkaSender` in the legacy
|
|
adapter. Same gap, two doors.
|
|
|
|
The cases are separated rather than merged: selection is tested with a publisher supplied by the
|
|
test, and `noProviderSuppliesAProductionPublisher` states the gap as its own assertion, naming C3 and
|
|
instructing that it **invert rather than disappear** when C3 lands. Letting the missing publisher
|
|
fail the selection cases would have hidden a defect behind a defect.
|
|
|
|
### Not done here
|
|
|
|
C4 is the gate; C3 is the thing behind it. No membership changed: every messaging leaf still has
|
|
empty `runtime_memberships`, which is the registry's way of saying build-only. The wave's rule is
|
|
that the starter earns `app-bootstrap` membership **in the same change that proves a live broker
|
|
round trip**, so `local-messaging` and `local-messaging-outbox` stay non-blocking and
|
|
`ShippedRuntimeFacadePresenceTest.messagingPlatformFacadeIsShipped` stays the one Wave 2 entry in the
|
|
`wave0-red` report.
|
|
|
|
```
|
|
$ ./gradlew :messaging:messaging-spring-boot-starter:test 8 cases, 0 failures
|
|
$ ./gradlew test + the four architecture gates BUILD SUCCESSFUL
|
|
```
|
|
|
|
---
|
|
|
|
## C3 — the platform could not publish, and three separate things were why (MSG-INT-003)
|
|
|
|
`MessagingCoreAutoConfiguration` has consumed a `MessagePublisher` since it was written — the
|
|
dead-letter orchestrator, the blocking publisher, the reactive publisher and the batch publisher all
|
|
take one — and **no configuration produced one**. A selected transport failed on a missing bean
|
|
rather than publishing anywhere. The Compose lane found the same gap from the other side, as a
|
|
missing `KafkaSender` in the legacy adapter.
|
|
|
|
What made it hard to see is that nothing was *broken*. Three things were absent, each in a different
|
|
place:
|
|
|
|
| Absent | Where it should have been |
|
|
| --- | --- |
|
|
| the starter's dependency on `messaging-runtime-core` | `build.gradle` — the leaf holding `DefaultMessagePublisher` was not on the starter's path at all |
|
|
| `MessageCodecRegistry` implementation | anywhere — an interface the publisher's constructor named and nothing in the repository built |
|
|
| the producer, transport, access policy, admission controller and publisher beans | the starter's auto-configurations |
|
|
|
|
`DefaultMessagePublisher` and `KafkaMessagingTransport` existed the whole time and were unit-tested.
|
|
C3 is a wiring job with two small pieces written, not the transport implementation the spec's wording
|
|
suggests.
|
|
|
|
### The one real decision: what a deployment may publish to
|
|
|
|
`DestinationAccessPolicy` is three sets of destination names with a `denyAll()` factory, and neither
|
|
extreme is a usable default:
|
|
|
|
- **deny everything** and a correctly configured deployment assembles, starts and refuses every
|
|
publish, with an error naming a policy nobody knew they had to write;
|
|
- **allow everything** and the check is decoration — and the policy exists precisely because relying
|
|
on broker ACLs alone surfaces an accidental publish as a generic authorization error at runtime, in
|
|
the adapter, with no record of which module attempted it.
|
|
|
|
So the default is **the destinations the deployment declared**. Declaring a destination profile is
|
|
already an act of configuration — it states the ordering guarantee, the retry policy, the dead-letter
|
|
target — and a message to a destination nobody declared is not an access-control edge case; it is a
|
|
typo or a module reaching past its contract, which is what the check is for. Consume and administer
|
|
stay empty: a publisher's default has no business granting either.
|
|
|
|
### Two smaller decisions, both taken the same way as elsewhere this wave
|
|
|
|
**The broker address comes from `spring.kafka.bootstrap-servers`**, not from a second description
|
|
under `app.messaging.*` — one resource described twice is the defect already paid for in
|
|
`app.jpa-platform.datasource.*`, and the SMTP assembler took the same decision about
|
|
`spring.mail.*`. A selected broker with no address **fails at startup**: a producer built without one
|
|
silently defaults to `localhost:9092` and fails on the first publish, which is a deployment that
|
|
starts, reports healthy, and loses the first message somebody sends.
|
|
|
|
**`acks=all` and idempotence on.** `acks=1` loses an accepted publish to a leader failover, which is
|
|
exactly the outcome an outbox exists to prevent — inheriting that default would make the reliability
|
|
layer above it a formality.
|
|
|
|
### What the registry gate caught
|
|
|
|
Adding the dependency failed `verifyCleanArchitectureDependencies` immediately: the edge was not in
|
|
`modules.json`. That is the gate doing its job — the edge is now registered, in the same change that
|
|
needs it.
|
|
|
|
### The live round trip, and the four absences it had to cross
|
|
|
|
`MessagingLiveRoundTripQualificationTest` publishes through the assembled platform to a Testcontainers
|
|
Kafka and reads the bytes back with a consumer that shares nothing with the producing code but the
|
|
topic name. It passes, with `brokerAccepted=true`.
|
|
|
|
Getting one publish confirmed took crossing four separate absences, each stopping a message at a
|
|
different stage — and **no fake would have hit any of them**:
|
|
|
|
| Absent | Where the publish stopped |
|
|
| --- | --- |
|
|
| the starter's dependency on `messaging-runtime-core` | context assembly |
|
|
| `MessageCodecRegistry` implementation | context assembly |
|
|
| a declared message contract | `PUBLISH_PREPARATION_FAILED`, at encoding |
|
|
| `MessagingRuntime` implementation and its installation | `PUBLISH_RUNTIME_UNAVAILABLE`, after resolution, access and encoding |
|
|
|
|
The last is the same shape as the notification provider registry: a registry constructed empty that
|
|
nothing ever installed into, so a message got all the way to the wire and was refused there.
|
|
|
|
Two smaller decisions fell out. **An empty codec contract map is fail-closed** — a codec that accepted
|
|
an unregistered type would serialise whatever object it was handed onto a topic consumers read with a
|
|
different shape, so a deployment that publishes must declare what it publishes. And the contracts are
|
|
held in a named record rather than a bare `Map` bean, because a `Map<K, V>` injection point in Spring
|
|
means "every bean of type V, keyed by bean name" — a contract map registered as a bean is either
|
|
ignored or turned into something nobody wrote.
|
|
|
|
The destination registry taught the fixture two invariants on the way, which is the registry working:
|
|
a declared dead-letter destination must itself be registered, and it may not reference itself.
|
|
|
|
### Membership: qualified one, would ship eighteen
|
|
|
|
The wave's rule is exact — the starter earns membership in the same change unit that turns the round
|
|
trip green, and *"leaves that are unsupported or unqualified are excluded from both the starter's
|
|
dependencies and the registry."*
|
|
|
|
Adding the `app-bootstrap` edge surfaces **eighteen** leaves at once, which the plan predicted:
|
|
`messaging-{admin-api, admin-runtime, claim-check, cloudevents, core-api, inbox-jdbc-postgresql,
|
|
kafka, observability, outbox-jdbc-postgresql, policy, rabbit, reliability-api, runtime-core,
|
|
schema-api, schema-json, security, spring-boot-starter, transport-spi}`.
|
|
|
|
The round trip qualified **one transport**. Rabbit, the JDBC inbox and outbox reliability adapters and
|
|
the admin plane have no round trip. Promoting all eighteen on the strength of one Kafka publish is the
|
|
"wiring first, qualify later" the wave forbids, in the form that is easiest to rationalise: the code
|
|
is written, every test passes, and only the evidence for *this deployment* is missing.
|
|
|
|
So membership stays where it is and the fork is recorded rather than taken quietly:
|
|
|
|
- **qualify each** — a round trip per transport and per reliability adapter, the larger and more
|
|
honest path; or
|
|
- **trim the starter** to what one qualified transport needs, which changes what C4's provider
|
|
selection can select and removes the Rabbit path its contract test covers.
|
|
|
|
`MessagingMembershipQualificationTest` holds the state so neither happens by accident: the round trip
|
|
exists and asserts broker acceptance, no messaging leaf is a runtime member, and `app-bootstrap` does
|
|
not depend on the starter. `ShippedRuntimeFacadePresenceTest.messagingPlatformFacadeIsShipped` stays
|
|
the one Wave 2 entry in the `wave0-red` report — but its reason has changed from **"cannot assemble"**
|
|
to **"one of several transports qualified"**, which is a different and much smaller gap.
|
|
|
|
---
|
|
|
|
## C2 — the half that can be finished now, and why the other half is C3's
|
|
|
|
Two types declare `@ConfigurationProperties` on `app.messaging`: `MessagingSettings` in the outbound
|
|
adapter, which owns `broker`, and `MessagingProperties` in the starter, which owns everything else.
|
|
Spring binds both without complaint — each takes the fields it declares — so the split is invisible
|
|
at runtime and shows up only as two places to look, neither validating the other's view.
|
|
|
|
**The collapse is bound to C3's change unit, and saying so is the decision.** Deleting
|
|
`MessagingSettings` means the adapter stops selecting a broker and becomes a port bridge over the
|
|
platform's publish contract. It can only do that once the platform *has* a production publisher.
|
|
Removing the binding first would leave the adapter unable to select anything — a worse state than the
|
|
split it fixes, and precisely the "wiring first, qualifying later" the wave forbids.
|
|
|
|
What is finished is the guard that keeps the split honest and would catch the way C3 could go wrong
|
|
silently. `MessagingAuthorityContractTest` asserts the two owners by name, that **exactly one
|
|
production type implements `OutboxMessagePublishPort`** (two publishers emitting one event is the
|
|
dual write the wave forbids, and it reads as working — every message arrives, twice), and that the
|
|
adapter-local `MessagePublisher` reaches neither `application-core` nor `domain-core`.
|
|
|
|
A source scan rather than a context assertion, deliberately: no module sees both the adapter and the
|
|
starter. That is the boundary working — a test that could see both would be evidence it had gone.
|
|
|
|
### What C3 turns out to need
|
|
|
|
Reading the platform for C3's estimate changed the picture. The production publish path **already
|
|
exists**: `DefaultMessagePublisher` in `messaging-runtime-core`, `KafkaMessagingTransport` in
|
|
`messaging-kafka`, with a six-argument constructor whose collaborators all have concrete shapes.
|
|
C3 is a wiring job, not a write-from-scratch — but the starter does not depend on
|
|
`messaging-runtime-core` at all, and `MessageCodecRegistry` has an interface and no implementation
|
|
anywhere.
|
|
|
|
So C3 is: that missing dependency edge, a codec registry, a producer built from properties, a
|
|
transport bean, a publisher bean, four collaborator defaults each carrying a real policy decision
|
|
(what a default `DestinationAccessPolicy` permits is a security question, not a wiring one), then the
|
|
live round trip and the membership that only lands with it. Named here so the next session starts
|
|
from the shape rather than rediscovering it.
|
|
|
|
---
|
|
|
|
## B3 — a check conditioned on the bean whose absence it should report (MNG-INT-003)
|
|
|
|
`mongoPlatformStartupCheck` carries `@ConditionalOnBean(MongoTopologyProbe.class)`. Inside the
|
|
probe-present case it already fails closed — a partial set of inputs is refused rather than
|
|
half-validated, and its Javadoc records that earlier fix. The condition itself is the wider hole: a
|
|
deployment that enables the platform and supplies no probe gets **no validation at all**, silently.
|
|
Not supplying a bean is exactly what an operator who has not finished wiring will do.
|
|
|
|
**The capability flags were literals.** `true, true` went straight into `MongoStartupValidator`,
|
|
telling it that transactions and change streams were both wanted whatever the deployment had
|
|
configured — and the topology was then validated against that invented answer.
|
|
`MongoPlatformSettings` carried neither field. They are settings now: `transactions` is a subordinate
|
|
switch defaulting `false`, and `changeStreams` is **refused rather than stored** — the compact
|
|
constructor forces it to `false`, because the driver-side source is not shipped and accepting a flag
|
|
the platform cannot honour would leave an operator believing it took effect.
|
|
|
|
### Where the scope got settled, and by what
|
|
|
|
The first attempt required a probe whenever the module was on. It broke `MongoPersistenceConfigTest`
|
|
and would have broken the `local-mongo` lane — because **this repository ships no probe**. The probe
|
|
is built from the live data-plane client by the composition root that owns the connection, which is a
|
|
fork's decision, and the module's opt-in contract explicitly allows "switched on, no platform profile
|
|
configured yet" as a state that must start.
|
|
|
|
So the requirement is scoped to a platform that is actually configured — `profiles` non-empty — which
|
|
is the same line the settings record already draws. Fail-closed where the platform is in use,
|
|
unchanged where the module is merely enabled. Deliberately **not** conditioned on the probe: a
|
|
requirement that only applies when the thing it requires is present is not a requirement.
|
|
|
|
`local-mongo` passes unchanged.
|
|
|
|
### Recorded, not fixed
|
|
|
|
`verifyMongoApiSurface` was **already red at HEAD**, from a rename sweep this task did not make:
|
|
`MongoPersistenceProperties`→`MongoPersistenceSettings`, `MongoAdvancedProperties`→
|
|
`MongoAdvancedSettings`, `MongoPlatformProperties`→`MongoPlatformSettings`, plus Wave 1's
|
|
`MongoRootAutoConfiguration`. B3 added no public type. The approved update absorbed the four, noted
|
|
here for the same reason the GraphQL one was: a snapshot that has been red for a while stops being a
|
|
gate.
|
|
|
|
---
|
|
|
|
## B4 — a security check that could never fire (MNG-INT-004)
|
|
|
|
`MongoCredentialReference.fingerprint()` hashed `role.name() + '|' + secretReference`. The spec calls
|
|
this "the same secret reference used under two roles looks like two different credentials, defeating
|
|
the separation it was meant to enforce". Reading the callers makes it concrete and worse.
|
|
|
|
`MongoSecurityProfileValidator.requireDistinctCredentials(runtime, admin)` exists to refuse a
|
|
deployment where one credential opens both the runtime and the admin plane. It is **always** called
|
|
with two different roles — that is what runtime and admin mean. With the role in the hash,
|
|
`sameCredentialAs` was therefore always false. **The check could not reject anything.** A deployment
|
|
pointing both planes at one secret passed a validator written for precisely that case, and the
|
|
message it would have printed — "separating the planes means nothing if one credential opens both" —
|
|
was unreachable.
|
|
|
|
`MongoCredentialRotationPolicy` is the counter-evidence that the role was never meant to be part of
|
|
the identity: it compares roles on its own line, immediately after asking whether the credential is
|
|
the same, because those are two questions. Folding one into the other left the security check
|
|
answering neither.
|
|
|
|
The fingerprint is the secret reference alone now. Rotation is unaffected — it kept its own role
|
|
comparison throughout — and the fingerprint still reveals nothing: 16 hex characters of SHA-256 over
|
|
a reference that is itself not a secret.
|
|
|
|
---
|
|
|
|
## B1 — the Mongo namespace, where only the documentation had drifted
|
|
|
|
`spring.data.mongodb.*` is deprecated at error level in Spring Boot 4's metadata; the canonical
|
|
namespace is `spring.mongodb.*`. The runtime was never on the wrong one — every Compose lane supplies
|
|
`SPRING_MONGODB_URI` and `local-mongo` passes against a real single-node replica set — but
|
|
`MongoPersistenceSettings`' own Javadoc pointed operators at the deprecated key.
|
|
|
|
That is the worst place for the drift to sit. Somebody reads the class that owns the switch, sets the
|
|
property it names, and inherits a deprecation they did not choose. Two Javadocs, corrected.
|
|
|
|
`MongoNamespaceContractTest` strips Java comments before checking, so a sentence recording that the
|
|
old namespace is deprecated survives while a reference a compiler would act on does not — the same
|
|
distinction D1's test had to learn. Resources are checked whole: a key in a YAML file is never
|
|
commentary.
|
|
|
|
## A1 — two defects that cancelled each other out (JPA-INT-002)
|
|
|
|
`JpaDataSourceSettings` bound `app.jpa-platform.datasource.*` while the pool that serves requests is
|
|
built from `spring.datasource.hikari.*` — one pool with two descriptions, and a validator can pass
|
|
against the description that is not in use. That is the spec's finding.
|
|
|
|
Reading it at HEAD makes it worse and simpler at once. **The parallel namespace appears in no shipped
|
|
YAML and no row of the env-key registry**, so both of its fields were always null — and
|
|
`requirePoolBounds` throws on a null. Had anything called it, every deployment would have failed to
|
|
start.
|
|
|
|
Nothing called it. `validateStable` and `requirePoolBounds` were reachable only from their own unit
|
|
test. So: a validator nobody calls is a comment, and a validator nobody calls that would fail
|
|
everything if called is a comment holding a trap. **The reason the application started was the second
|
|
defect hiding the first.**
|
|
|
|
### What it validates now, and what it deliberately does not
|
|
|
|
The namespace is deleted and the validator reads the resolved `DataSource` — product and version from
|
|
a connection it opens, which also turns an unreachable database into a startup failure instead of a
|
|
failure at whoever sends the first request. It is invoked from `PersistenceJpaRootAutoConfiguration`
|
|
as an `InitializingBean`, so it runs exactly when JPA is on and never when it is off.
|
|
|
|
**Pool bounds are deliberately not re-checked there.** `HikariPoolConstraintValidator` already reads
|
|
`spring.datasource.hikari.*` — the namespace that actually builds the pool — and owns the acquisition
|
|
floor and the inter-knob constraints. Adding a second opinion on the same properties is how the
|
|
parallel namespace started. It also keeps HikariCP off app-bootstrap's production classpath, where
|
|
the build file deliberately declares it `testImplementation` only.
|
|
|
|
The product check follows the vendor selector rather than applying always: local development runs H2
|
|
by design, and `PersistenceVendorProdSafetyValidator` is what keeps that out of production. Demanding
|
|
PostgreSQL unconditionally would refuse every laptop.
|
|
|
|
### A third defect, found by wiring the second
|
|
|
|
Injecting the validator bean failed the lane: `JpaPlatformRuntimeAutoConfiguration` carries
|
|
`@ConditionalOnBean(DataSource.class)` on a plain `@Configuration` imported by the root — a condition
|
|
evaluated during configuration-class parsing, **before the datasource bean definition is
|
|
registered**. That class therefore drops out silently in the real application, taking the whole JPA
|
|
add-on layer with it: the retry coordinator, the safety guard, the platform report, and the validator
|
|
itself.
|
|
|
|
It was invisible because nothing depended on any of it. The check constructs its own validator —
|
|
depending on a bean from that class would make the check disappear for the same reason the thing it
|
|
checks disappeared — and the condition-ordering defect was then closed on its own footing, below.
|
|
|
|
### And why the existing test could not have caught it
|
|
|
|
`JpaPlatformRuntimeAutoConfigurationTest` registers the class through `AutoConfigurations.of(...)`,
|
|
where `@ConditionalOnBean` is evaluated **after** the datasource definition exists and therefore
|
|
answers yes. `PersistenceJpaRootAutoConfiguration` imports it as a plain `@Configuration`, where the
|
|
same annotation is evaluated during parsing and answers no.
|
|
|
|
So the test proved the class works in a registration shape the application does not use, and it had a
|
|
case — "without a DataSource, nothing is built" — whose green depended on exactly the annotation that
|
|
was deleting the layer in production. A test can be green, precise, and about a different program.
|
|
|
|
The class-level condition is removed rather than reordered: the class is reached only through the JPA
|
|
root, which already carries the master switch, so "is there a datasource" has been answered yes by
|
|
the time it is parsed. The method-level `@ConditionalOnBean`s stay — those are evaluated at
|
|
bean-definition time and are the pre-existing design. The no-datasource case now asserts what should
|
|
happen: **the context fails**, rather than quietly delivering less. `JpaPlatformAddonAssemblyTest`
|
|
holds the class-level annotation absent and names the eight beans, so a future re-addition is a
|
|
decision instead of a silent regression.
|
|
|
|
Eight beans — the Hibernate provider policy, the runtime role verifier, the platform composition, the
|
|
dangerous-configuration guard, the datasource validator, the platform report supplier, the retry
|
|
event listener and the platform startup check — assembled for the first time in this change.
|
|
|
|
A unit test could not have found any of this: nothing constructs the real root. The `local-jpa` lane
|
|
found each failure in turn.
|
|
|
|
```
|
|
$ ./gradlew test + the four architecture gates BUILD SUCCESSFUL
|
|
$ ./scripts/run-compose-runtime-smoke.sh --lane local-jpa passed
|
|
$ ./scripts/run-compose-runtime-smoke.sh --lane prod-smoke passed
|
|
$ ./scripts/run-compose-runtime-smoke.sh --lane off-local passed
|
|
```
|
|
|
|
---
|
|
|
|
## A2 — "it works locally" was about a different database (JPA-INT-003)
|
|
|
|
`local` defaulted to H2 with `create-drop` and Flyway off. So the sentence every developer says was a
|
|
statement about a datastore no other environment runs: migrations never applied, Hibernate wrote the
|
|
schema from the entities, and **a mapping that disagrees with the migration tree cannot be discovered
|
|
there at all** — a vendor that generates the schema from the entities has nothing to disagree with.
|
|
|
|
That is not theoretical. Two such disagreements shipped, and both were found by a Compose lane
|
|
minutes at a time rather than by a developer seconds at a time:
|
|
|
|
| Defect | Why H2 could not see it |
|
|
| --- | --- |
|
|
| `fs_cleanup_item` missing | its table exists only in a capability migration stream H2 never applies |
|
|
| `request_hash` `char(64)` vs `varchar(64)` | `create-drop` builds the column from the mapping, so the two agree by construction |
|
|
|
|
### The decision
|
|
|
|
`local` now resolves the Compose PostgreSQL this repository already ships — same vendor, same schema
|
|
owner, same `ddl-auto=validate` as `dev` and `prod`, differing only in address and credential.
|
|
`LocalJpaVendorParityTest` asserts that equality key by key and would fail the moment they drift.
|
|
|
|
**H2 is not removed.** It is the right tool for a laptop with no container, and deleting it would
|
|
cost a developer their inner loop to fix a problem they did not cause. It becomes
|
|
`./gradlew :app-bootstrap:bootRunH2` — a named task rather than a property, because choosing a
|
|
datastore whose schema comes from somewhere else should be visible in the command somebody typed. The
|
|
task carries every value that makes H2 safe together: its own vendor, `create-drop`, Flyway off
|
|
(the migration tree is PostgreSQL DDL, `DO $$` blocks and all), and `DB_CLOSE_DELAY=-1`, without
|
|
which the in-memory schema vanishes the first time the pool goes idle. It is in no release lane.
|
|
|
|
**The cost is stated rather than hidden:** a developer running `local` with JPA on now needs the
|
|
Compose PostgreSQL running, where before they needed nothing. That is the trade the wave asked for —
|
|
local and dev sharing vendor semantics rather than only the word "local".
|
|
|
|
### Two tests asserted the old contract
|
|
|
|
`ProfileSeparationContractTest` had cases named `localDefaultsToAnInMemoryDatabaseWithNoMigrations`
|
|
and `localKeepsTheInMemoryDatabaseAliveAcrossPoolIdleness`, both green, both pinning exactly what
|
|
this change removes. They were rewritten rather than deleted: the first now asserts the shipped
|
|
vendor, and the second follows `DB_CLOSE_DELAY=-1` to `bootRunH2`, where the database it protects now
|
|
lives. A contract test that pins a defect is still a contract test; what it needs is to move with the
|
|
decision, not to be silently dropped.
|
|
|
|
---
|
|
|
|
## D3 — INGEST_ONLY was delivering (NTF-INT-003)
|
|
|
|
`NotificationPlatformWorkerConfig` carried the master switch and nothing else, and both of its worker
|
|
beans call `start()` inside the factory method. So a deployment in `INGEST_ONLY` — the mode whose
|
|
entire purpose is to accept and store *without* delivering — started the dispatch scheduler, the
|
|
lease recovery pass, the provider-event replay worker and the reconciliation job. It accepted
|
|
requests and then tried to deliver them, which is the mode not existing.
|
|
|
|
The gate is a nested configuration the parent loads only in `SERVING`, not a condition repeated on
|
|
each bean. A per-bean condition is one a future bean can forget, and what forgetting costs here is
|
|
not a stray bean: it is a process delivering notifications that an operator deliberately put into a
|
|
drain. A configuration that is not loaded cannot be forgotten.
|
|
|
|
`matchIfMissing = true`, because `SERVING` is the shipped default. Making it false would turn an
|
|
absent property into a silent drain — the same failure in the other direction.
|
|
|
|
`NotificationWorkerLifecycleTest` asserts the structural fact rather than booting the platform, and
|
|
says why in the file: a full notification context needs a database, provider credentials and a
|
|
secrets resolver, and the capability has **no JPA persistence wired at all** yet, so a context test
|
|
would be a test that cannot run. The live thread count belongs to the
|
|
`local-notification-ingest` lane, which is where a running thread can actually be counted — and that
|
|
lane stays non-blocking until Section D wires the persistence.
|
|
|
|
---
|
|
|
|
## E3 — a CI gate that was red in a lane nobody runs (GQL-INT-004)
|
|
|
|
Three defects, all verified. Two are closed here; the third turned out to be discharged by Wave 3's
|
|
Compose lanes rather than by the Gradle task the spec sketched, and that is recorded rather than
|
|
duplicated.
|
|
|
|
### 1. The composition contract was asserting something false
|
|
|
|
`ConditionalTransportCompositionContractTest` asserted that all three opt-in transports have
|
|
`runtime_memberships: []`. The five-adapter contract put GraphQL on the single `app-bootstrap`
|
|
bootJar behind `APP_GRAPHQL_ENABLED`, so its membership is `[app-bootstrap]` by design — **the test
|
|
went red the moment Wave 1 landed and nobody saw it**, because this suite runs in
|
|
`conditionalTransportQualification` rather than in `test`.
|
|
|
|
`conditionalTransportQualification` is one of the two commands the CI quality job runs. So a gate CI
|
|
depends on had been failing for the whole of this effort, and the only reason it was not noticed is
|
|
that nothing runs it locally. A gate that is red in a lane nobody runs reports whatever the last
|
|
person to run it saw.
|
|
|
|
The two groups are asserted for what they now are: gRPC and WebSocket build-only, no membership;
|
|
GraphQL shipped and switch-gated, with membership **and** a master switch — a stronger claim carrying
|
|
a stronger obligation.
|
|
|
|
### 2. Class existence was never composition evidence
|
|
|
|
The other half asserted `Class.forName` resolves. A type resolving proves a jar is on a classpath and
|
|
says nothing about whether a composition assembles the transport or whether the switch gates it. The
|
|
test now names where that evidence actually lives — the `off-local`, `off-dev` and `off-prod` lanes
|
|
proving zero beans, sockets and routes with the switch off, and `local-graphql` proving a real
|
|
`/graphql` against a real identity provider with it on — and asserts those lanes are declared, so the
|
|
reference cannot rot into a comment.
|
|
|
|
### 3. CI never ran `graphqlStableTest`
|
|
|
|
`check` does not depend on it, so the lane's required-class guard — the check that its
|
|
module-boundary suite has not silently stopped being discovered — protected nothing in CI. The
|
|
quality job now runs `:adapter:inbound:graphql:graphqlStableTest` alongside
|
|
`conditionalTransportQualification` in one invocation, so the GraphQL lane cannot execute twice.
|
|
|
|
### What the spec asked for that was already done differently
|
|
|
|
The spec specifies a `:app-bootstrap:graphqlRuntimeQualification` task that runs the bootJar as a
|
|
child process, obtains a client-credentials token from a Keycloak container importing the tracked
|
|
realm artifact, and calls real HTTP `/graphql`. **Wave 3's `local-graphql` lane already does exactly
|
|
that**, in containers rather than in a Gradle task: the real image, the same realm artifact, a
|
|
service-account token, and `auth-smoke` calling the application. It is blocking, it runs in the
|
|
matrix, and it is what found the deployment-mode split-brain, the Boot introspection contradiction
|
|
and two Keycloak realm defects.
|
|
|
|
Building a second qualification of the same thing in a different harness would double the maintenance
|
|
and halve the attention each gets. `GraphqlHttpBoundaryQualificationTest` keeps its role as a module
|
|
contract test — its Basic Auth is test-only scaffolding for the transport boundary, not a claim about
|
|
the shipped composition, and the composition claim is now made by the lane.
|
|
|
|
```
|
|
$ ./gradlew :adapter:inbound:graphql:graphqlStableTest conditionalTransportQualification BUILD SUCCESSFUL
|
|
$ ./gradlew test + the four architecture gates BUILD SUCCESSFUL
|
|
```
|
|
|
|
---
|
|
|
|
## D6 — the recommended branch was not available (NTF-INT-007)
|
|
|
|
D6 requires choosing between encrypting the stored payload and restricting the variable types to
|
|
non-sensitive values, and recommends the restriction branch **"if and only if the variable types can
|
|
genuinely be restricted to non-sensitive values"**.
|
|
|
|
They cannot, and the reason is not a gap to be tightened. `NotificationVariable` is a closed algebra —
|
|
a real improvement over the `Map<String, Object>` it replaced — but `TextValue` holds an arbitrary
|
|
UTF-8 string up to 8 KiB because **the variables are the message's own content**: a reset code, an
|
|
order total, a delivery address, an appointment time. A restriction to "non-sensitive" would be
|
|
either unenforceable (a comment about a field designed to carry exactly that) or enforced and
|
|
useless (a type refusing free text does not restrict the capability, it deletes it).
|
|
|
|
So the recommendation's precondition is false and branch (a), encryption, is the required one. The
|
|
decision and its full scope are on file in `docs/notification/at-rest-threat-model.md`.
|
|
|
|
### When it lands, and the fact that decided that
|
|
|
|
**With the persistence wiring, not before** — because of something the spec did not have:
|
|
`NotificationJpaPersistenceFacade`, which assembles `JpaNotificationRequestStore`, is imported by
|
|
nothing. The notification capability has no JPA persistence at all, so **no deployment writes this
|
|
payload anywhere today**. The defect is real in the code and latent in the runtime.
|
|
|
|
Designing key rotation and a row migration for rows no deployment produces would be building the
|
|
migration before the table, and would settle the envelope's shape before the store that must read it
|
|
is wired — the order that produces an envelope the store cannot use.
|
|
|
|
### What was explicitly not done
|
|
|
|
Requiring `PAYLOAD_ENCRYPTION` in `INGEST_ONLY`. That secret is consumed by exactly two files, and
|
|
the one that uses it protects raw **callback** bodies; nothing on the accept path reads it. Demanding
|
|
it would make a deployment supply a key that protects nothing while the payload it appears to be
|
|
about stays in plaintext. This repository already has one defect of that exact shape —
|
|
`backend.graphql.cursor.key-ids`, which production refuses to start without and which no code signs a
|
|
cursor with. Two would make it a habit.
|
|
|
|
`NotificationPayloadAtRestContractTest` pins all four facts the decision rests on, including that the
|
|
accept path contains no encryption — so **that case fails the moment somebody adds it**, which is the
|
|
change the threat model is waiting for. Notification stays not-promoted-to-Stable, and its three
|
|
lanes stay non-blocking.
|
|
|
|
---
|
|
|
|
## D2 — the two pieces that meant SERVING could not work (NTF-INT-001)
|
|
|
|
`NotificationPlatformProviderConfig` collects `List<ProviderRuntimeAssembler>` and production main
|
|
source implemented that interface **nowhere**. A fully configured SMTP profile therefore produced no
|
|
runtime, no route and no error: requests reached durable acceptance and then found nothing eligible
|
|
to send them, which from outside reads as the platform silently dropping notifications.
|
|
|
|
Two things were missing, and only one of them was the one the spec named:
|
|
|
|
- **`SmtpDispatch` had no implementation.** The adapter above it, its MIME factory and its failure
|
|
classifier were all complete and unit-tested against fakes, so the SMTP family looked finished from
|
|
every angle except that nothing could send. `JavaMailSenderSmtpDispatch` is that send.
|
|
- **No assembler.** `SmtpProviderRuntimeAssembler` is the first production one.
|
|
|
|
The relay's address comes from Spring's own `spring.mail.*` through the injected `JavaMailSender`,
|
|
not from a second description on the provider profile — one resource described twice is the defect
|
|
this repository already paid for in `app.jpa-platform.datasource.*`. The profile owns what is
|
|
per-profile: timeout, concurrency, rate. Capabilities are declared as what SMTP has, which is none of
|
|
callback, status query, provider idempotency, batch, scheduling, cancel or collapse: handing a
|
|
message to a relay is the end of what a sender can observe, and a capability declared here is a
|
|
promise the dispatch loop acts on.
|
|
|
|
One thing the type system had already settled, found while writing the fixture: `SmtpProviderProperties.TlsMode`
|
|
has exactly `STARTTLS_REQUIRED` and `IMPLICIT_TLS`. **Plaintext SMTP is unrepresentable** — the
|
|
transport refuses an unencrypted relay by construction rather than by a validator somebody has to
|
|
remember to run.
|
|
|
|
### The signal that fired, and the one that told us to stop
|
|
|
|
`NotificationLegacyNamespaceRetirementTest` asserted the platform had **zero** assemblers, with the
|
|
instruction that "when D2 lands, this assertion fails — and that failure is the signal to retire the
|
|
namespace". It failed. Updated to the new truth, with what still blocks the R0 removal stated more
|
|
narrowly: the remaining families, and the persistence below.
|
|
|
|
Then the second half of D2 — registering the assembler, which needs the notification persistence
|
|
wired — produced the opposite signal.
|
|
|
|
## D6 completed — the envelope, then the wiring it was blocking
|
|
|
|
The wiring below was reverted because the payload was unprotected. Building the envelope was
|
|
therefore the way to unblock D2's registration, D5's handoff and the notification lanes, so it was
|
|
built next.
|
|
|
|
```
|
|
byte version always 1
|
|
byte keyIdLength 1..255 UTF-8 bytes
|
|
byte[] keyId
|
|
byte[12] nonce
|
|
byte[] ciphertext + GCM tag
|
|
```
|
|
|
|
**The key id is the reason there is a format at all.** This repository's callback protection stores
|
|
nonce and ciphertext and nothing else, so the day the active key changes, every row written under the
|
|
previous one is unreadable and nothing in the row can say which key it needed. That is not a rotation
|
|
story with a gap — it is the absence of one. `SecretMaterialProvider` already exposed `keyById`, so
|
|
the envelope carrying the id makes rotation a change of default rather than a data migration, and
|
|
`aRetiredKeyStillReads` proves it.
|
|
|
|
The header is passed as **AAD**, not merely prefixed: otherwise the key id is attacker-editable and an
|
|
envelope can be redirected at a key of the attacker's choosing. A failed decryption throws rather than
|
|
returning empty — a caller handed an empty payload renders every variable as nothing and sends
|
|
"Hello , your code is " to a real person, which is the failure delivered instead of reported. Unknown
|
|
key, wrong key and modified ciphertext collapse into one message, because distinguishing them tells an
|
|
attacker which of the three they achieved.
|
|
|
|
Applied at the **storage boundary** (`NotificationRecordMapper`), as a **required** constructor
|
|
argument. "At rest" means in the row, and the application necessarily holds the plaintext because it
|
|
has to render it; what is removed is the plaintext sitting in the column for as long as the request is
|
|
retained. Required rather than optional because that was the whole risk: the store existed, wiring it
|
|
was one import away, and nothing about its shape said the row it wrote held caller content
|
|
unprotected.
|
|
|
|
`V10__variables_payload_envelope_guard.sql` is a **guard, not a backfill**. No deployment of this
|
|
repository can have written such a row, so re-encrypting them would be migrating rows that cannot
|
|
exist — and a migration has no business holding key material. A fork that wired the store itself meets
|
|
its plaintext rows at migration time rather than one failed request at a time, where it looks like a
|
|
decryption bug instead of an un-migrated table. Accepting both shapes was rejected: a protection that
|
|
can be bypassed by writing plaintext is a control that announces itself and then declines to hold.
|
|
|
|
## Two defects the notification lane found once persistence was real
|
|
|
|
Wiring the facade made the notification entity scan active for the first time, and the
|
|
`local-notification-ingest` lane immediately found two things nothing else could.
|
|
|
|
**1. A Flyway customizer outranked the operator.** `PostgreSqlPersistenceConfig` called
|
|
`configuration.locations(...)` unconditionally, which *replaces* whatever Spring bound from
|
|
`spring.flyway.locations`. An operator could set `SPRING_FLYWAY_LOCATIONS`, watch Flyway report a
|
|
successful migration, and get only the vendor stream. The lane set seven locations and applied one.
|
|
Same shape as `application-local.yml`'s literal pins, same fix: contribute the value when nobody has
|
|
chosen one, stay out of the way when somebody has.
|
|
|
|
**2. The lanes' own Flyway setting could never have worked.** With the override gone and the
|
|
operator's value finally honoured, Flyway refused: *"Found more than one migration with version 1"* —
|
|
`db/migration/jpa/{core,fileserver,idempotency,inbox,notification-platform,outbox-polling,outbox-storage}`
|
|
each declare a V1, because **each is its own stream with its own history table by design**. Seven
|
|
lanes had been carrying a setting that was inert, and passing while their own configuration was
|
|
discarded.
|
|
|
|
The setting is removed from all nine lanes with the reason recorded in the contract file. Applying a
|
|
capability stream needs a Flyway execution *per stream* — the operator sequence both
|
|
`PostgreSqlNotificationPersistenceUnitIntegrationTest` and `FileserverRoundTripContractTest` already
|
|
perform in their fixtures — and that is what the notification lanes still need before they can block.
|
|
|
|
---
|
|
|
|
## The wiring that was reverted, and why that is the right outcome
|
|
|
|
Importing `NotificationJpaPersistenceFacade` from the JPA root makes `JpaNotificationRequestStore`
|
|
reachable. That store writes the accepted request's template variables to
|
|
`notification_request.variables_payload` **in plaintext** — caller content that can be a reset code,
|
|
an order total, an address (NTF-INT-007).
|
|
|
|
`NotificationPayloadAtRestContractTest` failed on exactly that: its case asserting the write path is
|
|
reachable from no composition is what turns the latent defect into a live one the moment somebody
|
|
wires it. It did its job on the first run.
|
|
|
|
The wave's rule is unambiguous — *"a P0 correctness or security finding on a runtime path is a
|
|
prerequisite, not a follow-up; connecting wiring over a known data-loss path is forbidden."* So the
|
|
wiring was reverted, not the test relaxed, and the reason is recorded at the import site rather than
|
|
in a commit message nobody reads at the point of decision.
|
|
|
|
This also **corrects the threat model's own sequencing**. It said the envelope "lands with the
|
|
wiring". Landing them together in one change means the plaintext write is reachable for the duration
|
|
of that change's review; the honest ordering is that the envelope lands **before or with** the
|
|
wiring, never after, and the contract test now enforces that by failing on the wiring alone.
|
|
|
|
**What did land** is the half that is safe on its own: the entity scan now travels with the stores it
|
|
serves, so when the facade is finally imported it cannot arrive without the mappings they need —
|
|
closing the "entity metadata nowhere" defect independently of the payload question.
|
|
|
|
D2 is therefore complete as far as it can be without D6's envelope: the platform can assemble a
|
|
production SMTP provider, and registering it is one import away, blocked by a security prerequisite
|
|
rather than by missing work.
|
|
|
|
---
|
|
|
|
## D4 — the retirement order is forced by which path works (NTF-INT-004)
|
|
|
|
Two namespaces own notification. The legacy R0 selectors live under `app.notification.*` —
|
|
`slack-webhook.enabled`, `google-email.enabled`, `routes.<channel>.<route>` — and the delivery
|
|
platform lives under `ca-skeleton.notification.platform.*`. Both hang off the same master switch, so
|
|
a deployment turning notification on configures one capability through two vocabularies, neither of
|
|
which validates the other.
|
|
|
|
The plan's instruction is to isolate R0 behind a migration shim and then remove it. Counting what
|
|
each path can actually do settles the sequence, and it is not a preference:
|
|
|
|
| Path | Can deliver? |
|
|
| --- | --- |
|
|
| R0 (`app.notification.*`) | **yes** — real `NotificationProvider` implementations (Google Email, Slack webhook) |
|
|
| Platform (`ca-skeleton.notification.platform.*`) | **no** — zero `ProviderRuntimeAssembler` implementations (NTF-INT-001), and no JPA persistence wired |
|
|
|
|
**R0 is currently the only path in this repository that can send a notification.** Retiring it first
|
|
would not be a retirement; it would delete the only working capability and leave the replacement
|
|
unable to take over. The wave forbids exactly this shape in the other direction — connecting wiring
|
|
over a known-broken path — and it is no better performed backwards.
|
|
|
|
So D4 pins the state instead of asserting it away: the two namespaces, which one is canonical, and
|
|
the asymmetry that fixes the order. `NotificationLegacyNamespaceRetirementTest` asserts that the
|
|
platform has **zero** assemblers, so **when D2 lands, that assertion fails** — and the failure is the
|
|
signal that the shim can be raised and the namespace removed, not a regression. The negative half of
|
|
NTF-INT-004, that no legacy selector is documented as an activation switch, is already enforced in
|
|
`MasterSwitchRegistryContractTest`; this test asserts it is still in force rather than duplicating
|
|
it, so removing it there is visible from the task that depends on it.
|
|
|
|
---
|
|
|
|
## E2 — a key that protects nothing (GQL-INT-003)
|
|
|
|
The spec's premise for E2 is that "auto-configuration and a startup validator existing is not
|
|
evidence that cost, authz, cursor, and idempotency policies apply". Checking each one separately
|
|
splits them in two.
|
|
|
|
**Cost and authorization do apply.** They are two of the three handlers in `GraphQlExecutionChain`,
|
|
reached through the platform's `Instrumentation`, and `GraphQlPlatformExecutionPathTest` already
|
|
proves on a real random-port request that a depth violation, an alias bomb, a complexity overrun and
|
|
an unauthorized coordinate are each rejected *before any resolver runs*. E2 does not repeat that.
|
|
|
|
**Cursor signing does not.** `backend.graphql.cursor.key-ids` is read in exactly two places:
|
|
|
|
| Reader | What it does with it |
|
|
| --- | --- |
|
|
| `GraphQlPlatformStartupValidator` | refuses to start production without it |
|
|
| `GraphQlPlatformActuatorEndpoint` | reports it back as configured |
|
|
|
|
Nothing signs a cursor with it. `HmacGraphQlCursorCodec` and `GraphQlCursorKeyRing` exist and are
|
|
unit-tested; the platform never constructs either. So production demands a key identity, an operator
|
|
supplies one, the operations endpoint confirms it is configured — and cursors stay exactly as
|
|
client-editable as they were, which is the thing the validator's own message says the key prevents.
|
|
|
|
Every signal an operator can see says this is on. That is what makes it worse than an unfinished
|
|
capability: `persisted operation` is also `modelled`, and nothing pretends otherwise.
|
|
|
|
**Mutation idempotency does not either** — `GraphQlMutationIdempotencyInterceptor` is referenced by
|
|
no configuration, so a repeated mutation is a repeated mutation.
|
|
|
|
### The decision: record, because closing it is a design question
|
|
|
|
`GraphQlCursorKeyRing.of` takes `Map<String, byte[]>`, and the settings contract says deliberately
|
|
that "the keys themselves never appear in configuration". So **where the key material comes from has
|
|
to be decided before anything can be wired** — a secrets question, not a wiring one, and the same
|
|
shape as NTF-INT-007's payload-encryption branch. Wiring a codec against key bytes pulled from
|
|
somewhere unconsidered would be the half-built envelope that section explicitly refuses.
|
|
|
|
`GraphQlPolicyRequestPathTest` pins all of it: what is wired, that a configured cursor key builds no
|
|
codec, that the idempotency interceptor is absent, and that the validator's demand **stays** — the
|
|
demand is right and the implementation is the missing half. The two absence cases invert when it
|
|
lands; the validator case does not change. The leaf's own capability-grade table gains both rows at
|
|
`modelled`, which is the table's stated purpose: never describe a capability above its grade.
|
|
|
|
```
|
|
$ ./gradlew :adapter:inbound:graphql:test --tests '*GraphQlPolicyRequestPathTest*' 4 cases, 0 failures
|
|
$ ./gradlew test + the four architecture gates BUILD SUCCESSFUL
|
|
```
|
|
|
|
---
|
|
|
|
## B5 — already closed, re-verified rather than assumed
|
|
|
|
The plan's Task B5 offers implement-or-demote for three release-manifest entries naming tasks no
|
|
build file registers. Re-reproducing it at current HEAD found the demotion already applied:
|
|
`MONGO-REL-010/011/012` sit in `experimental_contracts[]` with a `not_promoted_reason` each,
|
|
`ReleaseManifestTaskExistenceTest` reads only the blocking `contracts[]`, and
|
|
`verify-mongodb-advanced.sh` fails with a "not promoted" message instead of invoking a task that does
|
|
not exist. Recorded because the wave requires the decision on file, and because "already done" is
|
|
worth stating with the evidence rather than leaving a plan step ambiguous.
|