donghyeon.kang/clean-architecture-backend-template
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
Files
e98b56eb03ecab588c21fd1e7dbcaa493c1d8645
clean-architecture-backend-…/docs/superpowers/plans/2026-08-15-wave3-environment-and-infra.md
T

26 KiB
Raw Blame History

Wave 3 — Environment Separation and Infrastructure Smoke Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans. Steps use checkbox (- [ ]) syntax. Read 2026-08-15-five-adapter-runtime-remediation-index.md first. Entry criterion: Wave 2 complete — full test green, each adapter's one-on lane passing. Exception: Task 4 (Keycloak realm) is a prerequisite of Wave 2 Task E3 and may be pulled forward. If it was, mark it complete here and continue.

Goal: Separate example configuration from operator input, make a profileless deployment impossible, and build the Compose lane matrix — with its two canonical scripts, Keycloak realm, and MinIO round trip — that proves each activation combination actually runs.

Architecture: One JSON contract file (config/runtime/compose-profile-contracts.json) is the SSOT for every lane: its Compose profile, its file stack, its explicit Spring runtime, and its exact sorted service set. Two scripts are the only entry points — one static (verify-compose-profile-contracts.sh), one dynamic (run-compose-runtime-smoke.sh) — and CI calls those scripts rather than inlining fragments of them, so a lane cannot be half-run by a workflow that forgot a flag. Shared infrastructure lives in its own docker-compose.infra.yml, never mixed into an environment overlay. Each lane gets a unique Compose project, mode-0700 temp directory, mode-0600 secret files, sanitized evidence, and a trap-driven teardown scoped to that project alone.

Tech Stack: Docker Compose ≥ 2.24.4 (this machine: 5.4.0), PostgreSQL 16 with TLS, MongoDB replica set, Kafka, Mailpit, MinIO + mc, Keycloak with --import-realm, Bash.

Spec: 2026-08-15-five-adapter-runtime-remediation-review-design.md (§7 in full, §11 Wave 3, §12.4)

Global Constraints

Inherited from the index. Wave 3 adds:

  • A secret value never reaches Git, rendered config, a command line, JUnit XML, or an evidence artifact. Secrets are files created per run at mode 0600 and deleted on teardown.
  • up --wait applies only to long-running services. A one-shot client (auth-smoke, object-storage-smoke, notification-smoke, minio-init) is run with run --rm and must exit zero. A required one-shot that is missing, skipped, or non-zero fails the whole lane.
  • Teardown is scoped. down --volumes --remove-orphans runs against the lane's unique COMPOSE_PROJECT_NAME only. Never touch another project or a named volume outside the lane.
  • Reference docs are authoritative for merge and import semantics, not memory:
  • Static verification precedes dynamic. config and create must pass before any up.

File Structure

Created

File Responsibility
src/config/runtime/compose-profile-contracts.json The lane SSOT: id, Compose profile, file stack, Spring runtime, exact sorted service set, blocking flag.
docker-compose.infra.yml Every shared infrastructure service and one-shot smoke client. Owns nothing environment-specific.
docker-compose.prod-smoke.yml TLS PostgreSQL, prod env source, secret references. Test-only.
scripts/verify-compose-profile-contracts.sh The only static entry point.
scripts/run-compose-runtime-smoke.sh The only dynamic entry point.
infra/keycloak/realms/ca-skeleton-realm.json Reproducible realm import. No secret values.
infra/keycloak/entrypoint.sh Reads the secret file, exports it, execs kc.sh start-dev --import-realm.
infra/keycloak/smoke/auth-smoke.sh The one-shot client-credentials + protected-endpoint assertion.
infra/minio/smoke/object-storage-smoke.sh upload → HEAD → download → delete → wrong-credential rejection.
infra/minio/init/bucket-bootstrap.sh Bucket and minimum policy creation. Not a substitute for the round trip.
infra/notification/smoke/notification-smoke.sh Accept/ingest, Mailpit assertion, duplicate check.
src/.env.example Public key catalog with empty placeholders. Tracked.
src/.env.local.example Local opt-in combination example. Tracked.

Modified

File Change
src/app-bootstrap/src/main/resources/application.yml Remove the ${SPRING_PROFILES_ACTIVE:local} fallback.
docs/registries/env-keys.yaml SPRING_PROFILES_ACTIVE becomes a defaultless enum local|dev|prod.
src/app-bootstrap/build.gradle Replace the bootRun-only .env parsing with a single loader; pass an explicit profile.
src/build.gradle Rewrite verifyEnvKeys's input contract: registry + profile YAML + .env.example + generated metadata; never a gitignored operator .env.
docker-compose.yml App only; no infrastructure.
docker-compose.local.yml Local overlay; profiles for service selection; PostgreSQL moves to infra.
docker-compose.dev.yml tmpfs: !override [] then exactly one /var/tmp/heap bind mount; owns SPRING_PROFILES_ACTIVE=dev and its env source.
.gitignore Ignore src/.env* except the two .example files.
src/.env Untracked (git rm --cached). It is operator input, not a build input.
src/app-bootstrap/src/main/resources/logback-spring.xml Profile field reads the real active profile.
.github/workflows/ci-quality-gates.yml Call the two scripts; do not inline their commands.

Task 1: Make a profileless deployment impossible

Files:

  • Modify: src/app-bootstrap/src/main/resources/application.yml:22-24
  • Modify: docs/registries/env-keys.yaml (SPRING_PROFILES_ACTIVE row)
  • Create: src/app-bootstrap/src/main/java/dev/caskeleton/bootstrap/activation/RuntimeEnvironmentProfileValidator.java
  • Modify: src/app-bootstrap/build.gradle (bootRun passes an explicit profile)
  • Modify/replace: EnvProfileMatrixContractTest and any profileless-permitting test
  • Test: src/app-bootstrap/src/test/java/dev/caskeleton/bootstrap/activation/RuntimeEnvironmentProfileValidatorTest.java

Interfaces:

  • Produces: RuntimeEnvironmentProfileValidator, an EnvironmentPostProcessor ordered after MasterSwitchEnvironmentPostProcessor (Wave 1 Task 2). Tasks 5–8 assume exactly one environment profile is resolvable.

Context: Verified at HEAD: spring.profiles.active: ${SPRING_PROFILES_ACTIVE:local}. A bootJar started with no profile silently becomes local, which — before Wave 2 Task A2 — also meant H2 and create-drop. Deployable cardinality is exactly one.

Rules: missing ⇒ error; blank ⇒ error; unknown ⇒ error naming the three permitted values; multiple (local,prod) ⇒ error, and SPRING_PROFILES_ACTIVE is not treated as CSV; test ⇒ rejected for a deployable artifact, permitted only in a test-source context.

Feature selection must not be expressed as a supplementary Spring profile — the five master switches are for that. The validator therefore rejects any active profile outside the permitted set rather than ignoring extras.

  • Step 1: Write the validator test — one case per rule above, plus one asserting that a test-source context may still use test.
  • Step 2: Run to verify it fails.
  • Step 3: Remove the :local fallback from application.yml; write the validator; register it in META-INF/spring.factories next to the master-switch post-processor.
  • Step 4: Update the SPRING_PROFILES_ACTIVE registry row to a defaultless enum.
  • Step 5: Make bootRun pass an explicit profile so the developer convenience path stays usable without reintroducing an implicit default.
  • Step 6: Replace EnvProfileMatrixContractTest's local-fallback expectation with the new fail-closed contract. Do not delete coverage — rewrite it.
  • Step 7: Run ./gradlew :app-bootstrap:test --console=plain --no-daemon and ./gradlew verifyEnvKeys --console=plain --no-daemon.
  • Step 8: Commit.

Task 2: Separate example configuration from operator input

Files:

  • Create: src/.env.example, src/.env.local.example
  • Modify: .gitignore
  • Untrack: src/.env (git rm --cached src/.env — the human runs this)
  • Modify: src/build.gradle (verifyEnvKeys input contract, lines ~2209-2229, ~2263-2276)
  • Modify: src/app-bootstrap/build.gradle (single env loader, replacing the bootRun-only parser at ~251-272)
  • Test: src/app-bootstrap/src/test/java/dev/caskeleton/bootstrap/activation/EnvSourceSeparationTest.java

Context: Verified at HEAD: src/.env is tracked (git ls-files lists it), there is no .env.example, and .gitignore contains only three unrelated lines. The tracked file carries SPRING_PROFILES_ACTIVE=local, APP_DATASOURCE_DDL_AUTO=update, and local credentials — which is why dev inherits ddl-auto=update and fails.

verifyEnvKeys currently requires src/.env to exist and compares required placeholders against it. That contract is false in both directions: it passes only when a real secret file is present, and it would pass with no example file at all. The new SSOT is the env registry, the profile YAMLs, .env.example, and the generated configuration metadata. Gitignored operator .env* files and real secret values leave the build inputs entirely.

.gitignore addition:

src/.env*
!src/.env.example
!src/.env.local.example
  • Step 1: Write EnvSourceSeparationTest — asserts .env.example exists and every registry key appears in it; that no example value looks like a real secret (non-empty for a classification: secret row); and that src/.env is not tracked (git ls-files --error-unmatch src/.env must fail).
  • Step 2: Run to verify it fails.
  • Step 3: Generate .env.example from the registry — every key, secrets as empty placeholders or secret:// references. Write .env.local.example with a documented local opt-in combination.
  • Step 4: Update .gitignore; ask the human to run git rm --cached src/.env.
  • Step 5: Rewrite verifyEnvKeys's inputs; replace the bootRun-only .env parser with a loader that selects the file by environment, or with Spring's standard config import.
  • Step 6: Run ./gradlew verifyEnvKeys :app-bootstrap:test --console=plain --no-daemon.
  • Step 7: Commit.

Task 3: The Compose contract SSOT and the static verifier

Files:

  • Create: src/config/runtime/compose-profile-contracts.json
  • Create: scripts/verify-compose-profile-contracts.sh
  • Modify: docker-compose.yml, docker-compose.local.yml, docker-compose.dev.yml
  • Create: docker-compose.infra.yml, docker-compose.prod-smoke.yml

Interfaces:

  • Produces: the contract file, whose schema is { "minimumComposeVersion": "2.24.4", "lanes": [ { "id", "composeProfile"|null, "files": [...], "springRuntime", "services": [...sorted], "blocking": true } ] }. Task 7's runtime-smoke script and Wave 6's matrix both read it.

Context: The full lane table is spec §7.2 and is reproduced here as the exact content to encode. base, infra, local, dev, prod-smoke mean docker-compose.yml, docker-compose.infra.yml, docker-compose.local.yml, docker-compose.dev.yml, docker-compose.prod-smoke.yml, merged in the order listed.

lane Compose profile file stack Spring runtime services (sorted)
off-local — base+local local app
off-dev — base+dev dev app
off-prod — base+prod-smoke prod app
local-jpa local-jpa base+infra+local local app,db
local-mongo local-mongo base+infra+local local app,mongo,mongo-rs-init
local-messaging local-messaging base+infra+local local app,kafka
local-messaging-outbox local-messaging-outbox base+infra+local local app,db,kafka
local-notification-ingest local-notification-ingest base+infra+local local app,db,notification-smoke
local-notification-serving local-notification-serving base+infra+local local app,db,mailpit,notification-smoke
local-notification-handoff local-notification-handoff base+infra+local local app,db,mailpit,notification-smoke
local-graphql local-graphql base+infra+local local app,auth-smoke,keycloak
shared-infra-local shared-infra base+infra+local local app,auth-smoke,db,keycloak,minio,minio-init,object-storage-smoke
shared-infra-dev shared-infra base+infra+dev dev app,auth-smoke,db,keycloak,minio,minio-init,object-storage-smoke
prod-smoke prod-smoke base+infra+prod-smoke prod app,auth-smoke,db,keycloak,minio,minio-init,object-storage-smoke
all-adapters all-adapters base+infra+local local app,auth-smoke,db,kafka,keycloak,mailpit,mongo,mongo-rs-init,notification-smoke

Service membership rules:

  • auth-smoke ∈ local-graphql, shared-infra, prod-smoke, all-adapters.
  • object-storage-smoke ∈ shared-infra, prod-smoke.
  • notification-smoke ∈ the three local notification profiles and all-adapters.
  • minio-init bootstraps bucket and policy; it is not a substitute for the round trip.
  • A Compose profile selects services; it never implies a Spring profile.

The dev merge fix, using Compose ≥ 2.24.4 semantics: the dev overlay declares tmpfs: !override [] to replace the base tmpfs, then declares the /var/tmp/heap bind mount exactly once. Do not assume an empty sequence auto-deletes the base sequence — verify target uniqueness in the merged JSON, which is what the script's final check does.

The verifier checks, per lane:

  1. docker compose version --short ≥ the contract's minimumComposeVersion (semver compare);
  2. config --services for the lane's file stack (with --profile <name>, omitted for the three off lanes) equals the contract's sorted set exactly — not a superset;
  3. the rendered app service's SPRING_PROFILES_ACTIVE equals the lane's springRuntime;
  4. --profile '*' config --format json renders, and every service's volumes + tmpfs targets are unique within that service.
  • Step 1: Write the contract JSON encoding the table above.
  • Step 2: Restructure the Compose files: move PostgreSQL out of docker-compose.local.yml into docker-compose.infra.yml; add Mongo + mongo-rs-init, Kafka, Mailpit, MinIO + minio-init + object-storage-smoke, Keycloak + auth-smoke, notification-smoke; add Compose profiles: to each; create docker-compose.prod-smoke.yml; apply the !override fix to docker-compose.dev.yml and give it SPRING_PROFILES_ACTIVE=dev plus its env source.
  • Step 3: Write scripts/verify-compose-profile-contracts.sh implementing checks 1–4. set -euo pipefail; jq for JSON; exit non-zero with the lane id and the exact diff on any mismatch.
  • Step 4: Run ./scripts/verify-compose-profile-contracts.sh. Expected: all 15 lanes pass.
  • Step 5: Run cd src && ./gradlew :app-bootstrap:test --tests '*ComposeMergeCharacterizationTest*' — the dev case is now green; remove its @Tag("wave0-red").
  • Step 6: Pin the Compose minimum version in README.md and the CI workflow.
  • Step 7: Commit.

Task 4: The Keycloak realm and its acceptance

May be pulled forward — Wave 2 Task E3 depends on this artifact.

Files:

  • Create: infra/keycloak/realms/ca-skeleton-realm.json
  • Create: infra/keycloak/entrypoint.sh
  • Create: infra/keycloak/smoke/auth-smoke.sh
  • Modify: docker-compose.infra.yml, src/app-bootstrap/src/main/resources/application-local.yml

Context: The realm defines ca-skeleton-api as a confidential client with client authentication and a service account enabled, and with standard flow and direct access grant disabled. The service account carries realm role user and client role graphql-query; an audience mapper puts ca-skeleton-api into aud. Authentication for smoke is OAuth 2.0 client_credentials — one method, no alternatives. No test user, no password grant, no direct access grant.

The seven acceptance checks (spec §7.3):

  1. realm ca-skeleton imported;
  2. client/audience ca-skeleton-api exists;
  3. the application's roles and the role/permission claim mapping exist;
  4. a token is issued via the service account's client credentials;
  5. the token has non-blank sub, exact iss, aud=ca-skeleton-api, realm_access.roles containing user, and resource_access.ca-skeleton-api.roles containing graphql-query;
  6. public health succeeds unauthenticated; protected REST and GraphQL succeed only with a valid token;
  7. wrong realm, wrong audience, and expired token are rejected with the expected safe error contract.

The issuer trap, and why one hostname is not enough. application-local.yml:68-76,142-145 defaults the issuer to localhost:8081. That resolves on the host and, inside the app container, points at the app itself. JWKS discovery is lazy (JwtDecoderConfig.java:25-58), so startup succeeds and the error only appears at the first protected request. Do not assume one hostname resolves everywhere:

  • bootJar qualification (Wave 2 E3): inject Testcontainers' mapped Keycloak URL into both the token endpoint and the app issuer — the same single URL on both sides.
  • Compose smoke: put app and auth-smoke on the same network and inject http://keycloak:8080/realms/ca-skeleton into both.

A token obtained from one URL and validated against another is not evidence, and neither is a successful startup.

Secret handling. The qualification script creates a URL-safe random secret file at mode 0600 per run and mounts it as a Compose/Testcontainers secret. entrypoint.sh reads /run/secrets/keycloak-graphql-smoke-client-secret, exports it as a process-local KEYCLOAK_GRAPHQL_SMOKE_CLIENT_SECRET, and execs /opt/keycloak/bin/kc.sh start-dev --import-realm. The realm JSON contains only the ${KEYCLOAK_GRAPHQL_SMOKE_CLIENT_SECRET} reference. The file is removed on teardown.

  • Steps 1–8: build the realm, entrypoint, and smoke script; wire the Compose service; run ./scripts/verify-compose-profile-contracts.sh and then the local-graphql lane; confirm all seven checks; confirm no secret value appears in any rendered config or artifact (grep -r the evidence directory for the generated value must find nothing).

Task 5: MinIO bucket bootstrap and a real object round trip

Files:

  • Create: infra/minio/init/bucket-bootstrap.sh, infra/minio/smoke/object-storage-smoke.sh
  • Modify: docker-compose.infra.yml

Context: MinIO readiness is not success. minio-init creates the test bucket and minimum policy; object-storage-smoke is a black-box one-shot that consumes the lane's endpoint, bucket, and secret file and performs, in order and with no step skippable:

  1. upload known bytes to a random object key;
  2. HEAD and verify size and checksum;
  3. download and verify byte equality;
  4. delete and verify not-found;
  5. attempt the same operations with a deliberately wrong credential and verify rejection.

Results are written to minio-roundtrip.json in the lane's evidence directory, with no secrets.

Boundaries: the existing object-storage qualification owns its own Testcontainers and random credentials, so it is not evidence about this Compose service — keep it, but give the Compose lane a separate name and separate artifacts. Local or static credentials are never passed to prod-smoke. minio-init succeeding is never accepted in place of the round trip. Whether object storage joins the app-bootstrap runtime is a separate decision from the five master switches; if it does not, this smoke client is a release fixture, not a production bean.

The client image is pinned by digest.

  • Steps 1–6: build both scripts, wire the services, run the shared-infra-local lane, verify the artifact, confirm no secret leaked, commit.

Task 6: The notification smoke client and the stateful handoff lane

Files:

  • Create: infra/notification/smoke/notification-smoke.sh
  • Modify: docker-compose.infra.yml

Context: Three lanes use this client. local-notification-ingest proves durable accept with zero provider beans and zero workers. local-notification-serving proves a real Mailpit delivery.

local-notification-handoff is a composite stateful lane, not two lanes concatenated. The same project, the same PostgreSQL service, and the same named volume persist across four phases:

  1. Start DB and app with INGEST_ONLY phase env. Store an accept request using credential-free Mailpit route metadata. Record the request ID and route version in evidence. Confirm via the activation report that provider beans/calls and worker threads are all zero.
  2. Stop only the app, cleanly. Do not bring down the DB or the volume.
  3. In the same project, recreate the app with SERVING phase env and reference-provider settings (--force-recreate), and bring Mailpit to ready.
  4. Re-run the smoke with the phase-1 request ID and frozen route version. Assert exactly one Mailpit message, a terminal DB state, and the same route version. Wait at least one more dispatch poll window and assert duplicates are still zero.

Only after both phases and the intermediate app exit succeed does the lane proceed to shared evidence collection and teardown. Deleting the volume after phase 1, or copying rows into a second project, is not handoff evidence.

  • Steps 1–7: build the client, encode the phases in the runtime-smoke wrapper (Task 7), run the lane, verify the evidence, commit.

Task 7: The runtime-smoke wrapper

Files:

  • Create: scripts/run-compose-runtime-smoke.sh

Interfaces:

  • Produces: --matrix <contract.json> (all blocking lanes, zero-discovery and zero-skip) and --lane <id> (focused reproduction only — never a substitute for a matrix run). Wave 6 runs the matrix form.

Context: The wrapper enforces this order internally so no human and no CI job can skip a step:

  1. Create a per-lane, per-run COMPOSE_PROJECT_NAME and a mode-0700 temp directory; write env and secret files at mode 0600. Fail if the evidence directory already exists — never reuse one.
  2. Run verify-compose-profile-contracts.sh, then config, then create.
  3. up --wait the long-running services only; check app health/readiness and the resolved activation report from Wave 1's adapteractivation endpoint.
  4. run --rm each one-shot the lane declares (auth-smoke, object-storage-smoke, notification-smoke); for JPA lanes, assert the app's migration/schema/TLS report. A required one-shot that is missing, skipped, or non-zero fails the lane.
  5. Write to src/app-bootstrap/build/evidence/runtime-smoke/<lane>/<run-id>/: manifest.json, the Compose and service-set digest, activation/health, DB migration/TLS, sanitized Keycloak claims, the MinIO round trip, and a warning/error summary. Never a raw token, URI credential, secret value, or rendered secret.
  6. On both success and failure: collect sanitized logs and container exits first, then in a trap run down --volumes --remove-orphans against this project only, and delete the temp env and secret files. Never touch another project or an outside named volume.

The local-notification-handoff phase sequence from Task 6 lives here.

  • Steps 1–8: write it, run --lane off-local first, then --lane local-jpa, then the full --matrix, verifying evidence and teardown each time. Confirm with docker ps -a and docker volume ls that nothing outside the lane's project was touched.

Task 8: Wire CI to the scripts

Files:

  • Modify: .github/workflows/ci-quality-gates.yml

Context: CI calls the two scripts and nothing else for Compose work. Verified at HEAD, the quality job runs ./gradlew check verifyPublicPathSnapshot verifyDependencyLocks and ./gradlew conditionalTransportQualification, with no Compose verification at all. Inlining wrapper fragments would let a workflow silently run a lane without its one-shots, and past evidence must never be aggregated as a current pass.

  • Steps 1–4: add the two script invocations, run the workflow (or act/a branch push), confirm both execute and fail loudly on a deliberately broken lane, commit.

Wave 3 Exit Criteria

  • ./scripts/verify-compose-profile-contracts.sh — all 15 lanes pass.
  • ./scripts/run-compose-runtime-smoke.sh --matrix src/config/runtime/compose-profile-contracts.json — every blocking lane passes with zero discovery failures and zero skips, including prod-smoke actually starting TLS DB + app + Keycloak + MinIO and running both one-shots.
  • git ls-files src/.env returns nothing; src/.env.example and src/.env.local.example are tracked.
  • A profileless bootJar start fails; local,prod fails; stage fails; each of local, dev, prod succeeds.
  • cd src && ./gradlew verifyEnvKeys --console=plain --no-daemon — green with the new input contract, and green with src/.env absent.
  • ./gradlew wave0RedReport — only the Wave 4 warning entry remains.
  • No secret value appears anywhere under src/app-bootstrap/build/evidence/ (grep -r the generated values finds nothing).

What Wave 3 explicitly does not do

  • No warning removal (Wave 4) and no build-logic extraction (Wave 5).
  • No promotion of object storage into the app-bootstrap runtime — that decision is separate from the five master switches and is not made here.
  • No reuse of local MinIO or Keycloak credentials in prod-smoke.
  • No acceptance of minio-init success as round-trip evidence, of Keycloak readiness as realm evidence, or of a successful startup as issuer evidence.
Reference in New Issue View Git Blame Copy Permalink