tech-log-backend/docs/superpowers/plans/2026-08-10-redis-optionality-and-composition.md

# Redis Optionality and Composition Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development
> (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use
> checkbox (`- [ ]`) syntax for tracking.

**Goal:** Make Redis genuinely optional at both ends — `APP_REDIS_ENABLED=false` loads, binds,
validates and allocates nothing Redis-shaped, and `APP_REDIS_ENABLED=true` assembles a validated,
fail-fast Redis runtime — and close the SDK correctness defects that must not be wired live.

**Architecture:** A single conditional composition root (`RedisSdkAutoConfiguration`) owns
`RedisSdkSettings`, its validation, its secret/credential resolution, and its resource loading.
Nothing Redis-shaped is registered by the global `@ConfigurationPropertiesScan`. Secret requirements
move from the unconditional bootstrap list into that conditional owner. The SDK stays an
implementation detail of the `adapter:outbound:cache-redis` leaf; provider-neutral semantic ports
are re-implemented on top of it in a later phase.

**Tech Stack:** Java 21, Spring Boot 4.0.0, Lettuce, Gradle (fail-closed 19-leaf registry), JUnit 5,
AssertJ, ArchUnit.

## Status — 2026-08-10

| Review item | State | Where |
| --- | --- | --- |
| P1 #1 optionality (settings/validation half) | done | `RedisSdkAutoConfiguration`, `RedisSdkSettings`, `RedisOptionalityContractTest` |
| P1 #1 optionality (client/runtime half) | done | Phase D: `RedisTopologyClientFactory`, `RedisRuntimeOwner`, `RedisStartupProbe`, health contributors |
| P1 #2 production Redis secrets | done | `SecretSourceValidator`, `RedisActivationValidator` |
| P1 #3 env SSOT for the 34 settings | done | `env-keys.yaml`, `verifyEnvKeys` check E |
| P1 #4 semantic adapters | 4 of 5 | rate-limit, lease, idempotency V2, cache done. **Session is blocked, not deferred**: no provider-neutral session contract exists in `application-core` or `shared-contract` — it was deleted with the previous generation and the bootstrap references it only by bean name. Restoring it is a contract design task, not a port implementation, and the review does not specify that contract. |
| P1 #5 counter TTL | done | `AtomicCounterScripts` |
| P1 #6 transaction slot (aggregate check) | done | `LettuceRedisTransactionOperations.AttemptSlot` |
| P1 #6 transaction exclusive connection lease | done | typed `RedisLease` with `invalidate()`; the TRANSACTION lane is bounded and a poisoned connection is never pooled |
| P1 #7 telemetry isolation | done | `NoThrowObservationSink`, all three executors |
| P1 #8 topology lane fail-closed | done | `cache-redis/build.gradle` |
| P1 #9 README three-state split | done | `cache-redis/README.md` |
| TLS lane | done | `infra/redis-sdk/tls/compose.yml`, plaintext port off, certificates generated at start-up |
| P1 #9 PR/nightly/RC release gates | done | `redis-sdk-topology.yml` PR/schedule/RC matrix + evidence artifacts; gate promoted from `delegated-pending` |
| P1 #10 Netty floor | done | `ext['netty.version'] = '4.2.17.Final'`, all lockfiles |
| Phase B3 orphan configuration removal | done | 4 blocks removed from `application.yml`, 33 `.env` keys dropped, registry rows deprecated |

### P2/P3 hardening

| Item | State | Where |
| --- | --- | --- |
| Multi-key permit dead branch | done | `CommandPolicyGuard.requirePermits`; set algebra and blocking list now present a multi-key permit |
| Codec type safety | done | `RedisCodecRegistry` records the declared type and refuses a mismatched lookup |
| Error metadata on decode failure | done | `RedisFailureMetadata.storedDataCorruption`, deployment mode threaded from the caller |
| Pub/Sub codec per target | done | per-channel codec map; pattern subscriptions must agree on one codec |
| Pub/Sub backpressure | done | `SubscriptionFlux` bounded buffer + explicit overflow policy, decode failure terminates |
| Admin `CONFIG GET` | done | fixed allowlisted projection, secret-shaped values redacted, no caller pattern |
| Reply budget | done (consolidated) | dead `CommandPolicyGuard.validateReply` removed; `RedisOperationContext.requireReplyWithinBudget` is the single authority |
| Sentinel durability probe | done | `min-replicas-max-lag` now required alongside the replica count |
| Missing raw allowlist resource | done | `RedisSdkAutoConfiguration` opens it at startup |
| ACL fixture | done | `user default off`, fixture-only header, named-credential instructions |
| Readiness false-green | done | `validate-group-membership: true`, group names only contributors that exist |
| Dependency drift | done | unused `spring-data-redis`/`micrometer-core` removed, Reactor declared directly |
| JSON framing | done | control characters escaped, schema identifier constrained by regex |
| Connection lifecycle state machine | done | `RedisRuntimeOwner` `OPEN→DRAINING→CLOSED` |
| Gateway/`CommandRequest` visibility | **open** | needs `sdk.programmability`, `sdk.raw`, `sdk.admin` and `sdk.extensions` to stop constructing requests directly; a package restructuring, not a rename |
| Raw movable keys (`SORT BY/GET/STORE`) | done | `RawMovableKeys` settles SORT/SORT_RO locally including the STORE destination; BY/GET stay refused because their patterns cannot be namespace-checked, and an unknown option is a rejection rather than a guess |
| Batch observed-aggregate reply bytes | done | `BatchExecution` accumulates measured replies and fails the item that crosses the ceiling |

Residual limitation on P1 #6: keys queued inside the callback are only knowable after `MULTI`, so
the aggregate slot is enforced as each key becomes known — the offending command is refused before
it is written and the window is discarded, rather than the whole attempt being refused before
`WATCH`. Refusing before `WATCH` in every case needs a declared-keys transaction API, which Phase E
would revisit anyway.

## Global Constraints

- Registry SSOT for module identity, Gradle paths and allowed edges is
  `src/config/architecture/modules.json`. Never infer a Gradle path.
- Commit policy is `human-only`. Agents do not stage, commit, amend, or push.
- `domain-core` must stay free of framework/transport/database/cloud dependencies.
- `application-core` must never see an SDK type, a Redis key, a topology or a connection type.
- Global Redis activation is exactly one switch: `APP_REDIS_ENABLED`. `APP_CACHE_REDIS_ENABLED`
  must not be a second master switch.
- Every new `APP_*` key must land in all four places or `verifyEnvKeys` fails:
  `src/app-bootstrap/src/main/resources/application.yml`, `src/.env`,
  `docs/registries/env-keys.yaml`, and (when secret-classified)
  `docs/registries/secrets-classification.yaml`.
- `SecretsClassificationRegistryTest` asserts `SecretSourceValidator.REQUIRED_PROD_SECRETS` matches
  `docs/registries/secrets-classification.yaml` 1:1. Changing one requires changing the other.
- Netty floor: `4.2.16` or higher (CVE-2026-42577 epoll `<4.2.13`, CVE-2026-59901
  codec-compression `<4.2.16`).
- Topology lane modes allowlist: exactly `STANDALONE`, `SENTINEL`, `CLUSTER`.
- Verification commands run from `src/`.

## Current-state facts this plan is written against

Established by direct inspection on 2026-08-10, working tree (not HEAD):

- `CaSkeletonApplication` scans `dev.caskeleton.adapter` for `@ConfigurationProperties`, so
  `RedisSdkSettings` (`ca-skeleton.capabilities.redis-sdk`) is registered with Redis off.
- `RedisSdkSettings.validate()` has no production caller.
- The `cache-redis` leaf has **no** `@Bean`, `@Configuration`, or `@AutoConfiguration` in main
  source: nothing constructs a client, connection, gateway, or health contributor.
- 240 tracked main-source files under `cache-redis` are deleted in the working tree; the SDK
  (~300 files under `…cache.redis.sdk`) is untracked. The semantic cache/session/idempotency/
  rate-limit/lease adapters are gone.
- `ca-skeleton.providers.redis.*`, `ca-skeleton.capabilities.cache.*`, and
  `ca-skeleton.security.redis-session.*` in `application.yml` bind to **no** Java type — orphan
  configuration from the previous generation.
- `SecretSourceValidator.REQUIRED_PROD_SECRETS` requires `APP_CACHE_REDIS_PASSWORD` and
  `APP_CACHE_REDIS_KEY_HMAC_SECRET` unconditionally in prod; the other Redis roles have
  conditional skips.
- `verifyEnvKeys` compares only the three text sets (`.env`, `application.yml` placeholders,
  `env-keys.yaml`); it never reads `spring-configuration-metadata.json`, so a typed property with
  no env name passes.
- `redisTopologyTest` builds its tag as `lane-${declaredMode}` from an unvalidated project
  property, with no mode allowlist and no positive test-count postcondition — an unknown mode
  selects zero tests and exits 0.
- `src/app-bootstrap/gradle.lockfile` pins `io.netty:*:4.2.7.Final` on
  `productionRuntimeClasspath`, and still carries a `redisCompositionTestRuntimeClasspath`
  configuration whose source set no longer exists.

---

## Phase A — Redis optionality (P1 #1, #2) and the dead second switch

### Task A1: Remove the unconditional production Redis secret requirement

**Files:**
- Modify: `src/app-bootstrap/src/main/java/dev/caskeleton/bootstrap/runtime/SecretSourceValidator.java`
- Modify: `docs/registries/secrets-classification.yaml`
- Test: `src/app-bootstrap/src/test/java/dev/caskeleton/bootstrap/runtime/SecretSourceValidatorTest.java`

**Interfaces:**
- Produces: `SecretSourceValidator.REQUIRED_PROD_SECRETS` without any `APP_CACHE_REDIS_*` entry;
  `isCacheRedisMaterial(String)` + `isRedisGloballyEnabled()` private helpers gating every
  remaining Redis-prefixed secret on `app.redis.enabled`.

- [ ] **Step 1: Write the failing test** — prod profile, Redis off, no Redis secrets present,
      validator must not throw.
- [ ] **Step 2: Run it and watch it fail** on the two cache secrets.
- [ ] **Step 3: Gate every Redis secret on `app.redis.enabled` plus its role selector.**
- [ ] **Step 4: Re-run the focused test class.**
- [ ] **Step 5: Update `secrets-classification.yaml` `required_in_prod` metadata to match.**

### Task A2: Stop the global scan from registering `RedisSdkSettings`

**Files:**
- Modify: `src/app-bootstrap/src/main/java/dev/caskeleton/bootstrap/CaSkeletonApplication.java`
  (exclude the SDK config package) **or** move `RedisSdkSettings` out of a scanned package —
  preferred: keep the class where it is and drop `@ConfigurationProperties` from it, binding it
  instead from the conditional configuration with `@ConfigurationProperties` on the `@Bean` method.
- Test: new bootstrap contract test asserting zero `RedisSdkSettings` beans when
  `app.redis.enabled` is absent or false.

### Task A3: `RedisSdkAutoConfiguration` — the ON/OFF composition root

**Files:**
- Create: `src/adapter/outbound/cache-redis/src/main/java/dev/caskeleton/adapter/outbound/cache/redis/sdk/config/RedisSdkAutoConfiguration.java`
- Create: `src/adapter/outbound/cache-redis/src/main/resources/META-INF/spring/org.springframework.boot.autoconfigure.AutoConfiguration.imports`
- Test: `…/sdk/config/RedisSdkAutoConfigurationTest.java` (ApplicationContextRunner)

Conditions: `@ConditionalOnProperty(prefix = "app.redis", name = "enabled", havingValue = "true")`.
Inside: bind settings, call `validate()` and fail the context on `IllegalStateException`, log
warnings, then (Phase D) build the topology client.

### Task A4: Retire `APP_CACHE_REDIS_ENABLED` as a second master switch

**Files:**
- Modify: `src/app-bootstrap/src/main/resources/application.yml` (add `app.redis.enabled`)
- Modify: `src/.env`, `docs/registries/env-keys.yaml`

---

## Phase B — env SSOT migration (P1 #3)

### Task B1: Register `APP_REDIS_ENABLED` and the 34 SDK settings

Names are fixed by the review's env contract table. Each `env-keys.yaml` row carries
`property`, `owner_module`, `type`, `default`, `secret`, `required_when`, and (where one exists)
`deprecated_alias` + `removal_deadline`.

### Task B2: Extend `verifyEnvKeys` to read `spring-configuration-metadata.json`

Bidirectional: a typed `app.redis.*` property with no registry row fails; a registry row whose
`property` matches no metadata entry fails.

### Task B3: Remove the orphan generations

Delete `ca-skeleton.providers.redis.*`, `ca-skeleton.capabilities.cache.*`, and
`ca-skeleton.security.redis-session.*` from `application.yml` once a migration table records the
old→new mapping; drop the now-orphaned `.env` keys; mark the registry rows deprecated rather than
deleting their metadata.

---

## Phase C — SDK correctness (P1 #5, #6, #7)

### Task C1: Atomic counter must not add a TTL to a pre-existing persistent key

**Files:**
- Modify: `…/sdk/lettuce/operations/AtomicCounterScripts.java`
- Test: `…/sdk/lettuce/operations/AtomicCounterScriptsTest.java`

Both scripts must record existence **before** the increment and apply the initial expiry only when
the key was absent:

```lua
local existed = redis.call('EXISTS', KEYS[1])
local value = redis.call('INCRBY', KEYS[1], ARGV[1])
if existed == 0 then
  if ARGV[3] == 'AT' then
    redis.call('PEXPIREAT', KEYS[1], ARGV[2])
  else
    redis.call('PEXPIRE', KEYS[1], ARGV[2])
  end
end
return value
```

### Task C2: Validate the transaction's whole key set against one slot

**Files:**
- Modify: `…/sdk/programmability/LettuceRedisTransactionOperations.java`
- Test: `…/sdk/programmability/LettuceRedisTransactionOperationsTest.java`

Collect watched + queued keys per attempt and validate the aggregate slot before `MULTI`, instead
of validating the WATCH bundle and each queued write independently.

### Task C3: A throwing observation sink must not fail a successful command

**Files:**
- Create: `…/sdk/lettuce/observability/NoThrowObservationSink.java`
- Modify: `…/sdk/lettuce/command/SyncRedisCommandExecutor.java`
- Modify: `…/sdk/lettuce/command/ReactiveRedisCommandExecutor.java`
- Test: `…/sdk/lettuce/command/ObservationIsolationTest.java`

---

## Phase D — Runtime composition (P1 #4 prerequisite, deferred)

Topology strategy (standalone/sentinel/cluster), authentication/TLS, shared vs dedicated
connection lanes, lifecycle owner, capability/durability probe, health contributors.

## Phase E — Semantic adapter restoration (P1 #4, deferred)

Re-implement the provider-neutral ports on top of the SDK: cache, session, idempotency V2,
rate-limit, efficiency-only lease. This is the restoration of the 240 deleted files' behaviour and
is the largest single body of work in this plan.

## Phase F — Release gates, evidence and dependencies (P1 #8, #9, #10)

### Task F1: `redisTopologyTest` fails closed

Mode allowlist, `failOnNoDiscoveredTests = true`, per-lane required tag/class presence, and a
`>= 1` executed-test postcondition.

### Task F2: Netty floor `4.2.16`

Add a platform constraint, regenerate every lockfile, rerun the dependency scan.

### Task F3: README status split

`API implemented` / `Spring composition implemented` / `production-qualified` as three separate
states.

## Phase G — P2/P3 hardening (deferred)

Gateway/request visibility, multi-key permit dead branch, connection lifecycle state machine,
reply budgets, admin `CONFIG GET` projection, pub/sub codec mapping and backpressure, codec type
safety, error metadata, raw movable keys, Sentinel durability probe, ACL fixture, readiness
false-green, missing raw resource, dependency drift, JSON framing.

---

## Round 2 — the defects a real server found that this plan did not

Everything above was written before any of it had run against Redis. A second review started four
Docker lanes, wired the production code to them, and found that several items marked done were
done in the sense that the code existed, not in the sense that it worked. What follows is what that
round changed, and what it changed because of.

### The readiness group could not start at all

`management.endpoint.health.group.readiness.include` named `redisRequired`, a contributor that only
exists when a correctness role selected Redis. Boot validates group membership and does **not**
tolerate a conditional member being absent, so every Redis-off and cache-only deployment failed at
startup with `Included health contributor 'redisRequired' in group 'readiness' does not exist`. The
comment in `application.yml` asserted the opposite.

The group now names only unconditional contributors, and
`RedisReadinessGroupPostProcessor` appends `redisRequired` from `RedisCorrectnessRoles` — the same
predicate the bean's `@Conditional` asks, so membership and existence cannot drift.
`RedisReadinessGroupPostProcessorTest` boots a real Actuator context in each of the three shapes;
putting the name back in the shipped file makes two of them fail exactly as production did.

### Redis on composed no capability

`APP_REDIS_ENABLED=true` produced a client, an owner and a health contributor. Every semantic port
count was zero, so a deployment that selected `redis` for its rate limiter started, reported
healthy, and had no rate limiter. `RedisCapabilityConfig` composes cache, rate limit, lease and the
owner-safe idempotency store, each on its own selector.

The idempotency guard was also counting `application.idempotency.IdempotencyStorePortV2`, which no
provider implements — the implemented contract is the one in `…idempotency.v2`. Selecting `redis`
therefore required a bean nothing could supply. Driving the V2 store from an executor remains
outstanding and is named as such rather than covered by a guard that cannot see it.

### Four key prefixes, and an ACL that matched none of them

Each capability joined its own `namespace-application` / `namespace-environment` pair in its own
order, so the cache wrote `ca-skeleton:prod:…` while the ACL granted `~prod:*`. `CapabilityKeyspace`
renders every capability below one `RedisNamespace`, and the per-capability namespace keys are
deprecated.

The scripted capabilities also ran `EVALSHA` on the application account, which does not have it.
Lanes now carry a `RedisCredentialRole`; the topology factory builds one client per configured
role, so the `SCRIPT` lane authenticates as the advanced account and the account that reads a cache
entry still cannot execute a script. `LiveRedisSemanticPortsTest` proves both directions against a
real server.

### Cluster transactions were impossible, and multi-key WATCH was refused

`beginTransaction()` on a live cluster failed by design: every lane opened the slot-routing
connection, which cannot own a window. `RedisTransactionRunner` derives a routing key and pins the
lane to the node that owns the slot. Fixing that surfaced a second defect a cluster was not needed
for — `watch()` presented no multi-key permit, so watching more than one key was rejected
unconditionally, which is most optimistic transactions.

### The fixtures could not fail

Every ACL account was `nopass`, which accepts any password: every assertion about authentication
passed for the same reason a typo would have. The accounts carry real passwords and a wrong one is
now asserted to produce `WRONGPASS`. The cluster lane's readiness helper checked
`CLUSTER INFO` unauthenticated, so it never matched, never exited, and `up --wait` returned while
slots were still being assigned; a `ready` gate now blocks on `cluster_state:ok`.

### TLS was reachable only by hand

`tls` is a lane of `redisTopologyTest` and of the CI matrix. Trust material resolved with
`new File(...)` broke `classpath:` references, and resolving it purely through the resource loader
breaks mounted paths — both shapes are ordinary, and both are supported.

### Gates that could report success for a lane they did not run

`afterTest` fires for skipped tests too, so the "ran something" check could be satisfied by a run
that skipped everything. Lanes now declare the classes they exist to run and a floor for the
executed count, and a skipped test fails the run. `verifyEnvKeys` gained a check for registered
keys that nothing reads — no typed property, no yaml reference, no `.env` entry, no Java consumer —
which found eight orphaned Redis keys beyond the two the review named.

### Verified

| Lane | Result |
| --- | --- |
| standalone | 25 tests |
| sentinel | 27 tests |
| cluster | 29 tests, including a same-slot transaction and a cross-slot refusal |
| tls | 4 tests, filesystem and classpath CA |

Repository: 3594 tests, 0 failures. `verifyCleanArchitectureDependencies`,
`verifyPublicPathSnapshot`, `verifyEnvKeys`, `CleanArchitectureTest`, `verify-gate-matrix.sh`
(37 gates) and `verify-gradle-wrapper.sh` all pass.

### Still open

- **Session port.** No provider-neutral session contract exists in `application-core` or
  `shared-contract`; it went with the previous generation. That is a contract to design, not a port
  to implement, and inventing one here would be guessing at its shape.
- **V2 idempotency executor.** `IdempotencyExecutorV2` targets a contract no provider implements.
- **Gateway / `CommandRequest` visibility.** Narrowing it is a package restructuring across
  `sdk.programmability`, `sdk.raw`, `sdk.admin` and `sdk.extensions`, not an access-modifier change.