tech-log-backend/docs/superpowers/plans/2026-08-10-redis-optionality-and-composition.md

Redis Optionality and Composition Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Make Redis genuinely optional at both ends — APP_REDIS_ENABLED=false loads, binds, validates and allocates nothing Redis-shaped, and APP_REDIS_ENABLED=true assembles a validated, fail-fast Redis runtime — and close the SDK correctness defects that must not be wired live.

Architecture: A single conditional composition root (RedisSdkAutoConfiguration) owns RedisSdkSettings, its validation, its secret/credential resolution, and its resource loading. Nothing Redis-shaped is registered by the global @ConfigurationPropertiesScan. Secret requirements move from the unconditional bootstrap list into that conditional owner. The SDK stays an implementation detail of the adapter:outbound:cache-redis leaf; provider-neutral semantic ports are re-implemented on top of it in a later phase.

Tech Stack: Java 21, Spring Boot 4.0.0, Lettuce, Gradle (fail-closed 19-leaf registry), JUnit 5, AssertJ, ArchUnit.

Status — 2026-08-10

Review item	State	Where
P1 #1 optionality (settings/validation half)	done	RedisSdkAutoConfiguration, RedisSdkSettings, RedisOptionalityContractTest
P1 #1 optionality (client/runtime half)	done	Phase D: RedisTopologyClientFactory, RedisRuntimeOwner, RedisStartupProbe, health contributors
P1 #2 production Redis secrets	done	SecretSourceValidator, RedisActivationValidator
P1 #3 env SSOT for the 34 settings	done	env-keys.yaml, verifyEnvKeys check E
P1 #4 semantic adapters	4 of 5	rate-limit, lease, idempotency V2, cache done. Session is blocked, not deferred: no provider-neutral session contract exists in application-core or shared-contract — it was deleted with the previous generation and the bootstrap references it only by bean name. Restoring it is a contract design task, not a port implementation, and the review does not specify that contract.
P1 #5 counter TTL	done	AtomicCounterScripts
P1 #6 transaction slot (aggregate check)	done	LettuceRedisTransactionOperations.AttemptSlot
P1 #6 transaction exclusive connection lease	done	typed RedisLease with invalidate(); the TRANSACTION lane is bounded and a poisoned connection is never pooled
P1 #7 telemetry isolation	done	NoThrowObservationSink, all three executors
P1 #8 topology lane fail-closed	done	cache-redis/build.gradle
P1 #9 README three-state split	done	cache-redis/README.md
TLS lane	done	infra/redis-sdk/tls/compose.yml, plaintext port off, certificates generated at start-up
P1 #9 PR/nightly/RC release gates	done	redis-sdk-topology.yml PR/schedule/RC matrix + evidence artifacts; gate promoted from delegated-pending
P1 #10 Netty floor	done	ext['netty.version'] = '4.2.17.Final', all lockfiles
Phase B3 orphan configuration removal	done	4 blocks removed from application.yml, 33 .env keys dropped, registry rows deprecated

P2/P3 hardening

Item	State	Where
Multi-key permit dead branch	done	CommandPolicyGuard.requirePermits; set algebra and blocking list now present a multi-key permit
Codec type safety	done	RedisCodecRegistry records the declared type and refuses a mismatched lookup
Error metadata on decode failure	done	RedisFailureMetadata.storedDataCorruption, deployment mode threaded from the caller
Pub/Sub codec per target	done	per-channel codec map; pattern subscriptions must agree on one codec
Pub/Sub backpressure	done	SubscriptionFlux bounded buffer + explicit overflow policy, decode failure terminates
Admin CONFIG GET	done	fixed allowlisted projection, secret-shaped values redacted, no caller pattern
Reply budget	done (consolidated)	dead CommandPolicyGuard.validateReply removed; RedisOperationContext.requireReplyWithinBudget is the single authority
Sentinel durability probe	done	min-replicas-max-lag now required alongside the replica count
Missing raw allowlist resource	done	RedisSdkAutoConfiguration opens it at startup
ACL fixture	done	user default off, fixture-only header, named-credential instructions
Readiness false-green	done	validate-group-membership: true, group names only contributors that exist
Dependency drift	done	unused spring-data-redis/micrometer-core removed, Reactor declared directly
JSON framing	done	control characters escaped, schema identifier constrained by regex
Connection lifecycle state machine	done	RedisRuntimeOwner OPEN→DRAINING→CLOSED
Gateway/CommandRequest visibility	open	needs sdk.programmability, sdk.raw, sdk.admin and sdk.extensions to stop constructing requests directly; a package restructuring, not a rename
Raw movable keys (SORT BY/GET/STORE)	done	RawMovableKeys settles SORT/SORT_RO locally including the STORE destination; BY/GET stay refused because their patterns cannot be namespace-checked, and an unknown option is a rejection rather than a guess
Batch observed-aggregate reply bytes	done	BatchExecution accumulates measured replies and fails the item that crosses the ceiling

Residual limitation on P1 #6: keys queued inside the callback are only knowable after MULTI, so the aggregate slot is enforced as each key becomes known — the offending command is refused before it is written and the window is discarded, rather than the whole attempt being refused before WATCH. Refusing before WATCH in every case needs a declared-keys transaction API, which Phase E would revisit anyway.

Global Constraints

• Registry SSOT for module identity, Gradle paths and allowed edges is src/config/architecture/modules.json. Never infer a Gradle path.
• Commit policy is human-only. Agents do not stage, commit, amend, or push.
• domain-core must stay free of framework/transport/database/cloud dependencies.
• application-core must never see an SDK type, a Redis key, a topology or a connection type.
• Global Redis activation is exactly one switch: APP_REDIS_ENABLED. APP_CACHE_REDIS_ENABLED must not be a second master switch.
• Every new APP_* key must land in all four places or verifyEnvKeys fails: src/app-bootstrap/src/main/resources/application.yml, src/.env, docs/registries/env-keys.yaml, and (when secret-classified) docs/registries/secrets-classification.yaml.
• SecretsClassificationRegistryTest asserts SecretSourceValidator.REQUIRED_PROD_SECRETS matches docs/registries/secrets-classification.yaml 1:1. Changing one requires changing the other.
• Netty floor: 4.2.16 or higher (CVE-2026-42577 epoll <4.2.13, CVE-2026-59901 codec-compression <4.2.16).
• Topology lane modes allowlist: exactly STANDALONE, SENTINEL, CLUSTER.
• Verification commands run from src/.

Current-state facts this plan is written against

Established by direct inspection on 2026-08-10, working tree (not HEAD):

• CaSkeletonApplication scans dev.caskeleton.adapter for @ConfigurationProperties, so RedisSdkSettings (ca-skeleton.capabilities.redis-sdk) is registered with Redis off.
• RedisSdkSettings.validate() has no production caller.
• The cache-redis leaf has no @Bean, @Configuration, or @AutoConfiguration in main source: nothing constructs a client, connection, gateway, or health contributor.
• 240 tracked main-source files under cache-redis are deleted in the working tree; the SDK (~300 files under …cache.redis.sdk) is untracked. The semantic cache/session/idempotency/ rate-limit/lease adapters are gone.
• ca-skeleton.providers.redis.*, ca-skeleton.capabilities.cache.*, and ca-skeleton.security.redis-session.* in application.yml bind to no Java type — orphan configuration from the previous generation.
• SecretSourceValidator.REQUIRED_PROD_SECRETS requires APP_CACHE_REDIS_PASSWORD and APP_CACHE_REDIS_KEY_HMAC_SECRET unconditionally in prod; the other Redis roles have conditional skips.
• verifyEnvKeys compares only the three text sets (.env, application.yml placeholders, env-keys.yaml); it never reads spring-configuration-metadata.json, so a typed property with no env name passes.
• redisTopologyTest builds its tag as lane-${declaredMode} from an unvalidated project property, with no mode allowlist and no positive test-count postcondition — an unknown mode selects zero tests and exits 0.
• src/app-bootstrap/gradle.lockfile pins io.netty:*:4.2.7.Final on productionRuntimeClasspath, and still carries a redisCompositionTestRuntimeClasspath configuration whose source set no longer exists.

---

Phase A — Redis optionality (P1 #1, #2) and the dead second switch

Task A1: Remove the unconditional production Redis secret requirement

Files:

• Modify: src/app-bootstrap/src/main/java/dev/caskeleton/bootstrap/runtime/SecretSourceValidator.java
• Modify: docs/registries/secrets-classification.yaml
• Test: src/app-bootstrap/src/test/java/dev/caskeleton/bootstrap/runtime/SecretSourceValidatorTest.java

Interfaces:

• Produces: SecretSourceValidator.REQUIRED_PROD_SECRETS without any APP_CACHE_REDIS_* entry; isCacheRedisMaterial(String) + isRedisGloballyEnabled() private helpers gating every remaining Redis-prefixed secret on app.redis.enabled.

• Step 1: Write the failing test — prod profile, Redis off, no Redis secrets present, validator must not throw.

• Step 2: Run it and watch it fail on the two cache secrets.

• Step 3: Gate every Redis secret on app.redis.enabled plus its role selector.

• Step 4: Re-run the focused test class.

• Step 5: Update secrets-classification.yaml required_in_prod metadata to match.

Task A2: Stop the global scan from registering RedisSdkSettings

Files:

• Modify: src/app-bootstrap/src/main/java/dev/caskeleton/bootstrap/CaSkeletonApplication.java (exclude the SDK config package) or move RedisSdkSettings out of a scanned package — preferred: keep the class where it is and drop @ConfigurationProperties from it, binding it instead from the conditional configuration with @ConfigurationProperties on the @Bean method.
• Test: new bootstrap contract test asserting zero RedisSdkSettings beans when app.redis.enabled is absent or false.

Task A3: RedisSdkAutoConfiguration — the ON/OFF composition root

Files:

• Create: src/adapter/outbound/cache-redis/src/main/java/dev/caskeleton/adapter/outbound/cache/redis/sdk/config/RedisSdkAutoConfiguration.java
• Create: src/adapter/outbound/cache-redis/src/main/resources/META-INF/spring/org.springframework.boot.autoconfigure.AutoConfiguration.imports
• Test: …/sdk/config/RedisSdkAutoConfigurationTest.java (ApplicationContextRunner)

Conditions: @ConditionalOnProperty(prefix = "app.redis", name = "enabled", havingValue = "true"). Inside: bind settings, call validate() and fail the context on IllegalStateException, log warnings, then (Phase D) build the topology client.

Task A4: Retire APP_CACHE_REDIS_ENABLED as a second master switch

Files:

• Modify: src/app-bootstrap/src/main/resources/application.yml (add app.redis.enabled)
• Modify: src/.env, docs/registries/env-keys.yaml

---

Phase B — env SSOT migration (P1 #3)

Task B1: Register APP_REDIS_ENABLED and the 34 SDK settings

Names are fixed by the review's env contract table. Each env-keys.yaml row carries property, owner_module, type, default, secret, required_when, and (where one exists) deprecated_alias + removal_deadline.

Task B2: Extend verifyEnvKeys to read spring-configuration-metadata.json

Bidirectional: a typed app.redis.* property with no registry row fails; a registry row whose property matches no metadata entry fails.

Task B3: Remove the orphan generations

Delete ca-skeleton.providers.redis.*, ca-skeleton.capabilities.cache.*, and ca-skeleton.security.redis-session.* from application.yml once a migration table records the old→new mapping; drop the now-orphaned .env keys; mark the registry rows deprecated rather than deleting their metadata.

---

Phase C — SDK correctness (P1 #5, #6, #7)

Task C1: Atomic counter must not add a TTL to a pre-existing persistent key

Files:

• Modify: …/sdk/lettuce/operations/AtomicCounterScripts.java
• Test: …/sdk/lettuce/operations/AtomicCounterScriptsTest.java

Both scripts must record existence before the increment and apply the initial expiry only when the key was absent:

local existed = redis.call('EXISTS', KEYS[1])
local value = redis.call('INCRBY', KEYS[1], ARGV[1])
if existed == 0 then
  if ARGV[3] == 'AT' then
    redis.call('PEXPIREAT', KEYS[1], ARGV[2])
  else
    redis.call('PEXPIRE', KEYS[1], ARGV[2])
  end
end
return value

Task C2: Validate the transaction's whole key set against one slot

Files:

• Modify: …/sdk/programmability/LettuceRedisTransactionOperations.java
• Test: …/sdk/programmability/LettuceRedisTransactionOperationsTest.java

Collect watched + queued keys per attempt and validate the aggregate slot before MULTI, instead of validating the WATCH bundle and each queued write independently.

Task C3: A throwing observation sink must not fail a successful command

Files:

• Create: …/sdk/lettuce/observability/NoThrowObservationSink.java
• Modify: …/sdk/lettuce/command/SyncRedisCommandExecutor.java
• Modify: …/sdk/lettuce/command/ReactiveRedisCommandExecutor.java
• Test: …/sdk/lettuce/command/ObservationIsolationTest.java

---

Phase D — Runtime composition (P1 #4 prerequisite, deferred)

Topology strategy (standalone/sentinel/cluster), authentication/TLS, shared vs dedicated connection lanes, lifecycle owner, capability/durability probe, health contributors.

Phase E — Semantic adapter restoration (P1 #4, deferred)

Re-implement the provider-neutral ports on top of the SDK: cache, session, idempotency V2, rate-limit, efficiency-only lease. This is the restoration of the 240 deleted files' behaviour and is the largest single body of work in this plan.

Phase F — Release gates, evidence and dependencies (P1 #8, #9, #10)

Task F1: redisTopologyTest fails closed

Mode allowlist, failOnNoDiscoveredTests = true, per-lane required tag/class presence, and a >= 1 executed-test postcondition.

Task F2: Netty floor 4.2.16

Add a platform constraint, regenerate every lockfile, rerun the dependency scan.

Task F3: README status split

API implemented / Spring composition implemented / production-qualified as three separate states.

Phase G — P2/P3 hardening (deferred)

Gateway/request visibility, multi-key permit dead branch, connection lifecycle state machine, reply budgets, admin CONFIG GET projection, pub/sub codec mapping and backpressure, codec type safety, error metadata, raw movable keys, Sentinel durability probe, ACL fixture, readiness false-green, missing raw resource, dependency drift, JSON framing.

---

Round 2 — the defects a real server found that this plan did not

Everything above was written before any of it had run against Redis. A second review started four Docker lanes, wired the production code to them, and found that several items marked done were done in the sense that the code existed, not in the sense that it worked. What follows is what that round changed, and what it changed because of.

The readiness group could not start at all

management.endpoint.health.group.readiness.include named redisRequired, a contributor that only exists when a correctness role selected Redis. Boot validates group membership and does not tolerate a conditional member being absent, so every Redis-off and cache-only deployment failed at startup with Included health contributor 'redisRequired' in group 'readiness' does not exist. The comment in application.yml asserted the opposite.

The group now names only unconditional contributors, and RedisReadinessGroupPostProcessor appends redisRequired from RedisCorrectnessRoles — the same predicate the bean's @Conditional asks, so membership and existence cannot drift. RedisReadinessGroupPostProcessorTest boots a real Actuator context in each of the three shapes; putting the name back in the shipped file makes two of them fail exactly as production did.

Redis on composed no capability

APP_REDIS_ENABLED=true produced a client, an owner and a health contributor. Every semantic port count was zero, so a deployment that selected redis for its rate limiter started, reported healthy, and had no rate limiter. RedisCapabilityConfig composes cache, rate limit, lease and the owner-safe idempotency store, each on its own selector.

The idempotency guard was also counting application.idempotency.IdempotencyStorePortV2, which no provider implements — the implemented contract is the one in …idempotency.v2. Selecting redis therefore required a bean nothing could supply. Driving the V2 store from an executor remains outstanding and is named as such rather than covered by a guard that cannot see it.

Four key prefixes, and an ACL that matched none of them

Each capability joined its own namespace-application / namespace-environment pair in its own order, so the cache wrote ca-skeleton:prod:… while the ACL granted ~prod:*. CapabilityKeyspace renders every capability below one RedisNamespace, and the per-capability namespace keys are deprecated.

The scripted capabilities also ran EVALSHA on the application account, which does not have it. Lanes now carry a RedisCredentialRole; the topology factory builds one client per configured role, so the SCRIPT lane authenticates as the advanced account and the account that reads a cache entry still cannot execute a script. LiveRedisSemanticPortsTest proves both directions against a real server.

Cluster transactions were impossible, and multi-key WATCH was refused

beginTransaction() on a live cluster failed by design: every lane opened the slot-routing connection, which cannot own a window. RedisTransactionRunner derives a routing key and pins the lane to the node that owns the slot. Fixing that surfaced a second defect a cluster was not needed for — watch() presented no multi-key permit, so watching more than one key was rejected unconditionally, which is most optimistic transactions.

The fixtures could not fail

Every ACL account was nopass, which accepts any password: every assertion about authentication passed for the same reason a typo would have. The accounts carry real passwords and a wrong one is now asserted to produce WRONGPASS. The cluster lane's readiness helper checked CLUSTER INFO unauthenticated, so it never matched, never exited, and up --wait returned while slots were still being assigned; a ready gate now blocks on cluster_state:ok.

TLS was reachable only by hand

tls is a lane of redisTopologyTest and of the CI matrix. Trust material resolved with new File(...) broke classpath: references, and resolving it purely through the resource loader breaks mounted paths — both shapes are ordinary, and both are supported.

Gates that could report success for a lane they did not run

afterTest fires for skipped tests too, so the "ran something" check could be satisfied by a run that skipped everything. Lanes now declare the classes they exist to run and a floor for the executed count, and a skipped test fails the run. verifyEnvKeys gained a check for registered keys that nothing reads — no typed property, no yaml reference, no .env entry, no Java consumer — which found eight orphaned Redis keys beyond the two the review named.

Verified

Lane	Result
standalone	25 tests
sentinel	27 tests
cluster	29 tests, including a same-slot transaction and a cross-slot refusal
tls	4 tests, filesystem and classpath CA

Repository: 3594 tests, 0 failures. verifyCleanArchitectureDependencies, verifyPublicPathSnapshot, verifyEnvKeys, CleanArchitectureTest, verify-gate-matrix.sh (37 gates) and verify-gradle-wrapper.sh all pass.

Still open

• Session port. No provider-neutral session contract exists in application-core or shared-contract; it went with the previous generation. That is a contract to design, not a port to implement, and inventing one here would be guessing at its shape.
• V2 idempotency executor. IdempotencyExecutorV2 targets a contract no provider implements.
• Gateway / CommandRequest visibility. Narrowing it is a package restructuring across sdk.programmability, sdk.raw, sdk.admin and sdk.extensions, not an access-modifier change.