The audit found ~80 placeholders, and the damaging ones were where the
measuring apparatus itself was prose rather than a command:
a6 "( curl ... ) & 를 20개 띄우고 wait" — the 22.2s headline came from this
a3 "<로그인 반복, sid 를 /tmp/sids 에>" — the whole RPO measurement
a3 "<sid 목록>" — the control it is compared against
a5 "<수신 파드IP>" — the injection
a8 writes /tmp/tok, reads /tmp/rt — self-inconsistent, sent an empty token
b3 $KC / $RT / $NEW never assigned
c2 bare kcadm.sh with no kubectl exec
a1 conntrack tuples written by hand, though the direction flips per restart
Each is now a shell-expandable form: pod IPs from jsonpath, the admin password
from the secret, ids from kcadm --format csv, conntrack tuples derived from
"conntrack -L" with awk rather than transcribed.
Then the rewritten commands were executed against the live cluster, and one
of them failed — the 20-way load generator, written as "kubectl run --rm -i",
lost its output stream twice in a row. That is a trap this series already hit
once, and the rewrite reintroduced it. A-6 now uses a resident probe pod that
collects into a file and is cat-ed once; verified 20/20 lines.
Evidence: docs/evidence/followup/05-command-reproducibility.txt
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The twelve-request result and the post-reauth value were printed in the document but never written to a file; the audit flagged it as a conclusion standing without evidence. Measured again with timestamps into 03-b4-role-propagation.txt, which also surfaced a 107-second clock skew between the browser host and the lab host that the file now records so the two timelines can be compared.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
An independent audit found ten documents printing values their evidence files do not contain. C-1 printed a session count of 0 where the evidence says 4, C-2 printed a success readback for a command that exited 1, and A-1 credited the conntrack flush with a split that the timestamps attribute to a pod restart four seconds earlier.
Also measured wal_writer_delay, which A-3 had asserted as matching without ever querying it, relabelled the A-6 control that moved 41 percent, noted A-8's nine-sample resolution, corrected D-1's RTO to the 41 seconds its own timeline shows, and added a correction banner to D-2. Every experiment document now links its evidence files with their real collection times, and the duplicate screenshots are documented as duplicates.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The forward upgrade to 26.7.3 was zero downtime across 87 samples, and since databasechangelog stayed at 210 the rollback to 26.7.0 also succeeded, which narrows D-2's conclusion: rolling back fails when the schema moved, not because of the version number. The row count is the check.
Role changes never reach the upstream through request repetition; the session is a snapshot taken at login and only a new session picks up the new claim. Auditing the docs also surfaced that Prometheus scrapes only keycloak, kubelet, node-exporter and itself, so the B-layer experiments have no metrics to screenshot rather than missing screenshots.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three SAN entries and no wildcard is the constraint that cost something real in B-7, where oauth2-proxy had to borrow Grafana's app2 hostname because a fourth name was not available. The served chain is four deep and verifies, so fullchain.pem is configured rather than the cert.pem mistake that only breaks clients without a cached intermediate.
The forced renewal and the reload behaviour could not be measured because sudo on the host asks for a password, the same silent failure first noticed in B-7. nginx reload is graceful by design, but this lab has repeatedly shown that by design is not the same as measured, so it is recorded as untested rather than assumed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every secret in the lab prints in four commands, while kubectl describe shows just a byte count and creates the impression that something is hidden. k3s reports encryption at rest disabled and the plaintext password is present in state.db, so one node disk carries the whole cluster's secrets, and inside the pod they are ordinary environment variables visible to exec, /proc and crash dumps.
The default service account cannot read secrets, which makes RBAC the one control doing real work here and the thing worth tightening.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Downgrading from 26.7.0 to 26.0 fails with liquibase ValidationFailedException on a changeset checksum, which is stricter than an unknown migration: the old version knows the changeset but its definition differs. The pod goes CrashLoopBackOff and never starts.
The StatefulSet stopped the rollout at the first pod, so the other kept serving and the front door stayed at 200, which replica 1 would not have done. The failed start never touched the schema, so restoring the image was enough; had the migration already applied, the D-1 database restore would have been the only way back.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Dropping the schema left Keycloak serving realm metadata and JWKS from its Infinispan cache, so the front door stayed at 200 while only the paths that read the database failed. That is a different shape from A-2, where the connection itself broke and readiness pulled the pods out of the Service; here the connection is fine and the tables are simply gone, which the health check does not notice.
Restoring the pg_dump took one second with zero errors and no pod restart, and the row counts matched the backup exactly, sessions included. The real RPO is the backup interval plus the synchronous_commit loss measured in A-3, and this dump sits in the host's /tmp, which is the same failure domain as the thing it protects.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Neither client had a backchannel logout URL and the BFF has no oidcLogout configuration, so the three candidate paths all answer 302, which is the authentication redirect rather than a handler. Setting the URL on the identity provider alone changed nothing: with a live session, logging the user out emptied the Keycloak side and left the Redis session untouched.
Reachability is not the blocker here, since a Keycloak pod fetches the app's public URL with a 200, but that is a property of this tailnet split-DNS lab and is the assumption most likely to fail in production, where it fails silently.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
One user session carries a client session per application, so visiting the second app skips the login screen. Deleting the identity provider session leaves both application sessions untouched and both apps keep serving, because the identity provider, the application session and the access token each have their own lifetime.
That inverts the B-2 finding: there the app session was cleared and the surviving SSO session let the user straight back in. Either way, clearing one side leaves the other. It also means an identity provider outage is a single point of failure for logging in, not for already-authenticated users, and the failure arrives late and all at once.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
oauth2-proxy carries the authorization request in a signed cookie, so the callback can land on a different replica and still succeed, which is the opposite of the BFF failure in B-0. Sharing is therefore just sharing one Secret.
Rotating it is all-or-nothing: --cookie-secret is singular, so there is no second key to read old tickets with, and the log shows both the validation failure and Error removing session, leaving the Redis session orphaned because the key cannot be derived from a ticket that will not decode.
Getting there required two diagnoses: the callback 502 came from the full session riding in Set-Cookie past nginx's buffer, and every earlier attempt to read nginx config returned nothing because sudo on the host asks for a password while the guests do not.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adding a higher-priority RSA provider leaves both kids in JWKS, so tokens signed before and after the rotation both validate. Deleting the old provider makes its tokens 401 immediately, and the resource server's JWKS cache does not buy a grace period because an unknown kid triggers a refetch.
The encryption key Q3 asks about does not exist yet, since B-2 showed the tokens are stored as plaintext JWTs, so the measured signing-key rotation is what its design has to copy: write with one key, read with several, and keep the overlap longer than the lifetime of anything signed with the old one.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Stopping Redis returns HTTP 000 rather than an error because the client waits on reconnect, and the pod keeps serving traffic because the redis health indicator is not in the readiness group even though /actuator/health returns 503. That is the mirror image of A-2, where Keycloak put its database check in readiness and the pods left the Service.
Turning on AOF with config set created the appendonlydir and still lost everything on pod deletion, because /data was the container filesystem; adding a PVC makes the same setting work. Volume first, persistence setting second.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two headers of the same name both arrive rather than one overwriting the other, because nginx only replaces headers it sets with proxy_set_header. A comma inside a role name is indistinguishable from the delimiter, and the size limit is a cliff: Tomcat returns 400 around 8KB and the connection dies around 16KB, so the same cause produces two different-looking failures.
Forged identity headers reach the upstream untouched while the JWT-protected paths return 401, which is Q4's own point that a header-fed upstream has nothing to verify against. By Q4's checklist that answer alone points at the BFF structure.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Five simultaneous refreshes with one token return a single 200, and that winner's new token is already dead. Reuse detection removes the client session while the user session stays, which is why the other responses read Session doesn't have required client rather than a reuse error.
Comparing policies shows rotation off passes all five and keeps the session, while raising refreshTokenMaxReuse to one still destroys it. Since no retry can recover a removed client session, Q2's own criterion resolves to a lock, and a database row lock is the natural place because its lifetime is tied to the connection.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Moving the authorized client to JdbcOAuth2AuthorizedClientService makes tokens work across replicas, so the session-in-Redis plus tokens-in-PostgreSQL split holds. The table then shows what sharing cannot fix: the primary key is (client_registration_id, principal_name) with no session in it, so a second login for the same user updates the same row rather than adding one.
The refresh token sits in bytea as the raw JWT, readable with convert_from, and logout clears only the Redis session while the plaintext token row and the Keycloak SSO session both survive. The schema itself failed silently first because the default DDL uses blob, which PostgreSQL does not have, and continue-on-error swallowed it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adding Spring Session Redis grows the context by 81 beans and swaps sessionRepository for RedisSessionRepository, while authorizedClientService stays InMemoryOAuth2AuthorizedClientService. The user then reads as logged in with principal labuser while accessTokenStoredOnServer is false, which is worse than being logged out.
Redis holds only the security context, serialized with Java native serialization, and the refresh token is not there to encrypt in the first place. Three problems on the way: Kubernetes service links overwrote REDIS_PORT with a tcp:// URL, the tests tried to reach Redis, and the resource server was never deployed so a DNS failure looked like a token failure.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The authorized client repository is AuthenticatedPrincipalOAuth2AuthorizedClientRepository, keyed by principal with no session id in it, which is the mechanism behind the sharing problem Q1 and Q3 describe. Sharing a store does not fix a lookup key.
Five problems on the way in: only build output was committed under bff/, a duplicate YAML key broke the image build and was invisible until the full log was captured, env placeholders without defaults broke the tests, actuator was behind the login redirect so a 200 was the login page, and the 117KB beans response failed through the proxy.
Deploying two replicas made the login itself fail before any experiment started, because the authorization request lives in per-instance memory and the callback lands elsewhere.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Disabling persistent-user-sessions moves the session from PostgreSQL into the cluster, and the A-1 and A-8 outcomes flip to 400 Session not active while a new login during database loss starts working. The control group in each case still returns 200, so the injections cut only what they were meant to cut.
This is the pair that makes the A layer legible: the conventional wisdom that sessions ride TCP 7800 is correct for Keycloak 24 and earlier, and the mistake is applying it to 26 without checking the version.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Nine samples through the restart all returned 200, the refresh token issued beforehand still works, and the session count is unchanged at 151 while both caches reset to zero. The updated last_session_refresh proves the write path recovered too, not just the response code.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Nine database round trips per login multiply the injected delay to 1.9 seconds, and connection pool queueing multiplies it again under twenty concurrent requests. The readiness probe joins the same queue and times out, so the node leaves the load balancer and pushes its load onto the one still standing.
Two injections missed first: the guest interface is enp1s0 rather than eth0, and a filter on it can never match a pod IP because flannel has already encapsulated the packet. The delay has to go on flannel.1, before encapsulation.
The predicted rise in optimistic lock conflicts did not happen, because logins insert new rows rather than contending for one. That belongs to B-3.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three injections failed first: kube-router keeps reinserting its chain above a hand-placed FORWARD rule, the JGroups connection direction had reversed since A-1, and only the raw table runs ahead of conntrack. Each failure looked like nothing happening.
Blocking one direction never partitioned the cluster because JGroups reconnected the other way before failure detection fired. Blocking both produced a real split brain with two coordinators in JGROUPS_PING, yet only the non-coordinator node reported DOWN, so the Service kept an endpoint and the front door stayed at 200. That answers the question A-1 left open.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Kubernetes keeps calling the node Ready for forty seconds while users already see failures, and the pod on the powered-off machine stays ready=true because its kubelet can no longer contradict itself. Eviction waits another five minutes, then the StatefulSet refuses to recreate its pod and the replacement Deployment pod cannot schedule because the local-path volume is pinned to the dead node.
Killing the server node instead shows the opposite shape: containerd keeps the workload running while the API server, Traefik and the observability stack disappear, so the outage is the missing path rather than the missing application. Traefik at one replica is the ingress single point of failure.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Keycloak commits the login INSERT with synchronous_commit off, so a crash loses whole sessions and not just refresh timestamps. Measured 4 of 153 lost, matching the default wal_writer_delay window.
Two injections failed silently first: --grace-period=0 --force lets the container runtime send SIGTERM so PostgreSQL flushes and shuts down cleanly, and SIGKILL to PID 1 from inside its own namespace is ignored by the kernel. Killing a backend makes the postmaster reinitialize, which is a real crash recovery.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Both pods go NotReady, the Service endpoint list empties and the front door returns 503, so adding Keycloak replicas buys nothing against database loss. The node holding the session in cache fails too, because a refresh writes LAST_SESSION_REFRESH. Recovery was automatic in about fifteen seconds with no restart, which is what readiness rather than liveness buys.
The observability finding matters as much: up stayed at 1 through a total outage, so alerting on it would have caught nothing. kube-state-metrics is missing and pod readiness is therefore not recorded as a metric.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Cutting TCP 7800 leaves cross-node refresh working (200), confirming sessions travel through PostgreSQL rather than the cluster transport. Logout is the opposite: the database row is deleted but the other node answers from its stale local cache, so the A-0 conclusion that invalidation rides the database is corrected here.
Two things the plan did not anticipate: a NetworkPolicy cannot sever an established connection because conntrack accepts it before policy evaluation, and Keycloak reports the partition through its readiness probe so the split node removes itself from the Service.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
PostgreSQL statement logging shows keycloak-1 reading and updating the session created on keycloak-0. The same transaction reveals optimistic locking via VERSION, SKIP LOCKED, and synchronous_commit turned off. Fixes the earlier concept note that credited Infinispan with cross-node propagation.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Experiment 0 with three probes: cross-node refresh/logout, cache counter deltas around a single login, and cache entry ownership. Each node caches only what it handled; cache totals sum exactly to the database count.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Control experiment isolates two independent causes: the nginx 443 block still emits X-Forwarded-Proto http, and Traefik rewrites forwarded headers regardless.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
