The guides were written before the skill existed and a scored audit found
three gaps. Fixed by a subagent running under the skill, with measured
values, versions, IPs and quoted output declared off limits.
The big one: every step showed a command and its output, and almost none
said which line to look at or what it meant. 64 interpretation pairs added
across the seven files, weighted where the reading is hardest — 20 in the
Keycloak stage, where a Secret existing and a pod having received it are
different facts.
Only the extracting form of curl appeared. Where the reader meets a response
for the first time the guides now open with curl -I or curl -v and name the
lines worth reading; -w '%{http_code}' survives only where the code is a
value being compared — two upstream nodes against each other, or the 900-run
control loop.
Listing Secret keys went from a three-stage pipe to kubectl describe secret,
which prints the key names and their byte counts in one native command
without exposing a value.
And a tool assumption: jq and yamllint are installed on neither the lab host
nor the guests. The guides now say so where JSON is read by eye, rather than
sending the reader to install something mid-diagnosis. cloud-init schema is
on the guests and is now the guest-side check.
Also removes a stray Playwright screenshot committed at the repository root
in 919547a; the evidence copy under docs/evidence/b7a-orphan-session/ is the
one the document references.
Four things the audit left standing are recorded in the agent's report rather
than papered over — notably that 04's reload measurements are stated without
a reproduction procedure, and that 05 and 06 reference each other as
prerequisites.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The experiment documents record what was found. These record what to type to
reproduce it, in folders per stage.
Two kinds of command are kept apart. 하기/확인 is what somebody actually types
at a terminal — short, one at a time. 근거를 재려면 is the long measuring form
this lab used to put evidence in a document, marked as not needed day to day.
The same split applies to curl: -I to look once, -w '%{http_code}' only when
comparing across repetitions.
No placeholders. Where a value is needed the command that produces it is
given, and secrets are checked by length rather than printed:
TOKEN=$(ssh kc-lab-1 'sudo cat /var/lib/rancher/k3s/server/node-token')
echo "${#TOKEN} 자"
Stage 05 verifies resources in layers, because a Secret existing and a pod
having received it are different facts: keys, then length, then the value
inside the container, then which env var came from which Secret. Same for
workloads — Deployment to ReplicaSet to Pod, with the seven ReplicaSets this
cluster actually carries as the worked example.
Two commands were wrong and re-running them caught it. kubectl get endpoints
prints a deprecation warning on v1.33+, so the guide uses describe svc and
EndpointSlice. And the Keycloak image has no curl, so reading metrics from
inside the container fails with exit 127 — the guide asks Prometheus instead,
or runs a throwaway curl pod.
Read-only checks were executed against the running lab and their output is
quoted verbatim. Creating commands could not be re-run without destroying the
lab, so they are the ones used at build time; the README says which is which.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
An audit against the standard the series set — concepts, procedure,
commands, architecture diagram, evidence table, terminal output — found the
three new experiments met it while twelve of the original ones had no
diagram at all: A-0, A-1, A-3, A-4, A-5, A-6, A-8, B-0, B-2, B-7, C-2, D-2.
Each now has one drawn from what that experiment actually found, not filler:
A-0 shows sharing going through PostgreSQL rather than between the caches;
A-3 the gap between the 200 and the WAL flush, with both failed injections;
A-5 the three silent injection failures; A-6 the two places latency is
multiplied; B-0 the repository keyed by principal with no session id; B-2
the primary key that causes the overwrite; D-2 why the rolling update
stopped the accident halfway.
Also corrected the index's stale claim of 11 experiments without a
screenshot — it is 14, and the reason is recorded: those experiments were
measured from terminals, the database and logs, and the observability stack
does not scrape Redis, the BFF or PostgreSQL, so there is no console to
photograph.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Each of A-7a, B-7a and D-4a had the concepts, procedure, commands and
terminal evidence but no diagram, which the rest of the series carries.
a7a-cache-temperature.svg the three cache states and the SQL each issues
b7a-orphan-lifecycle.svg ticket structure, accumulation across two
rotations, and why TTL is the only signal
d4a-hook-timeline.svg the 38m25s gap against the 1-2s one, on one axis
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
D-4 named the fix and never ran it. Running it:
/etc/letsencrypt/renewal-hooks/deploy/reload-nginx.sh
#!/bin/sh
nginx -t && nginx -s reload
certbot reports "Hook 'deploy-hook' ran", the master stays at PID 585 and
the worker is replaced 28829 -> 37252, and the served certificate changes.
One trap worth naming: certbot prefixes the hook output with "ran with error
output" whenever the hook writes anything to stderr, and nginx's routine
types_hash warning goes to stderr. Everything inside is success — "test is
successful", "signal process started". A monitor that greps for "error"
would read a working hook as a failure. The worker PID is what to check.
Timing needed a clock correction. test-server has NTP off and runs 106
seconds fast; the dev machine matches Google and the Let's Encrypt ACME
endpoint exactly. Corrected, the hook's nginx -t lands at 12:27:50 UTC
against the new certificate's SCT at 12:27:49.054 — one second. The
correction validates itself: uncorrected, the hook would have run 104
seconds before the certificate existed.
That same skew is why D-4's gap was reported 106 seconds short; corrected to
2305s in the previous commit.
Remaining and left alone: whether certbot-renew.timer performs a real
renewal, which cannot be tested for about 89 days.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The 2199 seconds reported for D-4 subtracted a test-server timestamp
(archive/cert2.pem mtime) from a dev-machine timestamp (the serial change
observed by the poll), without noting they are different clocks.
Checked against external references: the dev machine matches Google and the
Let's Encrypt ACME endpoint to the second, while test-server is 105 seconds
fast and reports NTPSynchronized=no. Three round-trip measurements put the
offset at +106.1s every time.
Corrected:
new certificate written to disk 08:20:27 UTC (mtime 17:22:13 KST - 106s)
actually served 08:58:52 UTC (dev observation, no correction)
gap 2305s = 38m25s
The correction validates itself in D-4a, where the new certificate's SCT —
signed by CT logs on their own accurate clock at 12:27:49.054 GMT — lands
one second before the skew-corrected hook time. Without the correction the
hook would appear to have run 104 seconds before the certificate existed.
Updated across the experiment doc, the index, the follow-up doc, the verdict
evidence file and the SVG.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
B-7 stopped at "could not delete the server-side session". The reason it
gave was right: the ticket carries the session id, the ticket is encrypted
with the cookie secret, so after a rotation the proxy cannot work out which
Redis key to remove. But that is a limitation of the proxy, not of Redis.
Measured across two rotations:
- The orphan does expire. TTL falls one second per second and is not
refreshed by requests (the startup log says refresh:disabled, and
--cookie-refresh is unset), so it dies exactly one hour after creation.
- An operator can delete it. `redis-cli del` returned 1, dbsize went 2 to 1,
and the live session answered /oauth2/userinfo with 200 immediately after.
- But nothing in Redis says which key is the orphan. Same name prefix, same
type, the same 3510 bytes, and the values are encrypted.
- TTL is the only signal, and because it is never refreshed it is an exact
function of creation time. Anything created before the rotation is an
orphan. Derived creation time 11:30:26 against the AuthSuccess log line at
11:30:27 — one second out. The rule was then run and removed the orphan
while leaving the live session.
- They accumulate: the session that survived the first rotation became the
orphan of the second.
The caveat is recorded too: turning on --cookie-refresh breaks the
derivation, and at that point flushing and forcing everyone to
re-authenticate is the more honest option.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A-7 left the cause as a hypothesis and said to confirm it with the statement
logging A-3 used. Doing that shows the hypothesis was wrong.
Login issues no SQL at all in volatile mode. Refresh issues exactly one
statement, and it is not the one A-7 guessed:
select cscme1_0.SCOPE_ID from CLIENT_SCOPE_CLIENT cscme1_0
where cscme1_0.CLIENT_ID=$1 and cscme1_0.DEFAULT_SCOPE=$2
parameters: $1 = '131a9912-...', $2 = 'f'
REVOKED_TOKEN never appears. DEFAULT_SCOPE='f' means this is the optional
client scope list, which refresh needs because it recomputes which scopes go
into the new access token.
The larger finding is that A-7's table was conditional and did not say so.
The single statement is issued once and then cached, so the same
configuration produces three different outcomes depending only on cache
warmth — all three reproduced:
fully cold login 400 (select ce1_0.ID from CLIENT ...)
CLIENT warm login 200, refresh 500 <- what A-7 measured
fully warm login 200, refresh 200
So "volatile means you can log in without a database" is also conditional:
on a cold start the client lookup itself fails. In each case the Keycloak
log names the failing SQL directly, so this is determined rather than
inferred.
A-7 keeps its original wording with a correction banner above it, so what
was mis-guessed and why stays visible.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
B-4 left role propagation open because oauth2-proxy was not deployed yet.
B-7 deployed it and the follow-up measured it: the value does not change
with request count, only when a new session is created.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
D-2 forward upgrade and B-4 role propagation were filled earlier. D-4 forced
renewal was the third, and it needed a person at the keyboard because the
host asks for a sudo password.
It turned out to hold the largest finding of the three: the renewal
succeeded and went unserved for 36 minutes 39 seconds, because nothing in
the setup reloads nginx — no ExecStartPost, three empty hook directories,
no nginx plugin. And the half of the plan's question that was still an
expectation is now a measurement: the reload is graceful, 8856 connections
with zero failures and an in-flight request surviving intact.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The forced renewal did work. What did not work was anything making nginx
notice.
disk cert2.pem written 2026-09-04 17:22:13 KST
network old serial through 08:58:47 UTC, new serial from 08:58:52 UTC
gap 2199 seconds, and the old certificate was observed 428 times in it
The gap closed only because a person ran `nginx -s reload`. Nothing else
would have: certbot-renew.service is just `ExecStart=/usr/bin/certbot -q
renew` with no ExecStartPost, all three of renewal-hooks/{deploy,post,pre}
are empty, and certbot 5.7.0 here has no nginx plugin (dns-cloudflare,
manual, null, standalone, webroot). Three paths, all empty. Any one of them
would have been enough.
nginx holds the certificate in memory from startup, and certbot swaps the
live/ symlink rather than the path — so the config still looks right while
the served certificate is stale. Master 585 and worker 586 sharing a start
time 22.4 hours old is what proved no reload had happened.
This is invisible for 88 days. The timer ran twice today and exited
0/SUCCESS both times, because a renewal is not due yet. The day it becomes
due, the symptom is an expired certificate and the log still says SUCCESS.
The other half of the plan's question now has a measurement rather than an
expectation. The reload is genuinely graceful: 8856 polled connections, zero
non-200, p95 205.7ms before against 204.3ms after. And an in-flight request
— 845KB pulled at 20k/s, so still transferring 12 seconds in when the reload
landed — completed with all 845361 bytes over a single connection.
The earlier note that nginx reload "is graceful by design but must not be
written that way without checking" was right to hold back. The neighbouring
belief, that renewing gets the new certificate served, was the one that was
false, and there was no way to tell in advance which it would be.
Adds diagrams/d4-renewal-gap.svg and evidence 09-13.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The forced renewal did not change the certificate being served — 161 serial
samples over 13 minutes, all identical, notBefore still Sep 3.
Two things were readable without sudo and both matter.
nginx has never been reloaded: master 585 and worker 586 share a start time
and both are 80529 seconds old. A reload keeps the master and replaces the
workers, so the original fork still being there means no reload has happened
since Sep 3 19:00. nginx reads the certificate at startup and holds it in
memory; without a reload it serves the old one no matter what is on disk.
And the unit that is supposed to renew has nothing that would reload it:
[Service]
Type=oneshot
ExecStart=/usr/bin/certbot -q renew
PrivateTmp=true
No ExecStartPost, no --deploy-hook. The timer runs twice a day and exited
0/SUCCESS both times today, which is precisely why this stays invisible —
the renewal is not due for 88 days, so nothing has failed yet.
What remains undetermined is whether the user's --force-renewal failed, or
succeeded and left new files that nginx never picked up. Telling those apart
needs /etc/letsencrypt, which is root-only. If it is the latter, that is the
most valuable thing this lab has produced: "renewal succeeded" and "the new
certificate is being served" are different events.
CT logs looked like a way around the permission problem and were not: the
served certificate carries two valid SCTs, yet crt.sh returns zero rows for
auth.hyeonworks.com. Same shape as A-2's `up` metric — the observing tool
sees a subset of the truth.
Also recorded: 76 in-flight failures at 08:15:04 are a local artifact, not a
server event. The concurrent 0.2s poll was clean across the same second,
num_connects was 0, and time_total was 50 microseconds. Not reproducible in
100 retries. The monitor now records curl's exit code so a recurrence is
diagnosable.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The forced renewal needs sudo on the host, and the host asks for a password.
That blocks the injection, not the experiment — the part worth doing first
was the control anyway.
Control 1 — new connections, 0.2s x 900 over 180s: 900/900 = 200, zero
failures. The noise floor is 0, so a single non-200 during the renewal can
be attributed to the renewal. Without this the observation would have been
uninterpretable, which is the mistake A-6 made calling a -41% control
"no effect" and A-8 made claiming zero-downtime from 9 samples.
Control 2 — the poll cannot answer the question the plan actually asked.
TLS handshakes were 900/900, meaning every request is a fresh connection,
so it measures "are new connections accepted", not "what happens to a
request already in flight". A separate device: the 845KB admin console
bundle pulled at --limit-rate 20k, holding one request open for 42 seconds.
Baseline 200 / 845361 bytes / 1 connection.
Monitors are running under setsid, and stop three minutes after the
certificate serial changes.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The audit found ~80 placeholders, and the damaging ones were where the
measuring apparatus itself was prose rather than a command:
a6 "( curl ... ) & 를 20개 띄우고 wait" — the 22.2s headline came from this
a3 "<로그인 반복, sid 를 /tmp/sids 에>" — the whole RPO measurement
a3 "<sid 목록>" — the control it is compared against
a5 "<수신 파드IP>" — the injection
a8 writes /tmp/tok, reads /tmp/rt — self-inconsistent, sent an empty token
b3 $KC / $RT / $NEW never assigned
c2 bare kcadm.sh with no kubectl exec
a1 conntrack tuples written by hand, though the direction flips per restart
Each is now a shell-expandable form: pod IPs from jsonpath, the admin password
from the secret, ids from kcadm --format csv, conntrack tuples derived from
"conntrack -L" with awk rather than transcribed.
Then the rewritten commands were executed against the live cluster, and one
of them failed — the 20-way load generator, written as "kubectl run --rm -i",
lost its output stream twice in a row. That is a trap this series already hit
once, and the rewrite reintroduced it. A-6 now uses a resident probe pod that
collects into a file and is cat-ed once; verified 20/20 lines.
Evidence: docs/evidence/followup/05-command-reproducibility.txt
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The twelve-request result and the post-reauth value were printed in the document but never written to a file; the audit flagged it as a conclusion standing without evidence. Measured again with timestamps into 03-b4-role-propagation.txt, which also surfaced a 107-second clock skew between the browser host and the lab host that the file now records so the two timelines can be compared.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
An independent audit found ten documents printing values their evidence files do not contain. C-1 printed a session count of 0 where the evidence says 4, C-2 printed a success readback for a command that exited 1, and A-1 credited the conntrack flush with a split that the timestamps attribute to a pod restart four seconds earlier.
Also measured wal_writer_delay, which A-3 had asserted as matching without ever querying it, relabelled the A-6 control that moved 41 percent, noted A-8's nine-sample resolution, corrected D-1's RTO to the 41 seconds its own timeline shows, and added a correction banner to D-2. Every experiment document now links its evidence files with their real collection times, and the duplicate screenshots are documented as duplicates.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Twelve SVG architecture diagrams cover the experiments whose documents had little or no structure drawing, embedded under a 구조 heading with a shared convention file. Seven documents carried their concepts under narrative headings and now have an explicit 개념 section so they can be found.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The forward upgrade to 26.7.3 was zero downtime across 87 samples, and since databasechangelog stayed at 210 the rollback to 26.7.0 also succeeded, which narrows D-2's conclusion: rolling back fails when the schema moved, not because of the version number. The row count is the check.
Role changes never reach the upstream through request repetition; the session is a snapshot taken at login and only a new session picks up the new claim. Auditing the docs also surfaced that Prometheus scrapes only keycloak, kubelet, node-exporter and itself, so the B-layer experiments have no metrics to screenshot rather than missing screenshots.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
One table per experiment with its branch and result, plus the nine injections that silently did nothing and the five predictions that turned out wrong.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three SAN entries and no wildcard is the constraint that cost something real in B-7, where oauth2-proxy had to borrow Grafana's app2 hostname because a fourth name was not available. The served chain is four deep and verifies, so fullchain.pem is configured rather than the cert.pem mistake that only breaks clients without a cached intermediate.
The forced renewal and the reload behaviour could not be measured because sudo on the host asks for a password, the same silent failure first noticed in B-7. nginx reload is graceful by design, but this lab has repeatedly shown that by design is not the same as measured, so it is recorded as untested rather than assumed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every secret in the lab prints in four commands, while kubectl describe shows just a byte count and creates the impression that something is hidden. k3s reports encryption at rest disabled and the plaintext password is present in state.db, so one node disk carries the whole cluster's secrets, and inside the pod they are ordinary environment variables visible to exec, /proc and crash dumps.
The default service account cannot read secrets, which makes RBAC the one control doing real work here and the thing worth tightening.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Downgrading from 26.7.0 to 26.0 fails with liquibase ValidationFailedException on a changeset checksum, which is stricter than an unknown migration: the old version knows the changeset but its definition differs. The pod goes CrashLoopBackOff and never starts.
The StatefulSet stopped the rollout at the first pod, so the other kept serving and the front door stayed at 200, which replica 1 would not have done. The failed start never touched the schema, so restoring the image was enough; had the migration already applied, the D-1 database restore would have been the only way back.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Dropping the schema left Keycloak serving realm metadata and JWKS from its Infinispan cache, so the front door stayed at 200 while only the paths that read the database failed. That is a different shape from A-2, where the connection itself broke and readiness pulled the pods out of the Service; here the connection is fine and the tables are simply gone, which the health check does not notice.
Restoring the pg_dump took one second with zero errors and no pod restart, and the row counts matched the backup exactly, sessions included. The real RPO is the backup interval plus the synchronous_commit loss measured in A-3, and this dump sits in the host's /tmp, which is the same failure domain as the thing it protects.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Neither client had a backchannel logout URL and the BFF has no oidcLogout configuration, so the three candidate paths all answer 302, which is the authentication redirect rather than a handler. Setting the URL on the identity provider alone changed nothing: with a live session, logging the user out emptied the Keycloak side and left the Redis session untouched.
Reachability is not the blocker here, since a Keycloak pod fetches the app's public URL with a 200, but that is a property of this tailnet split-DNS lab and is the assumption most likely to fail in production, where it fails silently.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
One user session carries a client session per application, so visiting the second app skips the login screen. Deleting the identity provider session leaves both application sessions untouched and both apps keep serving, because the identity provider, the application session and the access token each have their own lifetime.
That inverts the B-2 finding: there the app session was cleared and the surviving SSO session let the user straight back in. Either way, clearing one side leaves the other. It also means an identity provider outage is a single point of failure for logging in, not for already-authenticated users, and the failure arrives late and all at once.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
oauth2-proxy carries the authorization request in a signed cookie, so the callback can land on a different replica and still succeed, which is the opposite of the BFF failure in B-0. Sharing is therefore just sharing one Secret.
Rotating it is all-or-nothing: --cookie-secret is singular, so there is no second key to read old tickets with, and the log shows both the validation failure and Error removing session, leaving the Redis session orphaned because the key cannot be derived from a ticket that will not decode.
Getting there required two diagnoses: the callback 502 came from the full session riding in Set-Cookie past nginx's buffer, and every earlier attempt to read nginx config returned nothing because sudo on the host asks for a password while the guests do not.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adding a higher-priority RSA provider leaves both kids in JWKS, so tokens signed before and after the rotation both validate. Deleting the old provider makes its tokens 401 immediately, and the resource server's JWKS cache does not buy a grace period because an unknown kid triggers a refetch.
The encryption key Q3 asks about does not exist yet, since B-2 showed the tokens are stored as plaintext JWTs, so the measured signing-key rotation is what its design has to copy: write with one key, read with several, and keep the overlap longer than the lifetime of anything signed with the old one.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Stopping Redis returns HTTP 000 rather than an error because the client waits on reconnect, and the pod keeps serving traffic because the redis health indicator is not in the readiness group even though /actuator/health returns 503. That is the mirror image of A-2, where Keycloak put its database check in readiness and the pods left the Service.
Turning on AOF with config set created the appendonlydir and still lost everything on pod deletion, because /data was the container filesystem; adding a PVC makes the same setting work. Volume first, persistence setting second.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two headers of the same name both arrive rather than one overwriting the other, because nginx only replaces headers it sets with proxy_set_header. A comma inside a role name is indistinguishable from the delimiter, and the size limit is a cliff: Tomcat returns 400 around 8KB and the connection dies around 16KB, so the same cause produces two different-looking failures.
Forged identity headers reach the upstream untouched while the JWT-protected paths return 401, which is Q4's own point that a header-fed upstream has nothing to verify against. By Q4's checklist that answer alone points at the BFF structure.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Five simultaneous refreshes with one token return a single 200, and that winner's new token is already dead. Reuse detection removes the client session while the user session stays, which is why the other responses read Session doesn't have required client rather than a reuse error.
Comparing policies shows rotation off passes all five and keeps the session, while raising refreshTokenMaxReuse to one still destroys it. Since no retry can recover a removed client session, Q2's own criterion resolves to a lock, and a database row lock is the natural place because its lifetime is tied to the connection.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Moving the authorized client to JdbcOAuth2AuthorizedClientService makes tokens work across replicas, so the session-in-Redis plus tokens-in-PostgreSQL split holds. The table then shows what sharing cannot fix: the primary key is (client_registration_id, principal_name) with no session in it, so a second login for the same user updates the same row rather than adding one.
The refresh token sits in bytea as the raw JWT, readable with convert_from, and logout clears only the Redis session while the plaintext token row and the Keycloak SSO session both survive. The schema itself failed silently first because the default DDL uses blob, which PostgreSQL does not have, and continue-on-error swallowed it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adding Spring Session Redis grows the context by 81 beans and swaps sessionRepository for RedisSessionRepository, while authorizedClientService stays InMemoryOAuth2AuthorizedClientService. The user then reads as logged in with principal labuser while accessTokenStoredOnServer is false, which is worse than being logged out.
Redis holds only the security context, serialized with Java native serialization, and the refresh token is not there to encrypt in the first place. Three problems on the way: Kubernetes service links overwrote REDIS_PORT with a tcp:// URL, the tests tried to reach Redis, and the resource server was never deployed so a DNS failure looked like a token failure.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The authorized client repository is AuthenticatedPrincipalOAuth2AuthorizedClientRepository, keyed by principal with no session id in it, which is the mechanism behind the sharing problem Q1 and Q3 describe. Sharing a store does not fix a lookup key.
Five problems on the way in: only build output was committed under bff/, a duplicate YAML key broke the image build and was invisible until the full log was captured, env placeholders without defaults broke the tests, actuator was behind the login redirect so a 200 was the login page, and the 117KB beans response failed through the proxy.
Deploying two replicas made the login itself fail before any experiment started, because the authorization request lives in per-instance memory and the callback lands elsewhere.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Disabling persistent-user-sessions moves the session from PostgreSQL into the cluster, and the A-1 and A-8 outcomes flip to 400 Session not active while a new login during database loss starts working. The control group in each case still returns 200, so the injections cut only what they were meant to cut.
This is the pair that makes the A layer legible: the conventional wisdom that sessions ride TCP 7800 is correct for Keycloak 24 and earlier, and the mistake is applying it to 26 without checking the version.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Nine samples through the restart all returned 200, the refresh token issued beforehand still works, and the session count is unchanged at 151 while both caches reset to zero. The updated last_session_refresh proves the write path recovered too, not just the response code.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Nine database round trips per login multiply the injected delay to 1.9 seconds, and connection pool queueing multiplies it again under twenty concurrent requests. The readiness probe joins the same queue and times out, so the node leaves the load balancer and pushes its load onto the one still standing.
Two injections missed first: the guest interface is enp1s0 rather than eth0, and a filter on it can never match a pod IP because flannel has already encapsulated the packet. The delay has to go on flannel.1, before encapsulation.
The predicted rise in optimistic lock conflicts did not happen, because logins insert new rows rather than contending for one. That belongs to B-3.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three injections failed first: kube-router keeps reinserting its chain above a hand-placed FORWARD rule, the JGroups connection direction had reversed since A-1, and only the raw table runs ahead of conntrack. Each failure looked like nothing happening.
Blocking one direction never partitioned the cluster because JGroups reconnected the other way before failure detection fired. Blocking both produced a real split brain with two coordinators in JGROUPS_PING, yet only the non-coordinator node reported DOWN, so the Service kept an endpoint and the front door stayed at 200. That answers the question A-1 left open.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Kubernetes keeps calling the node Ready for forty seconds while users already see failures, and the pod on the powered-off machine stays ready=true because its kubelet can no longer contradict itself. Eviction waits another five minutes, then the StatefulSet refuses to recreate its pod and the replacement Deployment pod cannot schedule because the local-path volume is pinned to the dead node.
Killing the server node instead shows the opposite shape: containerd keeps the workload running while the API server, Traefik and the observability stack disappear, so the outage is the missing path rather than the missing application. Traefik at one replica is the ingress single point of failure.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Keycloak commits the login INSERT with synchronous_commit off, so a crash loses whole sessions and not just refresh timestamps. Measured 4 of 153 lost, matching the default wal_writer_delay window.
Two injections failed silently first: --grace-period=0 --force lets the container runtime send SIGTERM so PostgreSQL flushes and shuts down cleanly, and SIGKILL to PID 1 from inside its own namespace is ignored by the kernel. Killing a backend makes the postmaster reinitialize, which is a real crash recovery.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Both pods go NotReady, the Service endpoint list empties and the front door returns 503, so adding Keycloak replicas buys nothing against database loss. The node holding the session in cache fails too, because a refresh writes LAST_SESSION_REFRESH. Recovery was automatic in about fifteen seconds with no restart, which is what readiness rather than liveness buys.
The observability finding matters as much: up stayed at 1 through a total outage, so alerting on it would have caught nothing. kube-state-metrics is missing and pod readiness is therefore not recorded as a metric.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Cutting TCP 7800 leaves cross-node refresh working (200), confirming sessions travel through PostgreSQL rather than the cluster transport. Logout is the opposite: the database row is deleted but the other node answers from its stale local cache, so the A-0 conclusion that invalidation rides the database is corrected here.
Two things the plan did not anticipate: a NetworkPolicy cannot sever an established connection because conntrack accepts it before policy evaluation, and Keycloak reports the partition through its readiness probe so the split node removes itself from the Service.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Cross-checked each question's 남은 미지수, 다음 검증 and 제약 against the plan item by item. Adds B-0 (autoconfiguration actually chosen), B-6 (encryption key rotation) and B-7 (oauth2-proxy cookie secret rotation) as new experiments, plus lock-holder death, rotation-disabled comparison, partial-logout recovery, store latency and the Q4 design checklist. Restores the Redis persistence comparison and records the correct index URL.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Twenty experiments across four layers, each with a topology diagram marking where the fault goes in, the metrics to watch, a falsifiable prediction written before the run, and a pass/fail rule.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Builds up from HTTP statelessness to why session storage location determines the operational response, so the measurements have context to land in.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
PostgreSQL statement logging shows keycloak-1 reading and updating the session created on keycloak-0. The same transaction reveals optimistic locking via VERSION, SKIP LOCKED, and synchronous_commit turned off. Fixes the earlier concept note that credited Infinispan with cross-node propagation.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Experiment 0 with three probes: cross-node refresh/logout, cache counter deltas around a single login, and cache entry ownership. Each node caches only what it handled; cache totals sum exactly to the database count.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
