An audit against the standard the series set — concepts, procedure,
commands, architecture diagram, evidence table, terminal output — found the
three new experiments met it while twelve of the original ones had no
diagram at all: A-0, A-1, A-3, A-4, A-5, A-6, A-8, B-0, B-2, B-7, C-2, D-2.
Each now has one drawn from what that experiment actually found, not filler:
A-0 shows sharing going through PostgreSQL rather than between the caches;
A-3 the gap between the 200 and the WAL flush, with both failed injections;
A-5 the three silent injection failures; A-6 the two places latency is
multiplied; B-0 the repository keyed by principal with no session id; B-2
the primary key that causes the overwrite; D-2 why the rolling update
stopped the accident halfway.
Also corrected the index's stale claim of 11 experiments without a
screenshot — it is 14, and the reason is recorded: those experiments were
measured from terminals, the database and logs, and the observability stack
does not scrape Redis, the BFF or PostgreSQL, so there is no console to
photograph.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
D-4 named the fix and never ran it. Running it:
/etc/letsencrypt/renewal-hooks/deploy/reload-nginx.sh
#!/bin/sh
nginx -t && nginx -s reload
certbot reports "Hook 'deploy-hook' ran", the master stays at PID 585 and
the worker is replaced 28829 -> 37252, and the served certificate changes.
One trap worth naming: certbot prefixes the hook output with "ran with error
output" whenever the hook writes anything to stderr, and nginx's routine
types_hash warning goes to stderr. Everything inside is success — "test is
successful", "signal process started". A monitor that greps for "error"
would read a working hook as a failure. The worker PID is what to check.
Timing needed a clock correction. test-server has NTP off and runs 106
seconds fast; the dev machine matches Google and the Let's Encrypt ACME
endpoint exactly. Corrected, the hook's nginx -t lands at 12:27:50 UTC
against the new certificate's SCT at 12:27:49.054 — one second. The
correction validates itself: uncorrected, the hook would have run 104
seconds before the certificate existed.
That same skew is why D-4's gap was reported 106 seconds short; corrected to
2305s in the previous commit.
Remaining and left alone: whether certbot-renew.timer performs a real
renewal, which cannot be tested for about 89 days.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The 2199 seconds reported for D-4 subtracted a test-server timestamp
(archive/cert2.pem mtime) from a dev-machine timestamp (the serial change
observed by the poll), without noting they are different clocks.
Checked against external references: the dev machine matches Google and the
Let's Encrypt ACME endpoint to the second, while test-server is 105 seconds
fast and reports NTPSynchronized=no. Three round-trip measurements put the
offset at +106.1s every time.
Corrected:
new certificate written to disk 08:20:27 UTC (mtime 17:22:13 KST - 106s)
actually served 08:58:52 UTC (dev observation, no correction)
gap 2305s = 38m25s
The correction validates itself in D-4a, where the new certificate's SCT —
signed by CT logs on their own accurate clock at 12:27:49.054 GMT — lands
one second before the skew-corrected hook time. Without the correction the
hook would appear to have run 104 seconds before the certificate existed.
Updated across the experiment doc, the index, the follow-up doc, the verdict
evidence file and the SVG.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
B-7 stopped at "could not delete the server-side session". The reason it
gave was right: the ticket carries the session id, the ticket is encrypted
with the cookie secret, so after a rotation the proxy cannot work out which
Redis key to remove. But that is a limitation of the proxy, not of Redis.
Measured across two rotations:
- The orphan does expire. TTL falls one second per second and is not
refreshed by requests (the startup log says refresh:disabled, and
--cookie-refresh is unset), so it dies exactly one hour after creation.
- An operator can delete it. `redis-cli del` returned 1, dbsize went 2 to 1,
and the live session answered /oauth2/userinfo with 200 immediately after.
- But nothing in Redis says which key is the orphan. Same name prefix, same
type, the same 3510 bytes, and the values are encrypted.
- TTL is the only signal, and because it is never refreshed it is an exact
function of creation time. Anything created before the rotation is an
orphan. Derived creation time 11:30:26 against the AuthSuccess log line at
11:30:27 — one second out. The rule was then run and removed the orphan
while leaving the live session.
- They accumulate: the session that survived the first rotation became the
orphan of the second.
The caveat is recorded too: turning on --cookie-refresh breaks the
derivation, and at that point flushing and forcing everyone to
re-authenticate is the more honest option.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A-7 left the cause as a hypothesis and said to confirm it with the statement
logging A-3 used. Doing that shows the hypothesis was wrong.
Login issues no SQL at all in volatile mode. Refresh issues exactly one
statement, and it is not the one A-7 guessed:
select cscme1_0.SCOPE_ID from CLIENT_SCOPE_CLIENT cscme1_0
where cscme1_0.CLIENT_ID=$1 and cscme1_0.DEFAULT_SCOPE=$2
parameters: $1 = '131a9912-...', $2 = 'f'
REVOKED_TOKEN never appears. DEFAULT_SCOPE='f' means this is the optional
client scope list, which refresh needs because it recomputes which scopes go
into the new access token.
The larger finding is that A-7's table was conditional and did not say so.
The single statement is issued once and then cached, so the same
configuration produces three different outcomes depending only on cache
warmth — all three reproduced:
fully cold login 400 (select ce1_0.ID from CLIENT ...)
CLIENT warm login 200, refresh 500 <- what A-7 measured
fully warm login 200, refresh 200
So "volatile means you can log in without a database" is also conditional:
on a cold start the client lookup itself fails. In each case the Keycloak
log names the failing SQL directly, so this is determined rather than
inferred.
A-7 keeps its original wording with a correction banner above it, so what
was mis-guessed and why stays visible.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
D-2 forward upgrade and B-4 role propagation were filled earlier. D-4 forced
renewal was the third, and it needed a person at the keyboard because the
host asks for a sudo password.
It turned out to hold the largest finding of the three: the renewal
succeeded and went unserved for 36 minutes 39 seconds, because nothing in
the setup reloads nginx — no ExecStartPost, three empty hook directories,
no nginx plugin. And the half of the plan's question that was still an
expectation is now a measurement: the reload is graceful, 8856 connections
with zero failures and an in-flight request surviving intact.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The forced renewal did work. What did not work was anything making nginx
notice.
disk cert2.pem written 2026-09-04 17:22:13 KST
network old serial through 08:58:47 UTC, new serial from 08:58:52 UTC
gap 2199 seconds, and the old certificate was observed 428 times in it
The gap closed only because a person ran `nginx -s reload`. Nothing else
would have: certbot-renew.service is just `ExecStart=/usr/bin/certbot -q
renew` with no ExecStartPost, all three of renewal-hooks/{deploy,post,pre}
are empty, and certbot 5.7.0 here has no nginx plugin (dns-cloudflare,
manual, null, standalone, webroot). Three paths, all empty. Any one of them
would have been enough.
nginx holds the certificate in memory from startup, and certbot swaps the
live/ symlink rather than the path — so the config still looks right while
the served certificate is stale. Master 585 and worker 586 sharing a start
time 22.4 hours old is what proved no reload had happened.
This is invisible for 88 days. The timer ran twice today and exited
0/SUCCESS both times, because a renewal is not due yet. The day it becomes
due, the symptom is an expired certificate and the log still says SUCCESS.
The other half of the plan's question now has a measurement rather than an
expectation. The reload is genuinely graceful: 8856 polled connections, zero
non-200, p95 205.7ms before against 204.3ms after. And an in-flight request
— 845KB pulled at 20k/s, so still transferring 12 seconds in when the reload
landed — completed with all 845361 bytes over a single connection.
The earlier note that nginx reload "is graceful by design but must not be
written that way without checking" was right to hold back. The neighbouring
belief, that renewing gets the new certificate served, was the one that was
false, and there was no way to tell in advance which it would be.
Adds diagrams/d4-renewal-gap.svg and evidence 09-13.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
An independent audit found ten documents printing values their evidence files do not contain. C-1 printed a session count of 0 where the evidence says 4, C-2 printed a success readback for a command that exited 1, and A-1 credited the conntrack flush with a split that the timestamps attribute to a pod restart four seconds earlier.
Also measured wal_writer_delay, which A-3 had asserted as matching without ever querying it, relabelled the A-6 control that moved 41 percent, noted A-8's nine-sample resolution, corrected D-1's RTO to the 41 seconds its own timeline shows, and added a correction banner to D-2. Every experiment document now links its evidence files with their real collection times, and the duplicate screenshots are documented as duplicates.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Twelve SVG architecture diagrams cover the experiments whose documents had little or no structure drawing, embedded under a 구조 heading with a shared convention file. Seven documents carried their concepts under narrative headings and now have an explicit 개념 section so they can be found.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
One table per experiment with its branch and result, plus the nine injections that silently did nothing and the five predictions that turned out wrong.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
