Each of A-7a, B-7a and D-4a had the concepts, procedure, commands and
terminal evidence but no diagram, which the rest of the series carries.
a7a-cache-temperature.svg the three cache states and the SQL each issues
b7a-orphan-lifecycle.svg ticket structure, accumulation across two
rotations, and why TTL is the only signal
d4a-hook-timeline.svg the 38m25s gap against the 1-2s one, on one axis
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The 2199 seconds reported for D-4 subtracted a test-server timestamp
(archive/cert2.pem mtime) from a dev-machine timestamp (the serial change
observed by the poll), without noting they are different clocks.
Checked against external references: the dev machine matches Google and the
Let's Encrypt ACME endpoint to the second, while test-server is 105 seconds
fast and reports NTPSynchronized=no. Three round-trip measurements put the
offset at +106.1s every time.
Corrected:
new certificate written to disk 08:20:27 UTC (mtime 17:22:13 KST - 106s)
actually served 08:58:52 UTC (dev observation, no correction)
gap 2305s = 38m25s
The correction validates itself in D-4a, where the new certificate's SCT —
signed by CT logs on their own accurate clock at 12:27:49.054 GMT — lands
one second before the skew-corrected hook time. Without the correction the
hook would appear to have run 104 seconds before the certificate existed.
Updated across the experiment doc, the index, the follow-up doc, the verdict
evidence file and the SVG.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The forced renewal did work. What did not work was anything making nginx
notice.
disk cert2.pem written 2026-09-04 17:22:13 KST
network old serial through 08:58:47 UTC, new serial from 08:58:52 UTC
gap 2199 seconds, and the old certificate was observed 428 times in it
The gap closed only because a person ran `nginx -s reload`. Nothing else
would have: certbot-renew.service is just `ExecStart=/usr/bin/certbot -q
renew` with no ExecStartPost, all three of renewal-hooks/{deploy,post,pre}
are empty, and certbot 5.7.0 here has no nginx plugin (dns-cloudflare,
manual, null, standalone, webroot). Three paths, all empty. Any one of them
would have been enough.
nginx holds the certificate in memory from startup, and certbot swaps the
live/ symlink rather than the path — so the config still looks right while
the served certificate is stale. Master 585 and worker 586 sharing a start
time 22.4 hours old is what proved no reload had happened.
This is invisible for 88 days. The timer ran twice today and exited
0/SUCCESS both times, because a renewal is not due yet. The day it becomes
due, the symptom is an expired certificate and the log still says SUCCESS.
The other half of the plan's question now has a measurement rather than an
expectation. The reload is genuinely graceful: 8856 polled connections, zero
non-200, p95 205.7ms before against 204.3ms after. And an in-flight request
— 845KB pulled at 20k/s, so still transferring 12 seconds in when the reload
landed — completed with all 845361 bytes over a single connection.
The earlier note that nginx reload "is graceful by design but must not be
written that way without checking" was right to hold back. The neighbouring
belief, that renewing gets the new certificate served, was the one that was
false, and there was no way to tell in advance which it would be.
Adds diagrams/d4-renewal-gap.svg and evidence 09-13.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The forward upgrade to 26.7.3 was zero downtime across 87 samples, and since databasechangelog stayed at 210 the rollback to 26.7.0 also succeeded, which narrows D-2's conclusion: rolling back fails when the schema moved, not because of the version number. The row count is the check.
Role changes never reach the upstream through request repetition; the session is a snapshot taken at login and only a new session picks up the new claim. Auditing the docs also surfaced that Prometheus scrapes only keycloak, kubelet, node-exporter and itself, so the B-layer experiments have no metrics to screenshot rather than missing screenshots.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
