D-2 forward upgrade and B-4 role propagation were filled earlier. D-4 forced
renewal was the third, and it needed a person at the keyboard because the
host asks for a sudo password.
It turned out to hold the largest finding of the three: the renewal
succeeded and went unserved for 36 minutes 39 seconds, because nothing in
the setup reloads nginx — no ExecStartPost, three empty hook directories,
no nginx plugin. And the half of the plan's question that was still an
expectation is now a measurement: the reload is graceful, 8856 connections
with zero failures and an in-flight request surviving intact.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The forced renewal did work. What did not work was anything making nginx
notice.
disk cert2.pem written 2026-09-04 17:22:13 KST
network old serial through 08:58:47 UTC, new serial from 08:58:52 UTC
gap 2199 seconds, and the old certificate was observed 428 times in it
The gap closed only because a person ran `nginx -s reload`. Nothing else
would have: certbot-renew.service is just `ExecStart=/usr/bin/certbot -q
renew` with no ExecStartPost, all three of renewal-hooks/{deploy,post,pre}
are empty, and certbot 5.7.0 here has no nginx plugin (dns-cloudflare,
manual, null, standalone, webroot). Three paths, all empty. Any one of them
would have been enough.
nginx holds the certificate in memory from startup, and certbot swaps the
live/ symlink rather than the path — so the config still looks right while
the served certificate is stale. Master 585 and worker 586 sharing a start
time 22.4 hours old is what proved no reload had happened.
This is invisible for 88 days. The timer ran twice today and exited
0/SUCCESS both times, because a renewal is not due yet. The day it becomes
due, the symptom is an expired certificate and the log still says SUCCESS.
The other half of the plan's question now has a measurement rather than an
expectation. The reload is genuinely graceful: 8856 polled connections, zero
non-200, p95 205.7ms before against 204.3ms after. And an in-flight request
— 845KB pulled at 20k/s, so still transferring 12 seconds in when the reload
landed — completed with all 845361 bytes over a single connection.
The earlier note that nginx reload "is graceful by design but must not be
written that way without checking" was right to hold back. The neighbouring
belief, that renewing gets the new certificate served, was the one that was
false, and there was no way to tell in advance which it would be.
Adds diagrams/d4-renewal-gap.svg and evidence 09-13.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
An independent audit found ten documents printing values their evidence files do not contain. C-1 printed a session count of 0 where the evidence says 4, C-2 printed a success readback for a command that exited 1, and A-1 credited the conntrack flush with a split that the timestamps attribute to a pod restart four seconds earlier.
Also measured wal_writer_delay, which A-3 had asserted as matching without ever querying it, relabelled the A-6 control that moved 41 percent, noted A-8's nine-sample resolution, corrected D-1's RTO to the 41 seconds its own timeline shows, and added a correction banner to D-2. Every experiment document now links its evidence files with their real collection times, and the duplicate screenshots are documented as duplicates.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Twelve SVG architecture diagrams cover the experiments whose documents had little or no structure drawing, embedded under a 구조 heading with a shared convention file. Seven documents carried their concepts under narrative headings and now have an explicit 개념 section so they can be found.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
One table per experiment with its branch and result, plus the nine injections that silently did nothing and the five predictions that turned out wrong.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
