The forced renewal did not change the certificate being served — 161 serial
samples over 13 minutes, all identical, notBefore still Sep 3.
Two things were readable without sudo and both matter.
nginx has never been reloaded: master 585 and worker 586 share a start time
and both are 80529 seconds old. A reload keeps the master and replaces the
workers, so the original fork still being there means no reload has happened
since Sep 3 19:00. nginx reads the certificate at startup and holds it in
memory; without a reload it serves the old one no matter what is on disk.
And the unit that is supposed to renew has nothing that would reload it:
[Service]
Type=oneshot
ExecStart=/usr/bin/certbot -q renew
PrivateTmp=true
No ExecStartPost, no --deploy-hook. The timer runs twice a day and exited
0/SUCCESS both times today, which is precisely why this stays invisible —
the renewal is not due for 88 days, so nothing has failed yet.
What remains undetermined is whether the user's --force-renewal failed, or
succeeded and left new files that nginx never picked up. Telling those apart
needs /etc/letsencrypt, which is root-only. If it is the latter, that is the
most valuable thing this lab has produced: "renewal succeeded" and "the new
certificate is being served" are different events.
CT logs looked like a way around the permission problem and were not: the
served certificate carries two valid SCTs, yet crt.sh returns zero rows for
auth.hyeonworks.com. Same shape as A-2's `up` metric — the observing tool
sees a subset of the truth.
Also recorded: 76 in-flight failures at 08:15:04 are a local artifact, not a
server event. The concurrent 0.2s poll was clean across the same second,
num_connects was 0, and time_total was 50 microseconds. Not reproducible in
100 retries. The monitor now records curl's exit code so a recurrence is
diagnosable.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The forced renewal needs sudo on the host, and the host asks for a password.
That blocks the injection, not the experiment — the part worth doing first
was the control anyway.
Control 1 — new connections, 0.2s x 900 over 180s: 900/900 = 200, zero
failures. The noise floor is 0, so a single non-200 during the renewal can
be attributed to the renewal. Without this the observation would have been
uninterpretable, which is the mistake A-6 made calling a -41% control
"no effect" and A-8 made claiming zero-downtime from 9 samples.
Control 2 — the poll cannot answer the question the plan actually asked.
TLS handshakes were 900/900, meaning every request is a fresh connection,
so it measures "are new connections accepted", not "what happens to a
request already in flight". A separate device: the 845KB admin console
bundle pulled at --limit-rate 20k, holding one request open for 42 seconds.
Baseline 200 / 845361 bytes / 1 connection.
Monitors are running under setsid, and stop three minutes after the
certificate serial changes.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three SAN entries and no wildcard is the constraint that cost something real in B-7, where oauth2-proxy had to borrow Grafana's app2 hostname because a fourth name was not available. The served chain is four deep and verifies, so fullchain.pem is configured rather than the cert.pem mistake that only breaks clients without a cached intermediate.
The forced renewal and the reload behaviour could not be measured because sudo on the host asks for a password, the same silent failure first noticed in B-7. nginx reload is graceful by design, but this lab has repeatedly shown that by design is not the same as measured, so it is recorded as untested rather than assumed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
