The forced renewal did not change the certificate being served — 161 serial
samples over 13 minutes, all identical, notBefore still Sep 3.
Two things were readable without sudo and both matter.
nginx has never been reloaded: master 585 and worker 586 share a start time
and both are 80529 seconds old. A reload keeps the master and replaces the
workers, so the original fork still being there means no reload has happened
since Sep 3 19:00. nginx reads the certificate at startup and holds it in
memory; without a reload it serves the old one no matter what is on disk.
And the unit that is supposed to renew has nothing that would reload it:
[Service]
Type=oneshot
ExecStart=/usr/bin/certbot -q renew
PrivateTmp=true
No ExecStartPost, no --deploy-hook. The timer runs twice a day and exited
0/SUCCESS both times today, which is precisely why this stays invisible —
the renewal is not due for 88 days, so nothing has failed yet.
What remains undetermined is whether the user's --force-renewal failed, or
succeeded and left new files that nginx never picked up. Telling those apart
needs /etc/letsencrypt, which is root-only. If it is the latter, that is the
most valuable thing this lab has produced: "renewal succeeded" and "the new
certificate is being served" are different events.
CT logs looked like a way around the permission problem and were not: the
served certificate carries two valid SCTs, yet crt.sh returns zero rows for
auth.hyeonworks.com. Same shape as A-2's `up` metric — the observing tool
sees a subset of the truth.
Also recorded: 76 in-flight failures at 08:15:04 are a local artifact, not a
server event. The concurrent 0.2s poll was clean across the same second,
num_connects was 0, and time_total was 50 microseconds. Not reproducible in
100 retries. The monitor now records curl's exit code so a recurrence is
diagnosable.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
