Backups, the daily drift check and the k3s serving certificates have no alert. k3s renews certificates only when it starts within 120 days of expiry, and the two kube-prometheus-stack kubelet certificate alerts can never fire because the k3s kubelet exports no kubelet_certificate_manager_* metric. Add a PrometheusRule (platform-host-checks) for backup age, failed runs, missing metrics, drift left for 6h, a stale drift check, and serving certificate expiry (warning under 100 days, critical under 14). Add the two remaining host timers (certificate expiry via openssl on 6443 and 10250; the drift check against origin/main) and an installer that copies the scripts to a root-owned path, creates the textfile and work directories and enables all four timers. Disable the two dead kubelet certificate rules in kube-prometheus-stack. Risk: the metrics need the node-exporter textfile collector, a one-line change in W1's node-exporter values that was asked of W1. Until then PlatformHostBackupMetricsAbsent fires after a day, which is the point. Verification (desktop): - test-platform-host-checks-alerts.sh: promtool check and unit tests pass; with the k3s threshold changed from 2 to 5 days they fail. - render-observability-core.sh passes; the render has no Kubelet* certificate alerts and still has both KubeClientCertificateExpiration. - k3s-cert-expiry.sh as the desktop user reads both certificates. - install-host-checks.sh --check: systemd-analyze verify is clean.
26 lines
1.4 KiB
Bash
26 lines
1.4 KiB
Bash
#!/usr/bin/env bash
|
|
# Write the notAfter time of the k3s API server (6443) and kubelet (10250)
|
|
# serving certificates as Prometheus textfile metrics. No root needed.
|
|
# k3s renews these only when it starts and they are within 120 days of expiry
|
|
# (docs.k3s.io cli/certificate), and the kubelet exposes no certificate metric,
|
|
# so nothing else in the cluster watches them.
|
|
set -Eeuo pipefail
|
|
# shellcheck source=../lib/host-backup.sh
|
|
source "$(dirname -- "${BASH_SOURCE[0]}")/../lib/host-backup.sh"
|
|
|
|
lines=""
|
|
for target in apiserver:6443 kubelet:10250; do
|
|
name="${target%%:*}" port="${target##*:}"
|
|
end="$(echo | timeout 10 openssl s_client -connect "127.0.0.1:$port" 2>/dev/null | openssl x509 -noout -enddate 2>/dev/null | sed 's/^notAfter=//')" || true
|
|
[[ -n "$end" ]] || hb_fail "could not read the $name certificate on 127.0.0.1:$port"
|
|
lines+="platform_k3s_certificate_not_after_seconds{certificate=\"$name\"} $(date -d "$end" +%s)"$'\n'
|
|
hb_log "$name certificate notAfter $end"
|
|
done
|
|
hb_write_metrics platform_k3s_certificates <<PROM
|
|
# HELP platform_k3s_certificate_not_after_seconds notAfter of a k3s serving certificate (Unix time).
|
|
# TYPE platform_k3s_certificate_not_after_seconds gauge
|
|
${lines}# HELP platform_k3s_certificate_check_timestamp_seconds Unix time of the last certificate check.
|
|
# TYPE platform_k3s_certificate_check_timestamp_seconds gauge
|
|
platform_k3s_certificate_check_timestamp_seconds $(date +%s)
|
|
PROM
|