Backups, the daily drift check and the k3s serving certificates have no alert. k3s renews certificates only when it starts within 120 days of expiry, and the two kube-prometheus-stack kubelet certificate alerts can never fire because the k3s kubelet exports no kubelet_certificate_manager_* metric. Add a PrometheusRule (platform-host-checks) for backup age, failed runs, missing metrics, drift left for 6h, a stale drift check, and serving certificate expiry (warning under 100 days, critical under 14). Add the two remaining host timers (certificate expiry via openssl on 6443 and 10250; the drift check against origin/main) and an installer that copies the scripts to a root-owned path, creates the textfile and work directories and enables all four timers. Disable the two dead kubelet certificate rules in kube-prometheus-stack. Risk: the metrics need the node-exporter textfile collector, a one-line change in W1's node-exporter values that was asked of W1. Until then PlatformHostBackupMetricsAbsent fires after a day, which is the point. Verification (desktop): - test-platform-host-checks-alerts.sh: promtool check and unit tests pass; with the k3s threshold changed from 2 to 5 days they fail. - render-observability-core.sh passes; the render has no Kubelet* certificate alerts and still has both KubeClientCertificateExpiration. - k3s-cert-expiry.sh as the desktop user reads both certificates. - install-host-checks.sh --check: systemd-analyze verify is clean.
17 lines
596 B
Desktop File
17 lines
596 B
Desktop File
[Unit]
|
|
Description=Compare the cluster with platform-core origin/main (kubectl diff, server dry-run)
|
|
After=k3s.service network-online.target
|
|
Wants=network-online.target
|
|
|
|
[Service]
|
|
Type=oneshot
|
|
User=donghyeon
|
|
Environment=PLATFORM_HELM_BIN=/home/donghyeon/.local/bin/helm
|
|
Environment=PLATFORM_CHECKOUT=/home/donghyeon/workspace/platform
|
|
ExecStart=/bin/bash /usr/local/lib/platform-backup/validate/live-drift-check-main.sh
|
|
# Exit 1 means "drift found"; the metric and its alert report that. Only 2 (the
|
|
# check itself failed) marks the unit failed.
|
|
SuccessExitStatus=1
|
|
Nice=10
|
|
TimeoutStartSec=30min
|