Backups, the daily drift check and the k3s serving certificates have no alert. k3s renews certificates only when it starts within 120 days of expiry, and the two kube-prometheus-stack kubelet certificate alerts can never fire because the k3s kubelet exports no kubelet_certificate_manager_* metric. Add a PrometheusRule (platform-host-checks) for backup age, failed runs, missing metrics, drift left for 6h, a stale drift check, and serving certificate expiry (warning under 100 days, critical under 14). Add the two remaining host timers (certificate expiry via openssl on 6443 and 10250; the drift check against origin/main) and an installer that copies the scripts to a root-owned path, creates the textfile and work directories and enables all four timers. Disable the two dead kubelet certificate rules in kube-prometheus-stack. Risk: the metrics need the node-exporter textfile collector, a one-line change in W1's node-exporter values that was asked of W1. Until then PlatformHostBackupMetricsAbsent fires after a day, which is the point. Verification (desktop): - test-platform-host-checks-alerts.sh: promtool check and unit tests pass; with the k3s threshold changed from 2 to 5 days they fail. - render-observability-core.sh passes; the render has no Kubelet* certificate alerts and still has both KubeClientCertificateExpiration. - k3s-cert-expiry.sh as the desktop user reads both certificates. - install-host-checks.sh --check: systemd-analyze verify is clean.
17 lines
874 B
Bash
17 lines
874 B
Bash
#!/usr/bin/env bash
|
|
# Daily drift check against origin/main (not the working tree of the checkout),
|
|
# so local edits or a stale checkout do not hide or invent drift.
|
|
# Run by platform-live-drift-check.timer as the desktop user.
|
|
set -Eeuo pipefail
|
|
readonly CHECKOUT="${PLATFORM_CHECKOUT:-/home/donghyeon/workspace/platform}"
|
|
readonly TEXTFILE=/var/lib/node_exporter/textfile_collector/platform_live_drift.prom
|
|
|
|
git -C "$CHECKOUT" fetch --quiet origin main
|
|
work="$(mktemp -d /tmp/platform-drift-main.XXXXXX)"
|
|
trap 'rm -rf -- "$work"' EXIT
|
|
git -C "$CHECKOUT" archive origin/main | tar -x -C "$work"
|
|
printf 'origin/main %s\n' "$(git -C "$CHECKOUT" rev-parse --short origin/main)"
|
|
args=(--accepted "$work/scripts/validate/live-drift-accepted.txt")
|
|
[[ -d "$(dirname "$TEXTFILE")" ]] && args+=(--textfile "$TEXTFILE")
|
|
bash "$work/scripts/validate/live-drift-check.sh" "${args[@]}"
|