Backups, the daily drift check and the k3s serving certificates have no alert. k3s renews certificates only when it starts within 120 days of expiry, and the two kube-prometheus-stack kubelet certificate alerts can never fire because the k3s kubelet exports no kubelet_certificate_manager_* metric. Add a PrometheusRule (platform-host-checks) for backup age, failed runs, missing metrics, drift left for 6h, a stale drift check, and serving certificate expiry (warning under 100 days, critical under 14). Add the two remaining host timers (certificate expiry via openssl on 6443 and 10250; the drift check against origin/main) and an installer that copies the scripts to a root-owned path, creates the textfile and work directories and enables all four timers. Disable the two dead kubelet certificate rules in kube-prometheus-stack. Risk: the metrics need the node-exporter textfile collector, a one-line change in W1's node-exporter values that was asked of W1. Until then PlatformHostBackupMetricsAbsent fires after a day, which is the point. Verification (desktop): - test-platform-host-checks-alerts.sh: promtool check and unit tests pass; with the k3s threshold changed from 2 to 5 days they fail. - render-observability-core.sh passes; the render has no Kubelet* certificate alerts and still has both KubeClientCertificateExpiration. - k3s-cert-expiry.sh as the desktop user reads both certificates. - install-host-checks.sh --check: systemd-analyze verify is clean.
89 lines
4.2 KiB
YAML
89 lines
4.2 KiB
YAML
apiVersion: monitoring.coreos.com/v1
|
|
kind: PrometheusRule
|
|
metadata:
|
|
name: platform-host-checks
|
|
namespace: observability
|
|
labels:
|
|
observability.hyeonworks.com/instance: home
|
|
app.kubernetes.io/part-of: platform
|
|
spec:
|
|
groups:
|
|
- name: platform-host-backups
|
|
rules:
|
|
- alert: PlatformK3sDatastoreBackupTooOld
|
|
expr: time() - max(platform_backup_last_success_timestamp_seconds{backup="k3s-datastore"}) > 2 * 86400
|
|
for: 1h
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: k3s datastore has not been backed up for more than 2 days
|
|
description: Check `journalctl -u platform-k3s-datastore-backup` on desktop. Daily timer, 03:30 KST.
|
|
- alert: PlatformGiteaBackupTooOld
|
|
expr: time() - max(platform_backup_last_success_timestamp_seconds{backup="gitea"}) > 8 * 86400
|
|
for: 1h
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: Gitea has not been backed up for more than 8 days
|
|
description: Check `journalctl -u platform-gitea-backup` on desktop. Weekly timer, Sunday 04:00 KST.
|
|
- alert: PlatformHostBackupFailed
|
|
expr: max by (backup) (platform_backup_last_run_success) == 0
|
|
for: 10m
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "The last {{ $labels.backup }} backup run failed"
|
|
description: "Check `journalctl -u platform-{{ $labels.backup }}-backup` on desktop."
|
|
- alert: PlatformHostBackupMetricsAbsent
|
|
expr: absent(platform_backup_last_run_timestamp_seconds{backup="k3s-datastore"}) or absent(platform_backup_last_run_timestamp_seconds{backup="gitea"})
|
|
for: 1d
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: Host backup metrics are missing
|
|
description: The backup timers or the node-exporter textfile collector (/var/lib/node_exporter/textfile_collector) are not running.
|
|
- name: platform-host-drift
|
|
rules:
|
|
- alert: PlatformLiveDriftDetected
|
|
expr: max(platform_live_drift_objects) > 0
|
|
for: 6h
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "{{ $value }} live objects differ from platform-core origin/main"
|
|
description: Run `scripts/validate/live-drift-check.sh --accepted scripts/validate/live-drift-accepted.txt` on desktop to list them.
|
|
- alert: PlatformLiveDriftCheckStale
|
|
expr: time() - max(platform_live_drift_last_run_timestamp_seconds) > 2 * 86400 or max(platform_live_drift_check_success) == 0
|
|
for: 1h
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: The daily live drift check has not completed
|
|
description: Check `journalctl -u platform-live-drift-check` on desktop.
|
|
- name: platform-k3s-certificates
|
|
rules:
|
|
- alert: PlatformK3sServingCertificateExpiry
|
|
expr: (min by (certificate) (platform_k3s_certificate_not_after_seconds) - time()) < 100 * 86400
|
|
for: 1h
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "k3s {{ $labels.certificate }} serving certificate expires in less than 100 days"
|
|
description: k3s renews certificates only when it starts within 120 days of expiry. Plan a k3s restart window (docs/runbooks).
|
|
- alert: PlatformK3sServingCertificateExpiry
|
|
expr: (min by (certificate) (platform_k3s_certificate_not_after_seconds) - time()) < 14 * 86400
|
|
for: 10m
|
|
labels:
|
|
severity: critical
|
|
annotations:
|
|
summary: "k3s {{ $labels.certificate }} serving certificate expires in less than 14 days"
|
|
description: Restart k3s (it renews certificates within 120 days of expiry) or run `k3s certificate rotate`.
|
|
- alert: PlatformK3sCertificateCheckStale
|
|
expr: time() - max(platform_k3s_certificate_check_timestamp_seconds) > 2 * 86400
|
|
for: 1h
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: The daily k3s certificate expiry check has not run
|
|
description: Check `journalctl -u platform-k3s-cert-expiry` on desktop.
|