Commit Graph
6 Commits
Author SHA1 Message Date
DongHyeonka 7fe38428e3 feat(observability): alert on the root disk before kubelet acts
The kubelet starts deleting unused images at 85% root use and evicts
pods at 90%. The root disk warning fired at 85% and the critical at
90%, so both arrived when the kubelet was already acting. On
2026-11-09 the warning fired for 33 hours, then an eviction stopped
the tech-log pods, the image GC deleted their imagePullPolicy: Never
images, and production went down.

Move the warning to 80% for 10m and the critical to 85% for 5m, and
add PlatformRootFilesystemFillingFast (critical, for 15m): the 6h
linear trend of free bytes falls below 10% of the disk, the eviction
point, within 6 hours. On 2026-11-07 the disk lost 17G in 9 hours;
this forecast would have fired hours before the 80% warning. The
info alert at 75% stays.

Add scripts/validate/test-platform-disk-alerts.sh, promtool unit
tests for the usage bands and the forecast with the pinned Prometheus
image. The existing rules test needs live metric inventories, so this
one runs offline.

Risk: the live rules change only when the observability access apply
runs; this commit does not touch the cluster. The forecast can fire
on a large build that the BuildKit GC removes minutes later; the 15m
hold and the 6h window damp that.

Verification: the new test failed against the old rules (no alert at
81%, 86% or for a 1-per-minute fill) and passes after the change;
promtool check rules finds 9 rules. test-observability-alerting.sh
passes its static contract, then stops at the helm render because
helm v3.19.4 is not installed on the desktop.
2026-11-12 07:28:03 +09:00
donghyeon.kang 28595c47e2 Merge pull request 'feat(keycloak): pin images and attach PriorityClasses for the restart window' (#7) from donghyeon.kang/w6-restart-window-pins into main 2026-11-11 16:48:42 +00:00
DongHyeonka de600a0bb1 feat(priority): attach PriorityClasses to Keycloak, Gitea, Loki, Tempo
The PriorityClasses exist in the repository but no pod uses them, so
every platform pod still evicts at priority 0.

Attach platform-critical to Keycloak (CR spec.scheduling) and Gitea
(chart priorityClassName), and platform-low to Loki (singleBinary) and
Tempo. The Keycloak CR also gets spec.image at the running 26.7.0
digest with startOptimized: false: with spec.image the operator
otherwise assumes a pre-built optimized image, and this is the stock
one. Grafana and pgAdmin have no top-level priorityClassName in their
charts and are left for later; platform-postgres is W1's file and
tech-log belongs to its deploy/ directory.

Risk: applying restarts Keycloak, Gitea, Loki and Tempo once, so it
belongs in the k3s restart window, after the PriorityClasses exist (a
pod that names a missing class is rejected). The accepted-drift list
holds these objects until 2026-11-30.

Verification (desktop): render-phase1/2 and render-observability-core
pass with the four priorityClassName values; kubectl diff (server
dry-run) against live shows only these fields.
2026-11-12 01:25:24 +09:00
DongHyeonka 012452cf78 feat(observability): alert on host backups, drift and k3s serving certs
Backups, the daily drift check and the k3s serving certificates have no
alert. k3s renews certificates only when it starts within 120 days of
expiry, and the two kube-prometheus-stack kubelet certificate alerts
can never fire because the k3s kubelet exports no
kubelet_certificate_manager_* metric.

Add a PrometheusRule (platform-host-checks) for backup age, failed
runs, missing metrics, drift left for 6h, a stale drift check, and
serving certificate expiry (warning under 100 days, critical under 14).
Add the two remaining host timers (certificate expiry via openssl on
6443 and 10250; the drift check against origin/main) and an installer
that copies the scripts to a root-owned path, creates the textfile and
work directories and enables all four timers. Disable the two dead
kubelet certificate rules in kube-prometheus-stack.

Risk: the metrics need the node-exporter textfile collector, a one-line
change in W1's node-exporter values that was asked of W1. Until then
PlatformHostBackupMetricsAbsent fires after a day, which is the point.

Verification (desktop):
- test-platform-host-checks-alerts.sh: promtool check and unit tests
  pass; with the k3s threshold changed from 2 to 5 days they fail.
- render-observability-core.sh passes; the render has no Kubelet*
  certificate alerts and still has both KubeClientCertificateExpiration.
- k3s-cert-expiry.sh as the desktop user reads both certificates.
- install-host-checks.sh --check: systemd-analyze verify is clean.
2026-11-12 01:13:02 +09:00
DongHyeonka af5a90bef8 fix(observability): let Prometheus accept samples after a clock jump
On 2026-11-11 the desktop booted with the RTC read as UTC while it held
KST, so the clock ran 9 hours ahead until NTP pulled it back. Prometheus
had written one hour of samples stamped 18:36-19:36Z into its TSDB head
during that window. With outOfOrderTimeWindow at the chart default of
0s, every later scrape was older than the head's valid range and was
dropped as "out of bounds". From 11:21Z no metric was stored, no rule
could fire on real data, and KubeAPIDown (an absent() rule) fired
falsely. It would have stayed blind until the wall clock passed the
bogus head time, about 7 hours later.

Set prometheusSpec.tsdb.outOfOrderTimeWindow to 10h. Samples up to 10
hours behind the head are now accepted, which covers the +9h KST-as-UTC
misread. The README records why.

Rejected: deleting the head WAL from the Retain PV. It works at once
but removes persisted data and needs the pod stopped. Rejected: waiting
for the clock to catch up. It leaves monitoring blind for hours during
the period when the cause of the incident is still being found.

Risk: out-of-order samples use an extra in-memory head and vertical
compaction only while they arrive; in normal operation none arrive. A
genuinely late exporter could now backfill up to 10h, which is harmless
on this single-node Prometheus. Preventing the clock jump itself is the
host change in infrastructure/host, not this commit.

Verification: the values file parses and holds
prometheus.prometheusSpec.tsdb.outOfOrderTimeWindow=10h. The live
Prometheus CR already carried tsdb.outOfOrderTimeWindow=0s from this
chart key, so the key maps to spec.tsdb. The same value was patched
onto the live CR: the operator wrote out_of_order_time_window: 10h, the
config reloaded at 13:04:37Z, "too old" warnings dropped to 0, all 34
targets report up, and KubeAPIDown resolved. The pinned renderer
(scripts/validate/render-observability-core.sh) was not run: it requires
Helm v3.19.4 and kustomize v5.8.1, which are not installed on the
laptop or the desktop.
2026-11-11 22:09:23 +09:00
donghyeon-ka 16c337bcc9 Add platform infrastructure configuration 2026-08-28 17:35:41 +09:00