fix(observability): let Prometheus accept samples after a clock jump

On 2026-11-11 the desktop booted with the RTC read as UTC while it held
KST, so the clock ran 9 hours ahead until NTP pulled it back. Prometheus
had written one hour of samples stamped 18:36-19:36Z into its TSDB head
during that window. With outOfOrderTimeWindow at the chart default of
0s, every later scrape was older than the head's valid range and was
dropped as "out of bounds". From 11:21Z no metric was stored, no rule
could fire on real data, and KubeAPIDown (an absent() rule) fired
falsely. It would have stayed blind until the wall clock passed the
bogus head time, about 7 hours later.

Set prometheusSpec.tsdb.outOfOrderTimeWindow to 10h. Samples up to 10
hours behind the head are now accepted, which covers the +9h KST-as-UTC
misread. The README records why.

Rejected: deleting the head WAL from the Retain PV. It works at once
but removes persisted data and needs the pod stopped. Rejected: waiting
for the clock to catch up. It leaves monitoring blind for hours during
the period when the cause of the incident is still being found.

Risk: out-of-order samples use an extra in-memory head and vertical
compaction only while they arrive; in normal operation none arrive. A
genuinely late exporter could now backfill up to 10h, which is harmless
on this single-node Prometheus. Preventing the clock jump itself is the
host change in infrastructure/host, not this commit.

Verification: the values file parses and holds
prometheus.prometheusSpec.tsdb.outOfOrderTimeWindow=10h. The live
Prometheus CR already carried tsdb.outOfOrderTimeWindow=0s from this
chart key, so the key maps to spec.tsdb. The same value was patched
onto the live CR: the operator wrote out_of_order_time_window: 10h, the
config reloaded at 13:04:37Z, "too old" warnings dropped to 0, all 34
targets report up, and KubeAPIDown resolved. The pinned renderer
(scripts/validate/render-observability-core.sh) was not run: it requires
Helm v3.19.4 and kustomize v5.8.1, which are not installed on the
laptop or the desktop.
This commit is contained in:
DongHyeonka
2026-11-11 22:09:23 +09:00
parent 89e81309b5
commit af5a90bef8
2 changed files with 9 additions and 0 deletions
@@ -13,3 +13,10 @@ Alertmanager source는 selector discovery 대신 global
`alertmanagerConfiguration.name=platform-alertmanager`를 참조한다. 참조 대상의
Slack Secret이 준비되기 전에는 이 변경을 live에 적용하지 않고 기존 null 구성을
유지한다.
Prometheus TSDB는 `outOfOrderTimeWindow: 10h`로 늦게 도착한 샘플을 받는다. 시계가
앞으로 튀었다가 되돌아오면 TSDB 머리에 미래 시각 샘플이 남고, 창이 0이면 그 시각이
지날 때까지 모든 수집이 `out of bounds`로 버려진다. 메트릭과 경보가 함께 멈추고,
`absent()` 경보(`KubeAPIDown` 등)는 거짓으로 울린다. 10시간은 재부팅 때 RTC의 KST를
UTC로 읽는 +9시간 오차를 덮는 값이다(2026-11-11 사고). 늦은 샘플이 없으면 추가 메모리는
거의 들지 않는다. 시계 자체를 막는 설정은 `infrastructure/host`가 맡는다.
@@ -219,6 +219,8 @@ prometheus:
scrapeInterval: 30s
evaluationInterval: 30s
walCompression: true
tsdb:
outOfOrderTimeWindow: 10h
automountServiceAccountToken: true
image:
registry: quay.io