From 403405973f0e344ce4de7523e3a8677b77d33723 Mon Sep 17 00:00:00 2001 From: DongHyeonka Date: Wed, 11 Nov 2026 23:52:13 +0900 Subject: [PATCH] fix(observability): stop throttling node-exporter node-exporter averages about 2m CPU, but each scrape runs its collectors in parallel. With a 250m limit (25ms per 100ms CFS period) 26% of its CFS periods were throttled over 30 minutes on 2026-11-11 (container_cpu_cfs_throttled_periods_total / cfs_periods_total), which delays scrapes and can make them time out. Raise the CPU limit to one core. The request stays 100m, so scheduling does not change, and a runaway collector is still capped. Removing the limit was rejected because every other workload in this repository sets one. Risk: none beyond one core of headroom on a 12-core node. The live DaemonSet changes only when the observability core is applied again. Verification: the throttling ratio and usage came from live Prometheus queries. helm template of prometheus-node-exporter 4.56.1 with these values renders limits.cpu "1". The effect after rollout is not measured yet. --- services/observability/node-exporter/values/home.yaml | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/services/observability/node-exporter/values/home.yaml b/services/observability/node-exporter/values/home.yaml index d994b14..c6df077 100644 --- a/services/observability/node-exporter/values/home.yaml +++ b/services/observability/node-exporter/values/home.yaml @@ -32,7 +32,11 @@ resources: cpu: 100m memory: 128Mi limits: - cpu: 250m + # Average use is about 2m, but each scrape runs the collectors in + # parallel. At 250m (25ms per 100ms CFS period) 26% of periods were + # throttled on 2026-11-11. One core removes the throttling and still caps + # a runaway collector. + cpu: "1" memory: 256Mi serviceAccount: