Files
platform-core/services/observability
DongHyeonka 403405973f fix(observability): stop throttling node-exporter
node-exporter averages about 2m CPU, but each scrape runs its
collectors in parallel. With a 250m limit (25ms per 100ms CFS period)
26% of its CFS periods were throttled over 30 minutes on 2026-11-11
(container_cpu_cfs_throttled_periods_total / cfs_periods_total), which
delays scrapes and can make them time out.

Raise the CPU limit to one core. The request stays 100m, so
scheduling does not change, and a runaway collector is still capped.
Removing the limit was rejected because every other workload in this
repository sets one.

Risk: none beyond one core of headroom on a 12-core node. The live
DaemonSet changes only when the observability core is applied again.

Verification: the throttling ratio and usage came from live
Prometheus queries. helm template of prometheus-node-exporter 4.56.1
with these values renders limits.cpu "1". The effect after rollout is
not measured yet.
2026-11-11 23:52:13 +09:00
..

Observability core

This root composes the internal observability workloads deployed after the namespace, CRD, and Local PV prerequisites:

  • kube-prometheus-stack without Grafana or its bundled node-exporter;
  • single-binary Loki and Tempo using isolated AIStor buckets;
  • Alloy for Pod logs and internal OTLP/gRPC traces;
  • a standalone node-exporter;
  • the exact core NetworkPolicy matrix, platform rules, scrape targets, verified dashboards, and source-only Alertmanager Slack routing.

It intentionally does not create public ingress, Grafana, OIDC, or credential Secrets. The Slack routing source references alertmanager-slack-webhook/url; while that Secret is absent, do not apply the AlertmanagerConfig, its kube-prometheus-stack reference, its public HTTPS NetworkPolicy, or alert rules. Namespace, CRD, and Local PV roots remain outside this aggregate because their protected apply order is different.