3 Commits
Author SHA1 Message Date
DongHyeonka d92dc5d163 feat(observability): read node-exporter textfile metrics
The host timers from the k3s maturity work (last Gitea dump and k3s
datastore backup, live drift count, API server and kubelet certificate
expiry) write their results as *.prom files. node-exporter does not
read any textfile directory, so Prometheus never sees them.

Pass --collector.textfile.directory pointing at
/var/lib/node_exporter/textfile_collector under the read-only host
root mount the chart already has. No volume is added. The host
installer creates the directory; until then the collector only
reports an error metric.

Verification: helm template of prometheus-node-exporter 4.56.1 with
these values renders the argument next to --path.rootfs=/host/root.
2026-11-12 07:29:48 +09:00
DongHyeonka bed20b456b fix(observability): stop throttling node-exporter
node-exporter averages about 2m CPU, but each scrape runs its
collectors in parallel. With a 250m limit (25ms per 100ms CFS period)
26% of its CFS periods were throttled over 30 minutes on 2026-11-11
(container_cpu_cfs_throttled_periods_total / cfs_periods_total), which
delays scrapes and can make them time out.

Raise the CPU limit to one core. The request stays 100m, so
scheduling does not change, and a runaway collector is still capped.
Removing the limit was rejected because every other workload in this
repository sets one.

Risk: none beyond one core of headroom on a 12-core node. The live
DaemonSet changes only when the observability core is applied again.

Verification: the throttling ratio and usage came from live
Prometheus queries. helm template of prometheus-node-exporter 4.56.1
with these values renders limits.cpu "1". The effect after rollout is
not measured yet.
2026-11-12 07:28:03 +09:00
donghyeon-ka 16c337bcc9 Add platform infrastructure configuration 2026-08-28 17:35:41 +09:00