602 lines
14 KiB
Markdown
602 lines
14 KiB
Markdown
# observability / health 예시
|
||
|
||
---
|
||
|
||
## 좋은 예시 1: ServiceMonitor (kube-prometheus-stack 표준)
|
||
|
||
```yaml
|
||
---
|
||
apiVersion: monitoring.coreos.com/v1
|
||
kind: ServiceMonitor
|
||
metadata:
|
||
name: auth-server
|
||
namespace: auth-prod
|
||
labels:
|
||
app.kubernetes.io/name: auth-server
|
||
app.kubernetes.io/instance: auth-server-prod
|
||
app.kubernetes.io/part-of: identity-platform
|
||
release: kube-prometheus-stack
|
||
spec:
|
||
namespaceSelector:
|
||
matchNames:
|
||
- auth-prod
|
||
selector:
|
||
matchLabels:
|
||
app.kubernetes.io/name: auth-server
|
||
app.kubernetes.io/instance: auth-server-prod
|
||
endpoints:
|
||
- port: metrics # named port (required)
|
||
path: /actuator/prometheus
|
||
scheme: http
|
||
interval: 30s
|
||
scrapeTimeout: 10s
|
||
honorLabels: false
|
||
relabelings:
|
||
- sourceLabels: [__meta_kubernetes_pod_name]
|
||
targetLabel: pod
|
||
- sourceLabels: [__meta_kubernetes_namespace]
|
||
targetLabel: namespace
|
||
- sourceLabels: [__meta_kubernetes_pod_label_app_kubernetes_io_version]
|
||
targetLabel: version
|
||
- action: labeldrop
|
||
regex: "pod_template_hash|controller_revision_hash"
|
||
metricRelabelings:
|
||
- sourceLabels: [__name__]
|
||
regex: "jvm_gc_pause_seconds_.*"
|
||
action: keep
|
||
- sourceLabels: [__name__]
|
||
regex: "debug_.*"
|
||
action: drop
|
||
```
|
||
|
||
**왜 좋은가:**
|
||
|
||
- `namespaceSelector` 명시로 암묵적 전체 허용 방지.
|
||
- `port: metrics` 는 Service/Deployment의 named port를 참조 → 포트 번호 변경에 내성.
|
||
- `interval / scrapeTimeout` 관계 유지 (timeout < interval).
|
||
- `relabelings` 로 pod / namespace / version label 정리, noise label drop.
|
||
- `metricRelabelings` 로 불필요 metric drop (cardinality / storage 절감).
|
||
- `release: kube-prometheus-stack` label 로 Operator가 선택.
|
||
|
||
---
|
||
|
||
## 좋은 예시 2: ServiceMonitor with bearer token (Vault telemetry)
|
||
|
||
```yaml
|
||
---
|
||
apiVersion: v1
|
||
kind: Secret
|
||
metadata:
|
||
name: vault-metrics-token
|
||
namespace: vault
|
||
type: Opaque
|
||
stringData:
|
||
token: "hvs.xxxx.prometheus-readonly"
|
||
---
|
||
apiVersion: monitoring.coreos.com/v1
|
||
kind: ServiceMonitor
|
||
metadata:
|
||
name: vault
|
||
namespace: vault
|
||
labels:
|
||
app.kubernetes.io/name: vault
|
||
app.kubernetes.io/instance: vault-prod
|
||
release: kube-prometheus-stack
|
||
spec:
|
||
namespaceSelector:
|
||
matchNames:
|
||
- vault
|
||
selector:
|
||
matchLabels:
|
||
app.kubernetes.io/name: vault
|
||
app.kubernetes.io/instance: vault-prod
|
||
endpoints:
|
||
- port: https
|
||
path: /v1/sys/metrics
|
||
params:
|
||
format: ["prometheus"]
|
||
scheme: https
|
||
interval: 30s
|
||
scrapeTimeout: 10s
|
||
bearerTokenSecret:
|
||
name: vault-metrics-token
|
||
key: token
|
||
tlsConfig:
|
||
insecureSkipVerify: false
|
||
ca:
|
||
secret:
|
||
name: vault-ca
|
||
key: ca.crt
|
||
serverName: vault.vault.svc
|
||
relabelings:
|
||
- sourceLabels: [__meta_kubernetes_pod_name]
|
||
targetLabel: pod
|
||
```
|
||
|
||
**왜 좋은가:**
|
||
|
||
- Vault `/sys/metrics` 는 read token 필수. `bearerTokenSecret` 참조로 Operator가 주입.
|
||
- TLS CA pinning + serverName 으로 MitM 방지.
|
||
- `params` 로 Prometheus format 요청.
|
||
|
||
---
|
||
|
||
## 좋은 예시 3: PodMonitor (Service 없는 워크로드)
|
||
|
||
```yaml
|
||
---
|
||
apiVersion: monitoring.coreos.com/v1
|
||
kind: PodMonitor
|
||
metadata:
|
||
name: batch-worker
|
||
namespace: batch
|
||
labels:
|
||
release: kube-prometheus-stack
|
||
spec:
|
||
namespaceSelector:
|
||
matchNames:
|
||
- batch
|
||
selector:
|
||
matchLabels:
|
||
app.kubernetes.io/name: batch-worker
|
||
podMetricsEndpoints:
|
||
- port: metrics
|
||
path: /metrics
|
||
interval: 30s
|
||
scrapeTimeout: 10s
|
||
relabelings:
|
||
- sourceLabels: [__meta_kubernetes_pod_name]
|
||
targetLabel: pod
|
||
```
|
||
|
||
**왜 좋은가:**
|
||
|
||
- Job / headless workload처럼 Service 뒤에 없는 경우 PodMonitor로 직접 pod 매칭.
|
||
|
||
---
|
||
|
||
## 좋은 예시 4: Annotation-based fallback (Operator 없는 환경 only)
|
||
|
||
```yaml
|
||
---
|
||
apiVersion: v1
|
||
kind: Service
|
||
metadata:
|
||
name: legacy-app
|
||
namespace: legacy
|
||
annotations:
|
||
prometheus.io/scrape: "true"
|
||
prometheus.io/port: "8081"
|
||
prometheus.io/path: "/metrics"
|
||
prometheus.io/scheme: "http"
|
||
spec:
|
||
selector:
|
||
app.kubernetes.io/name: legacy-app
|
||
ports:
|
||
- name: http
|
||
port: 80
|
||
targetPort: 8080
|
||
- name: metrics
|
||
port: 8081
|
||
targetPort: 8081
|
||
```
|
||
|
||
**왜 좋은가 (조건부):**
|
||
|
||
- kube-prometheus-stack이 없는 legacy 환경에서만 유효.
|
||
- Operator가 있으면 ServiceMonitor로 전환.
|
||
|
||
---
|
||
|
||
## 좋은 예시 5: NetworkPolicy — prometheus namespace만 metrics scrape 허용
|
||
|
||
```yaml
|
||
---
|
||
apiVersion: networking.k8s.io/v1
|
||
kind: NetworkPolicy
|
||
metadata:
|
||
name: auth-server-default-deny
|
||
namespace: auth-prod
|
||
spec:
|
||
podSelector:
|
||
matchLabels:
|
||
app.kubernetes.io/name: auth-server
|
||
policyTypes: ["Ingress", "Egress"]
|
||
ingress: []
|
||
egress: []
|
||
---
|
||
apiVersion: networking.k8s.io/v1
|
||
kind: NetworkPolicy
|
||
metadata:
|
||
name: auth-server-allow-metrics
|
||
namespace: auth-prod
|
||
spec:
|
||
podSelector:
|
||
matchLabels:
|
||
app.kubernetes.io/name: auth-server
|
||
policyTypes: ["Ingress"]
|
||
ingress:
|
||
- from:
|
||
- namespaceSelector:
|
||
matchLabels:
|
||
kubernetes.io/metadata.name: monitoring
|
||
podSelector:
|
||
matchLabels:
|
||
app.kubernetes.io/name: prometheus
|
||
ports:
|
||
- port: metrics
|
||
protocol: TCP
|
||
---
|
||
apiVersion: networking.k8s.io/v1
|
||
kind: NetworkPolicy
|
||
metadata:
|
||
name: auth-server-allow-http-from-ingress
|
||
namespace: auth-prod
|
||
spec:
|
||
podSelector:
|
||
matchLabels:
|
||
app.kubernetes.io/name: auth-server
|
||
policyTypes: ["Ingress"]
|
||
ingress:
|
||
- from:
|
||
- namespaceSelector:
|
||
matchLabels:
|
||
kubernetes.io/metadata.name: ingress-nginx
|
||
ports:
|
||
- port: http
|
||
protocol: TCP
|
||
```
|
||
|
||
**왜 좋은가:**
|
||
|
||
- default-deny → allow-list 패턴.
|
||
- metrics port는 monitoring namespace의 prometheus pod만.
|
||
- http port는 ingress controller namespace만.
|
||
|
||
---
|
||
|
||
## 좋은 예시 6: JSON structured log (Spring Boot logback)
|
||
|
||
```xml
|
||
<!-- logback-spring.xml -->
|
||
<configuration>
|
||
<appender name="JSON" class="ch.qos.logback.core.ConsoleAppender">
|
||
<encoder class="net.logstash.logback.encoder.LogstashEncoder">
|
||
<includeMdcKeyName>trace_id</includeMdcKeyName>
|
||
<includeMdcKeyName>span_id</includeMdcKeyName>
|
||
<includeMdcKeyName>request_id</includeMdcKeyName>
|
||
<customFields>{"service":"auth-server"}</customFields>
|
||
</encoder>
|
||
</appender>
|
||
<root level="INFO">
|
||
<appender-ref ref="JSON"/>
|
||
</root>
|
||
</configuration>
|
||
```
|
||
|
||
Actual output:
|
||
|
||
```json
|
||
{"timestamp":"2026-04-16T09:31:42.017Z","level":"INFO","service":"auth-server","trace_id":"4bf92f3577b34da6a3ce929d0e0e4736","span_id":"00f067aa0ba902b7","logger":"c.e.auth.LoginController","thread":"http-nio-8080-exec-3","message":"login success","user_id_hash":"ab12..."}
|
||
```
|
||
|
||
**왜 좋은가:**
|
||
|
||
- ISO 8601 UTC timestamp.
|
||
- trace_id / span_id 가 MDC에서 자동 주입 → Tempo / Jaeger와 correlate.
|
||
- service label이 customFields로 고정.
|
||
- user_id는 hashed → cardinality/PII 안전.
|
||
|
||
---
|
||
|
||
## 좋은 예시 7: OpenTelemetry Collector (DaemonSet agent + Deployment gateway)
|
||
|
||
```yaml
|
||
---
|
||
apiVersion: opentelemetry.io/v1beta1
|
||
kind: OpenTelemetryCollector
|
||
metadata:
|
||
name: otel-agent
|
||
namespace: observability
|
||
spec:
|
||
mode: daemonset
|
||
image: otel/opentelemetry-collector-contrib:0.101.0
|
||
config:
|
||
receivers:
|
||
otlp:
|
||
protocols:
|
||
grpc:
|
||
endpoint: 0.0.0.0:4317
|
||
http:
|
||
endpoint: 0.0.0.0:4318
|
||
processors:
|
||
batch:
|
||
send_batch_size: 1024
|
||
timeout: 5s
|
||
k8sattributes:
|
||
passthrough: false
|
||
extract:
|
||
metadata:
|
||
- k8s.pod.name
|
||
- k8s.namespace.name
|
||
- k8s.node.name
|
||
exporters:
|
||
otlp/gateway:
|
||
endpoint: otel-gateway.observability.svc:4317
|
||
tls:
|
||
insecure: true
|
||
service:
|
||
pipelines:
|
||
traces:
|
||
receivers: [otlp]
|
||
processors: [k8sattributes, batch]
|
||
exporters: [otlp/gateway]
|
||
metrics:
|
||
receivers: [otlp]
|
||
processors: [k8sattributes, batch]
|
||
exporters: [otlp/gateway]
|
||
---
|
||
apiVersion: opentelemetry.io/v1beta1
|
||
kind: OpenTelemetryCollector
|
||
metadata:
|
||
name: otel-gateway
|
||
namespace: observability
|
||
spec:
|
||
mode: deployment
|
||
replicas: 3
|
||
image: otel/opentelemetry-collector-contrib:0.101.0
|
||
config:
|
||
receivers:
|
||
otlp:
|
||
protocols:
|
||
grpc:
|
||
endpoint: 0.0.0.0:4317
|
||
processors:
|
||
batch:
|
||
send_batch_size: 2048
|
||
timeout: 5s
|
||
tail_sampling:
|
||
decision_wait: 10s
|
||
policies:
|
||
- name: errors-keep
|
||
type: status_code
|
||
status_code: { status_codes: [ERROR] }
|
||
- name: slow-keep
|
||
type: latency
|
||
latency: { threshold_ms: 500 }
|
||
- name: default-10pct
|
||
type: probabilistic
|
||
probabilistic: { sampling_percentage: 10 }
|
||
attributes/redact:
|
||
actions:
|
||
- key: http.request.header.authorization
|
||
action: delete
|
||
- key: user.email
|
||
action: hash
|
||
exporters:
|
||
otlp/tempo:
|
||
endpoint: tempo.observability.svc:4317
|
||
tls:
|
||
insecure: true
|
||
prometheusremotewrite:
|
||
endpoint: http://prometheus.monitoring.svc:9090/api/v1/write
|
||
service:
|
||
pipelines:
|
||
traces:
|
||
receivers: [otlp]
|
||
processors: [attributes/redact, tail_sampling, batch]
|
||
exporters: [otlp/tempo]
|
||
metrics:
|
||
receivers: [otlp]
|
||
processors: [batch]
|
||
exporters: [prometheusremotewrite]
|
||
```
|
||
|
||
**왜 좋은가:**
|
||
|
||
- agent (DaemonSet) → gateway (Deployment) 2단 구조.
|
||
- gateway에서 tail-based sampling (error + slow + 10% 나머지).
|
||
- PII redaction을 gateway에서 중앙 처리.
|
||
- agent가 node-local이라 app은 localhost endpoint만 알면 됨.
|
||
|
||
---
|
||
|
||
## 좋은 예시 8: Loki + Grafana Alloy DaemonSet (log shipping)
|
||
|
||
```yaml
|
||
---
|
||
apiVersion: v1
|
||
kind: ConfigMap
|
||
metadata:
|
||
name: alloy-config
|
||
namespace: observability
|
||
data:
|
||
config.alloy: |
|
||
discovery.kubernetes "pods" {
|
||
role = "pod"
|
||
}
|
||
discovery.relabel "pods" {
|
||
targets = discovery.kubernetes.pods.targets
|
||
rule {
|
||
source_labels = ["__meta_kubernetes_namespace"]
|
||
target_label = "namespace"
|
||
}
|
||
rule {
|
||
source_labels = ["__meta_kubernetes_pod_label_app_kubernetes_io_name"]
|
||
target_label = "service"
|
||
}
|
||
}
|
||
loki.source.kubernetes "pods" {
|
||
targets = discovery.relabel.pods.output
|
||
forward_to = [loki.write.default.receiver]
|
||
}
|
||
loki.write "default" {
|
||
endpoint {
|
||
url = "http://loki.observability.svc:3100/loki/api/v1/push"
|
||
}
|
||
}
|
||
```
|
||
|
||
**왜 좋은가:**
|
||
|
||
- Alloy DaemonSet이 node-level log tail.
|
||
- label은 namespace / service 두 개로 제한 (cardinality 안전).
|
||
|
||
---
|
||
|
||
## 좋은 예시 9: `kubectl events` (1.27+ stable)
|
||
|
||
```bash
|
||
# cluster-wide live watch, warnings only
|
||
kubectl events -A --types=Warning --watch
|
||
|
||
# specific pod
|
||
kubectl events -n auth-prod --for pod/auth-server-abc123
|
||
|
||
# last hour
|
||
kubectl events -n auth-prod --since=1h
|
||
```
|
||
|
||
**왜 좋은가:**
|
||
|
||
- `--for` 로 특정 오브젝트 event 만 필터링.
|
||
- `--watch` 가 `get events -w` 보다 안정적.
|
||
- timestamp sort 기본 제공.
|
||
|
||
---
|
||
|
||
## 나쁜 예시 1: `/metrics` 를 Ingress로 외부 공개
|
||
|
||
```yaml
|
||
apiVersion: networking.k8s.io/v1
|
||
kind: Ingress
|
||
spec:
|
||
rules:
|
||
- host: auth.example.com
|
||
http:
|
||
paths:
|
||
- path: /metrics # BAD
|
||
pathType: Prefix
|
||
backend:
|
||
service:
|
||
name: auth-server
|
||
port:
|
||
number: 8081
|
||
```
|
||
|
||
**문제:**
|
||
|
||
- Prometheus metric으로 내부 구조 / error rate / version 노출.
|
||
- DoS vector (scrape 비용).
|
||
- audit / compliance 위반.
|
||
|
||
**Fix:** metrics port는 외부 비공개, NetworkPolicy로 monitoring namespace만 허용.
|
||
|
||
---
|
||
|
||
## 나쁜 예시 2: high-cardinality label
|
||
|
||
```yaml
|
||
# app code
|
||
http_requests_total{user_id="12345", path="/users/12345/orders/98765", request_id="a1b2c3..."}
|
||
```
|
||
|
||
**문제:**
|
||
|
||
- user_id × path × request_id = 수백만 time series → Prometheus OOM.
|
||
- query 성능 붕괴.
|
||
|
||
**Fix:**
|
||
|
||
```
|
||
http_requests_total{route="/users/:id/orders/:id", method="GET", status_class="2xx"}
|
||
```
|
||
|
||
- route template 화, status는 bucket (2xx/4xx/5xx).
|
||
- user_id 는 logging에만, metric label 금지.
|
||
|
||
---
|
||
|
||
## 나쁜 예시 3: ServiceMonitor에 namespaceSelector 없음
|
||
|
||
```yaml
|
||
apiVersion: monitoring.coreos.com/v1
|
||
kind: ServiceMonitor
|
||
spec:
|
||
selector:
|
||
matchLabels:
|
||
app: my-app
|
||
# namespaceSelector 없음 → Operator 설정에 따라 전체 cluster scan
|
||
endpoints:
|
||
- port: metrics
|
||
```
|
||
|
||
**문제:**
|
||
|
||
- 암묵적으로 너무 넓은 범위 (Operator 설정에 따라 다름).
|
||
- 동일 label 를 다른 namespace에서 쓰면 의도치 않은 scrape.
|
||
|
||
**Fix:** `namespaceSelector.matchNames` 명시.
|
||
|
||
---
|
||
|
||
## 나쁜 예시 4: probe가 `/metrics` 사용
|
||
|
||
```yaml
|
||
readinessProbe:
|
||
httpGet:
|
||
path: /metrics # BAD
|
||
port: 8081
|
||
periodSeconds: 5
|
||
```
|
||
|
||
**문제:**
|
||
|
||
- `/metrics` 는 비용이 큰 endpoint (모든 registry dump).
|
||
- periodSeconds 5초 × N pod = unnecessary load.
|
||
- readiness 의미와 무관.
|
||
|
||
**Fix:** `/actuator/health/readiness` 같은 전용 shallow endpoint.
|
||
|
||
---
|
||
|
||
## 나쁜 예시 5: 로그에 access token 그대로
|
||
|
||
```
|
||
2026-04-16T09:32:11.002 INFO Exchanging code for token: access_token=eyJhbGciOi...
|
||
```
|
||
|
||
**문제:**
|
||
|
||
- token이 log index에 그대로 저장 → 유출 리스크.
|
||
- 중앙 로그 시스템 (OpenSearch / Loki) 에 영구 보관.
|
||
|
||
**Fix:**
|
||
|
||
- 애플리케이션에서 token 값 로깅 금지.
|
||
- 중앙 파이프라인에 regex redaction (`access_token=[^ ]+` → `access_token=***`).
|
||
- debug 로그에서도 masking.
|
||
|
||
---
|
||
|
||
## 나쁜 예시 6: 로그를 PVC / file로 적재
|
||
|
||
```yaml
|
||
volumeMounts:
|
||
- name: app-logs
|
||
mountPath: /var/log/app # BAD
|
||
volumes:
|
||
- name: app-logs
|
||
persistentVolumeClaim:
|
||
claimName: app-logs-pvc
|
||
```
|
||
|
||
**문제:**
|
||
|
||
- 컨테이너 표준 (stdout/stderr) 위반.
|
||
- Pod 삭제 시 로그 손실 또는 orphan PVC.
|
||
- `kubectl logs` 로 안 보임.
|
||
- node log agent가 수집 못 함.
|
||
|
||
**Fix:** stdout/stderr로 출력, DaemonSet agent가 수집.
|