init: 폴더구조 설계 및 인프라 설계
This commit is contained in:
DongHyeonka
@@ -0,0 +1,601 @@
|
||||
# observability / health 예시
|
||||
|
||||
---
|
||||
|
||||
## 좋은 예시 1: ServiceMonitor (kube-prometheus-stack 표준)
|
||||
|
||||
```yaml
|
||||
---
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: ServiceMonitor
|
||||
metadata:
|
||||
name: auth-server
|
||||
namespace: auth-prod
|
||||
labels:
|
||||
app.kubernetes.io/name: auth-server
|
||||
app.kubernetes.io/instance: auth-server-prod
|
||||
app.kubernetes.io/part-of: identity-platform
|
||||
release: kube-prometheus-stack
|
||||
spec:
|
||||
namespaceSelector:
|
||||
matchNames:
|
||||
- auth-prod
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: auth-server
|
||||
app.kubernetes.io/instance: auth-server-prod
|
||||
endpoints:
|
||||
- port: metrics # named port (required)
|
||||
path: /actuator/prometheus
|
||||
scheme: http
|
||||
interval: 30s
|
||||
scrapeTimeout: 10s
|
||||
honorLabels: false
|
||||
relabelings:
|
||||
- sourceLabels: [__meta_kubernetes_pod_name]
|
||||
targetLabel: pod
|
||||
- sourceLabels: [__meta_kubernetes_namespace]
|
||||
targetLabel: namespace
|
||||
- sourceLabels: [__meta_kubernetes_pod_label_app_kubernetes_io_version]
|
||||
targetLabel: version
|
||||
- action: labeldrop
|
||||
regex: "pod_template_hash|controller_revision_hash"
|
||||
metricRelabelings:
|
||||
- sourceLabels: [__name__]
|
||||
regex: "jvm_gc_pause_seconds_.*"
|
||||
action: keep
|
||||
- sourceLabels: [__name__]
|
||||
regex: "debug_.*"
|
||||
action: drop
|
||||
```
|
||||
|
||||
**왜 좋은가:**
|
||||
|
||||
- `namespaceSelector` 명시로 암묵적 전체 허용 방지.
|
||||
- `port: metrics` 는 Service/Deployment의 named port를 참조 → 포트 번호 변경에 내성.
|
||||
- `interval / scrapeTimeout` 관계 유지 (timeout < interval).
|
||||
- `relabelings` 로 pod / namespace / version label 정리, noise label drop.
|
||||
- `metricRelabelings` 로 불필요 metric drop (cardinality / storage 절감).
|
||||
- `release: kube-prometheus-stack` label 로 Operator가 선택.
|
||||
|
||||
---
|
||||
|
||||
## 좋은 예시 2: ServiceMonitor with bearer token (Vault telemetry)
|
||||
|
||||
```yaml
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: vault-metrics-token
|
||||
namespace: vault
|
||||
type: Opaque
|
||||
stringData:
|
||||
token: "hvs.xxxx.prometheus-readonly"
|
||||
---
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: ServiceMonitor
|
||||
metadata:
|
||||
name: vault
|
||||
namespace: vault
|
||||
labels:
|
||||
app.kubernetes.io/name: vault
|
||||
app.kubernetes.io/instance: vault-prod
|
||||
release: kube-prometheus-stack
|
||||
spec:
|
||||
namespaceSelector:
|
||||
matchNames:
|
||||
- vault
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: vault
|
||||
app.kubernetes.io/instance: vault-prod
|
||||
endpoints:
|
||||
- port: https
|
||||
path: /v1/sys/metrics
|
||||
params:
|
||||
format: ["prometheus"]
|
||||
scheme: https
|
||||
interval: 30s
|
||||
scrapeTimeout: 10s
|
||||
bearerTokenSecret:
|
||||
name: vault-metrics-token
|
||||
key: token
|
||||
tlsConfig:
|
||||
insecureSkipVerify: false
|
||||
ca:
|
||||
secret:
|
||||
name: vault-ca
|
||||
key: ca.crt
|
||||
serverName: vault.vault.svc
|
||||
relabelings:
|
||||
- sourceLabels: [__meta_kubernetes_pod_name]
|
||||
targetLabel: pod
|
||||
```
|
||||
|
||||
**왜 좋은가:**
|
||||
|
||||
- Vault `/sys/metrics` 는 read token 필수. `bearerTokenSecret` 참조로 Operator가 주입.
|
||||
- TLS CA pinning + serverName 으로 MitM 방지.
|
||||
- `params` 로 Prometheus format 요청.
|
||||
|
||||
---
|
||||
|
||||
## 좋은 예시 3: PodMonitor (Service 없는 워크로드)
|
||||
|
||||
```yaml
|
||||
---
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: PodMonitor
|
||||
metadata:
|
||||
name: batch-worker
|
||||
namespace: batch
|
||||
labels:
|
||||
release: kube-prometheus-stack
|
||||
spec:
|
||||
namespaceSelector:
|
||||
matchNames:
|
||||
- batch
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: batch-worker
|
||||
podMetricsEndpoints:
|
||||
- port: metrics
|
||||
path: /metrics
|
||||
interval: 30s
|
||||
scrapeTimeout: 10s
|
||||
relabelings:
|
||||
- sourceLabels: [__meta_kubernetes_pod_name]
|
||||
targetLabel: pod
|
||||
```
|
||||
|
||||
**왜 좋은가:**
|
||||
|
||||
- Job / headless workload처럼 Service 뒤에 없는 경우 PodMonitor로 직접 pod 매칭.
|
||||
|
||||
---
|
||||
|
||||
## 좋은 예시 4: Annotation-based fallback (Operator 없는 환경 only)
|
||||
|
||||
```yaml
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: legacy-app
|
||||
namespace: legacy
|
||||
annotations:
|
||||
prometheus.io/scrape: "true"
|
||||
prometheus.io/port: "8081"
|
||||
prometheus.io/path: "/metrics"
|
||||
prometheus.io/scheme: "http"
|
||||
spec:
|
||||
selector:
|
||||
app.kubernetes.io/name: legacy-app
|
||||
ports:
|
||||
- name: http
|
||||
port: 80
|
||||
targetPort: 8080
|
||||
- name: metrics
|
||||
port: 8081
|
||||
targetPort: 8081
|
||||
```
|
||||
|
||||
**왜 좋은가 (조건부):**
|
||||
|
||||
- kube-prometheus-stack이 없는 legacy 환경에서만 유효.
|
||||
- Operator가 있으면 ServiceMonitor로 전환.
|
||||
|
||||
---
|
||||
|
||||
## 좋은 예시 5: NetworkPolicy — prometheus namespace만 metrics scrape 허용
|
||||
|
||||
```yaml
|
||||
---
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: auth-server-default-deny
|
||||
namespace: auth-prod
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: auth-server
|
||||
policyTypes: ["Ingress", "Egress"]
|
||||
ingress: []
|
||||
egress: []
|
||||
---
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: auth-server-allow-metrics
|
||||
namespace: auth-prod
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: auth-server
|
||||
policyTypes: ["Ingress"]
|
||||
ingress:
|
||||
- from:
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
kubernetes.io/metadata.name: monitoring
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: prometheus
|
||||
ports:
|
||||
- port: metrics
|
||||
protocol: TCP
|
||||
---
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: auth-server-allow-http-from-ingress
|
||||
namespace: auth-prod
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: auth-server
|
||||
policyTypes: ["Ingress"]
|
||||
ingress:
|
||||
- from:
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
kubernetes.io/metadata.name: ingress-nginx
|
||||
ports:
|
||||
- port: http
|
||||
protocol: TCP
|
||||
```
|
||||
|
||||
**왜 좋은가:**
|
||||
|
||||
- default-deny → allow-list 패턴.
|
||||
- metrics port는 monitoring namespace의 prometheus pod만.
|
||||
- http port는 ingress controller namespace만.
|
||||
|
||||
---
|
||||
|
||||
## 좋은 예시 6: JSON structured log (Spring Boot logback)
|
||||
|
||||
```xml
|
||||
<!-- logback-spring.xml -->
|
||||
<configuration>
|
||||
<appender name="JSON" class="ch.qos.logback.core.ConsoleAppender">
|
||||
<encoder class="net.logstash.logback.encoder.LogstashEncoder">
|
||||
<includeMdcKeyName>trace_id</includeMdcKeyName>
|
||||
<includeMdcKeyName>span_id</includeMdcKeyName>
|
||||
<includeMdcKeyName>request_id</includeMdcKeyName>
|
||||
<customFields>{"service":"auth-server"}</customFields>
|
||||
</encoder>
|
||||
</appender>
|
||||
<root level="INFO">
|
||||
<appender-ref ref="JSON"/>
|
||||
</root>
|
||||
</configuration>
|
||||
```
|
||||
|
||||
Actual output:
|
||||
|
||||
```json
|
||||
{"timestamp":"2026-04-16T09:31:42.017Z","level":"INFO","service":"auth-server","trace_id":"4bf92f3577b34da6a3ce929d0e0e4736","span_id":"00f067aa0ba902b7","logger":"c.e.auth.LoginController","thread":"http-nio-8080-exec-3","message":"login success","user_id_hash":"ab12..."}
|
||||
```
|
||||
|
||||
**왜 좋은가:**
|
||||
|
||||
- ISO 8601 UTC timestamp.
|
||||
- trace_id / span_id 가 MDC에서 자동 주입 → Tempo / Jaeger와 correlate.
|
||||
- service label이 customFields로 고정.
|
||||
- user_id는 hashed → cardinality/PII 안전.
|
||||
|
||||
---
|
||||
|
||||
## 좋은 예시 7: OpenTelemetry Collector (DaemonSet agent + Deployment gateway)
|
||||
|
||||
```yaml
|
||||
---
|
||||
apiVersion: opentelemetry.io/v1beta1
|
||||
kind: OpenTelemetryCollector
|
||||
metadata:
|
||||
name: otel-agent
|
||||
namespace: observability
|
||||
spec:
|
||||
mode: daemonset
|
||||
image: otel/opentelemetry-collector-contrib:0.101.0
|
||||
config:
|
||||
receivers:
|
||||
otlp:
|
||||
protocols:
|
||||
grpc:
|
||||
endpoint: 0.0.0.0:4317
|
||||
http:
|
||||
endpoint: 0.0.0.0:4318
|
||||
processors:
|
||||
batch:
|
||||
send_batch_size: 1024
|
||||
timeout: 5s
|
||||
k8sattributes:
|
||||
passthrough: false
|
||||
extract:
|
||||
metadata:
|
||||
- k8s.pod.name
|
||||
- k8s.namespace.name
|
||||
- k8s.node.name
|
||||
exporters:
|
||||
otlp/gateway:
|
||||
endpoint: otel-gateway.observability.svc:4317
|
||||
tls:
|
||||
insecure: true
|
||||
service:
|
||||
pipelines:
|
||||
traces:
|
||||
receivers: [otlp]
|
||||
processors: [k8sattributes, batch]
|
||||
exporters: [otlp/gateway]
|
||||
metrics:
|
||||
receivers: [otlp]
|
||||
processors: [k8sattributes, batch]
|
||||
exporters: [otlp/gateway]
|
||||
---
|
||||
apiVersion: opentelemetry.io/v1beta1
|
||||
kind: OpenTelemetryCollector
|
||||
metadata:
|
||||
name: otel-gateway
|
||||
namespace: observability
|
||||
spec:
|
||||
mode: deployment
|
||||
replicas: 3
|
||||
image: otel/opentelemetry-collector-contrib:0.101.0
|
||||
config:
|
||||
receivers:
|
||||
otlp:
|
||||
protocols:
|
||||
grpc:
|
||||
endpoint: 0.0.0.0:4317
|
||||
processors:
|
||||
batch:
|
||||
send_batch_size: 2048
|
||||
timeout: 5s
|
||||
tail_sampling:
|
||||
decision_wait: 10s
|
||||
policies:
|
||||
- name: errors-keep
|
||||
type: status_code
|
||||
status_code: { status_codes: [ERROR] }
|
||||
- name: slow-keep
|
||||
type: latency
|
||||
latency: { threshold_ms: 500 }
|
||||
- name: default-10pct
|
||||
type: probabilistic
|
||||
probabilistic: { sampling_percentage: 10 }
|
||||
attributes/redact:
|
||||
actions:
|
||||
- key: http.request.header.authorization
|
||||
action: delete
|
||||
- key: user.email
|
||||
action: hash
|
||||
exporters:
|
||||
otlp/tempo:
|
||||
endpoint: tempo.observability.svc:4317
|
||||
tls:
|
||||
insecure: true
|
||||
prometheusremotewrite:
|
||||
endpoint: http://prometheus.monitoring.svc:9090/api/v1/write
|
||||
service:
|
||||
pipelines:
|
||||
traces:
|
||||
receivers: [otlp]
|
||||
processors: [attributes/redact, tail_sampling, batch]
|
||||
exporters: [otlp/tempo]
|
||||
metrics:
|
||||
receivers: [otlp]
|
||||
processors: [batch]
|
||||
exporters: [prometheusremotewrite]
|
||||
```
|
||||
|
||||
**왜 좋은가:**
|
||||
|
||||
- agent (DaemonSet) → gateway (Deployment) 2단 구조.
|
||||
- gateway에서 tail-based sampling (error + slow + 10% 나머지).
|
||||
- PII redaction을 gateway에서 중앙 처리.
|
||||
- agent가 node-local이라 app은 localhost endpoint만 알면 됨.
|
||||
|
||||
---
|
||||
|
||||
## 좋은 예시 8: Loki + Grafana Alloy DaemonSet (log shipping)
|
||||
|
||||
```yaml
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: alloy-config
|
||||
namespace: observability
|
||||
data:
|
||||
config.alloy: |
|
||||
discovery.kubernetes "pods" {
|
||||
role = "pod"
|
||||
}
|
||||
discovery.relabel "pods" {
|
||||
targets = discovery.kubernetes.pods.targets
|
||||
rule {
|
||||
source_labels = ["__meta_kubernetes_namespace"]
|
||||
target_label = "namespace"
|
||||
}
|
||||
rule {
|
||||
source_labels = ["__meta_kubernetes_pod_label_app_kubernetes_io_name"]
|
||||
target_label = "service"
|
||||
}
|
||||
}
|
||||
loki.source.kubernetes "pods" {
|
||||
targets = discovery.relabel.pods.output
|
||||
forward_to = [loki.write.default.receiver]
|
||||
}
|
||||
loki.write "default" {
|
||||
endpoint {
|
||||
url = "http://loki.observability.svc:3100/loki/api/v1/push"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**왜 좋은가:**
|
||||
|
||||
- Alloy DaemonSet이 node-level log tail.
|
||||
- label은 namespace / service 두 개로 제한 (cardinality 안전).
|
||||
|
||||
---
|
||||
|
||||
## 좋은 예시 9: `kubectl events` (1.27+ stable)
|
||||
|
||||
```bash
|
||||
# cluster-wide live watch, warnings only
|
||||
kubectl events -A --types=Warning --watch
|
||||
|
||||
# specific pod
|
||||
kubectl events -n auth-prod --for pod/auth-server-abc123
|
||||
|
||||
# last hour
|
||||
kubectl events -n auth-prod --since=1h
|
||||
```
|
||||
|
||||
**왜 좋은가:**
|
||||
|
||||
- `--for` 로 특정 오브젝트 event 만 필터링.
|
||||
- `--watch` 가 `get events -w` 보다 안정적.
|
||||
- timestamp sort 기본 제공.
|
||||
|
||||
---
|
||||
|
||||
## 나쁜 예시 1: `/metrics` 를 Ingress로 외부 공개
|
||||
|
||||
```yaml
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: Ingress
|
||||
spec:
|
||||
rules:
|
||||
- host: auth.example.com
|
||||
http:
|
||||
paths:
|
||||
- path: /metrics # BAD
|
||||
pathType: Prefix
|
||||
backend:
|
||||
service:
|
||||
name: auth-server
|
||||
port:
|
||||
number: 8081
|
||||
```
|
||||
|
||||
**문제:**
|
||||
|
||||
- Prometheus metric으로 내부 구조 / error rate / version 노출.
|
||||
- DoS vector (scrape 비용).
|
||||
- audit / compliance 위반.
|
||||
|
||||
**Fix:** metrics port는 외부 비공개, NetworkPolicy로 monitoring namespace만 허용.
|
||||
|
||||
---
|
||||
|
||||
## 나쁜 예시 2: high-cardinality label
|
||||
|
||||
```yaml
|
||||
# app code
|
||||
http_requests_total{user_id="12345", path="/users/12345/orders/98765", request_id="a1b2c3..."}
|
||||
```
|
||||
|
||||
**문제:**
|
||||
|
||||
- user_id × path × request_id = 수백만 time series → Prometheus OOM.
|
||||
- query 성능 붕괴.
|
||||
|
||||
**Fix:**
|
||||
|
||||
```
|
||||
http_requests_total{route="/users/:id/orders/:id", method="GET", status_class="2xx"}
|
||||
```
|
||||
|
||||
- route template 화, status는 bucket (2xx/4xx/5xx).
|
||||
- user_id 는 logging에만, metric label 금지.
|
||||
|
||||
---
|
||||
|
||||
## 나쁜 예시 3: ServiceMonitor에 namespaceSelector 없음
|
||||
|
||||
```yaml
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: ServiceMonitor
|
||||
spec:
|
||||
selector:
|
||||
matchLabels:
|
||||
app: my-app
|
||||
# namespaceSelector 없음 → Operator 설정에 따라 전체 cluster scan
|
||||
endpoints:
|
||||
- port: metrics
|
||||
```
|
||||
|
||||
**문제:**
|
||||
|
||||
- 암묵적으로 너무 넓은 범위 (Operator 설정에 따라 다름).
|
||||
- 동일 label 를 다른 namespace에서 쓰면 의도치 않은 scrape.
|
||||
|
||||
**Fix:** `namespaceSelector.matchNames` 명시.
|
||||
|
||||
---
|
||||
|
||||
## 나쁜 예시 4: probe가 `/metrics` 사용
|
||||
|
||||
```yaml
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /metrics # BAD
|
||||
port: 8081
|
||||
periodSeconds: 5
|
||||
```
|
||||
|
||||
**문제:**
|
||||
|
||||
- `/metrics` 는 비용이 큰 endpoint (모든 registry dump).
|
||||
- periodSeconds 5초 × N pod = unnecessary load.
|
||||
- readiness 의미와 무관.
|
||||
|
||||
**Fix:** `/actuator/health/readiness` 같은 전용 shallow endpoint.
|
||||
|
||||
---
|
||||
|
||||
## 나쁜 예시 5: 로그에 access token 그대로
|
||||
|
||||
```
|
||||
2026-04-16T09:32:11.002 INFO Exchanging code for token: access_token=eyJhbGciOi...
|
||||
```
|
||||
|
||||
**문제:**
|
||||
|
||||
- token이 log index에 그대로 저장 → 유출 리스크.
|
||||
- 중앙 로그 시스템 (OpenSearch / Loki) 에 영구 보관.
|
||||
|
||||
**Fix:**
|
||||
|
||||
- 애플리케이션에서 token 값 로깅 금지.
|
||||
- 중앙 파이프라인에 regex redaction (`access_token=[^ ]+` → `access_token=***`).
|
||||
- debug 로그에서도 masking.
|
||||
|
||||
---
|
||||
|
||||
## 나쁜 예시 6: 로그를 PVC / file로 적재
|
||||
|
||||
```yaml
|
||||
volumeMounts:
|
||||
- name: app-logs
|
||||
mountPath: /var/log/app # BAD
|
||||
volumes:
|
||||
- name: app-logs
|
||||
persistentVolumeClaim:
|
||||
claimName: app-logs-pvc
|
||||
```
|
||||
|
||||
**문제:**
|
||||
|
||||
- 컨테이너 표준 (stdout/stderr) 위반.
|
||||
- Pod 삭제 시 로그 손실 또는 orphan PVC.
|
||||
- `kubectl logs` 로 안 보임.
|
||||
- node log agent가 수집 못 함.
|
||||
|
||||
**Fix:** stdout/stderr로 출력, DaemonSet agent가 수집.
|
||||
Reference in New Issue
Block a user