# observability / health 예시

---

## 좋은 예시 1: ServiceMonitor (kube-prometheus-stack 표준)

```yaml
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: auth-server
  namespace: auth-prod
  labels:
    app.kubernetes.io/name: auth-server
    app.kubernetes.io/instance: auth-server-prod
    app.kubernetes.io/part-of: identity-platform
    release: kube-prometheus-stack
spec:
  namespaceSelector:
    matchNames:
      - auth-prod
  selector:
    matchLabels:
      app.kubernetes.io/name: auth-server
      app.kubernetes.io/instance: auth-server-prod
  endpoints:
    - port: metrics            # named port (required)
      path: /actuator/prometheus
      scheme: http
      interval: 30s
      scrapeTimeout: 10s
      honorLabels: false
      relabelings:
        - sourceLabels: [__meta_kubernetes_pod_name]
          targetLabel: pod
        - sourceLabels: [__meta_kubernetes_namespace]
          targetLabel: namespace
        - sourceLabels: [__meta_kubernetes_pod_label_app_kubernetes_io_version]
          targetLabel: version
        - action: labeldrop
          regex: "pod_template_hash|controller_revision_hash"
      metricRelabelings:
        - sourceLabels: [__name__]
          regex: "jvm_gc_pause_seconds_.*"
          action: keep
        - sourceLabels: [__name__]
          regex: "debug_.*"
          action: drop
```

**왜 좋은가:**

- `namespaceSelector` 명시로 암묵적 전체 허용 방지.
- `port: metrics` 는 Service/Deployment의 named port를 참조 → 포트 번호 변경에 내성.
- `interval / scrapeTimeout` 관계 유지 (timeout < interval).
- `relabelings` 로 pod / namespace / version label 정리, noise label drop.
- `metricRelabelings` 로 불필요 metric drop (cardinality / storage 절감).
- `release: kube-prometheus-stack` label 로 Operator가 선택.

---

## 좋은 예시 2: ServiceMonitor with bearer token (Vault telemetry)

```yaml
---
apiVersion: v1
kind: Secret
metadata:
  name: vault-metrics-token
  namespace: vault
type: Opaque
stringData:
  token: "hvs.xxxx.prometheus-readonly"
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: vault
  namespace: vault
  labels:
    app.kubernetes.io/name: vault
    app.kubernetes.io/instance: vault-prod
    release: kube-prometheus-stack
spec:
  namespaceSelector:
    matchNames:
      - vault
  selector:
    matchLabels:
      app.kubernetes.io/name: vault
      app.kubernetes.io/instance: vault-prod
  endpoints:
    - port: https
      path: /v1/sys/metrics
      params:
        format: ["prometheus"]
      scheme: https
      interval: 30s
      scrapeTimeout: 10s
      bearerTokenSecret:
        name: vault-metrics-token
        key: token
      tlsConfig:
        insecureSkipVerify: false
        ca:
          secret:
            name: vault-ca
            key: ca.crt
        serverName: vault.vault.svc
      relabelings:
        - sourceLabels: [__meta_kubernetes_pod_name]
          targetLabel: pod
```

**왜 좋은가:**

- Vault `/sys/metrics` 는 read token 필수. `bearerTokenSecret` 참조로 Operator가 주입.
- TLS CA pinning + serverName 으로 MitM 방지.
- `params` 로 Prometheus format 요청.

---

## 좋은 예시 3: PodMonitor (Service 없는 워크로드)

```yaml
---
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: batch-worker
  namespace: batch
  labels:
    release: kube-prometheus-stack
spec:
  namespaceSelector:
    matchNames:
      - batch
  selector:
    matchLabels:
      app.kubernetes.io/name: batch-worker
  podMetricsEndpoints:
    - port: metrics
      path: /metrics
      interval: 30s
      scrapeTimeout: 10s
      relabelings:
        - sourceLabels: [__meta_kubernetes_pod_name]
          targetLabel: pod
```

**왜 좋은가:**

- Job / headless workload처럼 Service 뒤에 없는 경우 PodMonitor로 직접 pod 매칭.

---

## 좋은 예시 4: Annotation-based fallback (Operator 없는 환경 only)

```yaml
---
apiVersion: v1
kind: Service
metadata:
  name: legacy-app
  namespace: legacy
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "8081"
    prometheus.io/path: "/metrics"
    prometheus.io/scheme: "http"
spec:
  selector:
    app.kubernetes.io/name: legacy-app
  ports:
    - name: http
      port: 80
      targetPort: 8080
    - name: metrics
      port: 8081
      targetPort: 8081
```

**왜 좋은가 (조건부):**

- kube-prometheus-stack이 없는 legacy 환경에서만 유효.
- Operator가 있으면 ServiceMonitor로 전환.

---

## 좋은 예시 5: NetworkPolicy — prometheus namespace만 metrics scrape 허용

```yaml
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: auth-server-default-deny
  namespace: auth-prod
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: auth-server
  policyTypes: ["Ingress", "Egress"]
  ingress: []
  egress: []
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: auth-server-allow-metrics
  namespace: auth-prod
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: auth-server
  policyTypes: ["Ingress"]
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: monitoring
          podSelector:
            matchLabels:
              app.kubernetes.io/name: prometheus
      ports:
        - port: metrics
          protocol: TCP
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: auth-server-allow-http-from-ingress
  namespace: auth-prod
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: auth-server
  policyTypes: ["Ingress"]
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: ingress-nginx
      ports:
        - port: http
          protocol: TCP
```

**왜 좋은가:**

- default-deny → allow-list 패턴.
- metrics port는 monitoring namespace의 prometheus pod만.
- http port는 ingress controller namespace만.

---

## 좋은 예시 6: JSON structured log (Spring Boot logback)

```xml
<!-- logback-spring.xml -->
<configuration>
  <appender name="JSON" class="ch.qos.logback.core.ConsoleAppender">
    <encoder class="net.logstash.logback.encoder.LogstashEncoder">
      <includeMdcKeyName>trace_id</includeMdcKeyName>
      <includeMdcKeyName>span_id</includeMdcKeyName>
      <includeMdcKeyName>request_id</includeMdcKeyName>
      <customFields>{"service":"auth-server"}</customFields>
    </encoder>
  </appender>
  <root level="INFO">
    <appender-ref ref="JSON"/>
  </root>
</configuration>
```

Actual output:

```json
{"timestamp":"2026-04-16T09:31:42.017Z","level":"INFO","service":"auth-server","trace_id":"4bf92f3577b34da6a3ce929d0e0e4736","span_id":"00f067aa0ba902b7","logger":"c.e.auth.LoginController","thread":"http-nio-8080-exec-3","message":"login success","user_id_hash":"ab12..."}
```

**왜 좋은가:**

- ISO 8601 UTC timestamp.
- trace_id / span_id 가 MDC에서 자동 주입 → Tempo / Jaeger와 correlate.
- service label이 customFields로 고정.
- user_id는 hashed → cardinality/PII 안전.

---

## 좋은 예시 7: OpenTelemetry Collector (DaemonSet agent + Deployment gateway)

```yaml
---
apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
  name: otel-agent
  namespace: observability
spec:
  mode: daemonset
  image: otel/opentelemetry-collector-contrib:0.101.0
  config:
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
          http:
            endpoint: 0.0.0.0:4318
    processors:
      batch:
        send_batch_size: 1024
        timeout: 5s
      k8sattributes:
        passthrough: false
        extract:
          metadata:
            - k8s.pod.name
            - k8s.namespace.name
            - k8s.node.name
    exporters:
      otlp/gateway:
        endpoint: otel-gateway.observability.svc:4317
        tls:
          insecure: true
    service:
      pipelines:
        traces:
          receivers: [otlp]
          processors: [k8sattributes, batch]
          exporters: [otlp/gateway]
        metrics:
          receivers: [otlp]
          processors: [k8sattributes, batch]
          exporters: [otlp/gateway]
---
apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
  name: otel-gateway
  namespace: observability
spec:
  mode: deployment
  replicas: 3
  image: otel/opentelemetry-collector-contrib:0.101.0
  config:
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
    processors:
      batch:
        send_batch_size: 2048
        timeout: 5s
      tail_sampling:
        decision_wait: 10s
        policies:
          - name: errors-keep
            type: status_code
            status_code: { status_codes: [ERROR] }
          - name: slow-keep
            type: latency
            latency: { threshold_ms: 500 }
          - name: default-10pct
            type: probabilistic
            probabilistic: { sampling_percentage: 10 }
      attributes/redact:
        actions:
          - key: http.request.header.authorization
            action: delete
          - key: user.email
            action: hash
    exporters:
      otlp/tempo:
        endpoint: tempo.observability.svc:4317
        tls:
          insecure: true
      prometheusremotewrite:
        endpoint: http://prometheus.monitoring.svc:9090/api/v1/write
    service:
      pipelines:
        traces:
          receivers: [otlp]
          processors: [attributes/redact, tail_sampling, batch]
          exporters: [otlp/tempo]
        metrics:
          receivers: [otlp]
          processors: [batch]
          exporters: [prometheusremotewrite]
```

**왜 좋은가:**

- agent (DaemonSet) → gateway (Deployment) 2단 구조.
- gateway에서 tail-based sampling (error + slow + 10% 나머지).
- PII redaction을 gateway에서 중앙 처리.
- agent가 node-local이라 app은 localhost endpoint만 알면 됨.

---

## 좋은 예시 8: Loki + Grafana Alloy DaemonSet (log shipping)

```yaml
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: alloy-config
  namespace: observability
data:
  config.alloy: |
    discovery.kubernetes "pods" {
      role = "pod"
    }
    discovery.relabel "pods" {
      targets = discovery.kubernetes.pods.targets
      rule {
        source_labels = ["__meta_kubernetes_namespace"]
        target_label  = "namespace"
      }
      rule {
        source_labels = ["__meta_kubernetes_pod_label_app_kubernetes_io_name"]
        target_label  = "service"
      }
    }
    loki.source.kubernetes "pods" {
      targets    = discovery.relabel.pods.output
      forward_to = [loki.write.default.receiver]
    }
    loki.write "default" {
      endpoint {
        url = "http://loki.observability.svc:3100/loki/api/v1/push"
      }
    }
```

**왜 좋은가:**

- Alloy DaemonSet이 node-level log tail.
- label은 namespace / service 두 개로 제한 (cardinality 안전).

---

## 좋은 예시 9: `kubectl events` (1.27+ stable)

```bash
# cluster-wide live watch, warnings only
kubectl events -A --types=Warning --watch

# specific pod
kubectl events -n auth-prod --for pod/auth-server-abc123

# last hour
kubectl events -n auth-prod --since=1h
```

**왜 좋은가:**

- `--for` 로 특정 오브젝트 event 만 필터링.
- `--watch` 가 `get events -w` 보다 안정적.
- timestamp sort 기본 제공.

---

## 나쁜 예시 1: `/metrics` 를 Ingress로 외부 공개

```yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
spec:
  rules:
    - host: auth.example.com
      http:
        paths:
          - path: /metrics         # BAD
            pathType: Prefix
            backend:
              service:
                name: auth-server
                port:
                  number: 8081
```

**문제:**

- Prometheus metric으로 내부 구조 / error rate / version 노출.
- DoS vector (scrape 비용).
- audit / compliance 위반.

**Fix:** metrics port는 외부 비공개, NetworkPolicy로 monitoring namespace만 허용.

---

## 나쁜 예시 2: high-cardinality label

```yaml
# app code
http_requests_total{user_id="12345", path="/users/12345/orders/98765", request_id="a1b2c3..."}
```

**문제:**

- user_id × path × request_id = 수백만 time series → Prometheus OOM.
- query 성능 붕괴.

**Fix:**

```
http_requests_total{route="/users/:id/orders/:id", method="GET", status_class="2xx"}
```

- route template 화, status는 bucket (2xx/4xx/5xx).
- user_id 는 logging에만, metric label 금지.

---

## 나쁜 예시 3: ServiceMonitor에 namespaceSelector 없음

```yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
spec:
  selector:
    matchLabels:
      app: my-app
  # namespaceSelector 없음 → Operator 설정에 따라 전체 cluster scan
  endpoints:
    - port: metrics
```

**문제:**

- 암묵적으로 너무 넓은 범위 (Operator 설정에 따라 다름).
- 동일 label 를 다른 namespace에서 쓰면 의도치 않은 scrape.

**Fix:** `namespaceSelector.matchNames` 명시.

---

## 나쁜 예시 4: probe가 `/metrics` 사용

```yaml
readinessProbe:
  httpGet:
    path: /metrics     # BAD
    port: 8081
  periodSeconds: 5
```

**문제:**

- `/metrics` 는 비용이 큰 endpoint (모든 registry dump).
- periodSeconds 5초 × N pod = unnecessary load.
- readiness 의미와 무관.

**Fix:** `/actuator/health/readiness` 같은 전용 shallow endpoint.

---

## 나쁜 예시 5: 로그에 access token 그대로

```
2026-04-16T09:32:11.002 INFO  Exchanging code for token: access_token=eyJhbGciOi...
```

**문제:**

- token이 log index에 그대로 저장 → 유출 리스크.
- 중앙 로그 시스템 (OpenSearch / Loki) 에 영구 보관.

**Fix:**

- 애플리케이션에서 token 값 로깅 금지.
- 중앙 파이프라인에 regex redaction (`access_token=[^ ]+` → `access_token=***`).
- debug 로그에서도 masking.

---

## 나쁜 예시 6: 로그를 PVC / file로 적재

```yaml
volumeMounts:
  - name: app-logs
    mountPath: /var/log/app      # BAD
volumes:
  - name: app-logs
    persistentVolumeClaim:
      claimName: app-logs-pvc
```

**문제:**

- 컨테이너 표준 (stdout/stderr) 위반.
- Pod 삭제 시 로그 손실 또는 orphan PVC.
- `kubectl logs` 로 안 보임.
- node log agent가 수집 못 함.

**Fix:** stdout/stderr로 출력, DaemonSet agent가 수집.