donghyeon.kang/project-infra

Fork 0

Files

T

DongHyeonka f9c463f87a init: 폴더구조 설계 및 인프라 설계

2026-07-24 14:31:53 +09:00

20 KiB

Raw Blame History

Vault 예시

Vault 1.17+ + Helm chart hashicorp/vault + VSO 0.8+ 기준. 모든 manifest는 kubectl apply 적용 가능한 완전한 형태다.

좋은 예시 1: Helm values.yaml — HA Raft + auto-unseal + audit

# values/vault-prod.yaml
global:
  enabled: true
  tlsDisable: false

injector:
  enabled: true
  replicas: 2
  resources:
    requests:
      cpu: 100m
      memory: 128Mi
    limits:
      cpu: 500m
      memory: 256Mi

server:
  image:
    repository: hashicorp/vault
    tag: "1.17.6"

  resources:
    requests:
      cpu: 250m
      memory: 256Mi
    limits:
      cpu: "1"
      memory: 512Mi

  extraEnvironmentVars:
    VAULT_CACERT: /vault/tls/ca.crt
    VAULT_TLSCERT: /vault/tls/tls.crt
    VAULT_TLSKEY: /vault/tls/tls.key
    AWS_REGION: ap-northeast-2

  volumes:
    - name: vault-tls
      secret:
        secretName: vault-tls
  volumeMounts:
    - name: vault-tls
      mountPath: /vault/tls
      readOnly: true

  serviceAccount:
    create: true
    name: vault
    annotations:
      eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/vault-autounseal

  readinessProbe:
    enabled: true
    path: "/v1/sys/health?standbyok=true&perfstandbyok=true&uninitcode=204"
    port: 8200
    scheme: HTTPS
    failureThreshold: 2
    initialDelaySeconds: 5
    periodSeconds: 5
    timeoutSeconds: 3

  livenessProbe:
    enabled: true
    path: "/v1/sys/health?standbyok=true&sealedcode=204&uninitcode=204"
    port: 8200
    scheme: HTTPS
    failureThreshold: 3
    initialDelaySeconds: 60
    periodSeconds: 30
    timeoutSeconds: 3

  dataStorage:
    enabled: true
    size: 20Gi
    storageClass: ebs-gp3
    accessMode: ReadWriteOnce
    mountPath: /vault/data

  auditStorage:
    enabled: true
    size: 10Gi
    storageClass: ebs-gp3
    accessMode: ReadWriteOnce
    mountPath: /vault/audit

  service:
    enabled: true
    type: ClusterIP
    port: 8200
    targetPort: 8200

  ha:
    enabled: true
    replicas: 3
    apiAddr: "https://$(POD_IP):8200"
    clusterAddr: "https://$(HOSTNAME).vault-internal:8201"
    raft:
      enabled: true
      setNodeId: true
      config: |
        ui = true

        listener "tcp" {
          address         = "[::]:8200"
          cluster_address = "[::]:8201"
          tls_cert_file   = "/vault/tls/tls.crt"
          tls_key_file    = "/vault/tls/tls.key"
          tls_min_version = "tls13"
        }

        storage "raft" {
          path    = "/vault/data"

          retry_join {
            leader_api_addr         = "https://vault-0.vault-internal:8200"
            leader_ca_cert_file     = "/vault/tls/ca.crt"
            leader_client_cert_file = "/vault/tls/tls.crt"
            leader_client_key_file  = "/vault/tls/tls.key"
          }
          retry_join {
            leader_api_addr         = "https://vault-1.vault-internal:8200"
            leader_ca_cert_file     = "/vault/tls/ca.crt"
            leader_client_cert_file = "/vault/tls/tls.crt"
            leader_client_key_file  = "/vault/tls/tls.key"
          }
          retry_join {
            leader_api_addr         = "https://vault-2.vault-internal:8200"
            leader_ca_cert_file     = "/vault/tls/ca.crt"
            leader_client_cert_file = "/vault/tls/tls.crt"
            leader_client_key_file  = "/vault/tls/tls.key"
          }
        }

        seal "awskms" {
          region     = "ap-northeast-2"
          kms_key_id = "alias/vault-autounseal"
        }

        service_registration "kubernetes" {}

        telemetry {
          prometheus_retention_time = "24h"
          disable_hostname          = true
        }

  affinity: |
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchLabels:
              app.kubernetes.io/name: vault
              component: server
          topologyKey: kubernetes.io/hostname

  topologySpreadConstraints: |
    - maxSkew: 1
      topologyKey: topology.kubernetes.io/zone
      whenUnsatisfiable: ScheduleAnyway
      labelSelector:
        matchLabels:
          app.kubernetes.io/name: vault
          component: server

왜 좋은가:

ha.enabled=true + raft.enabled=true + raft.setNodeId=true 3종 필수 플래그
listener와 storage raft stanza가 8200/8201 모두 바인드, cluster_address 명시 → peer replication 성립
seal "awskms"로 auto-unseal, Pod 재시작 시 수동 개입 불필요
auditStorage.enabled=true → audit 전용 PVC 분리 (dataStorage 오염 방지)
IRSA(eks.amazonaws.com/role-arn)로 KMS 접근 권한 위임 (static IAM key 없음)

나쁜 예시 1: chart 기본값 standalone

helm install vault hashicorp/vault --namespace vault --create-namespace

문제:

기본은 standalone + file storage → single pod, PVC 1개, HA 없음, snapshot restore로만 복구
Shamir 수동 unseal → pod 재시작마다 운영자 개입
audit device 미활성 → 감사 로그 없음
chart 문서 자체가 "not suitable for production"이라 명시

좋은 예시 2: Service — 8200 + 8201 둘 다 expose

Helm chart가 자동 생성하지만, 수제 Service 예시:

---
apiVersion: v1
kind: Service
metadata:
  name: vault
  namespace: vault
  labels:
    app.kubernetes.io/name: vault
    app.kubernetes.io/instance: vault-prod
spec:
  type: ClusterIP
  selector:
    app.kubernetes.io/name: vault
    component: server
  ports:
    - name: https
      port: 8200
      targetPort: 8200
      protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
  name: vault-internal
  namespace: vault
  labels:
    app.kubernetes.io/name: vault
spec:
  type: ClusterIP
  clusterIP: None
  publishNotReadyAddresses: true
  selector:
    app.kubernetes.io/name: vault
    component: server
  ports:
    - name: https
      port: 8200
      targetPort: 8200
    - name: https-internal
      port: 8201
      targetPort: 8201

왜 좋은가:

vault-internal headless + publishNotReadyAddresses: true → Raft peer가 unseal 전에도 서로 발견 가능
8201 포트 expose → peer-to-peer Raft replication 성립 (누락 시 leader election 영구 실패)
사용자용 vault Service는 8200만 노출

나쁜 예시 2: 8201 누락

spec:
  ports:
    - port: 8200
      targetPort: 8200

문제:

Raft peer가 8201로 서로 통신해야 하는데 Service가 expose하지 않음
vault operator raft list-peers에서 follower가 리더로 못 붙음
증상: 단일 노드만 unsealed, 나머지는 "storage: IO error" 로그 루프

좋은 예시 3: Kubernetes auth bootstrap + role

# 1. Kubernetes auth method 활성화
vault auth enable kubernetes

# 2. Vault가 Kubernetes TokenReview API를 호출하기 위한 설정
# (Vault Pod 내부에서 실행하거나 reviewer SA의 JWT를 주입)
vault write auth/kubernetes/config \
  token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
  kubernetes_host="https://kubernetes.default.svc.cluster.local" \
  kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt \
  disable_iss_validation=false

# 3. Policy 생성 (auth-server가 읽을 수 있는 경로만)
vault policy write auth-server-read - <<'EOF'
path "kv/data/auth-server/*" {
  capabilities = ["read"]
}
path "database/creds/auth-server" {
  capabilities = ["read"]
}
EOF

# 4. Role 생성 — 특정 SA + namespace에만 바인딩
vault write auth/kubernetes/role/auth-server \
  bound_service_account_names=auth-server \
  bound_service_account_namespaces=auth-prod \
  policies=auth-server-read \
  ttl=1h \
  max_ttl=24h \
  audience=vault

왜 좋은가:

TokenReview JWT를 명시적으로 구성 → Vault가 SA 토큰 유효성 검증 가능
Policy는 kv/data/auth-server/*, database/creds/auth-server만 허용 (최소 권한)
Role은 auth-prod namespace의 auth-server SA에만 바인딩
audience=vault로 projected token의 audience 검증 (token confusion 방어)

나쁜 예시 3: wildcard role

vault write auth/kubernetes/role/all-apps \
  bound_service_account_names="*" \
  bound_service_account_namespaces="*" \
  policies=default \
  ttl=720h

문제:

모든 namespace의 모든 SA가 로그인 가능 → 한 워크로드 침해가 전체 Vault 접근으로 확대
TTL 30일은 token revocation window가 너무 김
default policy가 넓으면 실질적인 접근 제어 상실

좋은 예시 4: VSO — 클러스터 수준 연결 + 앱 namespace auth

---
apiVersion: v1
kind: Namespace
metadata:
  name: vault-secrets-operator-system
---
# 1) 클러스터 전체 1개 VaultConnection
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultConnection
metadata:
  name: default
  namespace: vault-secrets-operator-system
spec:
  address: https://vault.vault.svc.cluster.local:8200
  tlsServerName: vault.vault.svc.cluster.local
  caCertSecretRef: vault-ca
  skipTLSVerify: false
  timeout: 60s
---
# 2) 앱 namespace의 ServiceAccount
apiVersion: v1
kind: ServiceAccount
metadata:
  name: auth-server
  namespace: auth-prod
---
# 3) 앱 namespace의 VaultAuth (Vault Kubernetes auth role로 로그인)
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultAuth
metadata:
  name: default
  namespace: auth-prod
spec:
  vaultConnectionRef: vault-secrets-operator-system/default
  method: kubernetes
  mount: kubernetes
  kubernetes:
    role: auth-server
    serviceAccount: auth-server
    audiences:
      - vault
    tokenExpirationSeconds: 600

왜 좋은가:

VaultConnection 1개를 operator namespace에 두고, 앱 namespace에서 cross-reference
VaultAuth.method: kubernetes가 Vault의 auth/kubernetes/role/auth-server를 호출
audiences: [vault]로 projected SA token의 audience 바인딩
tokenExpirationSeconds: 600 → projected token 10분마다 rotate

좋은 예시 5: VSO — Static / Dynamic / PKI secret

---
# KV v2에서 정적 secret 동기화
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
  name: auth-server-config
  namespace: auth-prod
spec:
  vaultAuthRef: default
  mount: kv
  path: auth-server/config
  type: kv-v2
  refreshAfter: 30m
  hmacSecretData: true
  rolloutRestartTargets:
    - kind: Deployment
      name: auth-server
  destination:
    name: auth-server-config
    create: true
    overwrite: true
---
# Postgres dynamic credential (TTL 1h)
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultDynamicSecret
metadata:
  name: auth-server-db
  namespace: auth-prod
spec:
  vaultAuthRef: default
  mount: database
  path: creds/auth-server
  renewalPercent: 67
  rolloutRestartTargets:
    - kind: Deployment
      name: auth-server
  destination:
    name: auth-server-db
    create: true
    overwrite: true
    transformation:
      excludeRaw: true
      templates:
        DATABASE_URL:
          text: 'postgresql://{{ .Secrets.username }}:{{ .Secrets.password }}@auth-db-rw.auth-prod.svc.cluster.local:5432/authdb?sslmode=require'
---
# PKI 인증서 발급
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultPKISecret
metadata:
  name: auth-server-cert
  namespace: auth-prod
spec:
  vaultAuthRef: default
  mount: pki_int
  role: auth-server
  commonName: auth-server.auth-prod.svc.cluster.local
  altNames:
    - auth-server
    - auth-server.auth-prod
  ipSans: []
  ttl: 720h
  revoke: true
  clear: true
  expiryOffset: 120h
  destination:
    name: auth-server-cert
    create: true
    type: kubernetes.io/tls

왜 좋은가:

세 패턴(정적 KV, 동적 DB credential, PKI cert)을 한 namespace에서 일관되게 선언
rolloutRestartTargets로 secret 갱신 시 consumer Deployment 자동 롤링 재시작
renewalPercent: 67 → TTL 67% 경과 시 갱신 (default는 보통 70%)
transformation.templates로 연결 문자열 포맷 변환 (앱이 username/password 파싱 안 해도 됨)

좋은 예시 6: Vault Agent Injector (init-only 모드)

apiVersion: apps/v1
kind: Deployment
metadata:
  name: legacy-app
  namespace: legacy
spec:
  replicas: 2
  selector:
    matchLabels:
      app: legacy-app
  template:
    metadata:
      labels:
        app: legacy-app
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/role: "legacy-app"
        vault.hashicorp.com/agent-pre-populate-only: "true"
        vault.hashicorp.com/agent-inject-secret-db.env: "database/creds/legacy-app"
        vault.hashicorp.com/agent-inject-template-db.env: |
          {{ with secret "database/creds/legacy-app" -}}
          DATABASE_USERNAME={{ .Data.username }}
          DATABASE_PASSWORD={{ .Data.password }}
          {{- end }}
        vault.hashicorp.com/agent-inject-file-db.env: "db.env"
        vault.hashicorp.com/agent-limits-cpu: "200m"
        vault.hashicorp.com/agent-limits-mem: "128Mi"
        vault.hashicorp.com/agent-requests-cpu: "50m"
        vault.hashicorp.com/agent-requests-mem: "64Mi"
    spec:
      serviceAccountName: legacy-app
      securityContext:
        runAsNonRoot: true
        runAsUser: 10001
        runAsGroup: 10001
        fsGroup: 10001
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: app
          image: registry.example.com/legacy-app:1.4.2
          command: ["sh", "-c", "source /vault/secrets/db.env && exec /app/run"]
          resources:
            requests: { cpu: 100m, memory: 128Mi }
            limits:   { memory: 256Mi }
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop: ["ALL"]
          volumeMounts:
            - { name: tmp, mountPath: /tmp }
      volumes:
        - name: tmp
          emptyDir: {}

왜 좋은가:

agent-pre-populate-only: "true" → init container만 돌고 sidecar 없음 → 2 pod 당 컨테이너 1개 절감
etcd에 Kubernetes Secret 생성 없음 (annotation에 명시적 destination 없음; in-memory volume)
template로 .env 포맷 렌더링 → legacy 앱이 그대로 소비

나쁜 예시 4: Injector + long-lived sidecar + 무한 renew

annotations:
  vault.hashicorp.com/agent-inject: "true"
  vault.hashicorp.com/role: "legacy-app"
  vault.hashicorp.com/agent-inject-secret-creds: "database/creds/legacy-app"
  # agent-pre-populate-only 없음 → sidecar 상시 실행
  # agent-limits-* 없음 → sidecar가 limit 없이 메모리 증가

문제:

sidecar가 Pod 수명 내내 상주 → 1000 서비스 x 3 replica = 3000 추가 컨테이너
resource limit 미지정 → OOM cascading
VSO로 대체 가능한데 Injector를 default로 쓰면 운영 복잡도 증가

좋은 예시 7: Raft snapshot CronJob

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: vault-snapshot
  namespace: vault
---
apiVersion: batch/v1
kind: CronJob
metadata:
  name: vault-raft-snapshot
  namespace: vault
spec:
  schedule: "0 2 * * *"
  concurrencyPolicy: Forbid
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 3
  jobTemplate:
    spec:
      backoffLimit: 2
      template:
        spec:
          restartPolicy: OnFailure
          serviceAccountName: vault-snapshot
          securityContext:
            runAsNonRoot: true
            runAsUser: 100
            runAsGroup: 1000
            fsGroup: 1000
            seccompProfile:
              type: RuntimeDefault
          containers:
            - name: snapshot
              image: hashicorp/vault:1.17.6
              env:
                - name: VAULT_ADDR
                  value: https://vault.vault.svc.cluster.local:8200
                - name: VAULT_CACERT
                  value: /vault/tls/ca.crt
                - name: VAULT_TOKEN
                  valueFrom:
                    secretKeyRef:
                      name: vault-snapshot-token
                      key: token
                - name: AWS_REGION
                  value: ap-northeast-2
              command:
                - sh
                - -c
                - |
                  set -eu
                  TS=$(date -u +%Y%m%dT%H%M%SZ)
                  SNAP=/tmp/vault-${TS}.snap
                  vault operator raft snapshot save "${SNAP}"
                  aws s3 cp "${SNAP}" "s3://vault-backup.example.com/daily/vault-${TS}.snap" \
                    --sse aws:kms --sse-kms-key-id alias/vault-backup
                  rm -f "${SNAP}"
              volumeMounts:
                - name: vault-tls
                  mountPath: /vault/tls
                  readOnly: true
                - name: tmp
                  mountPath: /tmp
              resources:
                requests:
                  cpu: 100m
                  memory: 128Mi
                limits:
                  cpu: 500m
                  memory: 512Mi
              securityContext:
                allowPrivilegeEscalation: false
                readOnlyRootFilesystem: true
                capabilities:
                  drop: ["ALL"]
          volumes:
            - name: vault-tls
              secret:
                secretName: vault-tls
            - name: tmp
              emptyDir: {}

왜 좋은가:

매일 02:00 UTC raft snapshot save 실행
결과를 S3 SSE-KMS로 off-cluster 보관 (PVC와 독립적 failure domain)
짧은 TTL snapshot token을 별도 Secret로 주입 (root token 미사용)
concurrencyPolicy: Forbid로 snapshot 중복 방지

좋은 예시 8: ServiceMonitor + Prometheus policy

---
# Vault policy: Prometheus가 /v1/sys/metrics 읽기 전용
# (이 정책은 Vault 내부에 생성)
# vault policy write prometheus-metrics - <<EOF
# path "sys/metrics" { capabilities = ["read"] }
# EOF
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: vault
  namespace: vault
  labels:
    app.kubernetes.io/name: vault
    release: kube-prometheus-stack
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: vault
  endpoints:
    - port: https
      scheme: https
      path: /v1/sys/metrics
      params:
        format:
          - prometheus
      interval: 30s
      scrapeTimeout: 10s
      bearerTokenSecret:
        name: prometheus-vault-token
        key: token
      tlsConfig:
        ca:
          secret:
            name: vault-ca
            key: ca.crt
        serverName: vault.vault.svc.cluster.local

왜 좋은가:

telemetry { prometheus_retention_time = "24h" } stanza와 매칭
Prometheus가 전용 Vault token으로 sys/metrics만 read (최소 권한)
TLS serverName 명시로 hostname 검증

나쁜 예시 5: Vault Ingress 외부 공개

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: vault
spec:
  rules:
    - host: vault.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: vault
                port:
                  number: 8200

문제:

Vault는 일반 외부 서비스가 아님 — /v1/auth/*, /v1/sys/* 가 인터넷에 노출되면 brute-force / DoS 표면 확대
root token / unseal key가 UI에서 한 번이라도 취급되면 공격 가치가 매우 큼
관리자 접근은 VPN / port-forward / OIDC 보호된 별도 bastion 경로로

좋은 예시 9: PodDisruptionBudget + Restricted securityContext

---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: vault
  namespace: vault
spec:
  minAvailable: 2
  unhealthyPodEvictionPolicy: AlwaysAllow
  selector:
    matchLabels:
      app.kubernetes.io/name: vault
      component: server

그리고 values.yaml에서:

server:
  statefulSet:
    securityContext:
      pod:
        runAsNonRoot: true
        runAsUser: 100
        runAsGroup: 1000
        fsGroup: 1000
        seccompProfile:
          type: RuntimeDefault
      container:
        allowPrivilegeEscalation: false
        readOnlyRootFilesystem: true
        capabilities:
          drop:
            - ALL
          add:
            - IPC_LOCK

왜 좋은가:

Raft 3-node quorum 유지: minAvailable: 2 → 한 번에 1 pod만 drain 가능
IPC_LOCK capability는 Vault의 mlockall을 허용 (swap으로 secret 유출 방지) — 그 외 capability 전부 drop
Restricted PSS 전체 충족

20 KiB Raw Blame History

Vault 예시

좋은 예시 1: Helm values.yaml — HA Raft + auto-unseal + audit

나쁜 예시 1: chart 기본값 standalone

좋은 예시 2: Service — 8200 + 8201 둘 다 expose

나쁜 예시 2: 8201 누락

좋은 예시 3: Kubernetes auth bootstrap + role

나쁜 예시 3: wildcard role

좋은 예시 4: VSO — 클러스터 수준 연결 + 앱 namespace auth

좋은 예시 5: VSO — Static / Dynamic / PKI secret

좋은 예시 6: Vault Agent Injector (init-only 모드)

나쁜 예시 4: Injector + long-lived sidecar + 무한 renew

좋은 예시 7: Raft snapshot CronJob

좋은 예시 8: ServiceMonitor + Prometheus policy

나쁜 예시 5: Vault Ingress 외부 공개

좋은 예시 9: PodDisruptionBudget + Restricted securityContext

20 KiB

Raw Blame History