project-infra/docs/examples/infra/config-and-secrets.md

config / secrets 예시

모든 YAML은 kubectl apply --dry-run=server -f - 기준 clean. VSO는 Helm chart hashicorp/vault-secrets-operator로 vault-secrets-operator namespace에 설치되어 있고, Vault는 vault namespace(https://vault.vault.svc:8200)에서 기동 중이며, Kubernetes auth method(auth/kubernetes)가 활성화되어 있다고 가정한다.

---

좋은 예시 1: 비기밀 ConfigMap (hash-suffixed by Kustomize)

apiVersion: v1
kind: ConfigMap
metadata:
  name: auth-server-config
  namespace: auth-prod
  labels:
    app.kubernetes.io/name: auth-server
    app.kubernetes.io/part-of: identity-platform
data:
  application.yaml: |
    server:
      port: 8080
      shutdown: graceful
    management:
      endpoints:
        web:
          base-path: /actuator
          exposure:
            include: health,info,prometheus
      server:
        port: 9090
    spring:
      main:
        banner-mode: off
      datasource:
        hikari:
          maximum-pool-size: 20
          connection-timeout: 5000
    logging:
      level:
        root: INFO
        com.example.auth: INFO

왜 좋은가:

• 비밀값은 하나도 없다(username/password/url 제외). Hikari pool size, log level, actuator 경로 같은 operational config만.
• Kustomize configMapGenerator로 hash suffix를 붙이면 Deployment가 자동 rollout.

---

좋은 예시 2: VSO 전체 스택 (VaultConnection + VaultAuth + VaultStaticSecret + VaultDynamicSecret + VaultPKISecret)

apiVersion: v1
kind: Namespace
metadata:
  name: auth-prod
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/enforce-version: v1.29
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: auth-server
  namespace: auth-prod
automountServiceAccountToken: true
---
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultConnection
metadata:
  name: vault
  namespace: auth-prod
spec:
  address: https://vault.vault.svc:8200
  skipTLSVerify: false
  caCertSecretRef: vault-ca-bundle
  headers: {}
---
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultAuth
metadata:
  name: auth-server
  namespace: auth-prod
spec:
  vaultConnectionRef: vault
  method: kubernetes
  mount: kubernetes
  kubernetes:
    role: auth-server
    serviceAccount: auth-server
    audiences:
      - vault
---
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
  name: auth-server-oidc-client
  namespace: auth-prod
spec:
  vaultAuthRef: auth-server
  mount: kv
  type: kv-v2
  path: identity/auth-server/prod/oidc
  refreshAfter: 1h
  destination:
    name: auth-server-oidc-client
    create: true
    type: Opaque
  rolloutRestartTargets:
    - kind: Deployment
      name: auth-server
---
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultDynamicSecret
metadata:
  name: auth-server-db
  namespace: auth-prod
spec:
  vaultAuthRef: auth-server
  mount: database
  path: creds/auth-server-role
  destination:
    name: auth-server-db
    create: true
    type: Opaque
    transformation:
      templates:
        DB_URL:
          text: 'jdbc:postgresql://identity-postgres.data-prod.svc:5432/auth?user={{ .Secrets.username }}&password={{ .Secrets.password }}&sslmode=require'
        DB_USERNAME:
          text: '{{ .Secrets.username }}'
        DB_PASSWORD:
          text: '{{ .Secrets.password }}'
  rolloutRestartTargets:
    - kind: Deployment
      name: auth-server
---
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultPKISecret
metadata:
  name: auth-server-internal-tls
  namespace: auth-prod
spec:
  vaultAuthRef: auth-server
  mount: pki_int
  role: auth-server
  commonName: auth-server.auth-prod.svc
  altNames:
    - auth-server.auth-prod.svc.cluster.local
    - auth-server
  ttl: 24h
  destination:
    name: auth-server-internal-tls
    create: true
    type: kubernetes.io/tls
  rolloutRestartTargets:
    - kind: Deployment
      name: auth-server

왜 좋은가:

• Vault가 source of truth. 모든 비밀이 kv/identity/auth-server/prod/* 또는 database/PKI engine에서 발급.
• VSO가 결과물을 표준 Kubernetes Secret(Opaque, kubernetes.io/tls)으로 materialize.
• Dynamic DB credential은 Postgres role에서 TTL 기반 자동 발급/폐기. Rotation 시 rolloutRestartTargets로 Deployment rolling restart.
• PKI Secret은 kubernetes.io/tls 타입 → Traefik/앱 TLS에 그대로 소비 가능.

---

좋은 예시 3: VSO Secret을 소비하는 Deployment (envFrom + volume 혼합)

apiVersion: apps/v1
kind: Deployment
metadata:
  name: auth-server
  namespace: auth-prod
  labels:
    app.kubernetes.io/name: auth-server
    app.kubernetes.io/part-of: identity-platform
    app.kubernetes.io/version: 1.42.0
    app.kubernetes.io/managed-by: argocd
spec:
  replicas: 6
  revisionHistoryLimit: 5
  selector:
    matchLabels:
      app.kubernetes.io/name: auth-server
      app.kubernetes.io/instance: auth-server
  template:
    metadata:
      labels:
        app.kubernetes.io/name: auth-server
        app.kubernetes.io/instance: auth-server
        app.kubernetes.io/part-of: identity-platform
        app.kubernetes.io/version: 1.42.0
    spec:
      serviceAccountName: auth-server
      automountServiceAccountToken: true
      securityContext:
        runAsNonRoot: true
        runAsUser: 10001
        runAsGroup: 10001
        fsGroup: 10001
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: auth-server
          image: registry.example.com/identity/auth-server@sha256:8f3c0a8c6b3a2a7a0f1b2c3d4e5f60718293a4b5c6d7e8f90a1b2c3d4e5f6071
          imagePullPolicy: IfNotPresent
          ports:
            - name: http
              containerPort: 8080
            - name: metrics
              containerPort: 9090
          envFrom:
            - configMapRef:
                name: auth-server-config
            - secretRef:
                name: auth-server-db
            - secretRef:
                name: auth-server-oidc-client
          volumeMounts:
            - name: internal-tls
              mountPath: /var/run/secrets/tls
              readOnly: true
            - name: appconfig
              mountPath: /workspace/config
              readOnly: true
            - name: tmp
              mountPath: /tmp
          resources:
            requests:
              cpu: 200m
              memory: 512Mi
            limits:
              cpu: "2"
              memory: 1Gi
          readinessProbe:
            httpGet:
              path: /actuator/health/readiness
              port: http
          livenessProbe:
            httpGet:
              path: /actuator/health/liveness
              port: http
          securityContext:
            runAsNonRoot: true
            runAsUser: 10001
            runAsGroup: 10001
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop: ["ALL"]
            seccompProfile:
              type: RuntimeDefault
        - name: metrics-exporter
          image: registry.example.com/platform/jmx-exporter@sha256:1111111111111111111111111111111111111111111111111111111111111111
          ports:
            - name: jmx-metrics
              containerPort: 9091
          resources:
            requests:
              cpu: 50m
              memory: 64Mi
            limits:
              cpu: 200m
              memory: 128Mi
          securityContext:
            runAsNonRoot: true
            runAsUser: 10001
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop: ["ALL"]
            seccompProfile:
              type: RuntimeDefault
      volumes:
        - name: internal-tls
          secret:
            secretName: auth-server-internal-tls
            defaultMode: 0400
        - name: appconfig
          configMap:
            name: auth-server-config
        - name: tmp
          emptyDir:
            medium: Memory
            sizeLimit: 64Mi
      imagePullSecrets:
        - name: registry-example-com

왜 좋은가:

• VSO가 생성한 auth-server-db, auth-server-oidc-client를 envFrom으로 소비. 앱 코드는 DB_USERNAME, DB_PASSWORD, OIDC_CLIENT_SECRET 환경변수를 읽기만 함.
• TLS private key는 volume(/var/run/secrets/tls, mode 0400)으로만 마운트. env 노출 없음.
• metrics-exporter sidecar에는 어떤 secret도 envFrom/volumeMount로 전달하지 않는다. Scope 최소화.
• image는 digest pin, imagePullPolicy: IfNotPresent.

---

나쁜 예시 1: plain Secret manifest + ConfigMap에 비밀 혼재

apiVersion: v1
kind: Secret
metadata:
  name: auth-server-db
  namespace: auth-prod
type: Opaque
stringData:
  username: prod-admin
  password: S3cur3P@ssw0rd!
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: auth-server-config
  namespace: auth-prod
data:
  application.yaml: |
    spring:
      datasource:
        url: jdbc:postgresql://prod-db:5432/auth
        username: prod-admin
        password: S3cur3P@ssw0rd!

문제:

• 운영 비밀이 Git에 평문으로 커밋된다. base64/stringData 여부와 무관.
• ConfigMap에 password가 들어가 있음 → RBAC configmaps:get 권한을 가진 모든 SA가 읽을 수 있음.
• secret source가 두 곳에 있어 회전 불가능.
• VSO/ESO/SealedSecrets 어느 경로에도 부합하지 않음.

---

좋은 예시 4: ImagePullSecret을 VSO로 Vault에서 sync

apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
  name: registry-example-com
  namespace: auth-prod
spec:
  vaultAuthRef: auth-server
  mount: kv
  type: kv-v2
  path: platform/registry/example-com
  refreshAfter: 24h
  destination:
    name: registry-example-com
    create: true
    type: kubernetes.io/dockerconfigjson
    transformation:
      templates:
        .dockerconfigjson:
          text: |
            {
              "auths": {
                "registry.example.com": {
                  "username": "{{ .Secrets.username }}",
                  "password": "{{ .Secrets.password }}",
                  "auth": "{{ printf "%s:%s" .Secrets.username .Secrets.password | b64enc }}"
                }
              }
            }
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: auth-server
  namespace: auth-prod
automountServiceAccountToken: true
imagePullSecrets:
  - name: registry-example-com

왜 좋은가:

• registry credential도 Vault가 SoT. 하드코딩 없음.
• VSO가 kubernetes.io/dockerconfigjson 타입 Secret을 생성. kubelet이 바로 인식.
• SA에 묶여 있어 Deployment마다 imagePullSecrets 반복 선언 불필요.

---

좋은 예시 5: cert-manager + VSO 비교 — Ingress TLS는 cert-manager, internal mTLS는 VSO PKI

cert-manager가 외부 공인 도메인용 kubernetes.io/tls Secret을 발급하고, VSO VaultPKISecret은 internal service mesh mTLS용 단기 인증서를 발급한다. 두 경로 모두 최종 형태는 kubernetes.io/tls Secret으로 동일하므로 앱은 secret name만 구분한다.

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: auth-example-com
  namespace: auth-prod
spec:
  secretName: auth-example-com-tls
  issuerRef:
    kind: ClusterIssuer
    name: letsencrypt-prod
  dnsNames:
    - auth.example.com
  duration: 2160h
  renewBefore: 360h
---
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultPKISecret
metadata:
  name: auth-server-internal-tls
  namespace: auth-prod
spec:
  vaultAuthRef: auth-server
  mount: pki_int
  role: auth-server
  commonName: auth-server.auth-prod.svc
  ttl: 24h
  destination:
    name: auth-server-internal-tls
    create: true
    type: kubernetes.io/tls
  rolloutRestartTargets:
    - kind: Deployment
      name: auth-server

왜 좋은가:

• 외부 ACME 인증서는 공인 CA(Let's Encrypt), 내부는 조직 CA(Vault PKI)로 분리.
• 둘 다 같은 Secret 타입이라 Traefik/앱이 동일하게 소비 가능.
• VSO PKI는 24h TTL로 짧게 회전 → lateral movement window 최소화.

---

좋은 예시 6: Vault Agent Injector가 K8s Secret 없이 파일로 템플릿 렌더링

apiVersion: apps/v1
kind: Deployment
metadata:
  name: legacy-report-generator
  namespace: reports-prod
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: report-generator
  template:
    metadata:
      labels:
        app.kubernetes.io/name: report-generator
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/role: "report-generator"
        vault.hashicorp.com/agent-inject-secret-report.conf: "kv/data/reports/smtp"
        vault.hashicorp.com/agent-inject-template-report.conf: |
          {{- with secret "kv/data/reports/smtp" -}}
          [smtp]
          host = {{ .Data.data.host }}
          port = {{ .Data.data.port }}
          username = {{ .Data.data.username }}
          password = {{ .Data.data.password }}
          {{- end }}
        vault.hashicorp.com/secret-volume-path-report.conf: "/vault/secrets"
        vault.hashicorp.com/agent-inject-containers: "report-generator"
        vault.hashicorp.com/agent-run-as-user: "10001"
        vault.hashicorp.com/agent-run-as-group: "10001"
    spec:
      serviceAccountName: report-generator
      automountServiceAccountToken: true
      securityContext:
        runAsNonRoot: true
        runAsUser: 10001
        runAsGroup: 10001
        fsGroup: 10001
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: report-generator
          image: registry.example.com/reports/generator@sha256:2222222222222222222222222222222222222222222222222222222222222222
          ports:
            - name: http
              containerPort: 8080
          volumeMounts:
            - name: tmp
              mountPath: /tmp
          resources:
            requests:
              cpu: 100m
              memory: 256Mi
            limits:
              cpu: 500m
              memory: 512Mi
          securityContext:
            runAsNonRoot: true
            runAsUser: 10001
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop: ["ALL"]
            seccompProfile:
              type: RuntimeDefault
      volumes:
        - name: tmp
          emptyDir:
            medium: Memory
            sizeLimit: 32Mi

왜 좋은가:

• K8s Secret object가 생성되지 않는다. RBAC audit이 Secret API 호출 없이 Vault audit log로 대체된다.
• Vault Agent sidecar가 tmpfs에 템플릿 렌더링 → 앱은 파일만 읽음.
• legacy 앱이 INI/TOML 포맷 설정 파일을 요구할 때 적합.

VSO vs Vault Agent Injector:

항목	VSO	Vault Agent Injector
결과	K8s Secret	Pod tmpfs 파일
K8s API 노출	Secret object 존재	없음
소비 방식	envFrom/volume	file read
회전 시	rolloutRestartTargets	Agent re-render(인메모리)
복잡도	낮음(CRD만)	높음(sidecar/init)
권장	운영 기본	템플릿/legacy 앱

---

나쁜 예시 2: Vault Injector annotation을 모든 컨테이너에 적용 + env 렌더링

metadata:
  annotations:
    vault.hashicorp.com/agent-inject: "true"
    vault.hashicorp.com/role: "auth-server"
    vault.hashicorp.com/agent-inject-secret-db: "kv/data/auth-server/db"
    vault.hashicorp.com/agent-inject-template-db: |
      {{- with secret "kv/data/auth-server/db" -}}
      export DB_USERNAME={{ .Data.data.username }}
      export DB_PASSWORD={{ .Data.data.password }}
      {{- end }}

문제:

• agent-inject-containers 미지정 → sidecar(metrics, proxy) 포함 모든 컨테이너의 /vault/secrets가 보임.
• export DB_PASSWORD=...를 source로 읽는 launcher 스크립트 → process env로 비밀이 흘러 /proc/<pid>/environ 노출.
• dynamic lease renew를 활용하지 못하고, 회전 시 rollout trigger 없음.

---

좋은 예시 7: SealedSecret (VSO 미도입 환경/bootstrap)

apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  name: vault-bootstrap-token
  namespace: vault
spec:
  encryptedData:
    token: AgCd9sK... (public key로 암호화된 blob)
  template:
    metadata:
      name: vault-bootstrap-token
      namespace: vault
    type: Opaque

왜 좋은가:

• Git 커밋 가능(public key로 암호화, cluster controller만 복호화).
• VSO 자체를 기동하기 위한 bootstrap credential(Vault root token, unseal key 대신 KMS auto-unseal 권장)에 적합.
• SealedSecrets controller가 Secret을 namespace에 materialize.

주의:

• 운영에서 VSO가 기동되면 SealedSecrets 경로는 최소화. 이중 source of truth 방지.
• Key 회전은 controller의 sealing key rotation 절차 준수.

---

좋은 예시 8: EncryptionConfiguration for Secret at-rest (API Server 레벨)

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - kms:
          apiVersion: v2
          name: platform-kms-v2
          endpoint: unix:///var/run/kmsplugin/socket.sock
          timeout: 3s
      - aescbc:
          keys:
            - name: fallback-2026-q1
              secret: c2VjcmV0LTMyLWJ5dGUtZmFsbGJhY2sta2V5LTIwMjZxMS1leGFtcGxl
      - identity: {}

K3s 활성화:

# /etc/rancher/k3s/config.yaml
secrets-encryption: true
kube-apiserver-arg:
  - "encryption-provider-config=/etc/rancher/k3s/encryption-config.yaml"
  - "audit-policy-file=/etc/rancher/k3s/audit-policy.yaml"
  - "audit-log-path=/var/log/k3s-audit.log"

왜 좋은가:

• KMS v2 provider가 primary → envelope encryption, 키는 KMS 외부에 존재.
• aescbc는 fallback. identity는 마지막(평문), 기존 Secret을 재암호화하기 전 decryption용.
• K3s config.yaml에 secrets-encryption: true로 선언. 서버 재시작 후 kubectl get secrets -A -o json | kubectl replace -f -로 기존 Secret 재암호화.

---

나쁜 예시 3: Kustomize secretGenerator로 운영 비밀 literal

# overlays/prod/kustomization.yaml
secretGenerator:
  - name: auth-server-db
    literals:
      - username=prod-admin
      - password=S3cur3P@ssw0rd!

문제:

• 운영 비밀이 Git에 literal 평문 저장.
• Kustomize hash suffix는 비밀 보호가 아님.
• 회전 시 매번 Git 커밋 필요(감사/리뷰 시 비밀 노출).
• 운영은 VSO/ESO/SealedSecrets 경로로만 비밀을 배포해야 한다.