donghyeon.kang/project-infra
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
Files
be2f8e4863f9445782a52024e6a28ab4c6b9d679
project-infra/docs/examples/infra/kustomize.md
T

15 KiB
Raw Blame History

Kustomize 예시

모든 예시는 Kustomize v5 문법 기준. 렌더 검증:

kubectl kustomize <dir> | kubectl apply --server-side --field-manager=ci --dry-run=server -f -

좋은 예시 1: catalog unit의 base / components / overlays 구조

gitops/
  apps/
    identity-auth/
      base/
        kustomization.yaml
        deployment.yaml
        service.yaml
        servicemonitor.yaml
        pdb.yaml
        hpa.yaml
      components/
        with-topology-spread-zone/
          kustomization.yaml
          patch.yaml
        with-pdb-tier1/
          kustomization.yaml
          patch.yaml
      overlays/
        prod/kr-main/
          kustomization.yaml
          patches/
            auth-resources.yaml
            auth-ingress-host.yaml
  clusters/
    prod/kr-main/
      kustomization.yaml

# gitops/apps/identity-auth/base/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - deployment.yaml
  - service.yaml
  - servicemonitor.yaml
  - pdb.yaml
  - hpa.yaml
labels:
  - pairs:
      app.kubernetes.io/name: auth
      app.kubernetes.io/component: api
      app.kubernetes.io/part-of: identity-platform
    includeSelectors: false
    includeTemplates: true

왜 좋은가:

  • base가 환경·region을 모른다 (namespace / replicas / host / image tag 전부 없음)
  • labels: (v5) 사용, commonLabels 없음 → selector immutability 안전
  • includeTemplates: true로 Pod label에는 전파되어 observability 쿼리 가능
  • selector에 들어가는 label은 base의 Deployment 내부에서 명시적으로 고정

좋은 예시 2: base Deployment (완전 apply-ready)

# gitops/apps/identity-auth/base/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: auth
  labels:
    app.kubernetes.io/name: auth
    app.kubernetes.io/instance: auth
    app.kubernetes.io/component: api
    app.kubernetes.io/part-of: identity-platform
    app.kubernetes.io/managed-by: argocd
  annotations:
    example.com/owner-email: identity-sre@example.com
spec:
  replicas: 2
  revisionHistoryLimit: 5
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 0
  selector:
    matchLabels:
      app.kubernetes.io/name: auth
      app.kubernetes.io/instance: auth
      app.kubernetes.io/component: api
  template:
    metadata:
      labels:
        app.kubernetes.io/name: auth
        app.kubernetes.io/instance: auth
        app.kubernetes.io/component: api
        app.kubernetes.io/part-of: identity-platform
        app.kubernetes.io/managed-by: argocd
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "8081"
        prometheus.io/path: "/actuator/prometheus"
    spec:
      serviceAccountName: auth
      automountServiceAccountToken: false
      securityContext:
        runAsNonRoot: true
        runAsUser: 10001
        runAsGroup: 10001
        fsGroup: 10001
        seccompProfile:
          type: RuntimeDefault
      terminationGracePeriodSeconds: 45
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels:
              app.kubernetes.io/name: auth
              app.kubernetes.io/instance: auth
              app.kubernetes.io/component: api
      containers:
        - name: auth
          image: registry.example.com/auth:placeholder
          imagePullPolicy: IfNotPresent
          ports:
            - name: http
              containerPort: 8080
              protocol: TCP
            - name: management
              containerPort: 8081
              protocol: TCP
          env:
            - name: JAVA_TOOL_OPTIONS
              value: "-XX:MaxRAMPercentage=75 -XX:+UseG1GC"
          envFrom:
            - configMapRef:
                name: auth-config
            - secretRef:
                name: auth-secrets
          resources:
            requests:
              cpu: 250m
              memory: 512Mi
            limits:
              cpu: "2"
              memory: 1Gi
          startupProbe:
            httpGet:
              path: /actuator/health/liveness
              port: management
            periodSeconds: 5
            failureThreshold: 30
          livenessProbe:
            httpGet:
              path: /actuator/health/liveness
              port: management
            periodSeconds: 10
            timeoutSeconds: 3
            failureThreshold: 3
          readinessProbe:
            httpGet:
              path: /actuator/health/readiness
              port: management
            periodSeconds: 5
            timeoutSeconds: 2
            failureThreshold: 3
          securityContext:
            runAsNonRoot: true
            runAsUser: 10001
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop:
                - ALL
          volumeMounts:
            - name: tmp
              mountPath: /tmp
            - name: cache
              mountPath: /app/cache
      volumes:
        - name: tmp
          emptyDir:
            sizeLimit: 64Mi
        - name: cache
          emptyDir:
            sizeLimit: 256Mi

왜 좋은가:

  • image는 :placeholder, overlay의 images:가 digest로 patch → base는 버전 모름
  • revisionHistoryLimit: 5 → 대규모 cluster에서 ReplicaSet 누적 방지
  • PodSecurity restricted 호환 (non-root, seccomp RuntimeDefault, capabilities drop ALL, readOnlyRootFilesystem)
  • startup/liveness/readiness 3종이 타이밍 분리 (startup 150s, liveness 30s, readiness 15s 윈도우)
  • topologySpreadConstraints로 zone별 분산
  • automountServiceAccountToken: false (ServiceAccount token을 쓰지 않는 워크로드)

좋은 예시 3: overlay prod/kr-main — 환경 차이만

# gitops/apps/identity-auth/overlays/prod/kr-main/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: prod-identity-auth
resources:
  - ../../../base
components:
  - ../../../components/with-topology-spread-zone
  - ../../../components/with-pdb-tier1
labels:
  - pairs:
      example.com/environment: prod
      example.com/region: kr-main
      example.com/slo-tier: tier-1
    includeSelectors: false
    includeTemplates: true
images:
  - name: registry.example.com/auth
    digest: "sha256:f1a2b3c4d5e6f7081920aabbccddeeff00112233445566778899aabbccddeeff"
replicas:
  - name: auth
    count: 6
patches:
  - target:
      kind: Deployment
      name: auth
    path: patches/auth-resources.yaml
  - target:
      kind: Ingress
      name: auth-public
    patch: |-
      - op: replace
        path: /spec/rules/0/host
        value: auth.example.com
      - op: replace
        path: /spec/tls/0/hosts/0
        value: auth.example.com

# gitops/apps/identity-auth/overlays/prod/kr-main/patches/auth-resources.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: auth
  labels:
    app.kubernetes.io/name: auth
    app.kubernetes.io/instance: auth
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: auth
      app.kubernetes.io/instance: auth
  template:
    metadata:
      labels:
        app.kubernetes.io/name: auth
        app.kubernetes.io/instance: auth
    spec:
      securityContext:
        runAsNonRoot: true
        runAsUser: 10001
        runAsGroup: 10001
        fsGroup: 10001
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: auth
          image: registry.example.com/auth-server:1.24.0
          resources:
            requests:
              cpu: 500m
              memory: 1Gi
            limits:
              cpu: "2"
              memory: 2Gi
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop: ["ALL"]

왜 좋은가:

  • overlay 자체가 짧음 (base를 재작성하지 않음)
  • digest 기반 image pinning
  • components:로 zone spread + PDB tier-1을 재사용
  • labels: 사용, includeSelectors: false → selector immutability 안전
  • replicas override는 HPA minReplicas와 일치 (HPA base에서 minReplicas: 6으로 설정되어 있다고 가정)

좋은 예시 4: Kustomize Component — with-pdb-tier1

# gitops/apps/identity-auth/components/with-pdb-tier1/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1alpha1
kind: Component
resources:
  - pdb.yaml

# gitops/apps/identity-auth/components/with-pdb-tier1/pdb.yaml
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: auth
  labels:
    app.kubernetes.io/name: auth
    app.kubernetes.io/instance: auth
    app.kubernetes.io/component: api
    app.kubernetes.io/part-of: identity-platform
    app.kubernetes.io/managed-by: argocd
    example.com/slo-tier: tier-1
spec:
  minAvailable: 50%
  unhealthyPodEvictionPolicy: AlwaysAllow
  selector:
    matchLabels:
      app.kubernetes.io/name: auth
      app.kubernetes.io/instance: auth
      app.kubernetes.io/component: api

왜 좋은가:

  • kind: Component로 선언 → 여러 overlay에서 components: 키로 재사용
  • tier-1의 PDB 정책(50% minAvailable)이 단일 파일에 고정
  • 다른 tier는 별도 component (with-pdb-tier2, with-pdb-tier3)

좋은 예시 5: ConfigMap generator + hash suffix를 활용한 자동 rollout

# gitops/apps/identity-auth/base/kustomization.yaml (with generator)
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - deployment.yaml
  - service.yaml
configMapGenerator:
  - name: auth-config
    files:
      - application.yaml=config/application.yaml
      - logback.xml=config/logback.xml
    options:
      labels:
        app.kubernetes.io/name: auth
        app.kubernetes.io/component: config
generatorOptions:
  disableNameSuffixHash: false

왜 좋은가:

  • ConfigMap 내용 변경 시 hash suffix가 바뀜 → Deployment가 새 이름을 참조 → rolling update 자동 트리거
  • annotation 기반 "checksum" hack 불필요
  • Secret은 generator로 만들지 않고 External Secrets로 관리

좋은 예시 6: HPA v2 + behavior (base 리소스)

# gitops/apps/identity-auth/base/hpa.yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: auth
  labels:
    app.kubernetes.io/name: auth
    app.kubernetes.io/instance: auth
    app.kubernetes.io/component: api
    app.kubernetes.io/part-of: identity-platform
    app.kubernetes.io/managed-by: argocd
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: auth
  minReplicas: 2
  maxReplicas: 20
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70
    - type: Resource
      resource:
        name: memory
        target:
          type: Utilization
          averageUtilization: 80
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
        - type: Percent
          value: 25
          periodSeconds: 60
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
        - type: Percent
          value: 100
          periodSeconds: 30
        - type: Pods
          value: 4
          periodSeconds: 30
      selectPolicy: Max

왜 좋은가:

  • HPA v2 behavior로 scaleDown stabilization (5분) vs scaleUp aggressive (즉시) 분리
  • overlay는 minReplicas / maxReplicas만 override하고 behavior는 상속

나쁜 예시 1: commonLabels로 environment 주입 → selector immutable 에러

# gitops/apps/identity-auth/overlays/prod/kustomization.yaml  (BAD)
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: prod-identity-auth
resources:
  - ../../base
commonLabels:
  example.com/environment: prod

문제: commonLabels는 spec.selector.matchLabels에 자동 주입된다. 이미 live 상태인 Deployment/StatefulSet에 apply하면 The Deployment "auth" is invalid: spec.selector: Invalid value: ...: field is immutable 로 차단. 해결: labels: + includeSelectors: false로 교체.

나쁜 예시 2: overlay가 base를 거의 재작성

gitops/apps/identity-auth/base/deployment.yaml                 (150 lines)
gitops/apps/identity-auth/overlays/prod/deployment.yaml        (140 lines, 95% identical)
gitops/apps/identity-auth/overlays/staging/deployment.yaml     (140 lines)
gitops/apps/identity-auth/overlays/dev/deployment.yaml         (135 lines)

문제: overlay가 base의 95%를 복붙 + 몇 줄 수정. drift 발생 시점부터 base가 의미 없어진다. 해결: overlay는 patches: + images: + replicas: + labels:만 쓰고 전체 리소스는 base에서 가져온다.

나쁜 예시 3: 운영 secret을 secretGenerator로 plaintext Git 커밋

# gitops/apps/identity-auth/overlays/prod/kustomization.yaml  (BAD)
secretGenerator:
  - name: auth-secrets
    literals:
      - OAUTH_CLIENT_SECRET=s3cr3t-prod-value
      - DB_PASSWORD=prod-db-password

문제: plaintext secret이 Git에 박힌다. 해결: External Secrets Operator + Vault / AWS Secrets Manager / Bitwarden Secrets. 또는 SealedSecrets (public key encrypted).

나쁜 예시 4: patchesStrategicMerge / patchesJson6902 (deprecated)

# gitops/apps/identity-auth/overlays/prod/kustomization.yaml  (BAD, v5 deprecated)
patchesStrategicMerge:
  - patches/auth-resources.yaml
patchesJson6902:
  - target:
      group: apps
      version: v1
      kind: Deployment
      name: auth
    path: patches/auth-env.yaml

문제: 두 필드는 Kustomize v5에서 deprecated (여전히 동작하지만 신규 사용 금지). 하나의 patches: 필드로 통합되어 strategic merge + JSON patch 양쪽을 지원하므로 혼재할 이유 없음. 해결: patches: 단일 키 사용.

나쁜 예시 5: base에 환경 host / domain 고정

# gitops/apps/identity-auth/base/ingress.yaml  (BAD)
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: auth-public
spec:
  rules:
    - host: auth.example.com      # prod host hardcoded in base
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: auth
                port:
                  number: 8080

문제: base가 prod를 전제한다. dev/staging overlay가 host를 교체하려고 patches:를 추가해야 하고, base는 더 이상 환경 중립이 아니다. 해결: base에서는 host를 placeholder (auth.placeholder.invalid)로 두고 overlay patches:에서 주입.

나쁜 예시 6: bases: 사용 (v2.1에서 resources:로 통합됨)

# gitops/apps/identity-auth/overlays/prod/kustomization.yaml  (BAD)
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
bases:
  - ../../base

문제: bases:는 v2.1에서 resources:에 흡수됨. 신규 코드에서 사용 금지. 해결: resources: 사용.

나쁜 예시 7: HPA가 있는 Deployment에 overlay replicas:로 고정값 주입

# gitops/apps/identity-auth/overlays/prod/kustomization.yaml  (BAD — conflicts with HPA)
replicas:
  - name: auth
    count: 3

(한편 HPA는 minReplicas: 6 / maxReplicas: 20)

문제: Kustomize가 replicas: 3으로 apply → HPA가 즉시 6으로 끌어올림 → 매 ArgoCD sync마다 out-of-sync flap. 해결: HPA 활성 리소스에서는 overlay replicas:를 쓰지 않고, HPA minReplicas를 환경별로 patch.

Reference in New Issue View Git Blame Copy Permalink