553 lines
15 KiB
Markdown
553 lines
15 KiB
Markdown
# Kustomize 예시
|
|
|
|
모든 예시는 Kustomize v5 문법 기준. 렌더 검증:
|
|
|
|
```bash
|
|
kubectl kustomize <dir> | kubectl apply --server-side --field-manager=ci --dry-run=server -f -
|
|
```
|
|
|
|
---
|
|
|
|
## 좋은 예시 1: catalog unit의 base / components / overlays 구조
|
|
|
|
```text
|
|
gitops/
|
|
apps/
|
|
identity-auth/
|
|
base/
|
|
kustomization.yaml
|
|
deployment.yaml
|
|
service.yaml
|
|
servicemonitor.yaml
|
|
pdb.yaml
|
|
hpa.yaml
|
|
components/
|
|
with-topology-spread-zone/
|
|
kustomization.yaml
|
|
patch.yaml
|
|
with-pdb-tier1/
|
|
kustomization.yaml
|
|
patch.yaml
|
|
overlays/
|
|
prod/kr-main/
|
|
kustomization.yaml
|
|
patches/
|
|
auth-resources.yaml
|
|
auth-ingress-host.yaml
|
|
clusters/
|
|
prod/kr-main/
|
|
kustomization.yaml
|
|
```
|
|
|
|
```yaml
|
|
# gitops/apps/identity-auth/base/kustomization.yaml
|
|
apiVersion: kustomize.config.k8s.io/v1beta1
|
|
kind: Kustomization
|
|
resources:
|
|
- deployment.yaml
|
|
- service.yaml
|
|
- servicemonitor.yaml
|
|
- pdb.yaml
|
|
- hpa.yaml
|
|
labels:
|
|
- pairs:
|
|
app.kubernetes.io/name: auth
|
|
app.kubernetes.io/component: api
|
|
app.kubernetes.io/part-of: identity-platform
|
|
includeSelectors: false
|
|
includeTemplates: true
|
|
```
|
|
|
|
**왜 좋은가:**
|
|
|
|
- base가 환경·region을 모른다 (namespace / replicas / host / image tag 전부 없음)
|
|
- `labels:` (v5) 사용, `commonLabels` 없음 → selector immutability 안전
|
|
- `includeTemplates: true`로 Pod label에는 전파되어 observability 쿼리 가능
|
|
- selector에 들어가는 label은 base의 Deployment 내부에서 명시적으로 고정
|
|
|
|
---
|
|
|
|
## 좋은 예시 2: base Deployment (완전 apply-ready)
|
|
|
|
```yaml
|
|
# gitops/apps/identity-auth/base/deployment.yaml
|
|
apiVersion: apps/v1
|
|
kind: Deployment
|
|
metadata:
|
|
name: auth
|
|
labels:
|
|
app.kubernetes.io/name: auth
|
|
app.kubernetes.io/instance: auth
|
|
app.kubernetes.io/component: api
|
|
app.kubernetes.io/part-of: identity-platform
|
|
app.kubernetes.io/managed-by: argocd
|
|
annotations:
|
|
example.com/owner-email: identity-sre@example.com
|
|
spec:
|
|
replicas: 2
|
|
revisionHistoryLimit: 5
|
|
strategy:
|
|
type: RollingUpdate
|
|
rollingUpdate:
|
|
maxSurge: 25%
|
|
maxUnavailable: 0
|
|
selector:
|
|
matchLabels:
|
|
app.kubernetes.io/name: auth
|
|
app.kubernetes.io/instance: auth
|
|
app.kubernetes.io/component: api
|
|
template:
|
|
metadata:
|
|
labels:
|
|
app.kubernetes.io/name: auth
|
|
app.kubernetes.io/instance: auth
|
|
app.kubernetes.io/component: api
|
|
app.kubernetes.io/part-of: identity-platform
|
|
app.kubernetes.io/managed-by: argocd
|
|
annotations:
|
|
prometheus.io/scrape: "true"
|
|
prometheus.io/port: "8081"
|
|
prometheus.io/path: "/actuator/prometheus"
|
|
spec:
|
|
serviceAccountName: auth
|
|
automountServiceAccountToken: false
|
|
securityContext:
|
|
runAsNonRoot: true
|
|
runAsUser: 10001
|
|
runAsGroup: 10001
|
|
fsGroup: 10001
|
|
seccompProfile:
|
|
type: RuntimeDefault
|
|
terminationGracePeriodSeconds: 45
|
|
topologySpreadConstraints:
|
|
- maxSkew: 1
|
|
topologyKey: topology.kubernetes.io/zone
|
|
whenUnsatisfiable: ScheduleAnyway
|
|
labelSelector:
|
|
matchLabels:
|
|
app.kubernetes.io/name: auth
|
|
app.kubernetes.io/instance: auth
|
|
app.kubernetes.io/component: api
|
|
containers:
|
|
- name: auth
|
|
image: registry.example.com/auth:placeholder
|
|
imagePullPolicy: IfNotPresent
|
|
ports:
|
|
- name: http
|
|
containerPort: 8080
|
|
protocol: TCP
|
|
- name: management
|
|
containerPort: 8081
|
|
protocol: TCP
|
|
env:
|
|
- name: JAVA_TOOL_OPTIONS
|
|
value: "-XX:MaxRAMPercentage=75 -XX:+UseG1GC"
|
|
envFrom:
|
|
- configMapRef:
|
|
name: auth-config
|
|
- secretRef:
|
|
name: auth-secrets
|
|
resources:
|
|
requests:
|
|
cpu: 250m
|
|
memory: 512Mi
|
|
limits:
|
|
cpu: "2"
|
|
memory: 1Gi
|
|
startupProbe:
|
|
httpGet:
|
|
path: /actuator/health/liveness
|
|
port: management
|
|
periodSeconds: 5
|
|
failureThreshold: 30
|
|
livenessProbe:
|
|
httpGet:
|
|
path: /actuator/health/liveness
|
|
port: management
|
|
periodSeconds: 10
|
|
timeoutSeconds: 3
|
|
failureThreshold: 3
|
|
readinessProbe:
|
|
httpGet:
|
|
path: /actuator/health/readiness
|
|
port: management
|
|
periodSeconds: 5
|
|
timeoutSeconds: 2
|
|
failureThreshold: 3
|
|
securityContext:
|
|
runAsNonRoot: true
|
|
runAsUser: 10001
|
|
allowPrivilegeEscalation: false
|
|
readOnlyRootFilesystem: true
|
|
capabilities:
|
|
drop:
|
|
- ALL
|
|
volumeMounts:
|
|
- name: tmp
|
|
mountPath: /tmp
|
|
- name: cache
|
|
mountPath: /app/cache
|
|
volumes:
|
|
- name: tmp
|
|
emptyDir:
|
|
sizeLimit: 64Mi
|
|
- name: cache
|
|
emptyDir:
|
|
sizeLimit: 256Mi
|
|
```
|
|
|
|
**왜 좋은가:**
|
|
|
|
- image는 `:placeholder`, overlay의 `images:`가 digest로 patch → base는 버전 모름
|
|
- `revisionHistoryLimit: 5` → 대규모 cluster에서 ReplicaSet 누적 방지
|
|
- PodSecurity restricted 호환 (non-root, seccomp RuntimeDefault, capabilities drop ALL, readOnlyRootFilesystem)
|
|
- startup/liveness/readiness 3종이 타이밍 분리 (startup 150s, liveness 30s, readiness 15s 윈도우)
|
|
- topologySpreadConstraints로 zone별 분산
|
|
- `automountServiceAccountToken: false` (ServiceAccount token을 쓰지 않는 워크로드)
|
|
|
|
---
|
|
|
|
## 좋은 예시 3: overlay prod/kr-main — 환경 차이만
|
|
|
|
```yaml
|
|
# gitops/apps/identity-auth/overlays/prod/kr-main/kustomization.yaml
|
|
apiVersion: kustomize.config.k8s.io/v1beta1
|
|
kind: Kustomization
|
|
namespace: prod-identity-auth
|
|
resources:
|
|
- ../../../base
|
|
components:
|
|
- ../../../components/with-topology-spread-zone
|
|
- ../../../components/with-pdb-tier1
|
|
labels:
|
|
- pairs:
|
|
example.com/environment: prod
|
|
example.com/region: kr-main
|
|
example.com/slo-tier: tier-1
|
|
includeSelectors: false
|
|
includeTemplates: true
|
|
images:
|
|
- name: registry.example.com/auth
|
|
digest: "sha256:f1a2b3c4d5e6f7081920aabbccddeeff00112233445566778899aabbccddeeff"
|
|
replicas:
|
|
- name: auth
|
|
count: 6
|
|
patches:
|
|
- target:
|
|
kind: Deployment
|
|
name: auth
|
|
path: patches/auth-resources.yaml
|
|
- target:
|
|
kind: Ingress
|
|
name: auth-public
|
|
patch: |-
|
|
- op: replace
|
|
path: /spec/rules/0/host
|
|
value: auth.example.com
|
|
- op: replace
|
|
path: /spec/tls/0/hosts/0
|
|
value: auth.example.com
|
|
```
|
|
|
|
```yaml
|
|
# gitops/apps/identity-auth/overlays/prod/kr-main/patches/auth-resources.yaml
|
|
apiVersion: apps/v1
|
|
kind: Deployment
|
|
metadata:
|
|
name: auth
|
|
labels:
|
|
app.kubernetes.io/name: auth
|
|
app.kubernetes.io/instance: auth
|
|
spec:
|
|
selector:
|
|
matchLabels:
|
|
app.kubernetes.io/name: auth
|
|
app.kubernetes.io/instance: auth
|
|
template:
|
|
metadata:
|
|
labels:
|
|
app.kubernetes.io/name: auth
|
|
app.kubernetes.io/instance: auth
|
|
spec:
|
|
securityContext:
|
|
runAsNonRoot: true
|
|
runAsUser: 10001
|
|
runAsGroup: 10001
|
|
fsGroup: 10001
|
|
seccompProfile:
|
|
type: RuntimeDefault
|
|
containers:
|
|
- name: auth
|
|
image: registry.example.com/auth-server:1.24.0
|
|
resources:
|
|
requests:
|
|
cpu: 500m
|
|
memory: 1Gi
|
|
limits:
|
|
cpu: "2"
|
|
memory: 2Gi
|
|
securityContext:
|
|
allowPrivilegeEscalation: false
|
|
readOnlyRootFilesystem: true
|
|
capabilities:
|
|
drop: ["ALL"]
|
|
```
|
|
|
|
**왜 좋은가:**
|
|
|
|
- overlay 자체가 짧음 (base를 재작성하지 않음)
|
|
- digest 기반 image pinning
|
|
- `components:`로 zone spread + PDB tier-1을 재사용
|
|
- `labels:` 사용, `includeSelectors: false` → selector immutability 안전
|
|
- replicas override는 HPA minReplicas와 일치 (HPA base에서 `minReplicas: 6`으로 설정되어 있다고 가정)
|
|
|
|
---
|
|
|
|
## 좋은 예시 4: Kustomize Component — `with-pdb-tier1`
|
|
|
|
```yaml
|
|
# gitops/apps/identity-auth/components/with-pdb-tier1/kustomization.yaml
|
|
apiVersion: kustomize.config.k8s.io/v1alpha1
|
|
kind: Component
|
|
resources:
|
|
- pdb.yaml
|
|
```
|
|
|
|
```yaml
|
|
# gitops/apps/identity-auth/components/with-pdb-tier1/pdb.yaml
|
|
apiVersion: policy/v1
|
|
kind: PodDisruptionBudget
|
|
metadata:
|
|
name: auth
|
|
labels:
|
|
app.kubernetes.io/name: auth
|
|
app.kubernetes.io/instance: auth
|
|
app.kubernetes.io/component: api
|
|
app.kubernetes.io/part-of: identity-platform
|
|
app.kubernetes.io/managed-by: argocd
|
|
example.com/slo-tier: tier-1
|
|
spec:
|
|
minAvailable: 50%
|
|
unhealthyPodEvictionPolicy: AlwaysAllow
|
|
selector:
|
|
matchLabels:
|
|
app.kubernetes.io/name: auth
|
|
app.kubernetes.io/instance: auth
|
|
app.kubernetes.io/component: api
|
|
```
|
|
|
|
**왜 좋은가:**
|
|
|
|
- `kind: Component`로 선언 → 여러 overlay에서 `components:` 키로 재사용
|
|
- tier-1의 PDB 정책(50% minAvailable)이 단일 파일에 고정
|
|
- 다른 tier는 별도 component (`with-pdb-tier2`, `with-pdb-tier3`)
|
|
|
|
---
|
|
|
|
## 좋은 예시 5: ConfigMap generator + hash suffix를 활용한 자동 rollout
|
|
|
|
```yaml
|
|
# gitops/apps/identity-auth/base/kustomization.yaml (with generator)
|
|
apiVersion: kustomize.config.k8s.io/v1beta1
|
|
kind: Kustomization
|
|
resources:
|
|
- deployment.yaml
|
|
- service.yaml
|
|
configMapGenerator:
|
|
- name: auth-config
|
|
files:
|
|
- application.yaml=config/application.yaml
|
|
- logback.xml=config/logback.xml
|
|
options:
|
|
labels:
|
|
app.kubernetes.io/name: auth
|
|
app.kubernetes.io/component: config
|
|
generatorOptions:
|
|
disableNameSuffixHash: false
|
|
```
|
|
|
|
**왜 좋은가:**
|
|
|
|
- ConfigMap 내용 변경 시 hash suffix가 바뀜 → Deployment가 새 이름을 참조 → rolling update 자동 트리거
|
|
- annotation 기반 "checksum" hack 불필요
|
|
- Secret은 generator로 만들지 않고 External Secrets로 관리
|
|
|
|
---
|
|
|
|
## 좋은 예시 6: HPA v2 + behavior (base 리소스)
|
|
|
|
```yaml
|
|
# gitops/apps/identity-auth/base/hpa.yaml
|
|
apiVersion: autoscaling/v2
|
|
kind: HorizontalPodAutoscaler
|
|
metadata:
|
|
name: auth
|
|
labels:
|
|
app.kubernetes.io/name: auth
|
|
app.kubernetes.io/instance: auth
|
|
app.kubernetes.io/component: api
|
|
app.kubernetes.io/part-of: identity-platform
|
|
app.kubernetes.io/managed-by: argocd
|
|
spec:
|
|
scaleTargetRef:
|
|
apiVersion: apps/v1
|
|
kind: Deployment
|
|
name: auth
|
|
minReplicas: 2
|
|
maxReplicas: 20
|
|
metrics:
|
|
- type: Resource
|
|
resource:
|
|
name: cpu
|
|
target:
|
|
type: Utilization
|
|
averageUtilization: 70
|
|
- type: Resource
|
|
resource:
|
|
name: memory
|
|
target:
|
|
type: Utilization
|
|
averageUtilization: 80
|
|
behavior:
|
|
scaleDown:
|
|
stabilizationWindowSeconds: 300
|
|
policies:
|
|
- type: Percent
|
|
value: 25
|
|
periodSeconds: 60
|
|
scaleUp:
|
|
stabilizationWindowSeconds: 0
|
|
policies:
|
|
- type: Percent
|
|
value: 100
|
|
periodSeconds: 30
|
|
- type: Pods
|
|
value: 4
|
|
periodSeconds: 30
|
|
selectPolicy: Max
|
|
```
|
|
|
|
**왜 좋은가:**
|
|
|
|
- HPA v2 behavior로 scaleDown stabilization (5분) vs scaleUp aggressive (즉시) 분리
|
|
- overlay는 `minReplicas` / `maxReplicas`만 override하고 behavior는 상속
|
|
|
|
---
|
|
|
|
## 나쁜 예시 1: `commonLabels`로 environment 주입 → selector immutable 에러
|
|
|
|
```yaml
|
|
# gitops/apps/identity-auth/overlays/prod/kustomization.yaml (BAD)
|
|
apiVersion: kustomize.config.k8s.io/v1beta1
|
|
kind: Kustomization
|
|
namespace: prod-identity-auth
|
|
resources:
|
|
- ../../base
|
|
commonLabels:
|
|
example.com/environment: prod
|
|
```
|
|
|
|
**문제:** `commonLabels`는 `spec.selector.matchLabels`에 자동 주입된다. 이미 live 상태인 Deployment/StatefulSet에 apply하면 `The Deployment "auth" is invalid: spec.selector: Invalid value: ...: field is immutable` 로 차단. 해결: `labels:` + `includeSelectors: false`로 교체.
|
|
|
|
---
|
|
|
|
## 나쁜 예시 2: overlay가 base를 거의 재작성
|
|
|
|
```text
|
|
gitops/apps/identity-auth/base/deployment.yaml (150 lines)
|
|
gitops/apps/identity-auth/overlays/prod/deployment.yaml (140 lines, 95% identical)
|
|
gitops/apps/identity-auth/overlays/staging/deployment.yaml (140 lines)
|
|
gitops/apps/identity-auth/overlays/dev/deployment.yaml (135 lines)
|
|
```
|
|
|
|
**문제:** overlay가 base의 95%를 복붙 + 몇 줄 수정. drift 발생 시점부터 base가 의미 없어진다. 해결: overlay는 `patches:` + `images:` + `replicas:` + `labels:`만 쓰고 전체 리소스는 base에서 가져온다.
|
|
|
|
---
|
|
|
|
## 나쁜 예시 3: 운영 secret을 `secretGenerator`로 plaintext Git 커밋
|
|
|
|
```yaml
|
|
# gitops/apps/identity-auth/overlays/prod/kustomization.yaml (BAD)
|
|
secretGenerator:
|
|
- name: auth-secrets
|
|
literals:
|
|
- OAUTH_CLIENT_SECRET=s3cr3t-prod-value
|
|
- DB_PASSWORD=prod-db-password
|
|
```
|
|
|
|
**문제:** plaintext secret이 Git에 박힌다. 해결: External Secrets Operator + Vault / AWS Secrets Manager / Bitwarden Secrets. 또는 SealedSecrets (public key encrypted).
|
|
|
|
---
|
|
|
|
## 나쁜 예시 4: `patchesStrategicMerge` / `patchesJson6902` (deprecated)
|
|
|
|
```yaml
|
|
# gitops/apps/identity-auth/overlays/prod/kustomization.yaml (BAD, v5 deprecated)
|
|
patchesStrategicMerge:
|
|
- patches/auth-resources.yaml
|
|
patchesJson6902:
|
|
- target:
|
|
group: apps
|
|
version: v1
|
|
kind: Deployment
|
|
name: auth
|
|
path: patches/auth-env.yaml
|
|
```
|
|
|
|
**문제:** 두 필드는 Kustomize v5에서 deprecated (여전히 동작하지만 신규 사용 금지). 하나의 `patches:` 필드로 통합되어 strategic merge + JSON patch 양쪽을 지원하므로 혼재할 이유 없음. 해결: `patches:` 단일 키 사용.
|
|
|
|
---
|
|
|
|
## 나쁜 예시 5: base에 환경 host / domain 고정
|
|
|
|
```yaml
|
|
# gitops/apps/identity-auth/base/ingress.yaml (BAD)
|
|
apiVersion: networking.k8s.io/v1
|
|
kind: Ingress
|
|
metadata:
|
|
name: auth-public
|
|
spec:
|
|
rules:
|
|
- host: auth.example.com # prod host hardcoded in base
|
|
http:
|
|
paths:
|
|
- path: /
|
|
pathType: Prefix
|
|
backend:
|
|
service:
|
|
name: auth
|
|
port:
|
|
number: 8080
|
|
```
|
|
|
|
**문제:** base가 prod를 전제한다. dev/staging overlay가 host를 교체하려고 `patches:`를 추가해야 하고, base는 더 이상 환경 중립이 아니다. 해결: base에서는 host를 placeholder (`auth.placeholder.invalid`)로 두고 overlay `patches:`에서 주입.
|
|
|
|
---
|
|
|
|
## 나쁜 예시 6: `bases:` 사용 (v2.1에서 `resources:`로 통합됨)
|
|
|
|
```yaml
|
|
# gitops/apps/identity-auth/overlays/prod/kustomization.yaml (BAD)
|
|
apiVersion: kustomize.config.k8s.io/v1beta1
|
|
kind: Kustomization
|
|
bases:
|
|
- ../../base
|
|
```
|
|
|
|
**문제:** `bases:`는 v2.1에서 `resources:`에 흡수됨. 신규 코드에서 사용 금지. 해결: `resources:` 사용.
|
|
|
|
---
|
|
|
|
## 나쁜 예시 7: HPA가 있는 Deployment에 overlay `replicas:`로 고정값 주입
|
|
|
|
```yaml
|
|
# gitops/apps/identity-auth/overlays/prod/kustomization.yaml (BAD — conflicts with HPA)
|
|
replicas:
|
|
- name: auth
|
|
count: 3
|
|
```
|
|
|
|
(한편 HPA는 `minReplicas: 6 / maxReplicas: 20`)
|
|
|
|
**문제:** Kustomize가 `replicas: 3`으로 apply → HPA가 즉시 6으로 끌어올림 → 매 ArgoCD sync마다 `out-of-sync` flap. 해결: HPA 활성 리소스에서는 overlay `replicas:`를 쓰지 않고, HPA `minReplicas`를 환경별로 patch.
|