642 lines
17 KiB
Markdown
642 lines
17 KiB
Markdown
# storage / PVC 예시
|
|
|
|
모든 예시는 `kubectl apply -f` 로 바로 적용 가능한 완성 매니페스트다.
|
|
생략(`...`)이 있는 곳은 의도적으로 다른 문서로 위임한 부분이다.
|
|
|
|
---
|
|
|
|
## 좋은 예시 1: 운영 StorageClass 표준 세트 (WaitForFirstConsumer + Retain)
|
|
|
|
```yaml
|
|
---
|
|
apiVersion: storage.k8s.io/v1
|
|
kind: StorageClass
|
|
metadata:
|
|
name: fast-ssd-retain
|
|
labels:
|
|
app.kubernetes.io/part-of: platform-storage
|
|
storage.platform.io/tier: gold
|
|
annotations:
|
|
storage.platform.io/description: "prod stateful (DB, vault, object store). retain on PVC delete."
|
|
provisioner: driver.longhorn.io
|
|
parameters:
|
|
numberOfReplicas: "3"
|
|
staleReplicaTimeout: "30"
|
|
fsType: ext4
|
|
reclaimPolicy: Retain
|
|
volumeBindingMode: WaitForFirstConsumer
|
|
allowVolumeExpansion: true
|
|
---
|
|
apiVersion: storage.k8s.io/v1
|
|
kind: StorageClass
|
|
metadata:
|
|
name: standard-delete
|
|
labels:
|
|
app.kubernetes.io/part-of: platform-storage
|
|
storage.platform.io/tier: silver
|
|
annotations:
|
|
storage.platform.io/description: "dev/test, ephemeral, rebuild-safe data. deletes on PVC removal."
|
|
provisioner: driver.longhorn.io
|
|
parameters:
|
|
numberOfReplicas: "2"
|
|
fsType: ext4
|
|
reclaimPolicy: Delete
|
|
volumeBindingMode: WaitForFirstConsumer
|
|
allowVolumeExpansion: true
|
|
---
|
|
apiVersion: storage.k8s.io/v1
|
|
kind: StorageClass
|
|
metadata:
|
|
name: rwx-shared
|
|
labels:
|
|
app.kubernetes.io/part-of: platform-storage
|
|
storage.platform.io/tier: shared
|
|
provisioner: nfs.csi.k8s.io
|
|
parameters:
|
|
server: nfs.storage.svc.cluster.local
|
|
share: /exports/shared
|
|
reclaimPolicy: Retain
|
|
volumeBindingMode: Immediate # 네트워크 스토리지이고 topology 제약 없음 → 예외적으로 Immediate 허용
|
|
allowVolumeExpansion: true
|
|
mountOptions:
|
|
- nfsvers=4.1
|
|
- hard
|
|
- noatime
|
|
```
|
|
|
|
왜 좋은가:
|
|
- `volumeBindingMode: WaitForFirstConsumer` 기본, `Immediate`는 이유를 주석으로 명시
|
|
- `reclaimPolicy`가 데이터 등급에 따라 다르게 선언됨 (Retain / Delete)
|
|
- `allowVolumeExpansion: true` 기본
|
|
- 라벨/annotation으로 용도 구분
|
|
|
|
❌ 나쁜 예시 1: 기본값 의존 + Immediate 바인딩
|
|
|
|
```yaml
|
|
apiVersion: storage.k8s.io/v1
|
|
kind: StorageClass
|
|
metadata:
|
|
name: default
|
|
provisioner: driver.longhorn.io
|
|
# reclaimPolicy 미지정 → 기본 Delete (운영 데이터도 삭제됨)
|
|
# volumeBindingMode 미지정 → 기본 Immediate (topology 충돌 유발)
|
|
# allowVolumeExpansion 미지정 → 확장 불가
|
|
```
|
|
|
|
문제:
|
|
- `reclaimPolicy` 기본 `Delete`: 실수로 PVC를 지우면 PV와 데이터까지 사라진다
|
|
- `volumeBindingMode` 기본 `Immediate`: Pod가 스케줄되지 못하는 zone/node에 PV가 붙을 수 있다
|
|
- 확장 불가
|
|
|
|
---
|
|
|
|
## 좋은 예시 2: VolumeSnapshotClass를 StorageClass와 매칭
|
|
|
|
```yaml
|
|
---
|
|
apiVersion: snapshot.storage.k8s.io/v1
|
|
kind: VolumeSnapshotClass
|
|
metadata:
|
|
name: fast-ssd-snap-retain
|
|
labels:
|
|
app.kubernetes.io/part-of: platform-storage
|
|
velero.io/csi-volumesnapshot-class: "true"
|
|
driver: driver.longhorn.io
|
|
deletionPolicy: Retain
|
|
parameters:
|
|
type: bak
|
|
csi.storage.k8s.io/snapshotter-secret-name: longhorn-backup-secret
|
|
csi.storage.k8s.io/snapshotter-secret-namespace: longhorn-system
|
|
---
|
|
apiVersion: snapshot.storage.k8s.io/v1
|
|
kind: VolumeSnapshotClass
|
|
metadata:
|
|
name: standard-snap-delete
|
|
labels:
|
|
app.kubernetes.io/part-of: platform-storage
|
|
driver: driver.longhorn.io
|
|
deletionPolicy: Delete
|
|
```
|
|
|
|
왜 좋은가:
|
|
- snapshot class가 StorageClass 등급과 1:1 매칭
|
|
- 운영 데이터용은 `deletionPolicy: Retain`
|
|
- Velero가 인식하도록 `velero.io/csi-volumesnapshot-class: "true"` 라벨 부여
|
|
|
|
---
|
|
|
|
## 좋은 예시 3: PostgreSQL StatefulSet + PVC retention Retain
|
|
|
|
```yaml
|
|
---
|
|
apiVersion: v1
|
|
kind: Namespace
|
|
metadata:
|
|
name: data-prod
|
|
labels:
|
|
pod-security.kubernetes.io/enforce: restricted
|
|
---
|
|
apiVersion: apps/v1
|
|
kind: StatefulSet
|
|
metadata:
|
|
name: postgres
|
|
namespace: data-prod
|
|
labels:
|
|
app.kubernetes.io/name: postgres
|
|
app.kubernetes.io/instance: postgres-prod
|
|
app.kubernetes.io/component: database
|
|
app.kubernetes.io/part-of: auth-platform
|
|
app.kubernetes.io/managed-by: argocd
|
|
app.kubernetes.io/version: "16.4"
|
|
spec:
|
|
serviceName: postgres
|
|
replicas: 1
|
|
persistentVolumeClaimRetentionPolicy:
|
|
whenDeleted: Retain
|
|
whenScaled: Retain
|
|
selector:
|
|
matchLabels:
|
|
app.kubernetes.io/name: postgres
|
|
app.kubernetes.io/instance: postgres-prod
|
|
template:
|
|
metadata:
|
|
labels:
|
|
app.kubernetes.io/name: postgres
|
|
app.kubernetes.io/instance: postgres-prod
|
|
app.kubernetes.io/component: database
|
|
spec:
|
|
securityContext:
|
|
runAsNonRoot: true
|
|
runAsUser: 999
|
|
runAsGroup: 999
|
|
fsGroup: 999
|
|
fsGroupChangePolicy: OnRootMismatch
|
|
seccompProfile:
|
|
type: RuntimeDefault
|
|
containers:
|
|
- name: postgres
|
|
image: postgres@sha256:8a6b7c6f0e0b5e2b0e2b0e2b0e2b0e2b0e2b0e2b0e2b0e2b0e2b0e2b0e2b0e2b
|
|
imagePullPolicy: IfNotPresent
|
|
ports:
|
|
- name: pg
|
|
containerPort: 5432
|
|
env:
|
|
- name: POSTGRES_DB
|
|
value: auth
|
|
- name: PGDATA
|
|
value: /var/lib/postgresql/data/pgdata
|
|
envFrom:
|
|
- secretRef:
|
|
name: postgres-credentials
|
|
resources:
|
|
requests:
|
|
cpu: "500m"
|
|
memory: "1Gi"
|
|
limits:
|
|
cpu: "2"
|
|
memory: "4Gi"
|
|
readinessProbe:
|
|
exec:
|
|
command: ["pg_isready", "-U", "postgres"]
|
|
initialDelaySeconds: 10
|
|
periodSeconds: 5
|
|
livenessProbe:
|
|
exec:
|
|
command: ["pg_isready", "-U", "postgres"]
|
|
initialDelaySeconds: 30
|
|
periodSeconds: 10
|
|
securityContext:
|
|
allowPrivilegeEscalation: false
|
|
readOnlyRootFilesystem: true
|
|
capabilities:
|
|
drop: ["ALL"]
|
|
volumeMounts:
|
|
- name: data
|
|
mountPath: /var/lib/postgresql/data
|
|
- name: tmp
|
|
mountPath: /tmp
|
|
- name: run
|
|
mountPath: /var/run/postgresql
|
|
volumes:
|
|
- name: tmp
|
|
emptyDir: {}
|
|
- name: run
|
|
emptyDir: {}
|
|
volumeClaimTemplates:
|
|
- metadata:
|
|
name: data
|
|
labels:
|
|
app.kubernetes.io/name: postgres
|
|
app.kubernetes.io/instance: postgres-prod
|
|
backup.platform.io/tier: gold
|
|
spec:
|
|
accessModes: ["ReadWriteOnce"]
|
|
storageClassName: fast-ssd-retain
|
|
resources:
|
|
requests:
|
|
storage: 50Gi
|
|
```
|
|
|
|
왜 좋은가:
|
|
- `persistentVolumeClaimRetentionPolicy`가 명시적으로 `Retain`
|
|
- StorageClass `fast-ssd-retain`에 맞물리는 `RWO`
|
|
- `fsGroup` + `fsGroupChangePolicy` 설정
|
|
- restricted PSA 준수 (runAsNonRoot, readOnlyRootFilesystem, capabilities drop all)
|
|
- `emptyDir`로 tmp/run 분리 (PVC 남발 방지)
|
|
- 라벨 full set + backup tier 라벨
|
|
|
|
---
|
|
|
|
## 좋은 예시 4: 단일 PVC Deployment (RWO, replicas 1)
|
|
|
|
```yaml
|
|
---
|
|
apiVersion: v1
|
|
kind: PersistentVolumeClaim
|
|
metadata:
|
|
name: minio-data
|
|
namespace: object-prod
|
|
labels:
|
|
app.kubernetes.io/name: minio
|
|
app.kubernetes.io/component: object-store
|
|
backup.platform.io/tier: gold
|
|
spec:
|
|
accessModes: ["ReadWriteOnce"]
|
|
storageClassName: fast-ssd-retain
|
|
resources:
|
|
requests:
|
|
storage: 500Gi
|
|
---
|
|
apiVersion: apps/v1
|
|
kind: Deployment
|
|
metadata:
|
|
name: minio
|
|
namespace: object-prod
|
|
labels:
|
|
app.kubernetes.io/name: minio
|
|
app.kubernetes.io/instance: minio-prod
|
|
app.kubernetes.io/component: object-store
|
|
app.kubernetes.io/part-of: platform-storage
|
|
spec:
|
|
replicas: 1
|
|
strategy:
|
|
type: Recreate # RWO 단일 PVC이므로 RollingUpdate 금지
|
|
selector:
|
|
matchLabels:
|
|
app.kubernetes.io/name: minio
|
|
app.kubernetes.io/instance: minio-prod
|
|
template:
|
|
metadata:
|
|
labels:
|
|
app.kubernetes.io/name: minio
|
|
app.kubernetes.io/instance: minio-prod
|
|
spec:
|
|
securityContext:
|
|
runAsNonRoot: true
|
|
runAsUser: 1000
|
|
fsGroup: 1000
|
|
seccompProfile:
|
|
type: RuntimeDefault
|
|
containers:
|
|
- name: minio
|
|
image: quay.io/minio/minio@sha256:abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789
|
|
args: ["server", "/data", "--console-address", ":9001"]
|
|
ports:
|
|
- {name: s3, containerPort: 9000}
|
|
- {name: console, containerPort: 9001}
|
|
envFrom:
|
|
- secretRef:
|
|
name: minio-root-credentials
|
|
resources:
|
|
requests: {cpu: "250m", memory: "512Mi"}
|
|
limits: {cpu: "2", memory: "4Gi"}
|
|
readinessProbe:
|
|
httpGet: {path: /minio/health/ready, port: s3}
|
|
periodSeconds: 5
|
|
livenessProbe:
|
|
httpGet: {path: /minio/health/live, port: s3}
|
|
periodSeconds: 20
|
|
securityContext:
|
|
allowPrivilegeEscalation: false
|
|
readOnlyRootFilesystem: true
|
|
capabilities:
|
|
drop: ["ALL"]
|
|
volumeMounts:
|
|
- {name: data, mountPath: /data}
|
|
volumes:
|
|
- name: data
|
|
persistentVolumeClaim:
|
|
claimName: minio-data
|
|
```
|
|
|
|
왜 좋은가:
|
|
- StatefulSet 없이도 stable한 단일 writer 구성
|
|
- `strategy: Recreate`로 RWO 충돌 방지
|
|
- PVC와 StorageClass가 명시적으로 매칭
|
|
- backup tier 라벨 → Velero selector와 연동
|
|
|
|
❌ 나쁜 예시 2: RWO에 RollingUpdate + replicas 2
|
|
|
|
```yaml
|
|
spec:
|
|
replicas: 2
|
|
strategy:
|
|
type: RollingUpdate
|
|
template:
|
|
spec:
|
|
containers:
|
|
- volumeMounts:
|
|
- {name: data, mountPath: /data}
|
|
volumes:
|
|
- name: data
|
|
persistentVolumeClaim:
|
|
claimName: minio-data # RWO인데 두 Pod가 동시에 마운트 시도
|
|
```
|
|
|
|
문제:
|
|
- RWO PVC를 두 Pod가 동시에 잡을 수 없어 신규 Pod가 영원히 Pending
|
|
- RollingUpdate가 old→new 전환 시 마운트 충돌
|
|
- 해결: replicas=1 + Recreate, 또는 RWX, 또는 StatefulSet
|
|
|
|
---
|
|
|
|
## 좋은 예시 5: separate PVC (정당한 수명/복구 단위 차이)
|
|
|
|
```yaml
|
|
---
|
|
apiVersion: apps/v1
|
|
kind: StatefulSet
|
|
metadata:
|
|
name: archive-ledger
|
|
namespace: finance-prod
|
|
labels:
|
|
app.kubernetes.io/name: archive-ledger
|
|
app.kubernetes.io/instance: archive-ledger-prod
|
|
spec:
|
|
serviceName: archive-ledger
|
|
replicas: 3
|
|
selector:
|
|
matchLabels:
|
|
app.kubernetes.io/name: archive-ledger
|
|
app.kubernetes.io/instance: archive-ledger-prod
|
|
template:
|
|
metadata:
|
|
labels:
|
|
app.kubernetes.io/name: archive-ledger
|
|
app.kubernetes.io/instance: archive-ledger-prod
|
|
spec:
|
|
securityContext:
|
|
runAsNonRoot: true
|
|
runAsUser: 10001
|
|
runAsGroup: 10001
|
|
fsGroup: 10001
|
|
seccompProfile:
|
|
type: RuntimeDefault
|
|
containers:
|
|
- name: archive-ledger
|
|
image: registry.example.com/finance/archive-ledger@sha256:3fbc632167424a6d997e74f52b878d7cc478225cffac6bc977eedfe51c7f4e79
|
|
ports:
|
|
- { name: http, containerPort: 8080 }
|
|
resources:
|
|
requests: { cpu: 500m, memory: 1Gi }
|
|
limits: { memory: 2Gi }
|
|
securityContext:
|
|
allowPrivilegeEscalation: false
|
|
readOnlyRootFilesystem: true
|
|
capabilities:
|
|
drop: ["ALL"]
|
|
volumeMounts:
|
|
- { name: data, mountPath: /var/lib/ledger }
|
|
- { name: audit-archive, mountPath: /var/lib/ledger/audit }
|
|
volumeClaimTemplates:
|
|
- metadata:
|
|
name: data
|
|
labels:
|
|
backup.platform.io/tier: gold # 5분 RPO, 매일 snapshot
|
|
spec:
|
|
accessModes: ["ReadWriteOnce"]
|
|
storageClassName: fast-ssd-retain
|
|
resources:
|
|
requests: {storage: 500Gi}
|
|
- metadata:
|
|
name: audit-archive
|
|
labels:
|
|
backup.platform.io/tier: bronze # 24h RPO, 주 1회 snapshot, 7년 보존
|
|
spec:
|
|
accessModes: ["ReadWriteOnce"]
|
|
storageClassName: archive-retain
|
|
resources:
|
|
requests: {storage: 2Ti}
|
|
```
|
|
|
|
왜 좋은가:
|
|
- data와 audit-archive의 RPO/retention이 다름
|
|
- StorageClass도 다름 (SSD vs 아카이브)
|
|
- backup tier 라벨이 Velero schedule selector에 의해 다르게 잡힘
|
|
|
|
❌ 나쁜 예시 3: separate PVC 남발
|
|
|
|
```yaml
|
|
volumeClaimTemplates:
|
|
- {metadata: {name: logs}}
|
|
- {metadata: {name: tmp}}
|
|
- {metadata: {name: config-copy}}
|
|
- {metadata: {name: cache}}
|
|
```
|
|
|
|
문제:
|
|
- 로그/tmp/cache는 `emptyDir` 또는 stdout 대상
|
|
- PVC 4개는 수명 구분 없이 쪼갠 것 — 운영 복잡도만 증가
|
|
- snapshot/backup 단위가 파편화됨
|
|
|
|
---
|
|
|
|
## 좋은 예시 6: 파이프라인 전체 — PVC → Snapshot → Restore → Verify
|
|
|
|
아래 6개 블록은 순서대로 `kubectl apply` 한다.
|
|
|
|
### (1) PVC
|
|
|
|
```yaml
|
|
---
|
|
apiVersion: v1
|
|
kind: PersistentVolumeClaim
|
|
metadata:
|
|
name: app-data
|
|
namespace: app-prod
|
|
labels:
|
|
app.kubernetes.io/name: app
|
|
backup.platform.io/tier: gold
|
|
spec:
|
|
accessModes: ["ReadWriteOnce"]
|
|
storageClassName: fast-ssd-retain
|
|
resources:
|
|
requests: {storage: 20Gi}
|
|
```
|
|
|
|
### (2) VolumeSnapshotClass (전역 1회)
|
|
|
|
```yaml
|
|
---
|
|
apiVersion: snapshot.storage.k8s.io/v1
|
|
kind: VolumeSnapshotClass
|
|
metadata:
|
|
name: fast-ssd-snap-retain
|
|
labels:
|
|
velero.io/csi-volumesnapshot-class: "true"
|
|
driver: driver.longhorn.io
|
|
deletionPolicy: Retain
|
|
```
|
|
|
|
### (3) On-demand VolumeSnapshot
|
|
|
|
```yaml
|
|
---
|
|
apiVersion: snapshot.storage.k8s.io/v1
|
|
kind: VolumeSnapshot
|
|
metadata:
|
|
name: app-data-2026-04-16-pre-migration
|
|
namespace: app-prod
|
|
labels:
|
|
app.kubernetes.io/name: app
|
|
snapshot.platform.io/reason: pre-migration
|
|
spec:
|
|
volumeSnapshotClassName: fast-ssd-snap-retain
|
|
source:
|
|
persistentVolumeClaimName: app-data
|
|
```
|
|
|
|
### (4) Restore: snapshot을 소스로 하는 새 PVC
|
|
|
|
```yaml
|
|
---
|
|
apiVersion: v1
|
|
kind: PersistentVolumeClaim
|
|
metadata:
|
|
name: app-data-restored
|
|
namespace: app-prod
|
|
spec:
|
|
accessModes: ["ReadWriteOnce"]
|
|
storageClassName: fast-ssd-retain
|
|
resources:
|
|
requests: {storage: 20Gi}
|
|
dataSource:
|
|
name: app-data-2026-04-16-pre-migration
|
|
kind: VolumeSnapshot
|
|
apiGroup: snapshot.storage.k8s.io
|
|
```
|
|
|
|
### (5) Verify Job
|
|
|
|
```yaml
|
|
---
|
|
apiVersion: batch/v1
|
|
kind: Job
|
|
metadata:
|
|
name: app-data-restore-verify
|
|
namespace: app-prod
|
|
spec:
|
|
backoffLimit: 0
|
|
ttlSecondsAfterFinished: 3600
|
|
template:
|
|
spec:
|
|
restartPolicy: Never
|
|
securityContext:
|
|
runAsNonRoot: true
|
|
runAsUser: 1000
|
|
fsGroup: 1000
|
|
containers:
|
|
- name: verify
|
|
image: busybox@sha256:3fbc632167424a6d997e74f52b878d7cc478225cffac6bc977eedfe51c7f4e79
|
|
command:
|
|
- sh
|
|
- -c
|
|
- |
|
|
set -eu
|
|
test -d /data
|
|
COUNT=$(find /data -type f | wc -l)
|
|
echo "file_count=${COUNT}"
|
|
test "${COUNT}" -gt 0
|
|
resources:
|
|
requests: { cpu: 50m, memory: 64Mi }
|
|
limits: { memory: 128Mi }
|
|
securityContext:
|
|
allowPrivilegeEscalation: false
|
|
readOnlyRootFilesystem: true
|
|
capabilities: {drop: ["ALL"]}
|
|
volumeMounts:
|
|
- {name: data, mountPath: /data, readOnly: true}
|
|
volumes:
|
|
- name: data
|
|
persistentVolumeClaim:
|
|
claimName: app-data-restored
|
|
```
|
|
|
|
### (6) 최종 확인
|
|
|
|
```bash
|
|
kubectl -n app-prod get volumesnapshot,pvc,job
|
|
kubectl -n app-prod logs job/app-data-restore-verify
|
|
```
|
|
|
|
왜 좋은가:
|
|
- PVC → SnapshotClass → Snapshot → dataSource 기반 PVC restore → Job 검증의 end-to-end 흐름
|
|
- `deletionPolicy: Retain`으로 snapshot을 실수로 삭제해도 PV는 남음
|
|
- Job이 restricted PSA 준수, 이미지 digest 고정, `backoffLimit: 0`
|
|
|
|
---
|
|
|
|
## 나쁜 예시 4: hostPath를 운영 PV로 사용
|
|
|
|
```yaml
|
|
apiVersion: v1
|
|
kind: PersistentVolume
|
|
metadata:
|
|
name: pg-host
|
|
spec:
|
|
capacity: {storage: 50Gi}
|
|
accessModes: ["ReadWriteOnce"]
|
|
hostPath:
|
|
path: /data/postgres
|
|
```
|
|
|
|
문제:
|
|
- 노드 장애 = 데이터 손실
|
|
- snapshot / expansion / 다중 노드 스케줄링 전부 불가
|
|
- 운영 표준 아님
|
|
|
|
---
|
|
|
|
## 나쁜 예시 5: K3s local-path로 production Postgres
|
|
|
|
```yaml
|
|
volumeClaimTemplates:
|
|
- metadata: {name: data}
|
|
spec:
|
|
accessModes: ["ReadWriteOnce"]
|
|
storageClassName: local-path # K3s default
|
|
resources: {requests: {storage: 100Gi}}
|
|
```
|
|
|
|
문제:
|
|
- local-path는 snapshot 미지원 → Velero CSI snapshot 불가
|
|
- expansion 미지원 → 용량 부족 시 마이그레이션 필요
|
|
- 노드 pin → 노드 장애 시 Postgres 복구 불가
|
|
- 해결: Longhorn / OpenEBS / cloud CSI driver로 교체
|
|
|
|
---
|
|
|
|
## 나쁜 예시 6: StorageClass 생략
|
|
|
|
```yaml
|
|
spec:
|
|
accessModes: ["ReadWriteOnce"]
|
|
resources: {requests: {storage: 20Gi}}
|
|
# storageClassName 미지정 → 클러스터 default annotation 사용
|
|
```
|
|
|
|
문제:
|
|
- 어떤 tier를 기대했는지 선언에서 드러나지 않음
|
|
- 클러스터 default가 바뀌면 침묵적으로 다른 StorageClass로 바인딩
|
|
- 환경 간 재현 불가
|