donghyeon.kang/project-infra
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
Files
80f58cd9f2fa54aaf5692773658092074bf098df
project-infra/docs/examples/infra/storage-pvc.md
T

17 KiB
Raw Blame History

storage / PVC 예시

모든 예시는 kubectl apply -f 로 바로 적용 가능한 완성 매니페스트다. 생략(...)이 있는 곳은 의도적으로 다른 문서로 위임한 부분이다.

좋은 예시 1: 운영 StorageClass 표준 세트 (WaitForFirstConsumer + Retain)

---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast-ssd-retain
  labels:
    app.kubernetes.io/part-of: platform-storage
    storage.platform.io/tier: gold
  annotations:
    storage.platform.io/description: "prod stateful (DB, vault, object store). retain on PVC delete."
provisioner: driver.longhorn.io
parameters:
  numberOfReplicas: "3"
  staleReplicaTimeout: "30"
  fsType: ext4
reclaimPolicy: Retain
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: standard-delete
  labels:
    app.kubernetes.io/part-of: platform-storage
    storage.platform.io/tier: silver
  annotations:
    storage.platform.io/description: "dev/test, ephemeral, rebuild-safe data. deletes on PVC removal."
provisioner: driver.longhorn.io
parameters:
  numberOfReplicas: "2"
  fsType: ext4
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: rwx-shared
  labels:
    app.kubernetes.io/part-of: platform-storage
    storage.platform.io/tier: shared
provisioner: nfs.csi.k8s.io
parameters:
  server: nfs.storage.svc.cluster.local
  share: /exports/shared
reclaimPolicy: Retain
volumeBindingMode: Immediate   # 네트워크 스토리지이고 topology 제약 없음 → 예외적으로 Immediate 허용
allowVolumeExpansion: true
mountOptions:
  - nfsvers=4.1
  - hard
  - noatime

왜 좋은가:

  • volumeBindingMode: WaitForFirstConsumer 기본, Immediate는 이유를 주석으로 명시
  • reclaimPolicy가 데이터 등급에 따라 다르게 선언됨 (Retain / Delete)
  • allowVolumeExpansion: true 기본
  • 라벨/annotation으로 용도 구분

❌ 나쁜 예시 1: 기본값 의존 + Immediate 바인딩

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: default
provisioner: driver.longhorn.io
# reclaimPolicy 미지정 → 기본 Delete (운영 데이터도 삭제됨)
# volumeBindingMode 미지정 → 기본 Immediate (topology 충돌 유발)
# allowVolumeExpansion 미지정 → 확장 불가

문제:

  • reclaimPolicy 기본 Delete: 실수로 PVC를 지우면 PV와 데이터까지 사라진다
  • volumeBindingMode 기본 Immediate: Pod가 스케줄되지 못하는 zone/node에 PV가 붙을 수 있다
  • 확장 불가

좋은 예시 2: VolumeSnapshotClass를 StorageClass와 매칭

---
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
  name: fast-ssd-snap-retain
  labels:
    app.kubernetes.io/part-of: platform-storage
    velero.io/csi-volumesnapshot-class: "true"
driver: driver.longhorn.io
deletionPolicy: Retain
parameters:
  type: bak
  csi.storage.k8s.io/snapshotter-secret-name: longhorn-backup-secret
  csi.storage.k8s.io/snapshotter-secret-namespace: longhorn-system
---
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
  name: standard-snap-delete
  labels:
    app.kubernetes.io/part-of: platform-storage
driver: driver.longhorn.io
deletionPolicy: Delete

왜 좋은가:

  • snapshot class가 StorageClass 등급과 1:1 매칭
  • 운영 데이터용은 deletionPolicy: Retain
  • Velero가 인식하도록 velero.io/csi-volumesnapshot-class: "true" 라벨 부여

좋은 예시 3: PostgreSQL StatefulSet + PVC retention Retain

---
apiVersion: v1
kind: Namespace
metadata:
  name: data-prod
  labels:
    pod-security.kubernetes.io/enforce: restricted
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres
  namespace: data-prod
  labels:
    app.kubernetes.io/name: postgres
    app.kubernetes.io/instance: postgres-prod
    app.kubernetes.io/component: database
    app.kubernetes.io/part-of: auth-platform
    app.kubernetes.io/managed-by: argocd
    app.kubernetes.io/version: "16.4"
spec:
  serviceName: postgres
  replicas: 1
  persistentVolumeClaimRetentionPolicy:
    whenDeleted: Retain
    whenScaled: Retain
  selector:
    matchLabels:
      app.kubernetes.io/name: postgres
      app.kubernetes.io/instance: postgres-prod
  template:
    metadata:
      labels:
        app.kubernetes.io/name: postgres
        app.kubernetes.io/instance: postgres-prod
        app.kubernetes.io/component: database
    spec:
      securityContext:
        runAsNonRoot: true
        runAsUser: 999
        runAsGroup: 999
        fsGroup: 999
        fsGroupChangePolicy: OnRootMismatch
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: postgres
          image: postgres@sha256:8a6b7c6f0e0b5e2b0e2b0e2b0e2b0e2b0e2b0e2b0e2b0e2b0e2b0e2b0e2b0e2b
          imagePullPolicy: IfNotPresent
          ports:
            - name: pg
              containerPort: 5432
          env:
            - name: POSTGRES_DB
              value: auth
            - name: PGDATA
              value: /var/lib/postgresql/data/pgdata
          envFrom:
            - secretRef:
                name: postgres-credentials
          resources:
            requests:
              cpu: "500m"
              memory: "1Gi"
            limits:
              cpu: "2"
              memory: "4Gi"
          readinessProbe:
            exec:
              command: ["pg_isready", "-U", "postgres"]
            initialDelaySeconds: 10
            periodSeconds: 5
          livenessProbe:
            exec:
              command: ["pg_isready", "-U", "postgres"]
            initialDelaySeconds: 30
            periodSeconds: 10
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop: ["ALL"]
          volumeMounts:
            - name: data
              mountPath: /var/lib/postgresql/data
            - name: tmp
              mountPath: /tmp
            - name: run
              mountPath: /var/run/postgresql
      volumes:
        - name: tmp
          emptyDir: {}
        - name: run
          emptyDir: {}
  volumeClaimTemplates:
    - metadata:
        name: data
        labels:
          app.kubernetes.io/name: postgres
          app.kubernetes.io/instance: postgres-prod
          backup.platform.io/tier: gold
      spec:
        accessModes: ["ReadWriteOnce"]
        storageClassName: fast-ssd-retain
        resources:
          requests:
            storage: 50Gi

왜 좋은가:

  • persistentVolumeClaimRetentionPolicy가 명시적으로 Retain
  • StorageClass fast-ssd-retain에 맞물리는 RWO
  • fsGroup + fsGroupChangePolicy 설정
  • restricted PSA 준수 (runAsNonRoot, readOnlyRootFilesystem, capabilities drop all)
  • emptyDir로 tmp/run 분리 (PVC 남발 방지)
  • 라벨 full set + backup tier 라벨

좋은 예시 4: 단일 PVC Deployment (RWO, replicas 1)

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: minio-data
  namespace: object-prod
  labels:
    app.kubernetes.io/name: minio
    app.kubernetes.io/component: object-store
    backup.platform.io/tier: gold
spec:
  accessModes: ["ReadWriteOnce"]
  storageClassName: fast-ssd-retain
  resources:
    requests:
      storage: 500Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: minio
  namespace: object-prod
  labels:
    app.kubernetes.io/name: minio
    app.kubernetes.io/instance: minio-prod
    app.kubernetes.io/component: object-store
    app.kubernetes.io/part-of: platform-storage
spec:
  replicas: 1
  strategy:
    type: Recreate   # RWO 단일 PVC이므로 RollingUpdate 금지
  selector:
    matchLabels:
      app.kubernetes.io/name: minio
      app.kubernetes.io/instance: minio-prod
  template:
    metadata:
      labels:
        app.kubernetes.io/name: minio
        app.kubernetes.io/instance: minio-prod
    spec:
      securityContext:
        runAsNonRoot: true
        runAsUser: 1000
        fsGroup: 1000
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: minio
          image: quay.io/minio/minio@sha256:abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789
          args: ["server", "/data", "--console-address", ":9001"]
          ports:
            - {name: s3, containerPort: 9000}
            - {name: console, containerPort: 9001}
          envFrom:
            - secretRef:
                name: minio-root-credentials
          resources:
            requests: {cpu: "250m", memory: "512Mi"}
            limits: {cpu: "2", memory: "4Gi"}
          readinessProbe:
            httpGet: {path: /minio/health/ready, port: s3}
            periodSeconds: 5
          livenessProbe:
            httpGet: {path: /minio/health/live, port: s3}
            periodSeconds: 20
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop: ["ALL"]
          volumeMounts:
            - {name: data, mountPath: /data}
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: minio-data

왜 좋은가:

  • StatefulSet 없이도 stable한 단일 writer 구성
  • strategy: Recreate로 RWO 충돌 방지
  • PVC와 StorageClass가 명시적으로 매칭
  • backup tier 라벨 → Velero selector와 연동

❌ 나쁜 예시 2: RWO에 RollingUpdate + replicas 2

spec:
  replicas: 2
  strategy:
    type: RollingUpdate
  template:
    spec:
      containers:
        - volumeMounts:
            - {name: data, mountPath: /data}
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: minio-data   # RWO인데 두 Pod가 동시에 마운트 시도

문제:

  • RWO PVC를 두 Pod가 동시에 잡을 수 없어 신규 Pod가 영원히 Pending
  • RollingUpdate가 old→new 전환 시 마운트 충돌
  • 해결: replicas=1 + Recreate, 또는 RWX, 또는 StatefulSet

좋은 예시 5: separate PVC (정당한 수명/복구 단위 차이)

---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: archive-ledger
  namespace: finance-prod
  labels:
    app.kubernetes.io/name: archive-ledger
    app.kubernetes.io/instance: archive-ledger-prod
spec:
  serviceName: archive-ledger
  replicas: 3
  selector:
    matchLabels:
      app.kubernetes.io/name: archive-ledger
      app.kubernetes.io/instance: archive-ledger-prod
  template:
    metadata:
      labels:
        app.kubernetes.io/name: archive-ledger
        app.kubernetes.io/instance: archive-ledger-prod
    spec:
      securityContext:
        runAsNonRoot: true
        runAsUser: 10001
        runAsGroup: 10001
        fsGroup: 10001
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: archive-ledger
          image: registry.example.com/finance/archive-ledger@sha256:3fbc632167424a6d997e74f52b878d7cc478225cffac6bc977eedfe51c7f4e79
          ports:
            - { name: http, containerPort: 8080 }
          resources:
            requests: { cpu: 500m, memory: 1Gi }
            limits:   { memory: 2Gi }
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop: ["ALL"]
          volumeMounts:
            - { name: data, mountPath: /var/lib/ledger }
            - { name: audit-archive, mountPath: /var/lib/ledger/audit }
  volumeClaimTemplates:
    - metadata:
        name: data
        labels:
          backup.platform.io/tier: gold       # 5분 RPO, 매일 snapshot
      spec:
        accessModes: ["ReadWriteOnce"]
        storageClassName: fast-ssd-retain
        resources:
          requests: {storage: 500Gi}
    - metadata:
        name: audit-archive
        labels:
          backup.platform.io/tier: bronze     # 24h RPO, 주 1회 snapshot, 7년 보존
      spec:
        accessModes: ["ReadWriteOnce"]
        storageClassName: archive-retain
        resources:
          requests: {storage: 2Ti}

왜 좋은가:

  • data와 audit-archive의 RPO/retention이 다름
  • StorageClass도 다름 (SSD vs 아카이브)
  • backup tier 라벨이 Velero schedule selector에 의해 다르게 잡힘

❌ 나쁜 예시 3: separate PVC 남발

volumeClaimTemplates:
  - {metadata: {name: logs}}
  - {metadata: {name: tmp}}
  - {metadata: {name: config-copy}}
  - {metadata: {name: cache}}

문제:

  • 로그/tmp/cache는 emptyDir 또는 stdout 대상
  • PVC 4개는 수명 구분 없이 쪼갠 것 — 운영 복잡도만 증가
  • snapshot/backup 단위가 파편화됨

좋은 예시 6: 파이프라인 전체 — PVC → Snapshot → Restore → Verify

아래 6개 블록은 순서대로 kubectl apply 한다.

(1) PVC

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: app-data
  namespace: app-prod
  labels:
    app.kubernetes.io/name: app
    backup.platform.io/tier: gold
spec:
  accessModes: ["ReadWriteOnce"]
  storageClassName: fast-ssd-retain
  resources:
    requests: {storage: 20Gi}

(2) VolumeSnapshotClass (전역 1회)

---
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
  name: fast-ssd-snap-retain
  labels:
    velero.io/csi-volumesnapshot-class: "true"
driver: driver.longhorn.io
deletionPolicy: Retain

(3) On-demand VolumeSnapshot

---
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: app-data-2026-04-16-pre-migration
  namespace: app-prod
  labels:
    app.kubernetes.io/name: app
    snapshot.platform.io/reason: pre-migration
spec:
  volumeSnapshotClassName: fast-ssd-snap-retain
  source:
    persistentVolumeClaimName: app-data

(4) Restore: snapshot을 소스로 하는 새 PVC

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: app-data-restored
  namespace: app-prod
spec:
  accessModes: ["ReadWriteOnce"]
  storageClassName: fast-ssd-retain
  resources:
    requests: {storage: 20Gi}
  dataSource:
    name: app-data-2026-04-16-pre-migration
    kind: VolumeSnapshot
    apiGroup: snapshot.storage.k8s.io

(5) Verify Job

---
apiVersion: batch/v1
kind: Job
metadata:
  name: app-data-restore-verify
  namespace: app-prod
spec:
  backoffLimit: 0
  ttlSecondsAfterFinished: 3600
  template:
    spec:
      restartPolicy: Never
      securityContext:
        runAsNonRoot: true
        runAsUser: 1000
        fsGroup: 1000
      containers:
        - name: verify
          image: busybox@sha256:3fbc632167424a6d997e74f52b878d7cc478225cffac6bc977eedfe51c7f4e79
          command:
            - sh
            - -c
            - |
              set -eu
              test -d /data
              COUNT=$(find /data -type f | wc -l)
              echo "file_count=${COUNT}"
              test "${COUNT}" -gt 0
          resources:
            requests: { cpu: 50m, memory: 64Mi }
            limits:   { memory: 128Mi }
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities: {drop: ["ALL"]}
          volumeMounts:
            - {name: data, mountPath: /data, readOnly: true}
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: app-data-restored

(6) 최종 확인

kubectl -n app-prod get volumesnapshot,pvc,job
kubectl -n app-prod logs job/app-data-restore-verify

왜 좋은가:

  • PVC → SnapshotClass → Snapshot → dataSource 기반 PVC restore → Job 검증의 end-to-end 흐름
  • deletionPolicy: Retain으로 snapshot을 실수로 삭제해도 PV는 남음
  • Job이 restricted PSA 준수, 이미지 digest 고정, backoffLimit: 0

나쁜 예시 4: hostPath를 운영 PV로 사용

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pg-host
spec:
  capacity: {storage: 50Gi}
  accessModes: ["ReadWriteOnce"]
  hostPath:
    path: /data/postgres

문제:

  • 노드 장애 = 데이터 손실
  • snapshot / expansion / 다중 노드 스케줄링 전부 불가
  • 운영 표준 아님

나쁜 예시 5: K3s local-path로 production Postgres

volumeClaimTemplates:
  - metadata: {name: data}
    spec:
      accessModes: ["ReadWriteOnce"]
      storageClassName: local-path   # K3s default
      resources: {requests: {storage: 100Gi}}

문제:

  • local-path는 snapshot 미지원 → Velero CSI snapshot 불가
  • expansion 미지원 → 용량 부족 시 마이그레이션 필요
  • 노드 pin → 노드 장애 시 Postgres 복구 불가
  • 해결: Longhorn / OpenEBS / cloud CSI driver로 교체

나쁜 예시 6: StorageClass 생략

spec:
  accessModes: ["ReadWriteOnce"]
  resources: {requests: {storage: 20Gi}}
  # storageClassName 미지정 → 클러스터 default annotation 사용

문제:

  • 어떤 tier를 기대했는지 선언에서 드러나지 않음
  • 클러스터 default가 바뀌면 침묵적으로 다른 StorageClass로 바인딩
  • 환경 간 재현 불가
Reference in New Issue View Git Blame Copy Permalink