donghyeon.kang/project-infra
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
Files
main
project-infra/docs/examples/infra/backup-restore.md
T

15 KiB
Raw Permalink Blame History

backup / restore 예시

모든 예시는 실제 매니페스트로 kubectl apply -f 가능하다.

좋은 예시 1: Velero 설치 후 BackupStorageLocation / VolumeSnapshotLocation

---
apiVersion: velero.io/v1
kind: BackupStorageLocation
metadata:
  name: default
  namespace: velero
  labels:
    app.kubernetes.io/part-of: platform-backup
spec:
  provider: aws
  objectStorage:
    bucket: acme-prod-velero-backups
    prefix: k3s-prod
  config:
    region: us-east-1
    s3ForcePathStyle: "false"
    s3Url: https://s3.us-east-1.amazonaws.com
  default: true
  accessMode: ReadWrite
  credential:
    name: velero-s3-credentials
    key: cloud
---
apiVersion: velero.io/v1
kind: VolumeSnapshotLocation
metadata:
  name: csi-default
  namespace: velero
spec:
  provider: csi

왜 좋은가:

  • 백업 저장소가 클러스터 외부 S3 (같은 cluster MinIO에 넣지 않음)
  • credential은 별도 Secret
  • CSI snapshot location이 명시됨

❌ 나쁜 예시 1: 같은 cluster 안 MinIO를 백업 저장소로 사용

spec:
  provider: aws
  objectStorage:
    bucket: backups
  config:
    s3Url: http://minio.object-prod.svc.cluster.local:9000   # 같은 cluster!

문제:

  • cluster 장애 = 백업 동시 소실
  • MinIO 자체를 복구하려면 외부 백업이 또 필요 — 순환 의존

좋은 예시 2: Velero Schedule (tier별 분리, 30일 retention)

---
apiVersion: velero.io/v1
kind: Schedule
metadata:
  name: gold-daily
  namespace: velero
  labels:
    backup.platform.io/tier: gold
spec:
  schedule: "0 2 * * *"                     # 매일 02:00 UTC
  useOwnerReferencesInBackup: true
  template:
    ttl: 720h0m0s                           # 30일 retention
    includedNamespaces:
      - auth-prod
      - data-prod
      - object-prod
    includedResources:
      - persistentvolumeclaims
      - persistentvolumes
      - secrets
      - configmaps
      - deployments
      - statefulsets
      - services
      - ingresses
      - networkpolicies
    labelSelector:
      matchLabels:
        backup.platform.io/tier: gold
    snapshotVolumes: true
    defaultVolumesToFsBackup: false
    csiSnapshotTimeout: 30m
    storageLocation: default
    volumeSnapshotLocations:
      - csi-default
    hooks:
      resources:
        - name: postgres-consistent
          includedNamespaces: [data-prod]
          labelSelector:
            matchLabels:
              app.kubernetes.io/name: postgres
          pre:
            - exec:
                container: postgres
                command: ["/bin/sh", "-c", "psql -U postgres -c CHECKPOINT"]
                onError: Fail
                timeout: 2m
---
apiVersion: velero.io/v1
kind: Schedule
metadata:
  name: bronze-weekly-fsb
  namespace: velero
  labels:
    backup.platform.io/tier: bronze
spec:
  schedule: "0 3 * * 0"                     # 매주 일요일 03:00 UTC
  template:
    ttl: 2160h0m0s                          # 90일
    includedNamespaces: ["archive-prod"]
    labelSelector:
      matchLabels:
        backup.platform.io/tier: bronze
    snapshotVolumes: false
    defaultVolumesToFsBackup: true          # kopia/restic FSB
    storageLocation: default

왜 좋은가:

  • Schedule이 tier별로 분리되어 RPO/retention/도구를 구분
  • CSI snapshot(gold)과 FSB(bronze)를 목적에 맞게 선택
  • Postgres는 pre-hook으로 CHECKPOINT를 수행해 crash-consistent에 가까운 스냅샷 확보
  • labelSelector가 PVC의 backup.platform.io/tier와 매칭

좋은 예시 3: Velero Restore

---
apiVersion: velero.io/v1
kind: Restore
metadata:
  name: auth-prod-restore-2026-04-16
  namespace: velero
spec:
  backupName: gold-daily-20260415020000
  includedNamespaces: ["auth-prod"]
  restorePVs: true
  existingResourcePolicy: none              # 기존 리소스 보존, 누락된 것만 복원
  namespaceMapping:
    auth-prod: auth-prod-restore            # 검증용 별도 네임스페이스로 복원
  labelSelector:
    matchLabels:
      backup.platform.io/tier: gold

왜 좋은가:

  • 복원 대상이 auth-prod-restore로 분리되어 운영 영향 없이 검증 가능
  • existingResourcePolicy: none으로 실수 덮어쓰기 방지
  • restorePVs: true로 PVC/PV까지 함께 복원

좋은 예시 4: CloudNativePG Cluster + ScheduledBackup + Backup

---
apiVersion: v1
kind: Namespace
metadata:
  name: data-prod
  labels:
    pod-security.kubernetes.io/enforce: restricted
---
apiVersion: v1
kind: Secret
metadata:
  name: cnpg-s3-credentials
  namespace: data-prod
type: Opaque
stringData:
  ACCESS_KEY_ID: REPLACE_VIA_VSO
  ACCESS_SECRET_KEY: REPLACE_VIA_VSO
---
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: auth-pg
  namespace: data-prod
  labels:
    app.kubernetes.io/name: auth-pg
    app.kubernetes.io/component: database
    app.kubernetes.io/part-of: auth-platform
    backup.platform.io/tier: gold
spec:
  instances: 3
  imageName: ghcr.io/cloudnative-pg/postgresql:16.4-8
  primaryUpdateStrategy: unsupervised
  postgresql:
    parameters:
      shared_buffers: "512MB"
      max_connections: "200"
      wal_compression: "on"
      archive_timeout: "60s"
  bootstrap:
    initdb:
      database: auth
      owner: auth_app
      secret:
        name: auth-pg-app
  storage:
    size: 50Gi
    storageClass: fast-ssd-retain
  walStorage:
    size: 20Gi
    storageClass: fast-ssd-retain
  monitoring:
    enablePodMonitor: true
  resources:
    requests: {cpu: "500m", memory: "2Gi"}
    limits:   {cpu: "2",    memory: "4Gi"}
  backup:
    retentionPolicy: "30d"
    barmanObjectStore:
      destinationPath: s3://acme-prod-pg-backups/auth-pg
      endpointURL: https://s3.us-east-1.amazonaws.com
      s3Credentials:
        accessKeyId:
          name: cnpg-s3-credentials
          key: ACCESS_KEY_ID
        secretAccessKey:
          name: cnpg-s3-credentials
          key: ACCESS_SECRET_KEY
      wal:
        compression: gzip
        maxParallel: 8
      data:
        compression: gzip
        immediateCheckpoint: true
        jobs: 4
  affinity:
    podAntiAffinityType: required
    topologyKey: kubernetes.io/hostname
---
apiVersion: postgresql.cnpg.io/v1
kind: ScheduledBackup
metadata:
  name: auth-pg-daily
  namespace: data-prod
spec:
  schedule: "0 0 2 * * *"                   # 매일 02:00 (CNPG는 6-field cron)
  backupOwnerReference: self
  cluster:
    name: auth-pg
  method: barmanObjectStore
---
apiVersion: postgresql.cnpg.io/v1
kind: Backup
metadata:
  name: auth-pg-premigration-2026-04-16
  namespace: data-prod
spec:
  cluster:
    name: auth-pg
  method: barmanObjectStore

왜 좋은가:

  • Postgres 16, 3 instances, 자동 failover
  • WAL continuous archiving + daily base backup으로 RPO 5분 / PITR 가능
  • ScheduledBackup이 cron 기반 정기 백업, Backup이 on-demand (마이그레이션 직전 등)
  • enablePodMonitor로 Prometheus 연동
  • podAntiAffinity로 노드 분산
  • backup.retentionPolicy: 30d

❌ 나쁜 예시 2: StatefulSet + cron으로 pg_dump 하나만

apiVersion: batch/v1
kind: CronJob
metadata:
  name: pg-dump-nightly
spec:
  schedule: "0 3 * * *"
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: OnFailure
          securityContext:
            runAsNonRoot: true
            runAsUser: 999
            runAsGroup: 999
            fsGroup: 999
            seccompProfile:
              type: RuntimeDefault
          containers:
            - name: dump
              image: postgres:16
              command: ["sh", "-c", "pg_dumpall -U postgres > /backup/dump.sql"]
              resources:
                requests: { cpu: 100m, memory: 128Mi }
                limits:   { memory: 512Mi }
              securityContext:
                allowPrivilegeEscalation: false
                readOnlyRootFilesystem: true
                capabilities:
                  drop: ["ALL"]
              volumeMounts:
                - { name: tmp, mountPath: /tmp }
                - { name: backup, mountPath: /backup }
          volumes:
            - name: tmp
              emptyDir: {}
            - name: backup
              emptyDir: {}

문제:

  • PITR 불가 (base backup + WAL 아님)
  • single file → 대규모에서 restore 시간 폭증
  • logical dump는 replication slot / extension / large object 처리에 구멍
  • 같은 cluster의 PVC에 저장 시 장애 시 동시 소실

좋은 예시 5: CNPG PITR restore (bootstrap.recovery)

---
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: auth-pg-restore
  namespace: data-prod
spec:
  instances: 3
  imageName: ghcr.io/cloudnative-pg/postgresql:16.4-8
  storage:
    size: 50Gi
    storageClass: fast-ssd-retain
  walStorage:
    size: 20Gi
    storageClass: fast-ssd-retain
  bootstrap:
    recovery:
      source: auth-pg-source
      recoveryTarget:
        targetTime: "2026-04-16 09:45:00.00+00"
  externalClusters:
    - name: auth-pg-source
      barmanObjectStore:
        destinationPath: s3://acme-prod-pg-backups/auth-pg
        endpointURL: https://s3.us-east-1.amazonaws.com
        s3Credentials:
          accessKeyId:    {name: cnpg-s3-credentials, key: ACCESS_KEY_ID}
          secretAccessKey: {name: cnpg-s3-credentials, key: ACCESS_SECRET_KEY}
        wal:
          maxParallel: 8

왜 좋은가:

  • PITR을 declarative CRD로 표현
  • 원본 cluster를 건드리지 않고 별도 auth-pg-restore 클러스터로 복원
  • 특정 시점(targetTime)까지 WAL replay

좋은 예시 6: K3s etcd snapshot + S3 업로드

# /etc/rancher/k3s/config.yaml  (control-plane nodes)
etcd-snapshot-schedule-cron: "0 */6 * * *"
etcd-snapshot-retention: 28
etcd-s3: true
etcd-s3-endpoint: "s3.us-east-1.amazonaws.com"
etcd-s3-bucket: "acme-prod-k3s-etcd"
etcd-s3-folder: "prod-cluster-1"
etcd-s3-region: "us-east-1"
etcd-s3-access-key-file: /var/lib/rancher/k3s/server/etcd-s3-access
etcd-s3-secret-key-file: /var/lib/rancher/k3s/server/etcd-s3-secret
secrets-encryption: true

token 별도 보관 (예: 운영자 금고 / 외부 Vault):

/var/lib/rancher/k3s/server/token   → offline backup, 접근 로그 남김

왜 좋은가:

  • 6시간마다 etcd snapshot + S3 자동 업로드 + 28개 보관
  • secrets encryption 활성화로 snapshot 유출 시 노출 감소
  • server token을 snapshot과 같은 위치에 두지 않음

주의: 이 snapshot은 PVC 데이터를 포함하지 않는다. 반드시 Velero + CNPG backup과 병행.

좋은 예시 7: Vault raft snapshot (CronJob)

---
apiVersion: batch/v1
kind: CronJob
metadata:
  name: vault-raft-snapshot
  namespace: vault
spec:
  schedule: "0 */6 * * *"
  concurrencyPolicy: Forbid
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 3
  jobTemplate:
    spec:
      backoffLimit: 0
      template:
        spec:
          restartPolicy: Never
          serviceAccountName: vault-snapshot
          securityContext:
            runAsNonRoot: true
            runAsUser: 100
            runAsGroup: 1000
            fsGroup: 1000
            seccompProfile:
              type: RuntimeDefault
          containers:
            - name: snapshot
              image: hashicorp/vault@sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef
              env:
                - name: VAULT_ADDR
                  value: https://vault.vault.svc:8200
                - name: VAULT_TOKEN
                  valueFrom:
                    secretKeyRef:
                      name: vault-snapshot-token
                      key: token
              command:
                - sh
                - -c
                - |
                  set -eu
                  TS=$(date -u +%Y%m%dT%H%M%SZ)
                  vault operator raft snapshot save /snap/vault-${TS}.snap
                  aws s3 cp /snap/vault-${TS}.snap s3://acme-prod-vault-snap/ --sse aws:kms
              resources:
                requests: { cpu: 100m, memory: 128Mi }
                limits:   { memory: 512Mi }
              securityContext:
                allowPrivilegeEscalation: false
                readOnlyRootFilesystem: true
                capabilities: {drop: ["ALL"]}
              volumeMounts:
                - {name: snap, mountPath: /snap}
          volumes:
            - name: snap
              emptyDir: {}

왜 좋은가:

  • 6시간마다 raft snapshot + S3 (SSE-KMS) 업로드
  • snapshot용 scoped token 사용 (최소권한)
  • concurrencyPolicy: Forbid로 snapshot 중복 실행 방지

좋은 예시 8: MinIO bucket replication (DR)

# 소스 클러스터 MinIO에서
mc alias set src   https://minio.prod-a.acme.io $SRC_KEY $SRC_SECRET
mc alias set dst   https://minio.prod-b.acme.io $DST_KEY $DST_SECRET

mc admin replicate add src dst
mc version enable src/assets
mc version enable dst/assets
mc replicate add src/assets --remote-bucket dst/assets --replicate "delete,delete-marker,existing-objects,metadata-sync"

# DR 발생 시 (소스 완전 장애 후 복구)
mc replicate resync start src/assets --remote-bucket dst/assets

왜 좋은가:

  • bucket versioning이 replication 전제
  • resync로 DR 복구 경로 확보
  • mc mirror를 단독 DR 수단으로 사용하지 않음

❌ 나쁜 예시 3: mc mirror만 단독 사용

mc mirror --overwrite src/assets dst/assets   # 현재 객체만 동기화, 버전 이력 없음

문제:

  • 버전 이력 / 삭제 marker / metadata 누락
  • 랜섬웨어 / 실수 삭제 시 복구 불가

좋은 예시 9: Restore drill 기록 양식

# /runbooks/restore-drills/2026-Q1-auth-pg.yaml
drill:
  id: drill-2026-q1-auth-pg
  component: cloudnativepg:auth-pg
  tier: gold
  target_rpo: 5m
  target_rto: 30m
  executed_at: 2026-03-18T14:00:00Z
  executor: sre@acme.io
  source_backup: barman:auth-pg/base/20260318T020000
  restore_target_time: "2026-03-17 23:59:00+00"
  restore_cluster: auth-pg-drill
  result:
    status: success
    observed_rpo: 3m
    observed_rto: 22m
    verification_query: "select count(*) from users where created_at < '2026-03-17 23:59:00'"
    verification_result: 1842317
  issues:
    - description: "WAL fetch parallelism bumped from 4 to 8 for better RTO"
      action: "updated Cluster.spec.externalClusters[0].barmanObjectStore.wal.maxParallel to 8"
  next_drill_due: 2026-06-18

왜 좋은가:

  • RPO/RTO 목표 vs 실측을 같이 기록
  • 검증 query 결과까지 남김
  • 다음 drill 예정일이 명시 → 90일 초과 시 경보

나쁜 예시 4: "Git에 manifest 있으니 복구 완료"

✗ manifests are in Git
✗ so restore is solved

문제:

  • DB state, Vault state, MinIO objects, K3s cluster state 모두 복구 안 됨
  • Argo CD sync만으로는 runtime data가 돌아오지 않음

나쁜 예시 5: K3s etcd snapshot만 있으면 PVC도 복구된다고 오해

✗ k3s etcd-snapshot restore → all data back

문제:

  • etcd snapshot은 API object 선언만 복구. PVC 안의 파일은 복구 안 됨
  • 반드시 Velero + DB-level backup과 병행
Reference in New Issue View Git Blame Copy Permalink