565 lines
15 KiB
Markdown
565 lines
15 KiB
Markdown
# backup / restore 예시
|
|
|
|
모든 예시는 실제 매니페스트로 `kubectl apply -f` 가능하다.
|
|
|
|
---
|
|
|
|
## 좋은 예시 1: Velero 설치 후 BackupStorageLocation / VolumeSnapshotLocation
|
|
|
|
```yaml
|
|
---
|
|
apiVersion: velero.io/v1
|
|
kind: BackupStorageLocation
|
|
metadata:
|
|
name: default
|
|
namespace: velero
|
|
labels:
|
|
app.kubernetes.io/part-of: platform-backup
|
|
spec:
|
|
provider: aws
|
|
objectStorage:
|
|
bucket: acme-prod-velero-backups
|
|
prefix: k3s-prod
|
|
config:
|
|
region: us-east-1
|
|
s3ForcePathStyle: "false"
|
|
s3Url: https://s3.us-east-1.amazonaws.com
|
|
default: true
|
|
accessMode: ReadWrite
|
|
credential:
|
|
name: velero-s3-credentials
|
|
key: cloud
|
|
---
|
|
apiVersion: velero.io/v1
|
|
kind: VolumeSnapshotLocation
|
|
metadata:
|
|
name: csi-default
|
|
namespace: velero
|
|
spec:
|
|
provider: csi
|
|
```
|
|
|
|
왜 좋은가:
|
|
- 백업 저장소가 **클러스터 외부** S3 (같은 cluster MinIO에 넣지 않음)
|
|
- credential은 별도 Secret
|
|
- CSI snapshot location이 명시됨
|
|
|
|
❌ 나쁜 예시 1: 같은 cluster 안 MinIO를 백업 저장소로 사용
|
|
|
|
```yaml
|
|
spec:
|
|
provider: aws
|
|
objectStorage:
|
|
bucket: backups
|
|
config:
|
|
s3Url: http://minio.object-prod.svc.cluster.local:9000 # 같은 cluster!
|
|
```
|
|
|
|
문제:
|
|
- cluster 장애 = 백업 동시 소실
|
|
- MinIO 자체를 복구하려면 외부 백업이 또 필요 — 순환 의존
|
|
|
|
---
|
|
|
|
## 좋은 예시 2: Velero Schedule (tier별 분리, 30일 retention)
|
|
|
|
```yaml
|
|
---
|
|
apiVersion: velero.io/v1
|
|
kind: Schedule
|
|
metadata:
|
|
name: gold-daily
|
|
namespace: velero
|
|
labels:
|
|
backup.platform.io/tier: gold
|
|
spec:
|
|
schedule: "0 2 * * *" # 매일 02:00 UTC
|
|
useOwnerReferencesInBackup: true
|
|
template:
|
|
ttl: 720h0m0s # 30일 retention
|
|
includedNamespaces:
|
|
- auth-prod
|
|
- data-prod
|
|
- object-prod
|
|
includedResources:
|
|
- persistentvolumeclaims
|
|
- persistentvolumes
|
|
- secrets
|
|
- configmaps
|
|
- deployments
|
|
- statefulsets
|
|
- services
|
|
- ingresses
|
|
- networkpolicies
|
|
labelSelector:
|
|
matchLabels:
|
|
backup.platform.io/tier: gold
|
|
snapshotVolumes: true
|
|
defaultVolumesToFsBackup: false
|
|
csiSnapshotTimeout: 30m
|
|
storageLocation: default
|
|
volumeSnapshotLocations:
|
|
- csi-default
|
|
hooks:
|
|
resources:
|
|
- name: postgres-consistent
|
|
includedNamespaces: [data-prod]
|
|
labelSelector:
|
|
matchLabels:
|
|
app.kubernetes.io/name: postgres
|
|
pre:
|
|
- exec:
|
|
container: postgres
|
|
command: ["/bin/sh", "-c", "psql -U postgres -c CHECKPOINT"]
|
|
onError: Fail
|
|
timeout: 2m
|
|
---
|
|
apiVersion: velero.io/v1
|
|
kind: Schedule
|
|
metadata:
|
|
name: bronze-weekly-fsb
|
|
namespace: velero
|
|
labels:
|
|
backup.platform.io/tier: bronze
|
|
spec:
|
|
schedule: "0 3 * * 0" # 매주 일요일 03:00 UTC
|
|
template:
|
|
ttl: 2160h0m0s # 90일
|
|
includedNamespaces: ["archive-prod"]
|
|
labelSelector:
|
|
matchLabels:
|
|
backup.platform.io/tier: bronze
|
|
snapshotVolumes: false
|
|
defaultVolumesToFsBackup: true # kopia/restic FSB
|
|
storageLocation: default
|
|
```
|
|
|
|
왜 좋은가:
|
|
- Schedule이 tier별로 분리되어 RPO/retention/도구를 구분
|
|
- CSI snapshot(gold)과 FSB(bronze)를 목적에 맞게 선택
|
|
- Postgres는 pre-hook으로 `CHECKPOINT`를 수행해 crash-consistent에 가까운 스냅샷 확보
|
|
- `labelSelector`가 PVC의 `backup.platform.io/tier`와 매칭
|
|
|
|
---
|
|
|
|
## 좋은 예시 3: Velero Restore
|
|
|
|
```yaml
|
|
---
|
|
apiVersion: velero.io/v1
|
|
kind: Restore
|
|
metadata:
|
|
name: auth-prod-restore-2026-04-16
|
|
namespace: velero
|
|
spec:
|
|
backupName: gold-daily-20260415020000
|
|
includedNamespaces: ["auth-prod"]
|
|
restorePVs: true
|
|
existingResourcePolicy: none # 기존 리소스 보존, 누락된 것만 복원
|
|
namespaceMapping:
|
|
auth-prod: auth-prod-restore # 검증용 별도 네임스페이스로 복원
|
|
labelSelector:
|
|
matchLabels:
|
|
backup.platform.io/tier: gold
|
|
```
|
|
|
|
왜 좋은가:
|
|
- 복원 대상이 `auth-prod-restore`로 분리되어 운영 영향 없이 검증 가능
|
|
- `existingResourcePolicy: none`으로 실수 덮어쓰기 방지
|
|
- `restorePVs: true`로 PVC/PV까지 함께 복원
|
|
|
|
---
|
|
|
|
## 좋은 예시 4: CloudNativePG Cluster + ScheduledBackup + Backup
|
|
|
|
```yaml
|
|
---
|
|
apiVersion: v1
|
|
kind: Namespace
|
|
metadata:
|
|
name: data-prod
|
|
labels:
|
|
pod-security.kubernetes.io/enforce: restricted
|
|
---
|
|
apiVersion: v1
|
|
kind: Secret
|
|
metadata:
|
|
name: cnpg-s3-credentials
|
|
namespace: data-prod
|
|
type: Opaque
|
|
stringData:
|
|
ACCESS_KEY_ID: REPLACE_VIA_VSO
|
|
ACCESS_SECRET_KEY: REPLACE_VIA_VSO
|
|
---
|
|
apiVersion: postgresql.cnpg.io/v1
|
|
kind: Cluster
|
|
metadata:
|
|
name: auth-pg
|
|
namespace: data-prod
|
|
labels:
|
|
app.kubernetes.io/name: auth-pg
|
|
app.kubernetes.io/component: database
|
|
app.kubernetes.io/part-of: auth-platform
|
|
backup.platform.io/tier: gold
|
|
spec:
|
|
instances: 3
|
|
imageName: ghcr.io/cloudnative-pg/postgresql:16.4-8
|
|
primaryUpdateStrategy: unsupervised
|
|
postgresql:
|
|
parameters:
|
|
shared_buffers: "512MB"
|
|
max_connections: "200"
|
|
wal_compression: "on"
|
|
archive_timeout: "60s"
|
|
bootstrap:
|
|
initdb:
|
|
database: auth
|
|
owner: auth_app
|
|
secret:
|
|
name: auth-pg-app
|
|
storage:
|
|
size: 50Gi
|
|
storageClass: fast-ssd-retain
|
|
walStorage:
|
|
size: 20Gi
|
|
storageClass: fast-ssd-retain
|
|
monitoring:
|
|
enablePodMonitor: true
|
|
resources:
|
|
requests: {cpu: "500m", memory: "2Gi"}
|
|
limits: {cpu: "2", memory: "4Gi"}
|
|
backup:
|
|
retentionPolicy: "30d"
|
|
barmanObjectStore:
|
|
destinationPath: s3://acme-prod-pg-backups/auth-pg
|
|
endpointURL: https://s3.us-east-1.amazonaws.com
|
|
s3Credentials:
|
|
accessKeyId:
|
|
name: cnpg-s3-credentials
|
|
key: ACCESS_KEY_ID
|
|
secretAccessKey:
|
|
name: cnpg-s3-credentials
|
|
key: ACCESS_SECRET_KEY
|
|
wal:
|
|
compression: gzip
|
|
maxParallel: 8
|
|
data:
|
|
compression: gzip
|
|
immediateCheckpoint: true
|
|
jobs: 4
|
|
affinity:
|
|
podAntiAffinityType: required
|
|
topologyKey: kubernetes.io/hostname
|
|
---
|
|
apiVersion: postgresql.cnpg.io/v1
|
|
kind: ScheduledBackup
|
|
metadata:
|
|
name: auth-pg-daily
|
|
namespace: data-prod
|
|
spec:
|
|
schedule: "0 0 2 * * *" # 매일 02:00 (CNPG는 6-field cron)
|
|
backupOwnerReference: self
|
|
cluster:
|
|
name: auth-pg
|
|
method: barmanObjectStore
|
|
---
|
|
apiVersion: postgresql.cnpg.io/v1
|
|
kind: Backup
|
|
metadata:
|
|
name: auth-pg-premigration-2026-04-16
|
|
namespace: data-prod
|
|
spec:
|
|
cluster:
|
|
name: auth-pg
|
|
method: barmanObjectStore
|
|
```
|
|
|
|
왜 좋은가:
|
|
- Postgres 16, 3 instances, 자동 failover
|
|
- WAL continuous archiving + daily base backup으로 RPO 5분 / PITR 가능
|
|
- `ScheduledBackup`이 cron 기반 정기 백업, `Backup`이 on-demand (마이그레이션 직전 등)
|
|
- `enablePodMonitor`로 Prometheus 연동
|
|
- `podAntiAffinity`로 노드 분산
|
|
- `backup.retentionPolicy: 30d`
|
|
|
|
❌ 나쁜 예시 2: StatefulSet + cron으로 `pg_dump` 하나만
|
|
|
|
```yaml
|
|
apiVersion: batch/v1
|
|
kind: CronJob
|
|
metadata:
|
|
name: pg-dump-nightly
|
|
spec:
|
|
schedule: "0 3 * * *"
|
|
jobTemplate:
|
|
spec:
|
|
template:
|
|
spec:
|
|
restartPolicy: OnFailure
|
|
securityContext:
|
|
runAsNonRoot: true
|
|
runAsUser: 999
|
|
runAsGroup: 999
|
|
fsGroup: 999
|
|
seccompProfile:
|
|
type: RuntimeDefault
|
|
containers:
|
|
- name: dump
|
|
image: postgres:16
|
|
command: ["sh", "-c", "pg_dumpall -U postgres > /backup/dump.sql"]
|
|
resources:
|
|
requests: { cpu: 100m, memory: 128Mi }
|
|
limits: { memory: 512Mi }
|
|
securityContext:
|
|
allowPrivilegeEscalation: false
|
|
readOnlyRootFilesystem: true
|
|
capabilities:
|
|
drop: ["ALL"]
|
|
volumeMounts:
|
|
- { name: tmp, mountPath: /tmp }
|
|
- { name: backup, mountPath: /backup }
|
|
volumes:
|
|
- name: tmp
|
|
emptyDir: {}
|
|
- name: backup
|
|
emptyDir: {}
|
|
```
|
|
|
|
문제:
|
|
- PITR 불가 (base backup + WAL 아님)
|
|
- single file → 대규모에서 restore 시간 폭증
|
|
- logical dump는 replication slot / extension / large object 처리에 구멍
|
|
- 같은 cluster의 PVC에 저장 시 장애 시 동시 소실
|
|
|
|
---
|
|
|
|
## 좋은 예시 5: CNPG PITR restore (bootstrap.recovery)
|
|
|
|
```yaml
|
|
---
|
|
apiVersion: postgresql.cnpg.io/v1
|
|
kind: Cluster
|
|
metadata:
|
|
name: auth-pg-restore
|
|
namespace: data-prod
|
|
spec:
|
|
instances: 3
|
|
imageName: ghcr.io/cloudnative-pg/postgresql:16.4-8
|
|
storage:
|
|
size: 50Gi
|
|
storageClass: fast-ssd-retain
|
|
walStorage:
|
|
size: 20Gi
|
|
storageClass: fast-ssd-retain
|
|
bootstrap:
|
|
recovery:
|
|
source: auth-pg-source
|
|
recoveryTarget:
|
|
targetTime: "2026-04-16 09:45:00.00+00"
|
|
externalClusters:
|
|
- name: auth-pg-source
|
|
barmanObjectStore:
|
|
destinationPath: s3://acme-prod-pg-backups/auth-pg
|
|
endpointURL: https://s3.us-east-1.amazonaws.com
|
|
s3Credentials:
|
|
accessKeyId: {name: cnpg-s3-credentials, key: ACCESS_KEY_ID}
|
|
secretAccessKey: {name: cnpg-s3-credentials, key: ACCESS_SECRET_KEY}
|
|
wal:
|
|
maxParallel: 8
|
|
```
|
|
|
|
왜 좋은가:
|
|
- PITR을 declarative CRD로 표현
|
|
- 원본 cluster를 건드리지 않고 별도 `auth-pg-restore` 클러스터로 복원
|
|
- 특정 시점(`targetTime`)까지 WAL replay
|
|
|
|
---
|
|
|
|
## 좋은 예시 6: K3s etcd snapshot + S3 업로드
|
|
|
|
```ini
|
|
# /etc/rancher/k3s/config.yaml (control-plane nodes)
|
|
etcd-snapshot-schedule-cron: "0 */6 * * *"
|
|
etcd-snapshot-retention: 28
|
|
etcd-s3: true
|
|
etcd-s3-endpoint: "s3.us-east-1.amazonaws.com"
|
|
etcd-s3-bucket: "acme-prod-k3s-etcd"
|
|
etcd-s3-folder: "prod-cluster-1"
|
|
etcd-s3-region: "us-east-1"
|
|
etcd-s3-access-key-file: /var/lib/rancher/k3s/server/etcd-s3-access
|
|
etcd-s3-secret-key-file: /var/lib/rancher/k3s/server/etcd-s3-secret
|
|
secrets-encryption: true
|
|
```
|
|
|
|
token 별도 보관 (예: 운영자 금고 / 외부 Vault):
|
|
```
|
|
/var/lib/rancher/k3s/server/token → offline backup, 접근 로그 남김
|
|
```
|
|
|
|
왜 좋은가:
|
|
- 6시간마다 etcd snapshot + S3 자동 업로드 + 28개 보관
|
|
- secrets encryption 활성화로 snapshot 유출 시 노출 감소
|
|
- server token을 snapshot과 같은 위치에 두지 않음
|
|
|
|
**주의: 이 snapshot은 PVC 데이터를 포함하지 않는다. 반드시 Velero + CNPG backup과 병행.**
|
|
|
|
---
|
|
|
|
## 좋은 예시 7: Vault raft snapshot (CronJob)
|
|
|
|
```yaml
|
|
---
|
|
apiVersion: batch/v1
|
|
kind: CronJob
|
|
metadata:
|
|
name: vault-raft-snapshot
|
|
namespace: vault
|
|
spec:
|
|
schedule: "0 */6 * * *"
|
|
concurrencyPolicy: Forbid
|
|
successfulJobsHistoryLimit: 3
|
|
failedJobsHistoryLimit: 3
|
|
jobTemplate:
|
|
spec:
|
|
backoffLimit: 0
|
|
template:
|
|
spec:
|
|
restartPolicy: Never
|
|
serviceAccountName: vault-snapshot
|
|
securityContext:
|
|
runAsNonRoot: true
|
|
runAsUser: 100
|
|
runAsGroup: 1000
|
|
fsGroup: 1000
|
|
seccompProfile:
|
|
type: RuntimeDefault
|
|
containers:
|
|
- name: snapshot
|
|
image: hashicorp/vault@sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef
|
|
env:
|
|
- name: VAULT_ADDR
|
|
value: https://vault.vault.svc:8200
|
|
- name: VAULT_TOKEN
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: vault-snapshot-token
|
|
key: token
|
|
command:
|
|
- sh
|
|
- -c
|
|
- |
|
|
set -eu
|
|
TS=$(date -u +%Y%m%dT%H%M%SZ)
|
|
vault operator raft snapshot save /snap/vault-${TS}.snap
|
|
aws s3 cp /snap/vault-${TS}.snap s3://acme-prod-vault-snap/ --sse aws:kms
|
|
resources:
|
|
requests: { cpu: 100m, memory: 128Mi }
|
|
limits: { memory: 512Mi }
|
|
securityContext:
|
|
allowPrivilegeEscalation: false
|
|
readOnlyRootFilesystem: true
|
|
capabilities: {drop: ["ALL"]}
|
|
volumeMounts:
|
|
- {name: snap, mountPath: /snap}
|
|
volumes:
|
|
- name: snap
|
|
emptyDir: {}
|
|
```
|
|
|
|
왜 좋은가:
|
|
- 6시간마다 raft snapshot + S3 (SSE-KMS) 업로드
|
|
- snapshot용 scoped token 사용 (최소권한)
|
|
- `concurrencyPolicy: Forbid`로 snapshot 중복 실행 방지
|
|
|
|
---
|
|
|
|
## 좋은 예시 8: MinIO bucket replication (DR)
|
|
|
|
```bash
|
|
# 소스 클러스터 MinIO에서
|
|
mc alias set src https://minio.prod-a.acme.io $SRC_KEY $SRC_SECRET
|
|
mc alias set dst https://minio.prod-b.acme.io $DST_KEY $DST_SECRET
|
|
|
|
mc admin replicate add src dst
|
|
mc version enable src/assets
|
|
mc version enable dst/assets
|
|
mc replicate add src/assets --remote-bucket dst/assets --replicate "delete,delete-marker,existing-objects,metadata-sync"
|
|
|
|
# DR 발생 시 (소스 완전 장애 후 복구)
|
|
mc replicate resync start src/assets --remote-bucket dst/assets
|
|
```
|
|
|
|
왜 좋은가:
|
|
- bucket versioning이 replication 전제
|
|
- `resync`로 DR 복구 경로 확보
|
|
- `mc mirror`를 단독 DR 수단으로 사용하지 않음
|
|
|
|
❌ 나쁜 예시 3: `mc mirror`만 단독 사용
|
|
|
|
```bash
|
|
mc mirror --overwrite src/assets dst/assets # 현재 객체만 동기화, 버전 이력 없음
|
|
```
|
|
|
|
문제:
|
|
- 버전 이력 / 삭제 marker / metadata 누락
|
|
- 랜섬웨어 / 실수 삭제 시 복구 불가
|
|
|
|
---
|
|
|
|
## 좋은 예시 9: Restore drill 기록 양식
|
|
|
|
```yaml
|
|
# /runbooks/restore-drills/2026-Q1-auth-pg.yaml
|
|
drill:
|
|
id: drill-2026-q1-auth-pg
|
|
component: cloudnativepg:auth-pg
|
|
tier: gold
|
|
target_rpo: 5m
|
|
target_rto: 30m
|
|
executed_at: 2026-03-18T14:00:00Z
|
|
executor: sre@acme.io
|
|
source_backup: barman:auth-pg/base/20260318T020000
|
|
restore_target_time: "2026-03-17 23:59:00+00"
|
|
restore_cluster: auth-pg-drill
|
|
result:
|
|
status: success
|
|
observed_rpo: 3m
|
|
observed_rto: 22m
|
|
verification_query: "select count(*) from users where created_at < '2026-03-17 23:59:00'"
|
|
verification_result: 1842317
|
|
issues:
|
|
- description: "WAL fetch parallelism bumped from 4 to 8 for better RTO"
|
|
action: "updated Cluster.spec.externalClusters[0].barmanObjectStore.wal.maxParallel to 8"
|
|
next_drill_due: 2026-06-18
|
|
```
|
|
|
|
왜 좋은가:
|
|
- RPO/RTO 목표 vs 실측을 같이 기록
|
|
- 검증 query 결과까지 남김
|
|
- 다음 drill 예정일이 명시 → 90일 초과 시 경보
|
|
|
|
---
|
|
|
|
## 나쁜 예시 4: "Git에 manifest 있으니 복구 완료"
|
|
|
|
```
|
|
✗ manifests are in Git
|
|
✗ so restore is solved
|
|
```
|
|
|
|
문제:
|
|
- DB state, Vault state, MinIO objects, K3s cluster state 모두 복구 안 됨
|
|
- Argo CD sync만으로는 runtime data가 돌아오지 않음
|
|
|
|
---
|
|
|
|
## 나쁜 예시 5: K3s etcd snapshot만 있으면 PVC도 복구된다고 오해
|
|
|
|
```
|
|
✗ k3s etcd-snapshot restore → all data back
|
|
```
|
|
|
|
문제:
|
|
- etcd snapshot은 API object 선언만 복구. PVC 안의 파일은 복구 안 됨
|
|
- 반드시 Velero + DB-level backup과 병행
|