init: 폴더구조 설계 및 인프라 설계

2026-07-24 14:31:53 +09:00
parent 34ad612281
commit f9c463f87a
1839 changed files with 323096 additions and 1 deletions
@@ -0,0 +1,564 @@
+# backup / restore 예시
+
+모든 예시는 실제 매니페스트로 `kubectl apply -f` 가능하다.
+
+---
+
+## 좋은 예시 1: Velero 설치 후 BackupStorageLocation / VolumeSnapshotLocation
+
+```yaml
+---
+apiVersion: velero.io/v1
+kind: BackupStorageLocation
+metadata:
+  name: default
+  namespace: velero
+  labels:
+    app.kubernetes.io/part-of: platform-backup
+spec:
+  provider: aws
+  objectStorage:
+    bucket: acme-prod-velero-backups
+    prefix: k3s-prod
+  config:
+    region: us-east-1
+    s3ForcePathStyle: "false"
+    s3Url: https://s3.us-east-1.amazonaws.com
+  default: true
+  accessMode: ReadWrite
+  credential:
+    name: velero-s3-credentials
+    key: cloud
+---
+apiVersion: velero.io/v1
+kind: VolumeSnapshotLocation
+metadata:
+  name: csi-default
+  namespace: velero
+spec:
+  provider: csi
+```
+
+왜 좋은가:
+- 백업 저장소가 **클러스터 외부** S3 (같은 cluster MinIO에 넣지 않음)
+- credential은 별도 Secret
+- CSI snapshot location이 명시됨
+
+❌ 나쁜 예시 1: 같은 cluster 안 MinIO를 백업 저장소로 사용
+
+```yaml
+spec:
+  provider: aws
+  objectStorage:
+    bucket: backups
+  config:
+    s3Url: http://minio.object-prod.svc.cluster.local:9000   # 같은 cluster!
+```
+
+문제:
+- cluster 장애 = 백업 동시 소실
+- MinIO 자체를 복구하려면 외부 백업이 또 필요 — 순환 의존
+
+---
+
+## 좋은 예시 2: Velero Schedule (tier별 분리, 30일 retention)
+
+```yaml
+---
+apiVersion: velero.io/v1
+kind: Schedule
+metadata:
+  name: gold-daily
+  namespace: velero
+  labels:
+    backup.platform.io/tier: gold
+spec:
+  schedule: "0 2 * * *"                     # 매일 02:00 UTC
+  useOwnerReferencesInBackup: true
+  template:
+    ttl: 720h0m0s                           # 30일 retention
+    includedNamespaces:
+      - auth-prod
+      - data-prod
+      - object-prod
+    includedResources:
+      - persistentvolumeclaims
+      - persistentvolumes
+      - secrets
+      - configmaps
+      - deployments
+      - statefulsets
+      - services
+      - ingresses
+      - networkpolicies
+    labelSelector:
+      matchLabels:
+        backup.platform.io/tier: gold
+    snapshotVolumes: true
+    defaultVolumesToFsBackup: false
+    csiSnapshotTimeout: 30m
+    storageLocation: default
+    volumeSnapshotLocations:
+      - csi-default
+    hooks:
+      resources:
+        - name: postgres-consistent
+          includedNamespaces: [data-prod]
+          labelSelector:
+            matchLabels:
+              app.kubernetes.io/name: postgres
+          pre:
+            - exec:
+                container: postgres
+                command: ["/bin/sh", "-c", "psql -U postgres -c CHECKPOINT"]
+                onError: Fail
+                timeout: 2m
+---
+apiVersion: velero.io/v1
+kind: Schedule
+metadata:
+  name: bronze-weekly-fsb
+  namespace: velero
+  labels:
+    backup.platform.io/tier: bronze
+spec:
+  schedule: "0 3 * * 0"                     # 매주 일요일 03:00 UTC
+  template:
+    ttl: 2160h0m0s                          # 90일
+    includedNamespaces: ["archive-prod"]
+    labelSelector:
+      matchLabels:
+        backup.platform.io/tier: bronze
+    snapshotVolumes: false
+    defaultVolumesToFsBackup: true          # kopia/restic FSB
+    storageLocation: default
+```
+
+왜 좋은가:
+- Schedule이 tier별로 분리되어 RPO/retention/도구를 구분
+- CSI snapshot(gold)과 FSB(bronze)를 목적에 맞게 선택
+- Postgres는 pre-hook으로 `CHECKPOINT`를 수행해 crash-consistent에 가까운 스냅샷 확보
+- `labelSelector`가 PVC의 `backup.platform.io/tier`와 매칭
+
+---
+
+## 좋은 예시 3: Velero Restore
+
+```yaml
+---
+apiVersion: velero.io/v1
+kind: Restore
+metadata:
+  name: auth-prod-restore-2026-04-16
+  namespace: velero
+spec:
+  backupName: gold-daily-20260415020000
+  includedNamespaces: ["auth-prod"]
+  restorePVs: true
+  existingResourcePolicy: none              # 기존 리소스 보존, 누락된 것만 복원
+  namespaceMapping:
+    auth-prod: auth-prod-restore            # 검증용 별도 네임스페이스로 복원
+  labelSelector:
+    matchLabels:
+      backup.platform.io/tier: gold
+```
+
+왜 좋은가:
+- 복원 대상이 `auth-prod-restore`로 분리되어 운영 영향 없이 검증 가능
+- `existingResourcePolicy: none`으로 실수 덮어쓰기 방지
+- `restorePVs: true`로 PVC/PV까지 함께 복원
+
+---
+
+## 좋은 예시 4: CloudNativePG Cluster + ScheduledBackup + Backup
+
+```yaml
+---
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: data-prod
+  labels:
+    pod-security.kubernetes.io/enforce: restricted
+---
+apiVersion: v1
+kind: Secret
+metadata:
+  name: cnpg-s3-credentials
+  namespace: data-prod
+type: Opaque
+stringData:
+  ACCESS_KEY_ID: REPLACE_VIA_VSO
+  ACCESS_SECRET_KEY: REPLACE_VIA_VSO
+---
+apiVersion: postgresql.cnpg.io/v1
+kind: Cluster
+metadata:
+  name: auth-pg
+  namespace: data-prod
+  labels:
+    app.kubernetes.io/name: auth-pg
+    app.kubernetes.io/component: database
+    app.kubernetes.io/part-of: auth-platform
+    backup.platform.io/tier: gold
+spec:
+  instances: 3
+  imageName: ghcr.io/cloudnative-pg/postgresql:16.4-8
+  primaryUpdateStrategy: unsupervised
+  postgresql:
+    parameters:
+      shared_buffers: "512MB"
+      max_connections: "200"
+      wal_compression: "on"
+      archive_timeout: "60s"
+  bootstrap:
+    initdb:
+      database: auth
+      owner: auth_app
+      secret:
+        name: auth-pg-app
+  storage:
+    size: 50Gi
+    storageClass: fast-ssd-retain
+  walStorage:
+    size: 20Gi
+    storageClass: fast-ssd-retain
+  monitoring:
+    enablePodMonitor: true
+  resources:
+    requests: {cpu: "500m", memory: "2Gi"}
+    limits:   {cpu: "2",    memory: "4Gi"}
+  backup:
+    retentionPolicy: "30d"
+    barmanObjectStore:
+      destinationPath: s3://acme-prod-pg-backups/auth-pg
+      endpointURL: https://s3.us-east-1.amazonaws.com
+      s3Credentials:
+        accessKeyId:
+          name: cnpg-s3-credentials
+          key: ACCESS_KEY_ID
+        secretAccessKey:
+          name: cnpg-s3-credentials
+          key: ACCESS_SECRET_KEY
+      wal:
+        compression: gzip
+        maxParallel: 8
+      data:
+        compression: gzip
+        immediateCheckpoint: true
+        jobs: 4
+  affinity:
+    podAntiAffinityType: required
+    topologyKey: kubernetes.io/hostname
+---
+apiVersion: postgresql.cnpg.io/v1
+kind: ScheduledBackup
+metadata:
+  name: auth-pg-daily
+  namespace: data-prod
+spec:
+  schedule: "0 0 2 * * *"                   # 매일 02:00 (CNPG는 6-field cron)
+  backupOwnerReference: self
+  cluster:
+    name: auth-pg
+  method: barmanObjectStore
+---
+apiVersion: postgresql.cnpg.io/v1
+kind: Backup
+metadata:
+  name: auth-pg-premigration-2026-04-16
+  namespace: data-prod
+spec:
+  cluster:
+    name: auth-pg
+  method: barmanObjectStore
+```
+
+왜 좋은가:
+- Postgres 16, 3 instances, 자동 failover
+- WAL continuous archiving + daily base backup으로 RPO 5분 / PITR 가능
+- `ScheduledBackup`이 cron 기반 정기 백업, `Backup`이 on-demand (마이그레이션 직전 등)
+- `enablePodMonitor`로 Prometheus 연동
+- `podAntiAffinity`로 노드 분산
+- `backup.retentionPolicy: 30d`
+
+❌ 나쁜 예시 2: StatefulSet + cron으로 `pg_dump` 하나만
+
+```yaml
+apiVersion: batch/v1
+kind: CronJob
+metadata:
+  name: pg-dump-nightly
+spec:
+  schedule: "0 3 * * *"
+  jobTemplate:
+    spec:
+      template:
+        spec:
+          restartPolicy: OnFailure
+          securityContext:
+            runAsNonRoot: true
+            runAsUser: 999
+            runAsGroup: 999
+            fsGroup: 999
+            seccompProfile:
+              type: RuntimeDefault
+          containers:
+            - name: dump
+              image: postgres:16
+              command: ["sh", "-c", "pg_dumpall -U postgres > /backup/dump.sql"]
+              resources:
+                requests: { cpu: 100m, memory: 128Mi }
+                limits:   { memory: 512Mi }
+              securityContext:
+                allowPrivilegeEscalation: false
+                readOnlyRootFilesystem: true
+                capabilities:
+                  drop: ["ALL"]
+              volumeMounts:
+                - { name: tmp, mountPath: /tmp }
+                - { name: backup, mountPath: /backup }
+          volumes:
+            - name: tmp
+              emptyDir: {}
+            - name: backup
+              emptyDir: {}
+```
+
+문제:
+- PITR 불가 (base backup + WAL 아님)
+- single file → 대규모에서 restore 시간 폭증
+- logical dump는 replication slot / extension / large object 처리에 구멍
+- 같은 cluster의 PVC에 저장 시 장애 시 동시 소실
+
+---
+
+## 좋은 예시 5: CNPG PITR restore (bootstrap.recovery)
+
+```yaml
+---
+apiVersion: postgresql.cnpg.io/v1
+kind: Cluster
+metadata:
+  name: auth-pg-restore
+  namespace: data-prod
+spec:
+  instances: 3
+  imageName: ghcr.io/cloudnative-pg/postgresql:16.4-8
+  storage:
+    size: 50Gi
+    storageClass: fast-ssd-retain
+  walStorage:
+    size: 20Gi
+    storageClass: fast-ssd-retain
+  bootstrap:
+    recovery:
+      source: auth-pg-source
+      recoveryTarget:
+        targetTime: "2026-04-16 09:45:00.00+00"
+  externalClusters:
+    - name: auth-pg-source
+      barmanObjectStore:
+        destinationPath: s3://acme-prod-pg-backups/auth-pg
+        endpointURL: https://s3.us-east-1.amazonaws.com
+        s3Credentials:
+          accessKeyId:    {name: cnpg-s3-credentials, key: ACCESS_KEY_ID}
+          secretAccessKey: {name: cnpg-s3-credentials, key: ACCESS_SECRET_KEY}
+        wal:
+          maxParallel: 8
+```
+
+왜 좋은가:
+- PITR을 declarative CRD로 표현
+- 원본 cluster를 건드리지 않고 별도 `auth-pg-restore` 클러스터로 복원
+- 특정 시점(`targetTime`)까지 WAL replay
+
+---
+
+## 좋은 예시 6: K3s etcd snapshot + S3 업로드
+
+```ini
+# /etc/rancher/k3s/config.yaml  (control-plane nodes)
+etcd-snapshot-schedule-cron: "0 */6 * * *"
+etcd-snapshot-retention: 28
+etcd-s3: true
+etcd-s3-endpoint: "s3.us-east-1.amazonaws.com"
+etcd-s3-bucket: "acme-prod-k3s-etcd"
+etcd-s3-folder: "prod-cluster-1"
+etcd-s3-region: "us-east-1"
+etcd-s3-access-key-file: /var/lib/rancher/k3s/server/etcd-s3-access
+etcd-s3-secret-key-file: /var/lib/rancher/k3s/server/etcd-s3-secret
+secrets-encryption: true
+```
+
+token 별도 보관 (예: 운영자 금고 / 외부 Vault):
+```
+/var/lib/rancher/k3s/server/token   → offline backup, 접근 로그 남김
+```
+
+왜 좋은가:
+- 6시간마다 etcd snapshot + S3 자동 업로드 + 28개 보관
+- secrets encryption 활성화로 snapshot 유출 시 노출 감소
+- server token을 snapshot과 같은 위치에 두지 않음
+
+**주의: 이 snapshot은 PVC 데이터를 포함하지 않는다. 반드시 Velero + CNPG backup과 병행.**
+
+---
+
+## 좋은 예시 7: Vault raft snapshot (CronJob)
+
+```yaml
+---
+apiVersion: batch/v1
+kind: CronJob
+metadata:
+  name: vault-raft-snapshot
+  namespace: vault
+spec:
+  schedule: "0 */6 * * *"
+  concurrencyPolicy: Forbid
+  successfulJobsHistoryLimit: 3
+  failedJobsHistoryLimit: 3
+  jobTemplate:
+    spec:
+      backoffLimit: 0
+      template:
+        spec:
+          restartPolicy: Never
+          serviceAccountName: vault-snapshot
+          securityContext:
+            runAsNonRoot: true
+            runAsUser: 100
+            runAsGroup: 1000
+            fsGroup: 1000
+            seccompProfile:
+              type: RuntimeDefault
+          containers:
+            - name: snapshot
+              image: hashicorp/vault@sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef
+              env:
+                - name: VAULT_ADDR
+                  value: https://vault.vault.svc:8200
+                - name: VAULT_TOKEN
+                  valueFrom:
+                    secretKeyRef:
+                      name: vault-snapshot-token
+                      key: token
+              command:
+                - sh
+                - -c
+                - |
+                  set -eu
+                  TS=$(date -u +%Y%m%dT%H%M%SZ)
+                  vault operator raft snapshot save /snap/vault-${TS}.snap
+                  aws s3 cp /snap/vault-${TS}.snap s3://acme-prod-vault-snap/ --sse aws:kms
+              resources:
+                requests: { cpu: 100m, memory: 128Mi }
+                limits:   { memory: 512Mi }
+              securityContext:
+                allowPrivilegeEscalation: false
+                readOnlyRootFilesystem: true
+                capabilities: {drop: ["ALL"]}
+              volumeMounts:
+                - {name: snap, mountPath: /snap}
+          volumes:
+            - name: snap
+              emptyDir: {}
+```
+
+왜 좋은가:
+- 6시간마다 raft snapshot + S3 (SSE-KMS) 업로드
+- snapshot용 scoped token 사용 (최소권한)
+- `concurrencyPolicy: Forbid`로 snapshot 중복 실행 방지
+
+---
+
+## 좋은 예시 8: MinIO bucket replication (DR)
+
+```bash
+# 소스 클러스터 MinIO에서
+mc alias set src   https://minio.prod-a.acme.io $SRC_KEY $SRC_SECRET
+mc alias set dst   https://minio.prod-b.acme.io $DST_KEY $DST_SECRET
+
+mc admin replicate add src dst
+mc version enable src/assets
+mc version enable dst/assets
+mc replicate add src/assets --remote-bucket dst/assets --replicate "delete,delete-marker,existing-objects,metadata-sync"
+
+# DR 발생 시 (소스 완전 장애 후 복구)
+mc replicate resync start src/assets --remote-bucket dst/assets
+```
+
+왜 좋은가:
+- bucket versioning이 replication 전제
+- `resync`로 DR 복구 경로 확보
+- `mc mirror`를 단독 DR 수단으로 사용하지 않음
+
+❌ 나쁜 예시 3: `mc mirror`만 단독 사용
+
+```bash
+mc mirror --overwrite src/assets dst/assets   # 현재 객체만 동기화, 버전 이력 없음
+```
+
+문제:
+- 버전 이력 / 삭제 marker / metadata 누락
+- 랜섬웨어 / 실수 삭제 시 복구 불가
+
+---
+
+## 좋은 예시 9: Restore drill 기록 양식
+
+```yaml
+# /runbooks/restore-drills/2026-Q1-auth-pg.yaml
+drill:
+  id: drill-2026-q1-auth-pg
+  component: cloudnativepg:auth-pg
+  tier: gold
+  target_rpo: 5m
+  target_rto: 30m
+  executed_at: 2026-03-18T14:00:00Z
+  executor: sre@acme.io
+  source_backup: barman:auth-pg/base/20260318T020000
+  restore_target_time: "2026-03-17 23:59:00+00"
+  restore_cluster: auth-pg-drill
+  result:
+    status: success
+    observed_rpo: 3m
+    observed_rto: 22m
+    verification_query: "select count(*) from users where created_at < '2026-03-17 23:59:00'"
+    verification_result: 1842317
+  issues:
+    - description: "WAL fetch parallelism bumped from 4 to 8 for better RTO"
+      action: "updated Cluster.spec.externalClusters[0].barmanObjectStore.wal.maxParallel to 8"
+  next_drill_due: 2026-06-18
+```
+
+왜 좋은가:
+- RPO/RTO 목표 vs 실측을 같이 기록
+- 검증 query 결과까지 남김
+- 다음 drill 예정일이 명시 → 90일 초과 시 경보
+
+---
+
+## 나쁜 예시 4: "Git에 manifest 있으니 복구 완료"
+
+```
+✗ manifests are in Git
+✗ so restore is solved
+```
+
+문제:
+- DB state, Vault state, MinIO objects, K3s cluster state 모두 복구 안 됨
+- Argo CD sync만으로는 runtime data가 돌아오지 않음
+
+---
+
+## 나쁜 예시 5: K3s etcd snapshot만 있으면 PVC도 복구된다고 오해
+
+```
+✗ k3s etcd-snapshot restore → all data back
+```
+
+문제:
+- etcd snapshot은 API object 선언만 복구. PVC 안의 파일은 복구 안 됨
+- 반드시 Velero + DB-level backup과 병행