project-infra/docs/examples/infra/db-and-migration.md

db / migration 예시

모든 YAML은 kubectl apply 가능하다. 상세 Flyway Job 예시는 docs/examples/infra/flyway.md 참조.

---

좋은 예시 1: auth-server와 keycloak DB 경계 분리 (CNPG 2 cluster)

---
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: auth-pg
  namespace: data-prod
  labels:
    app.kubernetes.io/name: auth-pg
    app.kubernetes.io/part-of: auth-platform
    backup.platform.io/tier: gold
spec:
  instances: 3
  imageName: ghcr.io/cloudnative-pg/postgresql:16.4-8
  bootstrap:
    initdb:
      database: auth
      owner: auth_app
      secret: {name: auth-pg-app}
  storage:    {size: 50Gi, storageClass: fast-ssd-retain}
  walStorage: {size: 20Gi, storageClass: fast-ssd-retain}
  monitoring: {enablePodMonitor: true}
  backup:
    retentionPolicy: "30d"
    barmanObjectStore:
      destinationPath: s3://acme-prod-pg-backups/auth-pg
      s3Credentials:
        accessKeyId:     {name: cnpg-s3-credentials, key: ACCESS_KEY_ID}
        secretAccessKey: {name: cnpg-s3-credentials, key: ACCESS_SECRET_KEY}
      wal:  {compression: gzip, maxParallel: 8}
      data: {compression: gzip, immediateCheckpoint: true, jobs: 4}
---
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: keycloak-pg
  namespace: data-prod
  labels:
    app.kubernetes.io/name: keycloak-pg
    app.kubernetes.io/part-of: identity-platform
    backup.platform.io/tier: gold
spec:
  instances: 3
  imageName: ghcr.io/cloudnative-pg/postgresql:16.4-8
  bootstrap:
    initdb:
      database: keycloak
      owner: keycloak
      secret: {name: keycloak-pg-app}
  storage:    {size: 30Gi, storageClass: fast-ssd-retain}
  walStorage: {size: 10Gi, storageClass: fast-ssd-retain}
  monitoring: {enablePodMonitor: true}
  backup:
    retentionPolicy: "30d"
    barmanObjectStore:
      destinationPath: s3://acme-prod-pg-backups/keycloak-pg
      s3Credentials:
        accessKeyId:     {name: cnpg-s3-credentials, key: ACCESS_KEY_ID}
        secretAccessKey: {name: cnpg-s3-credentials, key: ACCESS_SECRET_KEY}
      wal:  {compression: gzip, maxParallel: 8}
      data: {compression: gzip, jobs: 4}

왜 좋은가:

• auth와 keycloak이 별도 CNPG cluster → 장애 / 업그레이드 영향 분리
• 각각 schema ownership이 분리되어 migration 파이프라인도 분리 가능
• 백업 destination path도 분리 → retention / 암호화 정책 독립

❌ 나쁜 예시 1: 하나의 cluster의 하나의 database에 두 서비스 schema

# single CNPG cluster, database=shared
# auth-server uses schema "auth"
# keycloak uses schema "keycloak"
# one Flyway project manages both

문제:

• 서비스별 업그레이드 / restore 영향 격리 불가
• Flyway history가 서로 섞임
• 한 서비스가 lock을 오래 잡으면 다른 서비스가 멈춤

---

좋은 예시 2: migration을 Helm hook으로 app보다 먼저 실행

---
apiVersion: batch/v1
kind: Job
metadata:
  name: auth-flyway-migrate
  namespace: auth-prod
  labels:
    app.kubernetes.io/name: auth-server
    app.kubernetes.io/component: db-migration
    app.kubernetes.io/managed-by: Helm
  annotations:
    "helm.sh/hook": "pre-upgrade,pre-install"
    "helm.sh/hook-weight": "-10"
    "helm.sh/hook-delete-policy": "before-hook-creation,hook-succeeded"
spec:
  parallelism: 1
  completions: 1
  backoffLimit: 0
  activeDeadlineSeconds: 1800
  ttlSecondsAfterFinished: 86400
  template:
    spec:
      restartPolicy: Never
      securityContext:
        runAsNonRoot: true
        runAsUser: 1000
        fsGroup: 1000
        seccompProfile: {type: RuntimeDefault}
      containers:
        - name: flyway
          image: flyway/flyway@sha256:7d9f7c4e2a1b4c5d6e7f8a9b0c1d2e3f4a5b6c7d8e9f0a1b2c3d4e5f6a7b8c9d
          args: ["-X", "migrate"]
          env:
            - {name: FLYWAY_URL,      value: "jdbc:postgresql://auth-pg-rw.data-prod.svc:5432/auth"}
            - {name: FLYWAY_USER,     value: "auth_app"}
            - {name: FLYWAY_LOCATIONS,           value: "filesystem:/flyway/sql"}
            - {name: FLYWAY_SCHEMAS,             value: "auth_server"}
            - {name: FLYWAY_DEFAULT_SCHEMA,      value: "auth_server"}
            - {name: FLYWAY_TABLE,               value: "flyway_schema_history"}
            - {name: FLYWAY_VALIDATE_ON_MIGRATE, value: "true"}
            - {name: FLYWAY_BASELINE_ON_MIGRATE, value: "false"}
            - {name: FLYWAY_CLEAN_DISABLED,      value: "true"}
            - name: FLYWAY_PASSWORD
              valueFrom: {secretKeyRef: {name: auth-pg-app, key: password}}
          resources:
            requests: {cpu: "100m", memory: "256Mi"}
            limits:   {cpu: "1",    memory: "1Gi"}
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities: {drop: ["ALL"]}
          volumeMounts:
            - {name: sql, mountPath: /flyway/sql, readOnly: true}
            - {name: tmp, mountPath: /tmp}
      volumes:
        - name: sql
          configMap: {name: auth-flyway-sql}
        - name: tmp
          emptyDir: {}

왜 좋은가:

• Helm hook으로 app install/upgrade보다 먼저 실행 (-10 weight)
• before-hook-creation,hook-succeeded 삭제 정책으로 이전 Job 깨끗이 정리
• cleanDisabled=true 명시 (실수로 flyway clean 방지)
• parallelism: 1, backoffLimit: 0, activeDeadlineSeconds: 1800
• digest pinning, restricted PSA

---

좋은 예시 3: Argo CD sync wave로 순서 지정

---
apiVersion: batch/v1
kind: Job
metadata:
  name: auth-flyway-migrate
  namespace: auth-prod
  labels:
    app.kubernetes.io/name: auth-server
    app.kubernetes.io/component: db-migration
  annotations:
    argocd.argoproj.io/sync-wave: "-1"
    argocd.argoproj.io/hook: Sync
    argocd.argoproj.io/hook-delete-policy: BeforeHookCreation
spec:
  parallelism: 1
  completions: 1
  backoffLimit: 0
  activeDeadlineSeconds: 1800
  ttlSecondsAfterFinished: 86400
  template:
    spec:
      restartPolicy: Never
      securityContext:
        runAsNonRoot: true
        runAsUser: 1000
        runAsGroup: 1000
        fsGroup: 1000
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: flyway
          image: flyway/flyway@sha256:7d9f7c4e2a1b4c5d6e7f8a9b0c1d2e3f4a5b6c7d8e9f0a1b2c3d4e5f6a7b8c9d
          args: ["-X", "migrate"]
          resources:
            requests: { cpu: 100m, memory: 256Mi }
            limits:   { memory: 1Gi }
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop: ["ALL"]
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: auth-server
  namespace: auth-prod
  labels:
    app.kubernetes.io/name: auth-server
    app.kubernetes.io/instance: auth-server-prod
  annotations:
    argocd.argoproj.io/sync-wave: "0"
spec:
  replicas: 3
  selector:
    matchLabels:
      app.kubernetes.io/name: auth-server
      app.kubernetes.io/instance: auth-server-prod
  template:
    metadata:
      labels:
        app.kubernetes.io/name: auth-server
        app.kubernetes.io/instance: auth-server-prod
    spec:
      securityContext:
        runAsNonRoot: true
        runAsUser: 10001
        runAsGroup: 10001
        fsGroup: 10001
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: auth-server
          image: registry.example.com/identity/auth-server:1.24.0
          resources:
            requests: { cpu: 500m, memory: 1Gi }
            limits:   { memory: 1536Mi }
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop: ["ALL"]

왜 좋은가:

• Argo CD는 sync-wave가 낮은 것부터 실행
• Helm hook과 혼용하지 않음 (한쪽만 사용)

❌ 나쁜 예시 2: Helm hook + Argo CD hook 혼용

annotations:
  "helm.sh/hook": "pre-upgrade"
  "argocd.argoproj.io/sync-wave": "-1"
  "argocd.argoproj.io/hook": Sync

문제:

• Argo CD가 Helm chart를 렌더링할 때 Helm hook을 일반 리소스로 취급해 sync 순서가 꼬임
• 실행이 중복되거나 누락됨
• 한 방식으로 통일할 것

---

좋은 예시 4: Expand → Migrate → Contract 3단계 릴리즈

배경

users 테이블의 email 컬럼 (NULL 허용)을 NOT NULL + 정규화된 email_canonical 컬럼으로 바꾸고 싶다.

Release 1 — Expand

V120__add_email_canonical_nullable.sql:

-- flyway:executeInTransaction=false
ALTER TABLE users ADD COLUMN email_canonical text;
CREATE INDEX CONCURRENTLY idx_users_email_canonical ON users(email_canonical);

V121__backfill_email_canonical.sql (같은 릴리즈 또는 별도 배치 Job):

UPDATE users
   SET email_canonical = lower(trim(email))
 WHERE email_canonical IS NULL
   AND email IS NOT NULL;

앱은 쓰기: email + email_canonical 둘 다 채움. 읽기: 여전히 email.

Release 2 — Migrate

앱 읽기 경로를 email_canonical로 전환. 새 가입/수정은 email_canonical만 보장.

V122__add_email_canonical_not_null.sql:

-- 이 시점에는 모든 row에 email_canonical이 채워져 있어야 함
ALTER TABLE users ALTER COLUMN email_canonical SET NOT NULL;
ALTER TABLE users ADD CONSTRAINT users_email_canonical_unique UNIQUE (email_canonical);

Release 3 — Contract

앱이 email 컬럼을 더 이상 읽지/쓰지 않는 버전으로 완전히 롤아웃된 뒤.

V130__drop_legacy_email_column.sql:

ALTER TABLE users DROP COLUMN email;

왜 좋은가:

• 각 릴리즈가 N-1 ↔ N 동시 운영 가능
• CREATE INDEX CONCURRENTLY는 -- flyway:executeInTransaction=false로 분리
• Contract는 backfill + 앱 전환이 모두 끝난 뒤 별도 릴리즈

❌ 나쁜 예시 3: 한 릴리즈에 expand + contract

-- V100__rename_email.sql
ALTER TABLE users RENAME COLUMN email TO email_old;
ALTER TABLE users ADD COLUMN email text NOT NULL DEFAULT '';
-- 앱이 어느 버전이든 장애 발생 가능

문제:

• rolling deploy 중간에 앱이 N-1 / N 모두 실행 → 컬럼 없음 / 이름 다름으로 에러
• rollback 시 DB 상태가 앞서가 있어 N-1 앱이 기동 안 됨

---

좋은 예시 5: PITR 복구 계획 (CNPG)

---
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: auth-pg-restore
  namespace: data-prod
spec:
  instances: 3
  imageName: ghcr.io/cloudnative-pg/postgresql:16.4-8
  storage:    {size: 50Gi, storageClass: fast-ssd-retain}
  walStorage: {size: 20Gi, storageClass: fast-ssd-retain}
  bootstrap:
    recovery:
      source: auth-pg-source
      recoveryTarget:
        targetTime: "2026-04-16 09:45:00+00"   # 잘못된 migration 직전
  externalClusters:
    - name: auth-pg-source
      barmanObjectStore:
        destinationPath: s3://acme-prod-pg-backups/auth-pg
        s3Credentials:
          accessKeyId:     {name: cnpg-s3-credentials, key: ACCESS_KEY_ID}
          secretAccessKey: {name: cnpg-s3-credentials, key: ACCESS_SECRET_KEY}
        wal: {maxParallel: 8}

왜 좋은가:

• 운영 cluster는 건드리지 않고 auth-pg-restore로 복원
• recoveryTarget.targetTime을 분단위로 지정
• 복원 후 검증 → 운영 전환은 별도 runbook

---

좋은 예시 6: non-transactional DDL을 별도 migration 파일로

V200__create_idx_users_last_login.sql:

-- flyway:executeInTransaction=false
-- Long-running DDL. Run in low-traffic window.
-- Runtime estimate: ~15min on 50M rows.
CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_users_last_login
    ON users(last_login_at);

왜 좋은가:

• CREATE INDEX CONCURRENTLY는 Postgres에서 트랜잭션 내 실행 불가
• Flyway 8.2+ executeInTransaction=false directive로 파일 단위 제어
• 주석에 runtime 추정치 / 영향 명시

❌ 나쁜 예시 4: 트랜잭션 내 CREATE INDEX CONCURRENTLY

-- V200__.sql (기본 트랜잭션 모드)
CREATE INDEX CONCURRENTLY idx_users_last_login ON users(last_login_at);
-- → ERROR: CREATE INDEX CONCURRENTLY cannot run inside a transaction block

문제:

• Flyway가 자동으로 트랜잭션을 열기 때문에 실패
• -- flyway:executeInTransaction=false가 필수

---

좋은 예시 7: 운영 절차 runbook snippet

# auth-server DB schema change — 2026-04-16 02:00 UTC maintenance window

## Pre-check (T-1d)
1. Pending migration 검토: 로컬 `flyway info`
2. PR review + migration 영향 분석 문서 작성 (expand/migrate/contract 단계)
3. Backup 상태 확인:
   kubectl -n data-prod get scheduledbackup auth-pg-daily
   kubectl -n data-prod get backup -l cnpg.io/cluster=auth-pg --sort-by=.metadata.creationTimestamp

## T-5min
1. On-demand backup:
   cat <<EOF | kubectl apply -f -
   apiVersion: postgresql.cnpg.io/v1
   kind: Backup
   metadata:
     name: auth-pg-pre-$(date -u +%Y%m%dT%H%M%SZ)
     namespace: data-prod
   spec:
     cluster: {name: auth-pg}
     method: barmanObjectStore
   EOF
2. Argo CD sync (dry-run):
   argocd app diff auth-server-prod

## Apply
1. argocd app sync auth-server-prod
   → Flyway Job이 sync-wave -1로 먼저 실행
   → Deployment는 wave 0에서 롤아웃
2. Flyway Job 로그 확인:
   kubectl -n auth-prod logs job/auth-flyway-migrate
3. Deployment rollout 확인:
   kubectl -n auth-prod rollout status deploy/auth-server

## Post-check
1. flyway info (적용 결과)
2. 앱 스모크 테스트
3. DB 메트릭 (slow query, error rate)
4. Next PITR recovery point 확인

왜 좋은가:

• migration 직전 on-demand backup
• migration → app rollout 순서가 선언 (sync-wave)으로 보장됨
• 실패 시 PITR 복구 지점이 명확

---

나쁜 예시 5: 앱 시작 시 자동 migration

# Spring Boot application.properties
spring.flyway.enabled=true
spring.flyway.baseline-on-migrate=true
# 앱이 기동될 때마다 Flyway migrate 수행

문제:

• replicas=3이면 3개 Pod가 동시에 migrate 시도 (Flyway advisory lock이 막아주지만 기동 latency 증가)
• app rollout 실패와 migration 실패가 섞임 — 원인 추적 어려움
• 신규 Pod가 기동되는 rolling restart 시에도 매번 validate 수행

---

나쁜 예시 6: pg_dump 하나만으로 운영 복구

apiVersion: batch/v1
kind: CronJob
metadata:
  name: pg-dump-nightly
spec:
  schedule: "0 3 * * *"
  # ... pg_dumpall > /backup/dump.sql

문제:

• PITR 불가, RPO = 24h
• replication slot / extension / large object 누락
• 대규모 DB에서 restore 시간 폭증
• 같은 cluster 안 PVC에 저장하면 동시 소실

---

나쁜 예시 7: U__ undo migration 작성

flyway/
  V120__add_column.sql
  U120__drop_column.sql     ← OSS Flyway는 실행 불가

문제:

• Flyway Community(OSS)는 undo 미지원 → flyway undo가 에러
• rollback 전략은 forward-only migration + PITR로 대체