# MinIO 예시 MinIO Operator + Tenant CRD (`minio.min.io/v2`) + KES + Vault transit 기준. 모든 manifest는 `kubectl apply` 적용 가능한 완전한 형태다. --- ## 좋은 예시 1: Namespace + Tenant configuration Secret ```yaml --- apiVersion: v1 kind: Namespace metadata: name: minio-prod labels: pod-security.kubernetes.io/enforce: restricted pod-security.kubernetes.io/audit: restricted app.kubernetes.io/part-of: storage-platform --- apiVersion: v1 kind: Secret metadata: name: minio-tenant-env namespace: minio-prod type: Opaque stringData: config.env: | export MINIO_ROOT_USER="REPLACE_VIA_VSO" export MINIO_ROOT_PASSWORD="REPLACE_VIA_VSO" export MINIO_STORAGE_CLASS_STANDARD="EC:4" export MINIO_STORAGE_CLASS_RRS="EC:2" export MINIO_BROWSER_REDIRECT_URL="https://minio-console.internal.example.com" export MINIO_SERVER_URL="https://s3.example.com" export MINIO_IDENTITY_OPENID_CONFIG_URL="https://auth.example.com/realms/platform/.well-known/openid-configuration" export MINIO_IDENTITY_OPENID_CLIENT_ID="minio" export MINIO_IDENTITY_OPENID_CLAIM_NAME="policy" export MINIO_IDENTITY_OPENID_SCOPES="openid,profile,email" export MINIO_PROMETHEUS_AUTH_TYPE="jwt" --- # 실 운영에서는 VSO가 이 Secret을 채움 apiVersion: secrets.hashicorp.com/v1beta1 kind: VaultStaticSecret metadata: name: minio-root-creds namespace: minio-prod spec: vaultAuthRef: default mount: kv path: minio/prod/root type: kv-v2 refreshAfter: 24h destination: name: minio-tenant-env create: false overwrite: true transformation: excludeRaw: true templates: config.env: text: | export MINIO_ROOT_USER="{{ .Secrets.username }}" export MINIO_ROOT_PASSWORD="{{ .Secrets.password }}" export MINIO_STORAGE_CLASS_STANDARD="EC:4" export MINIO_BROWSER_REDIRECT_URL="https://minio-console.internal.example.com" export MINIO_SERVER_URL="https://s3.example.com" export MINIO_IDENTITY_OPENID_CONFIG_URL="https://auth.example.com/realms/platform/.well-known/openid-configuration" export MINIO_IDENTITY_OPENID_CLIENT_ID="minio" export MINIO_IDENTITY_OPENID_CLIENT_SECRET="{{ .Secrets.oidc_client_secret }}" export MINIO_IDENTITY_OPENID_CLAIM_NAME="policy" export MINIO_IDENTITY_OPENID_SCOPES="openid,profile,email" export MINIO_PROMETHEUS_AUTH_TYPE="jwt" ``` **왜 좋은가:** - Tenant configuration은 **shell-source 형식**(`export KEY=VALUE`) Secret으로 전달 (Operator 규약) - Root credential을 Vault KV에서 VSO가 주입 — Git에 평문 없음 - OIDC 통합 (Keycloak), storage class EC:4, Prometheus JWT auth 한 파일에 고정 - `MINIO_SERVER_URL`로 외부 S3 endpoint 명시 (presigned URL 생성 시 사용) --- ## 좋은 예시 2: Tenant CR — 4 server × 4 volume + KES + TLS ```yaml apiVersion: minio.min.io/v2 kind: Tenant metadata: name: minio namespace: minio-prod labels: app.kubernetes.io/name: minio app.kubernetes.io/instance: minio-prod app.kubernetes.io/part-of: storage-platform app.kubernetes.io/managed-by: minio-operator annotations: prometheus.io/path: /minio/v2/metrics/cluster prometheus.io/port: "9000" prometheus.io/scrape: "true" spec: image: quay.io/minio/minio:RELEASE.2025-01-20T14-49-07Z imagePullPolicy: IfNotPresent mountPath: /export configuration: name: minio-tenant-env requestAutoCert: true certConfig: commonName: minio.minio-prod.svc.cluster.local organizationName: - example.com dnsNames: - minio.minio-prod.svc.cluster.local - "*.minio-hl.minio-prod.svc.cluster.local" - s3.example.com pools: - name: pool-0 servers: 4 volumesPerServer: 4 volumeClaimTemplate: metadata: name: data spec: accessModes: - ReadWriteOnce resources: requests: storage: 500Gi storageClassName: local-xfs-retain resources: requests: cpu: 500m memory: 2Gi limits: cpu: "4" memory: 8Gi securityContext: runAsUser: 1000 runAsGroup: 1000 runAsNonRoot: true fsGroup: 1000 fsGroupChangePolicy: OnRootMismatch seccompProfile: type: RuntimeDefault containerSecurityContext: runAsUser: 1000 runAsGroup: 1000 runAsNonRoot: true allowPrivilegeEscalation: false readOnlyRootFilesystem: true capabilities: drop: - ALL affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchLabels: v1.min.io/tenant: minio v1.min.io/pool: pool-0 topologyKey: kubernetes.io/hostname topologySpreadConstraints: - maxSkew: 1 topologyKey: topology.kubernetes.io/zone whenUnsatisfiable: ScheduleAnyway labelSelector: matchLabels: v1.min.io/tenant: minio tolerations: - key: storage operator: Equal value: dedicated effect: NoSchedule features: bucketDNS: false domains: console: https://minio-console.internal.example.com minio: - https://s3.example.com kes: image: quay.io/minio/kes:2025-01-16T16-24-39Z replicas: 2 kesSecret: name: kes-configuration imagePullPolicy: IfNotPresent resources: requests: cpu: 100m memory: 128Mi limits: cpu: 500m memory: 512Mi securityContext: runAsUser: 1000 runAsGroup: 1000 runAsNonRoot: true fsGroup: 1000 containerSecurityContext: runAsNonRoot: true allowPrivilegeEscalation: false readOnlyRootFilesystem: true capabilities: drop: - ALL prometheusOperator: true podManagementPolicy: Parallel exposeServices: minio: true console: false logging: anonymous: false json: true quiet: false --- apiVersion: policy/v1 kind: PodDisruptionBudget metadata: name: minio namespace: minio-prod spec: minAvailable: 3 unhealthyPodEvictionPolicy: AlwaysAllow selector: matchLabels: v1.min.io/tenant: minio ``` **왜 좋은가:** - `servers × volumesPerServer = 4 × 4 = 16` drive → erasure coding 최소 요건 충족, `EC:4` 기본 parity (4 drive 장애 허용) - `requestAutoCert: true` + `certConfig.dnsNames`로 Operator가 API/Console TLS 자동 발급 - `podAntiAffinity` hostname required → 한 node에 MinIO pod 복수 배치 금지 (EC 의미 보존) - KES가 별도 2 replica로 사이드카 없이 Deployment로 분리 (Tenant CR에서 관리됨) - `exposeServices.console: false` → Console은 Tenant Service에서 Ingress로 별도 처리만 허용 - `minAvailable: 3` → 4 server 중 1 동시 drain까지 허용 (write quorum 보존) --- ## 나쁜 예시 1: 단일 Deployment로 MinIO ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: minio spec: replicas: 1 template: spec: containers: - name: minio image: minio/minio args: ["server", "/data"] env: - name: MINIO_ROOT_USER value: minioadmin - name: MINIO_ROOT_PASSWORD value: minioadmin volumeMounts: - name: data mountPath: /data volumes: - name: data emptyDir: {} ``` **문제:** - Single-drive MinIO → erasure coding 없음, 1 drive 장애 = 전체 data loss - Deployment = 재시작 시 PVC binding 보장 없음, 복수 replica 시 동일 volume 충돌 - emptyDir → pod 재시작 시 모든 object 사라짐 - 기본 `minioadmin/minioadmin` credential → 공개 인터넷 스캐너가 수 분 내 탈취 - Operator + Tenant가 자동화하는 인증서, 서비스, headless, auto-restart를 전부 수제로 다시 만들어야 함 --- ## 좋은 예시 3: KES configuration + Vault transit ```yaml --- apiVersion: v1 kind: Secret metadata: name: kes-configuration namespace: minio-prod type: Opaque stringData: server-config.yaml: | version: v1 address: 0.0.0.0:7373 admin: identity: disabled tls: key: /tmp/kes/server.key cert: /tmp/kes/server.cert policy: minio-app: allow: - /v1/key/create/minio-* - /v1/key/generate/minio-* - /v1/key/decrypt/minio-* - /v1/key/bulk/decrypt/minio-* - /v1/key/list/minio-* - /v1/status - /v1/metrics - /v1/api identities: - ${MINIO_KES_IDENTITY} keystore: vault: endpoint: https://vault.vault.svc.cluster.local:8200 engine: transit version: v1 namespace: "" prefix: minio approle: id: ${VAULT_APPROLE_ID} secret: ${VAULT_APPROLE_SECRET} retry: 15s tls: ca: /tmp/kes/vault-ca.crt status: ping: 10s --- # Vault AppRole credential은 VSO 또는 별도 Secret으로 주입 apiVersion: secrets.hashicorp.com/v1beta1 kind: VaultStaticSecret metadata: name: kes-vault-approle namespace: minio-prod spec: vaultAuthRef: default mount: kv path: minio/kes/approle type: kv-v2 refreshAfter: 24h destination: name: kes-vault-approle create: true overwrite: true ``` 그리고 bucket에 SSE-KMS 적용: ```bash mc alias set minio https://s3.example.com $ROOT_USER $ROOT_PASS # Vault transit에 key 생성 mc admin kms key create minio minio-critical # bucket에 SSE-KMS 기본 적용 mc encrypt set sse-kms minio-critical minio/critical-bucket ``` **왜 좋은가:** - KES가 Vault transit을 key store로 사용 → master key는 Vault가 관리, MinIO는 DEK만 캐시 - KES policy로 `minio-*` prefix key만 access 허용 (최소 권한) - AppRole credential은 Vault → VSO → Secret 경로 - bucket level SSE-KMS → 업로드되는 모든 object가 per-object DEK로 자동 암호화 --- ## 나쁜 예시 2: KES 없이 평문 저장 ```yaml # Tenant CR spec: kes: {} # 미설정 # ... SSE 설정 없음 ``` ```bash mc cp secret.pdf minio/bucket/secret.pdf # object가 disk에 평문 저장 ``` **문제:** - PVC가 탈취되거나 물리 drive가 반출되면 평문 유출 - 감사/규제 요구(GDPR, PCI-DSS, ISO 27001) 위반 - SSE-S3를 대신 쓰더라도 master key가 MinIO 자체에 있어 키 라이프사이클 관리 불가 --- ## 좋은 예시 4: Probe — live + cluster-read Tenant CR이 자동으로 probe를 구성하지만, 커스텀 오버라이드가 필요할 때: ```yaml spec: pools: - name: pool-0 # ... containers: - name: minio livenessProbe: httpGet: path: /minio/health/live port: 9000 scheme: HTTPS initialDelaySeconds: 60 periodSeconds: 30 timeoutSeconds: 5 failureThreshold: 3 readinessProbe: httpGet: path: /minio/health/cluster/read port: 9000 scheme: HTTPS periodSeconds: 10 timeoutSeconds: 5 failureThreshold: 3 startupProbe: httpGet: path: /minio/health/live port: 9000 scheme: HTTPS periodSeconds: 5 failureThreshold: 60 timeoutSeconds: 5 ``` **왜 좋은가:** - `readinessProbe`는 `/minio/health/cluster/read` → **read quorum** 검사. rolling update 중에도 read가 가능하면 Service에 남아있음 - `/minio/health/cluster` (write quorum)을 readiness로 쓰면 rolling 재시작 시 pod가 전부 빠져 완전 unavailable - `livenessProbe`는 단순 프로세스 생존만 확인 → 일시적 quorum 상실로 pod 강제 재시작 방지 - HTTPS scheme (requestAutoCert과 일치) --- ## 나쁜 예시 3: readiness를 write quorum으로 ```yaml readinessProbe: httpGet: path: /minio/health/cluster port: 9000 periodSeconds: 5 failureThreshold: 1 ``` **문제:** - rolling update로 pod 1개를 재시작하면 write quorum이 일시적으로 무너져 살아있는 pod들도 NotReady - Service가 endpoint를 전부 제거 → **읽기도 불가능** - `failureThreshold: 1` + 5초 주기 → 한 번 느린 응답으로 pod 빠짐 --- ## 좋은 예시 5: Ingress — API는 공개, Console은 내부 ```yaml --- apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: minio-api namespace: minio-prod annotations: cert-manager.io/cluster-issuer: letsencrypt-prod nginx.ingress.kubernetes.io/proxy-body-size: "0" nginx.ingress.kubernetes.io/proxy-request-buffering: "off" nginx.ingress.kubernetes.io/proxy-buffering: "off" nginx.ingress.kubernetes.io/backend-protocol: HTTPS spec: ingressClassName: nginx tls: - hosts: - s3.example.com secretName: minio-api-ingress-tls rules: - host: s3.example.com http: paths: - path: / pathType: Prefix backend: service: name: minio port: number: 443 --- apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: minio-console namespace: minio-prod annotations: cert-manager.io/cluster-issuer: letsencrypt-prod nginx.ingress.kubernetes.io/whitelist-source-range: "10.0.0.0/8,192.168.0.0/16" nginx.ingress.kubernetes.io/auth-url: "https://oauth2-proxy.example.com/oauth2/auth" nginx.ingress.kubernetes.io/backend-protocol: HTTPS spec: ingressClassName: nginx-internal tls: - hosts: - minio-console.internal.example.com secretName: minio-console-ingress-tls rules: - host: minio-console.internal.example.com http: paths: - path: / pathType: Prefix backend: service: name: minio-console port: number: 9443 ``` **왜 좋은가:** - API Ingress는 `proxy-body-size: 0` + request/response buffering off → 대용량 multipart upload 지원 - Console은 내부 ingress class + IP whitelist + OIDC forward-auth 2중 보호 - `backend-protocol: HTTPS` → MinIO의 auto-cert TLS를 TLS passthrough 형태로 전달 (인증서 SAN 보존) --- ## 나쁜 예시 4: Console 외부 공개 ```yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: minio-all spec: rules: - host: minio.example.com http: paths: - path: / pathType: Prefix backend: service: name: minio port: number: 9090 ``` **문제:** - Console이 인터넷에 그대로 노출 → root/admin credential brute force 표면 확장 - OIDC forward-auth 없음 → 기본 login 페이지가 공격자에게 노출 - IP 제한 없음 - Bucket 목록, access key 관리, 사용자 관리가 모두 공개 domain에 위치 --- ## 좋은 예시 6: ServiceMonitor (Prometheus bearer-token) 먼저 MinIO 내부에서 scrape token 발급: ```bash mc admin prometheus generate minio cluster # 출력에 bearer token과 scrape config가 나옴 ``` 그 결과 token을 Secret로 저장: ```yaml --- apiVersion: v1 kind: Secret metadata: name: minio-prometheus-token namespace: minio-prod type: Opaque stringData: token: "eyJhbGciOiJIUzUxMiIsInR5cCI6IkpXVCJ9..." --- apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: minio namespace: minio-prod labels: app.kubernetes.io/name: minio release: kube-prometheus-stack spec: selector: matchLabels: v1.min.io/tenant: minio endpoints: - port: https-minio scheme: https path: /minio/v2/metrics/cluster interval: 30s scrapeTimeout: 10s bearerTokenSecret: name: minio-prometheus-token key: token tlsConfig: insecureSkipVerify: false ca: secret: name: minio-tls key: ca.crt serverName: minio.minio-prod.svc.cluster.local ``` **왜 좋은가:** - `MINIO_PROMETHEUS_AUTH_TYPE=jwt` 와 매칭 (기본값) - `/minio/v2/metrics/cluster`는 cluster-wide view (replication lag, bucket 사용량, API latency) - TLS 검증 유지 (`insecureSkipVerify: false`, CA bundle 제공) - 외부 노출 없이 내부 scrape만 --- ## 좋은 예시 7: Bucket 초기화 (Job) — versioning + Object Lock + lifecycle ```yaml --- apiVersion: v1 kind: ConfigMap metadata: name: minio-bootstrap namespace: minio-prod data: init.sh: | #!/bin/sh set -eu mc alias set minio https://minio.minio-prod.svc.cluster.local "$MINIO_ROOT_USER" "$MINIO_ROOT_PASSWORD" --api S3v4 # Object Lock은 bucket 생성 시점에만 활성화 가능 mc mb --with-lock minio/critical-audit || true mc retention set --default COMPLIANCE 2555d minio/critical-audit # 7년 보관 # Versioning + lifecycle mc mb minio/app-data || true mc version enable minio/app-data mc ilm add --expire-noncurrent-days 90 minio/app-data mc ilm add --expire-incomplete-upload-days 7 minio/app-data # SSE-KMS 기본 적용 mc encrypt set sse-kms minio-app-key minio/app-data mc encrypt set sse-kms minio-critical-key minio/critical-audit # Service account 발급 (앱 전용, 최소 권한 policy) mc admin policy create minio auth-server-rw /policies/auth-server-rw.json mc admin user svcacct add minio "$MINIO_ROOT_USER" \ --access-key "$AUTH_SERVER_ACCESS_KEY" \ --secret-key "$AUTH_SERVER_SECRET_KEY" \ --policy /policies/auth-server-rw.json || true echo "bootstrap complete" auth-server-rw.json: | { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": ["s3:PutObject", "s3:GetObject", "s3:DeleteObject", "s3:ListBucket"], "Resource": ["arn:aws:s3:::app-data/*", "arn:aws:s3:::app-data"] } ] } --- apiVersion: batch/v1 kind: Job metadata: name: minio-bootstrap namespace: minio-prod spec: backoffLimit: 3 ttlSecondsAfterFinished: 86400 template: spec: restartPolicy: OnFailure serviceAccountName: minio-bootstrap containers: - name: mc image: quay.io/minio/mc:RELEASE.2025-01-17T23-25-50Z command: ["/bin/sh", "/scripts/init.sh"] resources: requests: { cpu: 100m, memory: 128Mi } limits: { memory: 256Mi } env: - name: MINIO_ROOT_USER valueFrom: secretKeyRef: name: minio-root-creds key: username - name: MINIO_ROOT_PASSWORD valueFrom: secretKeyRef: name: minio-root-creds key: password - name: AUTH_SERVER_ACCESS_KEY valueFrom: secretKeyRef: name: auth-server-minio-svcacct key: access_key - name: AUTH_SERVER_SECRET_KEY valueFrom: secretKeyRef: name: auth-server-minio-svcacct key: secret_key volumeMounts: - name: scripts mountPath: /scripts - name: policies mountPath: /policies securityContext: runAsNonRoot: true runAsUser: 1000 allowPrivilegeEscalation: false readOnlyRootFilesystem: true capabilities: drop: - ALL volumes: - name: scripts configMap: name: minio-bootstrap defaultMode: 0755 items: - key: init.sh path: init.sh - name: policies configMap: name: minio-bootstrap items: - key: auth-server-rw.json path: auth-server-rw.json ``` **왜 좋은가:** - `mc mb --with-lock`은 bucket 생성 시점에만 Object Lock 활성화 가능 — Job이 그 타이밍을 보장 - COMPLIANCE 모드 7년 retention = 감사/규제 요구 충족 (root도 bypass 불가) - `app-data` bucket은 versioning + lifecycle (90일 noncurrent expire + 7일 incomplete abort) - service account는 특정 bucket prefix만 접근 가능한 policy로 제한 - `backoffLimit: 3` + idempotent 명령 (`|| true`) → 재실행 안전 --- ## 나쁜 예시 5: mc mirror만으로 DR ```bash # 매일 자정 crontab mc mirror minio/critical remote-minio/critical-backup ``` **문제:** - `mc mirror`는 **현재 object만** 동기화 — 버전 히스토리 유실 - Object Lock 상태, bucket policy, IAM 설정 미복제 - 메타데이터 중 일부(tag, legal hold) 누락 - RPO = 1일 (하루 단위 손실), replication은 async ms 단위 RPO - DR 연습(resync 절차) 불가 대안: `mc admin replicate add` (site replication, IAM + bucket + object 전부 async 동기화). --- ## 좋은 예시 8: Bucket replication ```bash # source alias 설정 mc alias set source https://minio.minio-prod.svc.cluster.local $SRC_USER $SRC_PASS mc alias set target https://minio.minio-dr.svc.cluster.local $TGT_USER $TGT_PASS # target에 replication 전용 user + policy mc admin policy create target replication-target /policies/replication.json mc admin user add target replication-bot $(openssl rand -hex 16) mc admin policy attach target replication-target --user replication-bot # source에서 remote target 등록 mc replicate add source/app-data \ --remote-bucket https://replication-bot:PASS@minio.minio-dr.svc.cluster.local/app-data \ --replicate "delete,delete-marker,existing-objects,metadata-sync" \ --priority 1 ``` `replication.json`: ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:GetBucketVersioning", "s3:PutBucketVersioning", "s3:GetReplicationConfiguration", "s3:ReplicateObject", "s3:ReplicateDelete", "s3:ReplicateTags", "s3:GetObjectVersion", "s3:GetObjectVersionTagging", "s3:GetObjectVersionForReplication" ], "Resource": ["arn:aws:s3:::app-data/*", "arn:aws:s3:::app-data"] } ] } ``` **왜 좋은가:** - `existing-objects` 옵션으로 기존 데이터 backfill - `delete` + `delete-marker`로 삭제도 복제 (true mirror) - 전용 replication user + 최소 권한 policy - async 복제, bucket versioning 전제 --- ## 좋은 예시 9: Keycloak OIDC STS 로그인 Keycloak에 `minio` client 생성 후: ```bash # 앱에서 JWT를 받은 다음 curl -X POST https://s3.example.com/ \ -d "Action=AssumeRoleWithWebIdentity" \ -d "Version=2011-06-15" \ -d "WebIdentityToken=${KEYCLOAK_ID_TOKEN}" \ -d "DurationSeconds=3600" ``` 응답의 `AccessKeyId`, `SecretAccessKey`, `SessionToken`을 S3 SDK에 주입. ```bash # AWS CLI 예시 aws configure set aws_access_key_id "$STS_ACCESS_KEY" aws configure set aws_secret_access_key "$STS_SECRET_KEY" aws configure set aws_session_token "$STS_SESSION_TOKEN" aws s3 ls s3://app-data/ --endpoint-url https://s3.example.com ``` **왜 좋은가:** - 앱/사용자는 Keycloak에 로그인만 하면 됨 — MinIO에 user 등록 불필요 - 임시 credential (1시간 TTL) → 유출 시 피해 제한 - JWT의 `policy` claim이 MinIO 정책과 자동 매핑 - 장기 access key 배포 없음