# MinIO 예시 MinIO Operator + Tenant CRD (`minio.min.io/v2`) + KES + Vault transit 기준. 모든 manifest는 `kubectl apply` 적용 가능한 완전한 형태다. --- ## 좋은 예시 1: Namespace + Tenant configuration Secret ```yaml --- apiVersion: v1 kind: Namespace metadata: name: minio-prod labels: pod-security.kubernetes.io/enforce: restricted pod-security.kubernetes.io/audit: restricted app.kubernetes.io/part-of: storage-platform --- apiVersion: v1 kind: Secret metadata: name: minio-tenant-env namespace: minio-prod type: Opaque stringData: config.env: | export MINIO_ROOT_USER="REPLACE_VIA_VSO" export MINIO_ROOT_PASSWORD="REPLACE_VIA_VSO" export MINIO_STORAGE_CLASS_STANDARD="EC:4" export MINIO_STORAGE_CLASS_RRS="EC:2" export MINIO_BROWSER_REDIRECT_URL="https://minio-console.internal.example.com" export MINIO_SERVER_URL="https://s3.example.com" export MINIO_IDENTITY_OPENID_CONFIG_URL="https://auth.example.com/realms/platform/.well-known/openid-configuration" export MINIO_IDENTITY_OPENID_CLIENT_ID="minio" export MINIO_IDENTITY_OPENID_CLAIM_NAME="policy" export MINIO_IDENTITY_OPENID_SCOPES="openid,profile,email" export MINIO_PROMETHEUS_AUTH_TYPE="jwt" --- # 실 운영에서는 VSO가 이 Secret을 채움 apiVersion: secrets.hashicorp.com/v1beta1 kind: VaultStaticSecret metadata: name: minio-root-creds namespace: minio-prod spec: vaultAuthRef: default mount: kv path: minio/prod/root type: kv-v2 refreshAfter: 24h destination: name: minio-tenant-env create: false overwrite: true transformation: excludeRaw: true templates: config.env: text: | export MINIO_ROOT_USER="{{ .Secrets.username }}" export MINIO_ROOT_PASSWORD="{{ .Secrets.password }}" export MINIO_STORAGE_CLASS_STANDARD="EC:4" export MINIO_BROWSER_REDIRECT_URL="https://minio-console.internal.example.com" export MINIO_SERVER_URL="https://s3.example.com" export MINIO_IDENTITY_OPENID_CONFIG_URL="https://auth.example.com/realms/platform/.well-known/openid-configuration" export MINIO_IDENTITY_OPENID_CLIENT_ID="minio" export MINIO_IDENTITY_OPENID_CLIENT_SECRET="{{ .Secrets.oidc_client_secret }}" export MINIO_IDENTITY_OPENID_CLAIM_NAME="policy" export MINIO_IDENTITY_OPENID_SCOPES="openid,profile,email" export MINIO_PROMETHEUS_AUTH_TYPE="jwt" ``` **왜 좋은가:** - Tenant configuration은 **shell-source 형식**(`export KEY=VALUE`) Secret으로 전달 (Operator 규약) - Root credential을 Vault KV에서 VSO가 주입 — Git에 평문 없음 - OIDC 통합 (Keycloak), storage class EC:4, Prometheus JWT auth 한 파일에 고정 - `MINIO_SERVER_URL`로 외부 S3 endpoint 명시 (presigned URL 생성 시 사용) --- ## 좋은 예시 2: Tenant CR — 4 server × 4 volume + KES + TLS ```yaml apiVersion: minio.min.io/v2 kind: Tenant metadata: name: minio namespace: minio-prod labels: app.kubernetes.io/name: minio app.kubernetes.io/instance: minio-prod app.kubernetes.io/part-of: storage-platform app.kubernetes.io/managed-by: minio-operator annotations: prometheus.io/path: /minio/v2/metrics/cluster prometheus.io/port: "9000" prometheus.io/scrape: "true" spec: image: quay.io/minio/minio:RELEASE.2025-01-20T14-49-07Z imagePullPolicy: IfNotPresent mountPath: /export configuration: name: minio-tenant-env requestAutoCert: true certConfig: commonName: minio.minio-prod.svc.cluster.local organizationName: - example.com dnsNames: - minio.minio-prod.svc.cluster.local - "*.minio-hl.minio-prod.svc.cluster.local" - s3.example.com pools: - name: pool-0 servers: 4 volumesPerServer: 4 volumeClaimTemplate: metadata: name: data spec: accessModes: - ReadWriteOnce resources: requests: storage: 500Gi storageClassName: local-xfs-retain resources: requests: cpu: 500m memory: 2Gi limits: cpu: "4" memory: 8Gi securityContext: runAsUser: 1000 runAsGroup: 1000 runAsNonRoot: true fsGroup: 1000 fsGroupChangePolicy: OnRootMismatch seccompProfile: type: RuntimeDefault containerSecurityContext: runAsUser: 1000 runAsGroup: 1000 runAsNonRoot: true allowPrivilegeEscalation: false readOnlyRootFilesystem: true capabilities: drop: - ALL affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchLabels: v1.min.io/tenant: minio v1.min.io/pool: pool-0 topologyKey: kubernetes.io/hostname topologySpreadConstraints: - maxSkew: 1 topologyKey: topology.kubernetes.io/zone whenUnsatisfiable: ScheduleAnyway labelSelector: matchLabels: v1.min.io/tenant: minio tolerations: - key: storage operator: Equal value: dedicated effect: NoSchedule features: bucketDNS: false domains: console: https://minio-console.internal.example.com minio: - https://s3.example.com kes: image: quay.io/minio/kes:2025-01-16T16-24-39Z replicas: 2 kesSecret: name: kes-configuration imagePullPolicy: IfNotPresent resources: requests: cpu: 100m memory: 128Mi limits: cpu: 500m memory: 512Mi securityContext: runAsUser: 1000 runAsGroup: 1000 runAsNonRoot: true fsGroup: 1000 containerSecurityContext: runAsNonRoot: true allowPrivilegeEscalation: false readOnlyRootFilesystem: true capabilities: drop: - ALL prometheusOperator: true podManagementPolicy: Parallel exposeServices: minio: true console: false logging: anonymous: false json: true quiet: false --- apiVersion: policy/v1 kind: PodDisruptionBudget metadata: name: minio namespace: minio-prod spec: minAvailable: 3 unhealthyPodEvictionPolicy: AlwaysAllow selector: matchLabels: v1.min.io/tenant: minio ``` **왜 좋은가:** - `servers × volumesPerServer = 4 × 4 = 16` drive → erasure coding 최소 요건 충족, `EC:4` 기본 parity (4 drive 장애 허용) - `requestAutoCert: true` + `certConfig.dnsNames`로 Operator가 API/Console TLS 자동 발급 - `podAntiAffinity` hostname required → 한 node에 MinIO pod 복수 배치 금지 (EC 의미 보존) - KES가 별도 2 replica로 사이드카 없이 Deployment로 분리 (Tenant CR에서 관리됨) - `exposeServices.console: false` → Console은 Tenant Service에서 Ingress로 별도 처리만 허용 - `minAvailable: 3` → 4 server 중 1 동시 drain까지 허용 (write quorum 보존) --- ## 나쁜 예시 1: 단일 Deployment로 MinIO ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: minio spec: replicas: 1 template: spec: containers: - name: minio image: minio/minio args: ["server", "/data"] env: - name: MINIO_ROOT_USER value: minioadmin - name: MINIO_ROOT_PASSWORD value: minioadmin volumeMounts: - name: data mountPath: /data volumes: - name: data emptyDir: {} ``` **문제:** - Single-drive MinIO → erasure coding 없음, 1 drive 장애 = 전체 data loss - Deployment = 재시작 시 PVC binding 보장 없음, 복수 replica 시 동일 volume 충돌 - emptyDir → pod 재시작 시 모든 object 사라짐 - 기본 `minioadmin/minioadmin` credential → 공개 인터넷 스캐너가 수 분 내 탈취 - Operator + Tenant가 자동화하는 인증서, 서비스, headless, auto-restart를 전부 수제로 다시 만들어야 함 --- ## 좋은 예시 3: KES configuration + Vault transit ```yaml --- apiVersion: v1 kind: Secret metadata: name: kes-configuration namespace: minio-prod type: Opaque stringData: server-config.yaml: | version: v1 address: 0.0.0.0:7373 admin: identity: disabled tls: key: /tmp/kes/server.key cert: /tmp/kes/server.cert policy: minio-app: allow: - /v1/key/create/minio-* - /v1/key/generate/minio-* - /v1/key/decrypt/minio-* - /v1/key/bulk/decrypt/minio-* - /v1/key/list/minio-* - /v1/status - /v1/metrics - /v1/api identities: - ${MINIO_KES_IDENTITY} keystore: vault: endpoint: https://vault.vault.svc.cluster.local:8200 engine: transit version: v1 namespace: "" prefix: minio approle: id: ${VAULT_APPROLE_ID} secret: ${VAULT_APPROLE_SECRET} retry: 15s tls: ca: /tmp/kes/vault-ca.crt status: ping: 10s --- # Vault AppRole credential은 VSO 또는 별도 Secret으로 주입 apiVersion: secrets.hashicorp.com/v1beta1 kind: VaultStaticSecret metadata: name: kes-vault-approle namespace: minio-prod spec: vaultAuthRef: default mount: kv path: minio/kes/approle type: kv-v2 refreshAfter: 24h destination: name: kes-vault-approle create: true overwrite: true ``` 그리고 bucket에 SSE-KMS 적용: ```bash mc alias set minio https://s3.example.com $ROOT_USER $ROOT_PASS # Vault transit에 key 생성 mc admin kms key create minio minio-critical # bucket에 SSE-KMS 기본 적용 mc encrypt set sse-kms minio-critical minio/critical-bucket ``` **왜 좋은가:** - KES가 Vault transit을 key store로 사용 → master key는 Vault가 관리, MinIO는 DEK만 캐시 - KES policy로 `minio-*` prefix key만 access 허용 (최소 권한) - AppRole credential은 Vault → VSO → Secret 경로 - bucket level SSE-KMS → 업로드되는 모든 object가 per-object DEK로 자동 암호화 --- ## 나쁜 예시 2: KES 없이 평문 저장 ```yaml # Tenant CR spec: kes: {} # 미설정 # ... SSE 설정 없음 ``` ```bash mc cp secret.pdf minio/bucket/secret.pdf # object가 disk에 평문 저장 ``` **문제:** - PVC가 탈취되거나 물리 drive가 반출되면 평문 유출 - 감사/규제 요구(GDPR, PCI-DSS, ISO 27001) 위반 - SSE-S3를 대신 쓰더라도 master key가 MinIO 자체에 있어 키 라이프사이클 관리 불가 --- ## 좋은 예시 4: Probe — live + cluster-read Tenant CR이 자동으로 probe를 구성하지만, 커스텀 오버라이드가 필요할 때: ```yaml spec: pools: - name: pool-0 # ... containers: - name: minio livenessProbe: httpGet: path: /minio/health/live port: 9000 scheme: HTTPS initialDelaySeconds: 60 periodSeconds: 30 timeoutSeconds: 5 failureThreshold: 3 readinessProbe: httpGet: path: /minio/health/cluster/read port: 9000 scheme: HTTPS periodSeconds: 10 timeoutSeconds: 5 failureThreshold: 3 startupProbe: httpGet: path: /minio/health/live port: 9000 scheme: HTTPS periodSeconds: 5 failureThreshold: 60 timeoutSeconds: 5 ``` **왜 좋은가:** - `readinessProbe`는 `/minio/health/cluster/read` → **read quorum** 검사. rolling update 중에도 read가 가능하면 Service에 남아있음 - `/minio/health/cluster` (write quorum)을 readiness로 쓰면 rolling 재시작 시 pod가 전부 빠져 완전 unavailable - `livenessProbe`는 단순 프로세스 생존만 확인 → 일시적 quorum 상실로 pod 강제 재시작 방지 - HTTPS scheme (requestAutoCert과 일치) --- ## 나쁜 예시 3: readiness를 write quorum으로 ```yaml readinessProbe: httpGet: path: /minio/health/cluster port: 9000 periodSeconds: 5 failureThreshold: 1 ``` **문제:** - rolling update로 pod 1개를 재시작하면 write quorum이 일시적으로 무너져 살아있는 pod들도 NotReady - Service가 endpoint를 전부 제거 → **읽기도 불가능** - `failureThreshold: 1` + 5초 주기 → 한 번 느린 응답으로 pod 빠짐 --- ## 좋은 예시 5: Ingress — API는 공개, Console은 내부 ```yaml --- apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: minio-api namespace: minio-prod annotations: cert-manager.io/cluster-issuer: letsencrypt-prod nginx.ingress.kubernetes.io/proxy-body-size: "0" nginx.ingress.kubernetes.io/proxy-request-buffering: "off" nginx.ingress.kubernetes.io/proxy-buffering: "off" nginx.ingress.kubernetes.io/backend-protocol: HTTPS spec: ingressClassName: nginx tls: - hosts: - s3.example.com secretName: minio-api-ingress-tls rules: - host: s3.example.com http: paths: - path: / pathType: Prefix backend: service: name: minio port: number: 443 --- apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: minio-console namespace: minio-prod annotations: cert-manager.io/cluster-issuer: letsencrypt-prod nginx.ingress.kubernetes.io/whitelist-source-range: "10.0.0.0/8,192.168.0.0/16" nginx.ingress.kubernetes.io/auth-url: "https://oauth2-proxy.example.com/oauth2/auth" nginx.ingress.kubernetes.io/backend-protocol: HTTPS spec: ingressClassName: nginx-internal tls: - hosts: - minio-console.internal.example.com secretName: minio-console-ingress-tls rules: - host: minio-console.internal.example.com http: paths: - path: / pathType: Prefix backend: service: name: minio-console port: number: 9443 ``` **왜 좋은가:** - API Ingress는 `proxy-body-size: 0` + request/response buffering off → 대용량 multipart upload 지원 - Console은 내부 ingress class + IP whitelist + OIDC forward-auth 2중 보호 - `backend-protocol: HTTPS` → MinIO의 auto-cert TLS를 TLS passthrough 형태로 전달 (인증서 SAN 보존) --- ## 나쁜 예시 4: Console 외부 공개 ```yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: minio-all spec: rules: - host: minio.example.com http: paths: - path: / pathType: Prefix backend: service: name: minio port: number: 9090 ``` **문제:** - Console이 인터넷에 그대로 노출 → root/admin credential brute force 표면 확장 - OIDC forward-auth 없음 → 기본 login 페이지가 공격자에게 노출 - IP 제한 없음 - Bucket 목록, access key 관리, 사용자 관리가 모두 공개 domain에 위치 --- ## 좋은 예시 6: ServiceMonitor (Prometheus bearer-token) 먼저 MinIO 내부에서 scrape token 발급: ```bash mc admin prometheus generate minio cluster # 출력에 bearer token과 scrape config가 나옴 ``` 그 결과 token을 Secret로 저장: ```yaml --- apiVersion: v1 kind: Secret metadata: name: minio-prometheus-token namespace: minio-prod type: Opaque stringData: token: "" --- apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: minio namespace: minio-prod labels: app.kubernetes.io/name: minio release: kube-prometheus-stack spec: selector: matchLabels: v1.min.io/tenant: minio endpoints: - port: https-minio scheme: https path: /minio/v2/metrics/cluster interval: 30s scrapeTimeout: 10s bearerTokenSecret: name: minio-prometheus-token key: token tlsConfig: insecureSkipVerify: false ca: secret: name: minio-tls key: ca.crt serverName: minio.minio-prod.svc.cluster.local ``` **왜 좋은가:** - `MINIO_PROMETHEUS_AUTH_TYPE=jwt` 와 매칭 (기본값) - `/minio/v2/metrics/cluster`는 cluster-wide view (replication lag, bucket 사용량, API latency) - TLS 검증 유지 (`insecureSkipVerify: false`, CA bundle 제공) - 외부 노출 없이 내부 scrape만 --- ## 좋은 예시 7: Bucket 초기화 (Job) — versioning + Object Lock + lifecycle ```yaml --- apiVersion: v1 kind: ConfigMap metadata: name: minio-bootstrap namespace: minio-prod data: init.sh: | #!/bin/sh set -eu mc --config-dir /tmp/mc alias import minio <