Add platform infrastructure configuration

2026-08-28 17:35:41 +09:00
parent fa76531e5b
commit 16c337bcc9
302 changed files with 83259 additions and 1 deletions
@@ -0,0 +1,38 @@
+# AIStor 정적 Local PersistentVolume
+
+상태(2026-07-23): 실제 클러스터 적용 및 바인딩 완료
+
+이 기반 구성은 `/srv/k3s/aistor`에 마운트된 전용 XFS 파일시스템을 하나의 정적
+Local PV에 연결한다.
+
+| 리소스 | 값 |
+| --- | --- |
+| PersistentVolume | `aistor-data-local-pv` |
+| StorageClass | `aistor-local-xfs-retain` |
+| Kubernetes에 표시되는 용량 | `900Gi` |
+| 접근 모드 | `ReadWriteOnce` |
+| Reclaim 정책 | `Retain` |
+| 바인딩 모드 | `WaitForFirstConsumer` |
+| 필수 노드 | `donghyeon-system-product-name` |
+
+ObjectStore pool은 이 StorageClass에 정확히 하나의 `900Gi` Claim을 요청한다.
+`WaitForFirstConsumer`는 해당 Claim과 Local PV node affinity를 함께 반영한다.
+StorageClass와 PV 모두 `Retain`을 사용하므로 Claim을 삭제해도 XFS 데이터는
+지워지지 않지만, 복구와 재바인딩은 수동 작업이다.
+
+적용하기 전에 호스트에서 다음 항목을 모두 확인한다.
+
+```sh
+findmnt -no SOURCE,TARGET,FSTYPE,OPTIONS /srv/k3s/aistor
+df -hT /srv/k3s/aistor
+kubectl get node donghyeon-system-product-name
+```
+
+첫 번째 명령은 전용 XFS 파티션이 정확한 경로에 마운트됐다고 출력해야 한다.
+PV의 `900Gi` 용량은 하드 파일시스템 quota가 아니라 Kubernetes 바인딩
+메타데이터다. 이 단일 노드 Local PV는 복제나 재해 복구를 제공하지 않는다.
+
+현재 `data0-minio-aistor-pool-0-0` PVC가 `aistor-data-local-pv`에 Bound 상태이며
+ObjectStore health는 `green`이다. `/srv/k3s/aistor`의 기존 XFS 메타데이터 사용량은
+AIStor 객체 데이터와 별개일 수 있으므로 `du` 합계와 `df` 사용량을 동일시하지
+않는다.
@@ -0,0 +1,27 @@
+apiVersion: v1
+kind: PersistentVolume
+metadata:
+  name: aistor-data-local-pv
+  labels:
+    app.kubernetes.io/part-of: platform
+    app.kubernetes.io/component: object-storage
+    platform.hyeonworks.com/storage-tier: hdd
+    platform.hyeonworks.com/filesystem: xfs
+spec:
+  capacity:
+    storage: 900Gi
+  volumeMode: Filesystem
+  accessModes:
+    - ReadWriteOnce
+  persistentVolumeReclaimPolicy: Retain
+  storageClassName: aistor-local-xfs-retain
+  local:
+    path: /srv/k3s/aistor
+  nodeAffinity:
+    required:
+      nodeSelectorTerms:
+        - matchExpressions:
+            - key: kubernetes.io/hostname
+              operator: In
+              values:
+                - donghyeon-system-product-name
@@ -0,0 +1,6 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+
+resources:
+  - storage-class.yaml
+  - aistor-data-pv.yaml
@@ -0,0 +1,12 @@
+apiVersion: storage.k8s.io/v1
+kind: StorageClass
+metadata:
+  name: aistor-local-xfs-retain
+  labels:
+    app.kubernetes.io/part-of: platform
+    app.kubernetes.io/component: object-storage
+    platform.hyeonworks.com/storage-tier: hdd
+provisioner: kubernetes.io/no-provisioner
+reclaimPolicy: Retain
+volumeBindingMode: WaitForFirstConsumer
+allowVolumeExpansion: false
@@ -0,0 +1,32 @@
+# Observability Local PersistentVolumes
+
+This kustomization reserves six workload-specific Local PV binding contracts on
+the root SSD. Each owner has a dedicated `kubernetes.io/no-provisioner`
+StorageClass with `WaitForFirstConsumer`, `Retain`, and volume expansion
+disabled. Every PV is `ReadWriteOnce`, retains its data after claim deletion,
+and is affined to `donghyeon-system-product-name`.
+
+| Owner | Host path | Capacity | StorageClass | Directory owner |
+| --- | --- | ---: | --- | --- |
+| Prometheus | `/srv/k3s/ssd/observability/prometheus` | `20Gi` | `ssd-local-observability-prometheus-retain` | `1000:2000` |
+| Grafana | `/srv/k3s/ssd/observability/grafana` | `2Gi` | `ssd-local-observability-grafana-retain` | `472:472` |
+| Alertmanager | `/srv/k3s/ssd/observability/alertmanager` | `1Gi` | `ssd-local-observability-alertmanager-retain` | `1000:2000` |
+| Alloy | `/srv/k3s/ssd/observability/alloy` | `2Gi` | `ssd-local-observability-alloy-retain` | `473:473` |
+| Loki | `/srv/k3s/ssd/observability/loki` | `5Gi` | `ssd-local-observability-loki-retain` | `10001:10001` |
+| Tempo | `/srv/k3s/ssd/observability/tempo` | `5Gi` | `ssd-local-observability-tempo-retain` | `10001:10001` |
+
+The declared 35 GiB is a Kubernetes binding contract, not an ext4 directory
+quota. Project quota is not active. All six directories share root-filesystem
+free space, so the later operational protections remain mandatory:
+
+- Prometheus `retentionSize=16GB` is not a whole-filesystem cap and does not cap
+  WAL plus head data.
+- Loki's 512 MB setting limits WAL replay RAM; it is not a disk quota.
+- Loki `disk_full_threshold=0.90` uses `Statfs` on the WAL filesystem.
+- Tempo 2.10.7 has no WAL byte limit or disk-full guard.
+- Root-filesystem 75%, 85%, and 90% alerts, apply/smoke `du` and `df` caps, and
+  acceptance of the root-90%-full ingest-stop runbook are still required.
+
+Run `bash scripts/bootstrap/prepare-observability-local-paths.sh` to inspect the
+read-only plan. The separately confirmed `--execute` mode prepares the exact
+paths at mode `0770`; it does not apply Kubernetes resources.
@@ -0,0 +1,6 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+
+resources:
+  - storage-classes.yaml
+  - persistent-volumes.yaml
@@ -0,0 +1,155 @@
+apiVersion: v1
+kind: PersistentVolume
+metadata:
+  name: observability-prometheus-local-pv
+  labels:
+    app.kubernetes.io/part-of: observability
+    app.kubernetes.io/component: prometheus
+spec:
+  capacity:
+    storage: 20Gi
+  volumeMode: Filesystem
+  accessModes:
+    - ReadWriteOnce
+  persistentVolumeReclaimPolicy: Retain
+  storageClassName: ssd-local-observability-prometheus-retain
+  local:
+    path: /srv/k3s/ssd/observability/prometheus
+  nodeAffinity:
+    required:
+      nodeSelectorTerms:
+        - matchExpressions:
+            - key: kubernetes.io/hostname
+              operator: In
+              values:
+                - donghyeon-system-product-name
+---
+apiVersion: v1
+kind: PersistentVolume
+metadata:
+  name: observability-grafana-local-pv
+  labels:
+    app.kubernetes.io/part-of: observability
+    app.kubernetes.io/component: grafana
+spec:
+  capacity:
+    storage: 2Gi
+  volumeMode: Filesystem
+  accessModes:
+    - ReadWriteOnce
+  persistentVolumeReclaimPolicy: Retain
+  storageClassName: ssd-local-observability-grafana-retain
+  local:
+    path: /srv/k3s/ssd/observability/grafana
+  nodeAffinity:
+    required:
+      nodeSelectorTerms:
+        - matchExpressions:
+            - key: kubernetes.io/hostname
+              operator: In
+              values:
+                - donghyeon-system-product-name
+---
+apiVersion: v1
+kind: PersistentVolume
+metadata:
+  name: observability-alertmanager-local-pv
+  labels:
+    app.kubernetes.io/part-of: observability
+    app.kubernetes.io/component: alertmanager
+spec:
+  capacity:
+    storage: 1Gi
+  volumeMode: Filesystem
+  accessModes:
+    - ReadWriteOnce
+  persistentVolumeReclaimPolicy: Retain
+  storageClassName: ssd-local-observability-alertmanager-retain
+  local:
+    path: /srv/k3s/ssd/observability/alertmanager
+  nodeAffinity:
+    required:
+      nodeSelectorTerms:
+        - matchExpressions:
+            - key: kubernetes.io/hostname
+              operator: In
+              values:
+                - donghyeon-system-product-name
+---
+apiVersion: v1
+kind: PersistentVolume
+metadata:
+  name: observability-alloy-local-pv
+  labels:
+    app.kubernetes.io/part-of: observability
+    app.kubernetes.io/component: alloy
+spec:
+  capacity:
+    storage: 2Gi
+  volumeMode: Filesystem
+  accessModes:
+    - ReadWriteOnce
+  persistentVolumeReclaimPolicy: Retain
+  storageClassName: ssd-local-observability-alloy-retain
+  local:
+    path: /srv/k3s/ssd/observability/alloy
+  nodeAffinity:
+    required:
+      nodeSelectorTerms:
+        - matchExpressions:
+            - key: kubernetes.io/hostname
+              operator: In
+              values:
+                - donghyeon-system-product-name
+---
+apiVersion: v1
+kind: PersistentVolume
+metadata:
+  name: observability-loki-local-pv
+  labels:
+    app.kubernetes.io/part-of: observability
+    app.kubernetes.io/component: loki
+spec:
+  capacity:
+    storage: 5Gi
+  volumeMode: Filesystem
+  accessModes:
+    - ReadWriteOnce
+  persistentVolumeReclaimPolicy: Retain
+  storageClassName: ssd-local-observability-loki-retain
+  local:
+    path: /srv/k3s/ssd/observability/loki
+  nodeAffinity:
+    required:
+      nodeSelectorTerms:
+        - matchExpressions:
+            - key: kubernetes.io/hostname
+              operator: In
+              values:
+                - donghyeon-system-product-name
+---
+apiVersion: v1
+kind: PersistentVolume
+metadata:
+  name: observability-tempo-local-pv
+  labels:
+    app.kubernetes.io/part-of: observability
+    app.kubernetes.io/component: tempo
+spec:
+  capacity:
+    storage: 5Gi
+  volumeMode: Filesystem
+  accessModes:
+    - ReadWriteOnce
+  persistentVolumeReclaimPolicy: Retain
+  storageClassName: ssd-local-observability-tempo-retain
+  local:
+    path: /srv/k3s/ssd/observability/tempo
+  nodeAffinity:
+    required:
+      nodeSelectorTerms:
+        - matchExpressions:
+            - key: kubernetes.io/hostname
+              operator: In
+              values:
+                - donghyeon-system-product-name
@@ -0,0 +1,53 @@
+apiVersion: storage.k8s.io/v1
+kind: StorageClass
+metadata:
+  name: ssd-local-observability-prometheus-retain
+provisioner: kubernetes.io/no-provisioner
+reclaimPolicy: Retain
+volumeBindingMode: WaitForFirstConsumer
+allowVolumeExpansion: false
+---
+apiVersion: storage.k8s.io/v1
+kind: StorageClass
+metadata:
+  name: ssd-local-observability-grafana-retain
+provisioner: kubernetes.io/no-provisioner
+reclaimPolicy: Retain
+volumeBindingMode: WaitForFirstConsumer
+allowVolumeExpansion: false
+---
+apiVersion: storage.k8s.io/v1
+kind: StorageClass
+metadata:
+  name: ssd-local-observability-alertmanager-retain
+provisioner: kubernetes.io/no-provisioner
+reclaimPolicy: Retain
+volumeBindingMode: WaitForFirstConsumer
+allowVolumeExpansion: false
+---
+apiVersion: storage.k8s.io/v1
+kind: StorageClass
+metadata:
+  name: ssd-local-observability-alloy-retain
+provisioner: kubernetes.io/no-provisioner
+reclaimPolicy: Retain
+volumeBindingMode: WaitForFirstConsumer
+allowVolumeExpansion: false
+---
+apiVersion: storage.k8s.io/v1
+kind: StorageClass
+metadata:
+  name: ssd-local-observability-loki-retain
+provisioner: kubernetes.io/no-provisioner
+reclaimPolicy: Retain
+volumeBindingMode: WaitForFirstConsumer
+allowVolumeExpansion: false
+---
+apiVersion: storage.k8s.io/v1
+kind: StorageClass
+metadata:
+  name: ssd-local-observability-tempo-retain
+provisioner: kubernetes.io/no-provisioner
+reclaimPolicy: Retain
+volumeBindingMode: WaitForFirstConsumer
+allowVolumeExpansion: false
@@ -0,0 +1,12 @@
+# pgAdmin SSD Local PV
+
+pgAdmin 설정 데이터베이스와 세션 상태를 위한 2Gi Local PV입니다.
+
+- 호스트 경로: /srv/k3s/ssd/pgadmin
+- 노드: donghyeon-system-product-name
+- StorageClass: ssd-local-pgadmin-retain
+- 회수 정책: Retain
+
+PVC나 배포를 제거해도 PV와 호스트 데이터는 자동 삭제하지 않습니다.
+호스트 디렉터리는 적용 스크립트가 소유권과 파일시스템 경계를 검증한 뒤
+생성합니다.
@@ -0,0 +1,6 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+
+resources:
+  - storage-class.yaml
+  - pgadmin-pv.yaml
@@ -0,0 +1,26 @@
+apiVersion: v1
+kind: PersistentVolume
+metadata:
+  name: pgadmin-data-local-pv
+  labels:
+    app.kubernetes.io/name: pgadmin4
+    app.kubernetes.io/part-of: platform
+    platform.hyeonworks.com/storage-tier: ssd
+spec:
+  capacity:
+    storage: 2Gi
+  volumeMode: Filesystem
+  accessModes:
+    - ReadWriteOnce
+  persistentVolumeReclaimPolicy: Retain
+  storageClassName: ssd-local-pgadmin-retain
+  local:
+    path: /srv/k3s/ssd/pgadmin
+  nodeAffinity:
+    required:
+      nodeSelectorTerms:
+        - matchExpressions:
+            - key: kubernetes.io/hostname
+              operator: In
+              values:
+                - donghyeon-system-product-name
@@ -0,0 +1,11 @@
+apiVersion: storage.k8s.io/v1
+kind: StorageClass
+metadata:
+  name: ssd-local-pgadmin-retain
+  labels:
+    app.kubernetes.io/part-of: platform
+    platform.hyeonworks.com/storage-tier: ssd
+provisioner: kubernetes.io/no-provisioner
+reclaimPolicy: Retain
+volumeBindingMode: WaitForFirstConsumer
+allowVolumeExpansion: false
@@ -0,0 +1,33 @@
+# SSD 정적 Local PersistentVolume
+
+첫 배포에서는 서버의 루트 SSD에 워크로드별 Local PV 두 개를 예약한다.
+
+| 워크로드 | StorageClass | 로컬 경로 | 선언 용량 |
+| --- | --- | --- | ---: |
+| 플랫폼 PostgreSQL | `ssd-local-postgres-retain` | `/srv/k3s/ssd/platform-postgres` | `20Gi` |
+| Gitea | `ssd-local-gitea-retain` | `/srv/k3s/ssd/gitea` | `50Gi` |
+
+별도의 StorageClass를 사용하여 PostgreSQL의 20 GiB Claim이 Gitea의 50 GiB 볼륨에
+잘못 바인딩되는 것을 방지한다. `WaitForFirstConsumer`를 사용하면 Scheduler가
+Local PV의 node affinity를 반영할 수 있다. 두 PV 모두 `Retain`을 사용하므로
+Claim을 삭제해도 디스크의 데이터는 삭제되지 않는다.
+
+PVC는 의도적으로 이 인프라 디렉터리가 아니라 소유자와 함께 둔다.
+
+- CloudNativePG는 `Cluster`의 storage 명세에서 PostgreSQL 인스턴스 PVC를 생성한다.
+  이 PVC는 `ssd-local-postgres-retain`과 `20Gi`를 요청해야 한다.
+- Gitea 서비스 배포가 직접 또는 공식 Chart를 통해 자체 PVC를 소유한다. 이 PVC는
+  `ssd-local-gitea-retain`과 `50Gi`를 요청해야 한다.
+
+PV를 생성하기 전에 `scripts/bootstrap/prepare-ssd-local-paths.sh`를 수동으로
+실행한다. 이 스크립트는 `/srv`가 루트 파일시스템에 있는지 확인한 후 정확히 두
+디렉터리만 준비한다.
+
+Local PV의 `capacity` 필드는 바인딩 메타데이터이며 파일시스템 quota가 아니다.
+PostgreSQL과 Gitea는 루트 SSD의 여유 공간을 공유한다. 루트 파일시스템을
+모니터링하고, 하드 제한이 필요해지면 파일시스템 또는 블록 장치 quota를 별도로
+추가한다.
+
+Local PV는 이 워크로드들을 `donghyeon-system-product-name` 노드에 종속시킨다.
+`Retain`은 자동 파일 삭제를 막지만 백업이나 고가용성을 제공하지 않는다. Released
+상태의 PV를 재사용하는 작업은 명시적인 복구 절차로 수행해야 한다.
@@ -0,0 +1,26 @@
+apiVersion: v1
+kind: PersistentVolume
+metadata:
+  name: gitea-local-pv
+  labels:
+    app.kubernetes.io/part-of: platform
+    app.kubernetes.io/component: source-control
+    platform.hyeonworks.com/storage-tier: ssd
+spec:
+  capacity:
+    storage: 50Gi
+  volumeMode: Filesystem
+  accessModes:
+    - ReadWriteOnce
+  persistentVolumeReclaimPolicy: Retain
+  storageClassName: ssd-local-gitea-retain
+  local:
+    path: /srv/k3s/ssd/gitea
+  nodeAffinity:
+    required:
+      nodeSelectorTerms:
+        - matchExpressions:
+            - key: kubernetes.io/hostname
+              operator: In
+              values:
+                - donghyeon-system-product-name
@@ -0,0 +1,7 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+
+resources:
+  - storage-classes.yaml
+  - platform-postgres-pv.yaml
+  - gitea-pv.yaml
@@ -0,0 +1,26 @@
+apiVersion: v1
+kind: PersistentVolume
+metadata:
+  name: platform-postgres-local-pv
+  labels:
+    app.kubernetes.io/part-of: platform
+    app.kubernetes.io/component: database
+    platform.hyeonworks.com/storage-tier: ssd
+spec:
+  capacity:
+    storage: 20Gi
+  volumeMode: Filesystem
+  accessModes:
+    - ReadWriteOnce
+  persistentVolumeReclaimPolicy: Retain
+  storageClassName: ssd-local-postgres-retain
+  local:
+    path: /srv/k3s/ssd/platform-postgres
+  nodeAffinity:
+    required:
+      nodeSelectorTerms:
+        - matchExpressions:
+            - key: kubernetes.io/hostname
+              operator: In
+              values:
+                - donghyeon-system-product-name
@@ -0,0 +1,23 @@
+apiVersion: storage.k8s.io/v1
+kind: StorageClass
+metadata:
+  name: ssd-local-postgres-retain
+  labels:
+    app.kubernetes.io/part-of: platform
+    platform.hyeonworks.com/storage-tier: ssd
+provisioner: kubernetes.io/no-provisioner
+reclaimPolicy: Retain
+volumeBindingMode: WaitForFirstConsumer
+allowVolumeExpansion: false
+---
+apiVersion: storage.k8s.io/v1
+kind: StorageClass
+metadata:
+  name: ssd-local-gitea-retain
+  labels:
+    app.kubernetes.io/part-of: platform
+    platform.hyeonworks.com/storage-tier: ssd
+provisioner: kubernetes.io/no-provisioner
+reclaimPolicy: Retain
+volumeBindingMode: WaitForFirstConsumer
+allowVolumeExpansion: false