Add platform infrastructure configuration

This commit is contained in:
donghyeon-ka
2026-08-28 17:35:41 +09:00
parent fa76531e5b
commit 16c337bcc9
302 changed files with 83259 additions and 1 deletions
+229
View File
@@ -0,0 +1,229 @@
# Gitea 서비스
Gitea는 같은 공식 Helm Chart와 공통 values를 사용하는 두 Kustomize 빌드 루트로
관리한다. 업스트림 Chart 템플릿은 복제하거나 수정하지 않는다.
| 용도 | 빌드 루트 | 검증 산출물 | 적용 진입점 |
| --- | --- | --- | --- |
| 신규 설치 baseline | `services/gitea` | `gitea.yaml` | `apply-phase1-gitea.sh` |
| OIDC·브랜딩 목표 상태 | `services/gitea/profiles/oidc` | `gitea-oidc.yaml` | `apply-gitea-oidc.sh` |
baseline 루트는 OIDC Secret, `id` host alias, Keycloak 전용 egress와 브랜딩을
포함하지 않는다. OIDC profile만 공통 기반에 이 네 요소를 추가한다.
상태(2026-07-23): Host Nginx public discovery 전환과 OIDC-enabled Gitea
rollout을 완료했다. 적용 manifest SHA-256은 `d25a757...a157`이며, 활성
`keycloak` OAuth2 source, 외부 인증 전용 가입 정책, authorization-code redirect와
아래 선언형 브랜딩의 자동 검증을 모두 통과했다. 실제 realm 사용자의 브라우저
login/callback/logout, 비상 관리자 실제 로그인과 Pod 재시작 뒤 설정 지속성은
별도 수용 시험으로 남아 있다.
## 고정된 업스트림 릴리스
| 항목 | 값 |
| --- | --- |
| Chart 저장소 | `https://dl.gitea.com/charts/` |
| Chart | `gitea` |
| Chart 버전 | `12.7.0` |
| Chart 패키지 SHA-256 | `5881ef9c59400bee2d5547e77c4cd0efb925143c2f5d93fb4f38446db76b0167` |
| Gitea 이미지 | `docker.gitea.com/gitea:1.27.0-rootless` |
Chart는 렌더링할 때 저장소 공통 `.gitignore`에서 제외한
`profiles/oidc/.helm/charts`에 내려받는다. baseline 루트와 OIDC profile이 이 고정
SHA 검증 캐시 하나를 각자의 `chartHome`으로 참조한다. values와 Patch는 모두
`services/gitea` 아래에 있으므로 Kustomize의 기본 `RootOnly` 로드 제한을 유지한다.
## baseline과 OIDC 로그인 정책
두 프로필이 공유하는 `profiles/oidc/values/baseline.yaml`은 다음 정책을 선언한다.
- 자체 가입을 닫고(`DISABLE_REGISTRATION=true`) 외부 인증 전용 가입도 끈다.
- 가입 버튼을 숨기고 기존 `gitea-admin` 로컬 비상 로그인을 유지한다.
- PostgreSQL, PVC, probe, resource와 restricted securityContext를 공통으로 둔다.
`profiles/oidc/values/oidc.yaml`은 baseline 위에 다음 항목만 덮어쓴다.
- `APP_NAME=Hyeonworks Git`으로 브라우저 제목과 애플리케이션 이름을 고정한다.
- `DISABLE_REGISTRATION=false``ALLOW_ONLY_EXTERNAL_REGISTRATION=true`를 함께 사용해
Keycloak 같은 외부 인증을 통한 신규 사용자 생성만 허용한다.
- `SHOW_REGISTRATION_BUTTON=false`로 Gitea 자체 공개 가입 버튼을 숨긴다.
- `oauth2_client.ENABLE_AUTO_REGISTRATION=true`, `USERNAME=preferred_username`,
`OPENID_CONNECT_SCOPES="profile email"`로 OIDC 최초 로그인 계정을 생성한다.
- baseline의 `ENABLE_PASSWORD_SIGNIN_FORM=true`를 유지해 Keycloak 장애 시에도 로컬
비상 관리자가 로그인할 수 있게 한다.
`ACCOUNT_LINKING=login`은 이메일이 같은 기존 계정을 자동 병합하지 않는다. 사용자가
기존 Gitea 자격 증명으로 한 번 더 확인해야 연결되므로, 이메일 일치만으로 계정을
탈취하는 위험을 줄인다.
## 의존성과 트래픽
- `gitea``platform-data` namespace를 먼저 적용한다.
- 이 서비스를 적용하기 전에 `/srv/k3s/ssd/gitea`를 준비하고 `gitea-local-pv`
기반 구성을 적용한다. `gitea-data` Claim은 `ssd-local-gitea-retain`을 통해 해당
PV에 바인딩되며 `50Gi`를 요청한다.
- 외부 Database endpoint는
`platform-postgres-rw.platform-data.svc.cluster.local:5432`다.
- Host Nginx에서 공개 TLS를 종료한다. Traefik은 HTTP를 받아 `gitea-http:3000`으로
전달하며, 이 Ingress에는 의도적으로 TLS 섹션이 없다.
- OIDC profile의 Gitea 서버가 discovery와 token endpoint에 접근할 때도 공개 issuer
`https://id.learn.hyeonworks.com/realms/hyeonworks`를 사용한다. 홈 라우터의 NAT
loopback에 의존하지 않도록 Pod의 `hostAliases`는 이 이름만 노드
`192.168.0.107`로 해석하고, 전용 NetworkPolicy가 해당 `/32`의 TCP 443만
허용한다. 연결은 Host Nginx TLS와 Traefik을 그대로 통과한다.
- 첫 배포에서는 Gitea SSH, Actions, Git LFS 및 Package Registry를 비활성화한다.
Chart `12.7.0`은 항상 SSH Service를 렌더링하므로 baseline 루트는 inline patch,
OIDC profile은 `patches/delete-ssh-service.yaml`로 Helm 렌더링 뒤
`gitea-ssh`를 삭제한다.
- 공통 egress는 클러스터 DNS, 플랫폼 PostgreSQL Pod 및 공개 IPv4 HTTPS 대상으로
제한한다. OIDC profile만 Host Nginx 노드 `/32`의 TCP 443을 추가한다. 내부
HTTPS, SMTP, SSH remote 및 사설 네트워크 webhook을 사용하려면 향후
NetworkPolicy를 명시적으로 변경해야 한다.
## 선언형 브랜딩
브랜딩은 Gitea 공식 custom path와 Chart의 volume 확장 필드만 사용한다.
Chart 템플릿과 Gitea 전체 페이지 템플릿은 복제하거나 수정하지 않는다.
```text
branding/
├── public/assets/
│ ├── css/hyeonworks.css
│ └── img/
│ ├── logo.svg
│ └── favicon.svg
└── templates/custom/
├── header.tmpl
└── extra_links.tmpl
```
`branding/kustomization.yaml``configMapGenerator`가 자산과 템플릿 ConfigMap을
각각 생성하고 `profiles/oidc/kustomization.yaml`만 이 구성요소를 포함한다. 이름
suffix hash를 끄지 않았으므로 파일 내용이 바뀌면 이름도 바뀌고, Kustomize가
Deployment의 ConfigMap 참조를 같은 해시 이름으로 치환한다.
`profiles/oidc/values/oidc.yaml`은 공식 Chart 확장 지점을 다음과 같이 사용한다.
- 자산 ConfigMap을 `/data/gitea/public/assets`에 마운트한다.
- 템플릿 ConfigMap을 `/data/gitea/templates/custom`에 마운트한다.
- 두 volume은 rootless Gitea 주 컨테이너에만 `readOnly: true`로 마운트하며,
init container에는 추가하지 않는다.
`header.tmpl`은 같은 origin의 CSS와 모바일 브라우저 색상만 추가한다.
`extra_links.tmpl`은 실제 Gitea 내부 경로인 `/explore/organizations`만 연결한다.
존재하지 않는 Learning 또는 Identity 서비스 링크는 미리 만들지 않는다.
전체 `home.tmpl`은 덮어쓰지 않아 Gitea 업그레이드 때 template drift 범위를 줄인다.
Gitea는 custom template을 시작할 때 읽으므로 변경 적용에는 Pod 재시작이 필요하다.
SVG 원본만 저장소에서 관리한다. Gitea가 참조하는 Open Graph, Apple touch icon 및
구형 브라우저용 PNG는 검증된 변환 절차를 추가하기 전까지 업스트림 기본 자산을
사용한다.
- [Gitea 1.27 커스터마이징 공식 문서](https://docs.gitea.com/administration/customizing-gitea)
- [Gitea Chart 12.7.0 공식 values](https://gitea.com/gitea/helm-chart/src/tag/v12.7.0/values.yaml)
## Secret 계약
credential Secret 매니페스트나 credential 값은 이 저장소에 두지 않는다.
`gitea-admin``gitea-db-credentials`는 baseline Deployment 최초 시작 전에,
`gitea-keycloak-oidc`는 OIDC-enabled Deployment rollout 전에 `gitea`
namespace에 있어야 한다.
### `gitea-admin`
| 속성 | 필수 값 |
| --- | --- |
| Kubernetes 유형 | `kubernetes.io/basic-auth` |
| `username` | 최초 로컬 비상 관리자 username |
| `password` | 강력한 일회용 관리자 비밀번호 |
Chart는 `initialOnlyRequireReset`을 사용하므로 최초 로그인 때 초기 비밀번호를
변경해야 하며 이후에는 조정하지 않는다.
### `gitea-db-credentials`
| 속성 | 필수 값 |
| --- | --- |
| Kubernetes 유형 | `kubernetes.io/basic-auth` |
| `username` | `gitea` |
| `password` | PostgreSQL `gitea` Role에 할당한 비밀번호 |
Kubernetes에서는 namespace를 넘어 Secret을 참조할 수 없다. `platform-data`에 있는
같은 이름의 Secret은 CloudNativePG/bootstrap 사본이다. `gitea` namespace의
Secret은 동일한 Database credential을 담아야 하며 Chart가 이 사본을 참조한다.
### `gitea-keycloak-oidc`
| 속성 | 필수 값 |
| --- | --- |
| Kubernetes 유형 | `Opaque` |
| `key` | `gitea` |
| `secret` | Keycloak이 생성한 confidential client secret |
`scripts/bootstrap/configure-keycloak-gitea-oidc.sh`가 Keycloak Admin REST API로
정확한 callback URI를 가진 client를 생성하거나 갱신하고 이 Secret을 만든다.
공식 Chart의 `gitea.oauth[].existingSecret``key``secret`을 환경 변수로만
초기화 컨테이너에 전달하고, 인증 소스 이름 `keycloak`을 반복 실행 시 갱신한다.
Secret payload는 values, 렌더 결과 또는 Git에 포함되지 않는다.
업스트림 Chart는 credential이 아닌 초기화 스크립트와 inline 설정이 담긴 Opaque
Secret도 렌더링한다. 이 생성 리소스들은 고정된 Chart의 일부이며 위 세 credential
계약의 값은 포함하지 않는다.
## 렌더링과 검사
Helm은 `PATH`에 있어야 하지만 릴리스를 직접 설치하는 데 사용하지 않는다. 플랫폼
저장소 루트에서 고정 Chart SHA를 확인하는 검증기로 렌더링한다.
```sh
PLATFORM_HELM_BIN=/home/donghyeon/.local/bin/helm \
bash scripts/validate/render-phase1.sh
```
검증기는 `gitea.yaml` baseline과 `gitea-oidc.yaml` 목표 상태를 모두 렌더한다.
두 결과에는 각각 Gitea Deployment 하나, `gitea-http`, `gitea-data`, Traefik
Ingress가 있어야 하고 `gitea-ssh`, NodePort, LoadBalancer와 내부 TLS는 없어야
한다. 두 Deployment 모두 `gitea-admin`, `gitea-db-credentials`를 참조만 하고
생성하지 않는다.
baseline은 `gitea-keycloak-oidc`, ID host alias, Keycloak 전용 NetworkPolicy,
브랜딩 ConfigMap과 외부 인증 전용 가입 설정이 없어야 한다. 반대로 OIDC 산출물은
`gitea-keycloak-oidc`를 값 노출 없이 참조하고 다음을 모두 만족해야 한다.
- 두 브랜딩 ConfigMap 이름에 content hash가 있고 Deployment 참조와 일치한다.
- `/data/gitea/public/assets``/data/gitea/templates/custom` 마운트는 read-only다.
- inline 설정에 외부 인증 전용 가입, OIDC 자동 계정 생성, `ACCOUNT_LINKING=login`,
로컬 비밀번호 로그인 유지가 모두 렌더링된다.
- OAuth 인증 소스 이름은 callback URI의 경로와 같은 `keycloak`이고 discovery
issuer는 `https://id.learn.hyeonworks.com/realms/hyeonworks`다.
## Keycloak OIDC 안전 적용
Host Nginx의 Keycloak discovery가 실제 JSON issuer를 반환하고 위 세 Secret 계약이
준비된 뒤 다음 전용 진입점으로 적용한다.
1. Host Nginx의 `id.learn.hyeonworks.com` 정적 hold를 Keycloak proxy 후보로
전환한다.
2. public discovery JSON의 `issuer`
`https://id.learn.hyeonworks.com/realms/hyeonworks`와 정확히 같은지 확인한다.
3. 다음 전용 스크립트로 OIDC 목표 상태와 브랜딩을 함께 적용한다.
```sh
PLATFORM_HELM_BIN=/home/donghyeon/.local/bin/helm \
bash scripts/bootstrap/apply-gitea-oidc.sh --execute
```
스크립트는 public discovery와 Secret의 type·key 이름만 먼저 확인하고, 고정 Chart
SHA 검증 결과 중 `gitea-oidc.yaml` 하나만 적용한다. 이후 Deployment·EndpointSlice,
활성 `keycloak` OAuth2 source, 외부 인증 전용 가입 정책, OIDC authorization-code
redirect와 브랜딩 자산 SHA-256을 검증한다. Secret 값, bearer token과 OIDC state가
포함된 전체 Location은 출력하지 않는다.
2026-07-23 실제 적용에서는 manifest SHA-256 `d25a757...a157`의 rollout과 위 자동
검사를 모두 통과했다. 이 redirect 검사는 realm 사용자의 실제 브라우저
login/callback/logout 시험을 대신하지 않는다. self-registration은 비활성이므로
named 관리자와 MFA를 준비한 뒤 시험 사용자를 별도로 생성한다. 로컬 비상 관리자
실제 로그인과 Pod 재시작 뒤 OAuth source·정책·브랜딩 지속성도 별도로 확인한다.
기존 Gitea에 `apply-phase1-gitea.sh`를 다시 실행해 OIDC 상태를 적용하거나
복구하지 않는다.
+25
View File
@@ -0,0 +1,25 @@
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: gitea-http
namespace: gitea
labels:
app.kubernetes.io/name: gitea
app.kubernetes.io/instance: gitea
app.kubernetes.io/component: source-control
app.kubernetes.io/part-of: platform
annotations:
traefik.ingress.kubernetes.io/router.entrypoints: web
spec:
ingressClassName: traefik
rules:
- host: git.learn.hyeonworks.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: gitea-http
port:
number: 3000
+7
View File
@@ -0,0 +1,7 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- pvc.yaml
- ingress.yaml
- network-policies.yaml
+161
View File
@@ -0,0 +1,161 @@
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: gitea-default-deny
namespace: gitea
labels:
app.kubernetes.io/name: gitea
app.kubernetes.io/instance: gitea
app.kubernetes.io/part-of: platform
spec:
podSelector:
matchLabels:
app.kubernetes.io/name: gitea
app.kubernetes.io/instance: gitea
policyTypes:
- Ingress
- Egress
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: gitea-allow-traefik-http
namespace: gitea
labels:
app.kubernetes.io/name: gitea
app.kubernetes.io/instance: gitea
app.kubernetes.io/part-of: platform
spec:
podSelector:
matchLabels:
app.kubernetes.io/name: gitea
app.kubernetes.io/instance: gitea
policyTypes:
- Ingress
ingress:
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
podSelector:
matchLabels:
app.kubernetes.io/name: traefik
ports:
- protocol: TCP
port: 3000
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: gitea-allow-dns
namespace: gitea
labels:
app.kubernetes.io/name: gitea
app.kubernetes.io/instance: gitea
app.kubernetes.io/part-of: platform
spec:
podSelector:
matchLabels:
app.kubernetes.io/name: gitea
app.kubernetes.io/instance: gitea
policyTypes:
- Egress
egress:
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
podSelector:
matchLabels:
k8s-app: kube-dns
ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: gitea-allow-platform-postgres
namespace: gitea
labels:
app.kubernetes.io/name: gitea
app.kubernetes.io/instance: gitea
app.kubernetes.io/part-of: platform
spec:
podSelector:
matchLabels:
app.kubernetes.io/name: gitea
app.kubernetes.io/instance: gitea
policyTypes:
- Egress
egress:
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: platform-data
podSelector:
matchLabels:
cnpg.io/cluster: platform-postgres
ports:
- protocol: TCP
port: 5432
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: gitea-allow-public-git-https
namespace: gitea
labels:
app.kubernetes.io/name: gitea
app.kubernetes.io/instance: gitea
app.kubernetes.io/part-of: platform
spec:
podSelector:
matchLabels:
app.kubernetes.io/name: gitea
app.kubernetes.io/instance: gitea
policyTypes:
- Egress
egress:
- to:
- ipBlock:
cidr: 0.0.0.0/0
except:
- 10.0.0.0/8
- 172.16.0.0/12
- 192.168.0.0/16
- 169.254.0.0/16
ports:
- protocol: TCP
port: 443
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: gitea-allow-prometheus-metrics
namespace: gitea
labels:
app.kubernetes.io/name: gitea
app.kubernetes.io/instance: gitea
app.kubernetes.io/part-of: platform
spec:
podSelector:
matchLabels:
app.kubernetes.io/name: gitea
app.kubernetes.io/instance: gitea
policyTypes:
- Ingress
ingress:
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: observability
podSelector:
matchLabels:
app.kubernetes.io/name: prometheus
app.kubernetes.io/instance: observability-core-kube-pr-prometheus
ports:
- protocol: TCP
port: 3000
+19
View File
@@ -0,0 +1,19 @@
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: gitea-data
namespace: gitea
labels:
app.kubernetes.io/name: gitea
app.kubernetes.io/instance: gitea
app.kubernetes.io/component: source-control
app.kubernetes.io/part-of: platform
spec:
accessModes:
- ReadWriteOnce
volumeMode: Filesystem
storageClassName: ssd-local-gitea-retain
volumeName: gitea-local-pv
resources:
requests:
storage: 50Gi
@@ -0,0 +1,15 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: gitea
configMapGenerator:
- name: gitea-branding-assets
files:
- hyeonworks.css=public/assets/css/hyeonworks.css
- logo.svg=public/assets/img/logo.svg
- favicon.svg=public/assets/img/favicon.svg
- name: gitea-branding-templates
files:
- header.tmpl=templates/custom/header.tmpl
- extra_links.tmpl=templates/custom/extra_links.tmpl
@@ -0,0 +1,59 @@
:root {
--hyeonworks-accent: #2563eb;
--hyeonworks-accent-hover: #1d4ed8;
--hyeonworks-focus: rgba(37, 99, 235, 0.35);
}
#navbar-logo {
gap: 0.5rem;
}
#navbar-logo img {
width: 2rem;
height: 2rem;
}
#navbar-logo::after {
content: "Hyeonworks Git";
font-weight: 700;
letter-spacing: -0.015em;
white-space: nowrap;
}
#navbar .hw-brand-link {
font-weight: 600;
}
#navbar .hw-brand-link:hover,
#navbar .hw-brand-link:focus-visible {
color: var(--hyeonworks-accent);
}
#navbar .hw-brand-link:focus-visible {
border-radius: 0.375rem;
outline: 3px solid var(--hyeonworks-focus);
outline-offset: -3px;
}
.ui.primary.button,
.ui.primary.buttons .button {
background-color: var(--hyeonworks-accent);
}
.ui.primary.button:hover,
.ui.primary.buttons .button:hover {
background-color: var(--hyeonworks-accent-hover);
}
@media (max-width: 767.98px) {
#navbar-logo::after {
display: none;
}
}
@media (prefers-reduced-motion: reduce) {
#navbar .hw-brand-link,
.ui.button {
transition: none;
}
}
@@ -0,0 +1,12 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 64 64" role="img" aria-labelledby="title">
<title id="title">Hyeonworks Git</title>
<defs>
<linearGradient id="brand-gradient" x1="6" y1="6" x2="58" y2="58" gradientUnits="userSpaceOnUse">
<stop stop-color="#38BDF8"/>
<stop offset="1" stop-color="#4F46E5"/>
</linearGradient>
</defs>
<rect x="3" y="3" width="58" height="58" rx="16" fill="url(#brand-gradient)"/>
<path d="M19 17v30M45 17v30M19 32h26" fill="none" stroke="#fff" stroke-width="6" stroke-linecap="round"/>
<path d="m14 25-6 7 6 7M50 25l6 7-6 7" fill="none" stroke="#BAE6FD" stroke-width="3" stroke-linecap="round" stroke-linejoin="round"/>
</svg>

After

Width:  |  Height:  |  Size: 695 B

@@ -0,0 +1,13 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 64 64" role="img" aria-labelledby="title description">
<title id="title">Hyeonworks Git</title>
<desc id="description">겹쳐진 코드 괄호와 H 문자를 사용한 Hyeonworks Git 로고</desc>
<defs>
<linearGradient id="brand-gradient" x1="6" y1="6" x2="58" y2="58" gradientUnits="userSpaceOnUse">
<stop stop-color="#38BDF8"/>
<stop offset="1" stop-color="#4F46E5"/>
</linearGradient>
</defs>
<rect x="3" y="3" width="58" height="58" rx="16" fill="url(#brand-gradient)"/>
<path d="M19 17v30M45 17v30M19 32h26" fill="none" stroke="#fff" stroke-width="6" stroke-linecap="round"/>
<path d="m14 25-6 7 6 7M50 25l6 7-6 7" fill="none" stroke="#BAE6FD" stroke-width="3" stroke-linecap="round" stroke-linejoin="round"/>
</svg>

After

Width:  |  Height:  |  Size: 810 B

@@ -0,0 +1 @@
<a class="item hw-brand-link" href="{{AppSubUrl}}/explore/organizations">조직</a>
@@ -0,0 +1,2 @@
<meta name="theme-color" content="#0f172a">
<link rel="stylesheet" href="{{AssetUrlPrefix}}/css/hyeonworks.css">
+40
View File
@@ -0,0 +1,40 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: gitea
helmGlobals:
chartHome: profiles/oidc/.helm/charts
helmCharts:
- name: gitea
repo: https://dl.gitea.com/charts/
version: 12.7.0
releaseName: gitea
namespace: gitea
valuesFile: profiles/oidc/values/baseline.yaml
includeCRDs: false
skipTests: true
resources:
- base
patches:
- path: profiles/oidc/patches/servicemonitor-job-label-patch.yaml
target:
group: monitoring.coreos.com
version: v1
kind: ServiceMonitor
name: gitea
namespace: gitea
- target:
group: ""
version: v1
kind: Service
name: gitea-ssh
patch: |-
$patch: delete
apiVersion: v1
kind: Service
metadata:
name: gitea-ssh
@@ -0,0 +1,39 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: gitea
helmGlobals:
chartHome: .helm/charts
helmCharts:
- name: gitea
repo: https://dl.gitea.com/charts/
version: 12.7.0
releaseName: gitea
namespace: gitea
valuesFile: values/baseline.yaml
additionalValuesFiles:
- values/oidc.yaml
includeCRDs: false
skipTests: true
resources:
- ../../base
- ../../branding
- network-policy.yaml
patches:
- path: patches/servicemonitor-job-label-patch.yaml
target:
group: monitoring.coreos.com
version: v1
kind: ServiceMonitor
name: gitea
namespace: gitea
- path: patches/delete-ssh-service.yaml
target:
group: ""
version: v1
kind: Service
name: gitea-ssh
@@ -0,0 +1,24 @@
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: gitea-allow-host-nginx-keycloak
namespace: gitea
labels:
app.kubernetes.io/name: gitea
app.kubernetes.io/instance: gitea
app.kubernetes.io/part-of: platform
spec:
podSelector:
matchLabels:
app.kubernetes.io/name: gitea
app.kubernetes.io/instance: gitea
policyTypes:
- Egress
egress:
- to:
- ipBlock:
# CoreDNS split DNS resolves the public issuer name to Host Nginx.
cidr: 192.168.0.107/32
ports:
- protocol: TCP
port: 443
@@ -0,0 +1,6 @@
$patch: delete
apiVersion: v1
kind: Service
metadata:
name: gitea-ssh
namespace: gitea
@@ -0,0 +1,7 @@
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: gitea
namespace: gitea
spec:
jobLabel: app.kubernetes.io/name
@@ -0,0 +1,189 @@
namespace: gitea
replicaCount: 1
image:
registry: docker.gitea.com
repository: gitea
tag: "1.27.0"
pullPolicy: IfNotPresent
rootless: true
# The official chart applies this context to its built-in init containers,
# configure-gitea, and the main container.
containerSecurityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
runAsNonRoot: true
seccompProfile:
type: RuntimeDefault
# A Local RWO volume cannot be attached to an old and a replacement Pod at the
# same time. Terminate the old Pod before creating its replacement.
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 0
maxUnavailable: 1
service:
http:
type: ClusterIP
port: 3000
clusterIP: ""
nodePort: null
ssh:
type: ClusterIP
port: 22
clusterIP: ""
nodePort: null
hostPort: null
# Routing is owned by ../../base/ingress.yaml. Host Nginx terminates TLS before
# forwarding requests to Traefik's HTTP entrypoint.
ingress:
enabled: false
route:
enabled: false
gatewayAPI:
enabled: false
persistence:
enabled: true
create: false
mount: true
claimName: gitea-data
gitea:
admin:
existingSecret: gitea-admin
email: admin@hyeonworks.com
passwordMode: initialOnlyRequireReset
config:
server:
PROTOCOL: http
DOMAIN: git.learn.hyeonworks.com
ROOT_URL: https://git.learn.hyeonworks.com/
HTTP_PORT: 3000
DISABLE_SSH: true
START_SSH_SERVER: false
LFS_START_SERVER: false
# Baseline allows only the pre-provisioned local emergency administrator.
# The guarded OIDC profile overrides the first two settings.
service:
DISABLE_REGISTRATION: true
ALLOW_ONLY_EXTERNAL_REGISTRATION: false
SHOW_REGISTRATION_BUTTON: false
ENABLE_PASSWORD_SIGNIN_FORM: true
database:
DB_TYPE: postgres
HOST: platform-postgres-rw.platform-data.svc.cluster.local:5432
NAME: gitea
SSL_MODE: disable
session:
PROVIDER: db
cache:
ADAPTER: memory
queue:
TYPE: level
actions:
ENABLED: false
packages:
ENABLED: false
repository:
DISABLED_REPO_UNITS: "repo.packages,repo.actions"
additionalConfigFromEnvs:
- name: GITEA__DATABASE__USER
valueFrom:
secretKeyRef:
name: gitea-db-credentials
key: username
- name: GITEA__DATABASE__PASSWD
valueFrom:
secretKeyRef:
name: gitea-db-credentials
key: password
metrics:
enabled: true
serviceMonitor:
enabled: true
additionalLabels:
observability.hyeonworks.com/instance: home
interval: 30s
scrapeTimeout: 10s
livenessProbe:
enabled: true
httpGet:
path: /api/healthz
port: http
initialDelaySeconds: 200
timeoutSeconds: 5
periodSeconds: 10
successThreshold: 1
failureThreshold: 10
readinessProbe:
enabled: true
httpGet:
path: /api/healthz
port: http
initialDelaySeconds: 5
timeoutSeconds: 5
periodSeconds: 10
successThreshold: 1
failureThreshold: 3
startupProbe:
enabled: true
httpGet:
path: /api/healthz
port: http
initialDelaySeconds: 10
timeoutSeconds: 5
periodSeconds: 10
successThreshold: 1
failureThreshold: 60
postgresql:
enabled: false
postgresql-ha:
enabled: false
valkey:
enabled: false
valkey-cluster:
enabled: false
resources:
requests:
cpu: 250m
memory: 512Mi
limits:
cpu: "2"
memory: 2Gi
initContainers:
resources:
requests:
cpu: 100m
memory: 128Mi
limits:
cpu: 500m
memory: 512Mi
test:
enabled: false
@@ -0,0 +1,49 @@
extraVolumes:
- name: branding-assets
configMap:
name: gitea-branding-assets
defaultMode: 292
items:
- key: hyeonworks.css
path: css/hyeonworks.css
- key: logo.svg
path: img/logo.svg
- key: favicon.svg
path: img/favicon.svg
- name: branding-templates
configMap:
name: gitea-branding-templates
defaultMode: 292
items:
- key: header.tmpl
path: header.tmpl
- key: extra_links.tmpl
path: extra_links.tmpl
extraContainerVolumeMounts:
- name: branding-assets
mountPath: /data/gitea/public/assets
readOnly: true
- name: branding-templates
mountPath: /data/gitea/templates/custom
readOnly: true
gitea:
oauth:
- name: keycloak
provider: openidConnect
existingSecret: gitea-keycloak-oidc
autoDiscoverUrl: https://id.learn.hyeonworks.com/realms/hyeonworks/.well-known/openid-configuration
config:
APP_NAME: Hyeonworks Git
service:
DISABLE_REGISTRATION: false
ALLOW_ONLY_EXTERNAL_REGISTRATION: true
oauth2_client:
ENABLE_AUTO_REGISTRATION: true
USERNAME: preferred_username
ACCOUNT_LINKING: login
OPENID_CONNECT_SCOPES: "profile email"
+111
View File
@@ -0,0 +1,111 @@
# Keycloak 서비스
이 Kustomization은 `keycloak` namespace에 Keycloak 26.7.0 인스턴스 하나를
선언한다. 이 인스턴스는 독립적인 OIDC Provider이며 인증 proxy나 Traefik
ForwardAuth middleware가 아니다.
외부 요청 경로는 다음과 같다.
```text
Client -> Host Nginx (TLS) -> Traefik web/HTTP -> keycloak-service:8080
```
상태(2026-07-23): Host Nginx의 Keycloak 공개 전환과 Gitea OIDC·브랜딩 rollout을
완료했다. Nginx 후보 SHA-256은
`5c5cd74b4992f537fd27c50cf2209573a80a9904e0b19b58c3154717be6ff4a5`, 백업은
`/etc/nginx/sites-available/learn-services.before-keycloak-20260723160519`이며
discovery issuer와 Gitea OAuth source·정책·redirect·브랜딩 자동 검증을 통과했다.
실제 realm 사용자의 브라우저 login/callback/logout은 별도 수용 시험으로 남아 있다.
Keycloak이 생성하는 Ingress는 비활성화한다. 저장소에서 관리하는 Ingress는
Traefik의 내부 HTTP entrypoint를 통해 `id.learn.hyeonworks.com`만 노출한다. 관리
포트 `9000`에는 Ingress나 NodePort가 없으며, NetworkPolicy의 출발지는 namespace
범위 Operator로 제한한다. TLS는 Host Nginx에서만 종료하므로 Keycloak은 Traefik이
선택한 HTTP 경로의 `xforwarded` header만 수락한다.
## 필수 Secret 계약
credential Secret 매니페스트나 값은 이 저장소에 보관하지 않는다. Keycloak
리소스를 생성하기 전에 동일하게 생성한 비밀번호를 사용하는
`keycloak-db-credentials`를 다음 두 namespace에 생성한다.
- `platform-data`: CloudNativePG의 `DatabaseRole`에서 사용
- `keycloak`: Keycloak 서버에서 사용
각 Secret의 유형은 `kubernetes.io/basic-auth`, username은 정확히 `keycloak`이어야
하며 `username``password` 키가 있어야 한다. `platform-data` 사본에는
`cnpg.io/reload: "true"` label도 있어야 한다.
인스턴스를 처음 조정할 때 Operator가 `keycloak` namespace에
`keycloak-initial-admin`을 생성한다. 이 생성된 Secret은 소스 관리하지 않는다.
서비스를 운영 준비 완료 상태로 판단하기 전에 bootstrap credential을 교체하고
MFA를 활성화한다.
## 의존성과 렌더링 순서
1. `keycloak`, `platform-data`, `cnpg-system` namespace가 존재한다.
2. CloudNativePG와 `platform-postgres`가 Ready 상태다.
3. `keycloak-db-credentials`의 두 사본이 모두 존재한다.
4. Keycloak Operator가 설치되어 Ready 상태다.
5. 이 서비스 Kustomization을 적용한다.
6. Ready 상태와 내부 Traefik routing을 확인한 후에만 Host Nginx에
`id.learn.hyeonworks.com`을 설정한다.
Helm 없이 렌더링한다.
```bash
kubectl kustomize services/keycloak
```
배포 후 다음을 검증한다.
```bash
kubectl -n keycloak wait --for=condition=Ready \
keycloak/keycloak --timeout=15m
kubectl -n keycloak get keycloak,pod,service,ingress,networkpolicy
curl --fail --silent --show-error \
https://id.learn.hyeonworks.com/realms/hyeonworks/.well-known/openid-configuration \
| jq --exit-status \
--arg issuer 'https://id.learn.hyeonworks.com/realms/hyeonworks' \
'.issuer == $issuer'
```
마지막 검사는 HTTP 성공 여부만 보지 않고 JSON의 `issuer`가 공개 issuer와 정확히
같은지 확인한다. 따라서 전환 전 Host Nginx의 정적 hold 응답이 HTTP 200을
반환하더라도 성공으로 오인하지 않는다.
TCP 9000을 대상으로 하는 Ingress, NodePort 또는 LoadBalancer가 없는지도 별도로
확인한다.
## 수동 bootstrap 진입점
Keycloak-only 적용은 AIStor와 분리한다. 저장소 루트에서 다음 순서를 사용하며
Secret payload는 명령 인자나 출력에 넣지 않는다.
```sh
kubectl apply --filename=infrastructure/namespaces/phase2/keycloak.yaml
bash scripts/bootstrap/create-keycloak-secrets.sh --generate --execute
bash scripts/bootstrap/apply-keycloak.sh --execute
bash scripts/bootstrap/configure-keycloak-gitea-oidc.sh --execute
```
각 스크립트는 context와 적용 범위를 다시 확인한다. 마지막 스크립트는
`hyeonworks` realm, confidential Gitea client와 Gitea OIDC Secret을 구성한다.
public discovery 전환은 클러스터 적용과 분리된 root 작업이며 실제 전환을
완료했다.
```sh
sudo bash scripts/bootstrap/apply-host-nginx-keycloak.sh --execute
```
상세 명령·출력·실패 경계는
[중앙 OIDC 실행 원장](../../../docs/platform/runbooks/2026-07-23-keycloak-gitea-oidc-cutover.md)에
기록한다.
## 공식 참고 문서
- <https://www.keycloak.org/operator/basic-deployment>
- <https://www.keycloak.org/operator/advanced-configuration>
- <https://www.keycloak.org/server/reverseproxy>
- <https://www.keycloak.org/server/hostname>
+23
View File
@@ -0,0 +1,23 @@
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: keycloak-http
labels:
app.kubernetes.io/name: keycloak
app.kubernetes.io/instance: keycloak
app.kubernetes.io/component: identity-provider
annotations:
traefik.ingress.kubernetes.io/router.entrypoints: web
spec:
ingressClassName: traefik
rules:
- host: id.learn.hyeonworks.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: keycloak-service
port:
number: 8080
+94
View File
@@ -0,0 +1,94 @@
apiVersion: k8s.keycloak.org/v2beta1
kind: Keycloak
metadata:
name: keycloak
labels:
app.kubernetes.io/name: keycloak
app.kubernetes.io/instance: keycloak
app.kubernetes.io/component: identity-provider
spec:
instances: 1
db:
vendor: postgres
host: platform-postgres-rw.platform-data.svc.cluster.local
port: 5432
database: keycloak
schema: public
usernameSecret:
name: keycloak-db-credentials
key: username
passwordSecret:
name: keycloak-db-credentials
key: password
poolInitialSize: 5
poolMinSize: 5
poolMaxSize: 20
hostname:
hostname: https://id.learn.hyeonworks.com
strict: true
http:
httpEnabled: true
httpPort: 8080
serviceHttpPort: 8080
proxy:
headers: xforwarded
ingress:
enabled: false
additionalOptions:
- name: metrics-enabled
value: "true"
serviceMonitor:
enabled: true
interval: 30s
scrapeTimeout: 10s
labels:
observability.hyeonworks.com/instance: home
networkPolicy:
enabled: true
http:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
podSelector:
matchLabels:
app.kubernetes.io/name: traefik
https:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
podSelector:
matchLabels:
app.kubernetes.io/name: traefik
management:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: keycloak
podSelector:
matchLabels:
app.kubernetes.io/name: keycloak-operator
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: observability
podSelector:
matchLabels:
app.kubernetes.io/name: prometheus
app.kubernetes.io/instance: observability-core-kube-pr-prometheus
resources:
requests:
cpu: 500m
memory: 1Gi
limits:
cpu: "2"
memory: 2Gi
transaction:
xaEnabled: false
@@ -0,0 +1,6 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- keycloak.yaml
- ingress.yaml
+13
View File
@@ -0,0 +1,13 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: keycloak
resources:
- base
labels:
- pairs:
app.kubernetes.io/part-of: platform
app.kubernetes.io/managed-by: kustomize
includeSelectors: false
@@ -0,0 +1,6 @@
# 홈 환경 Overlay
현재 홈 클러스터 구성은 `../base`의 안정적인 리소스로 직접 표현한다.
`services/keycloak`을 유일한 서비스 빌드 루트로 유지한다. 향후 홈 환경 전용
Patch는 이곳에 추가하고, 독립적으로 렌더링하는 두 번째 애플리케이션을 만들지 말고
서비스 루트 Kustomization에서 참조한다.
+81
View File
@@ -0,0 +1,81 @@
# MinIO AIStor ObjectStore
상태(2026-07-23):
- baseline ObjectStore와 인증 S3 스모크: 실제 적용 완료
- admin-oidc profile과 비공개 Console Ingress: 선언형 구현·dry-run 완료, live 적용 대기
공식 Chart를 수정하지 않고 Kustomize가 Helm을 렌더한다.
## 빌드 루트
- services/minio-aistor: 현재 live baseline 진입점
- services/minio-aistor/base/deployment: 공통 Helm baseline과 values
- services/minio-aistor/profiles/admin-oidc: baseline에 OIDC, Console Ingress와
추가 NetworkPolicy를 결합한 진입점
root와 profile이 같은 형제 baseline을 참조하므로 profile이 조상 root를 다시
포함하는 Kustomize 순환이 없다.
## 고정 원본
| 항목 | 값 |
|---|---|
| Chart 저장소 | https://helm.min.io/ |
| Chart | aistor-objectstore |
| 버전 | 1.0.16 |
| archive SHA-256 | 50ffa6a4e014cdc48566b237593baab41d039f103cdc50c1b1ac173b8d8bf71e |
검증기가 만드는 임시 Chart cache는
base/deployment/.helm/charts/aistor-objectstore-1.0.16 아래에만 존재하고 종료 시
제거한다.
## baseline 계약
- namespace: object-storage
- ObjectStore: minio-aistor
- pool/server/volume: 각각 1
- 저장소: aistor-local-xfs-retain, 900Gi RWO
- S3와 Console Service: ClusterIP
- baseline Ingress, NodePort, LoadBalancer: 없음
- pvcProtection과 Retain Local PV
- 기존 root 계정: 비상용으로 유지
Standalone이므로 erasure-code 이중화와 노드 failover는 없다.
## Secret 계약
baseline:
- object-storage/aistor-root-configuration: config.env
admin-oidc profile:
- object-storage/aistor-keycloak-oidc: client-id, client-secret
평문, Base64, checksum을 Git과 문서에 기록하지 않는다.
## 렌더와 적용
baseline 정적 렌더:
kubectl kustomize --enable-helm services/minio-aistor
admin profile 전체 정적 검증:
PLATFORM_HELM_BIN=/home/donghyeon/.local/bin/helm \
bash scripts/validate/render-admin-services.sh
admin profile 실제 적용은 private DNS, admin 인증서와 Keycloak client/group 준비
뒤 apply-admin-services.sh가 수행한다. S3 API에는 Ingress를 만들지 않는다.
현재 baseline은 Initialized/green이고 data0-minio-aistor-pool-0-0 PVC가
aistor-data-local-pv에 Bound다. S3 회귀 검사는 다음과 같다.
bash scripts/validate/aistor-s3-smoke.sh --execute
## 공식 문서
- https://docs.min.io/aistor/reference/kubernetes/object-store-helm-chart/
- https://docs.min.io/aistor/reference/kubernetes/pvc-protection/
- https://docs.min.io/aistor/administration/iam/identity/keycloak-identity/
+7
View File
@@ -0,0 +1,7 @@
# 기본 확장 지점
공통 ObjectStore Helm baseline은 base/deployment에 있다. 저장소, resource,
ClusterIP와 root 비상 계정 경계만 이 계층이 소유한다.
OIDC 환경 변수와 Console Ingress는 profiles/admin-oidc에서만 추가한다.
Chart 템플릿을 저장소에 복제하거나 직접 수정하지 않는다.
@@ -0,0 +1,6 @@
# AIStor ObjectStore baseline
AIStor ObjectStore의 공통 Helm baseline입니다. 저장소, root 비상 계정,
리소스 제한과 ClusterIP 경계만 정의합니다.
Console Ingress와 Keycloak OIDC는 profiles/admin-oidc에서만 추가합니다.
@@ -0,0 +1,17 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: object-storage
helmGlobals:
chartHome: .helm/charts
helmCharts:
- name: aistor-objectstore
repo: https://helm.min.io/
version: 1.0.16
releaseName: minio-aistor
namespace: object-storage
valuesFile: values/home.yaml
includeCRDs: false
skipTests: true
@@ -0,0 +1,67 @@
secrets:
name: aistor-root-configuration
existingSecret: true
disableRoot: false
objectStore:
name: minio-aistor
configuration:
name: aistor-root-configuration
pools:
- name: pool-0
servers: 1
volumesPerServer: 1
storageNamePrefix: data
size: 900Gi
storageClassName: aistor-local-xfs-retain
nodeSelector:
kubernetes.io/hostname: donghyeon-system-product-name
resources:
requests:
cpu: 500m
memory: 2Gi
limits:
cpu: "3"
memory: 8Gi
securityContext:
runAsUser: 1000
runAsGroup: 1000
runAsNonRoot: true
fsGroup: 1000
fsGroupChangePolicy: OnRootMismatch
containerSecurityContext:
runAsUser: 1000
runAsGroup: 1000
runAsNonRoot: true
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
seccompProfile:
type: RuntimeDefault
mountPath: /export
subPath: /data
certificates:
disableAutoCert: true
services:
minio:
serviceType: ClusterIP
console:
serviceType: ClusterIP
pvcProtection: true
prometheusOperator: false
sideCars:
resources:
requests:
cpu: 100m
memory: 128Mi
limits:
cpu: 500m
memory: 512Mi
+5
View File
@@ -0,0 +1,5 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- base/deployment
@@ -0,0 +1,8 @@
# 홈 환경 Overlay 예약 지점
현재 홈 환경의 공통 Helm values는
../../base/deployment/values/home.yaml에 있다. 실제 배포 진입점은 저장소 root와
profiles/admin-oidc다.
이 디렉터리는 이후 홈 환경 전용 patch가 필요할 때만 사용하며 현재 활성
Kustomization 리소스는 없다.
@@ -0,0 +1,16 @@
# AIStor Console OIDC 프로필
기존 ObjectStore baseline은 그대로 두고 다음 항목만 추가합니다.
- storage-admin.learn.hyeonworks.com Console Ingress
- Keycloak 표준 OIDC 환경 변수와 policy claim
- Traefik의 Console 접근과 Keycloak HTTPS egress
S3 API Service에는 Ingress를 만들지 않습니다. aistor-keycloak-oidc
Secret의 client-id, client-secret 값은 적용 스크립트가 만들며 Git에
저장하지 않습니다. 기존 root 계정은 비상용으로 유지합니다.
`MINIO_IDENTITY_OPENID_VENDOR=keycloak`은 사용하지 않습니다. 이 옵션은
AIStor가 Keycloak Admin API로 사용자를 추가 검증할 때 admin URL, realm과
service-account 권한을 요구합니다. 이 프로필은 Keycloak이 발급한 `policy`
claim을 표준 OIDC로 소비하므로 불필요한 Admin API 권한을 부여하지 않습니다.
@@ -0,0 +1,24 @@
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: minio-aistor-console
namespace: object-storage
labels:
app.kubernetes.io/name: minio-aistor
app.kubernetes.io/component: admin-console
app.kubernetes.io/part-of: platform
annotations:
traefik.ingress.kubernetes.io/router.entrypoints: web
spec:
ingressClassName: traefik
rules:
- host: storage-admin.learn.hyeonworks.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: minio-aistor-console
port:
number: 9090
@@ -0,0 +1,15 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../base/deployment
- console-ingress.yaml
- network-policies.yaml
patches:
- path: objectstore-oidc-patch.yaml
target:
group: aistor.min.io
version: v1
kind: ObjectStore
name: minio-aistor
@@ -0,0 +1,101 @@
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: object-storage-allow-traefik-console
namespace: object-storage
labels:
app.kubernetes.io/part-of: platform
app.kubernetes.io/component: admin-console
spec:
podSelector:
matchLabels:
aistor.min.io/objectStore: minio-aistor
policyTypes:
- Ingress
ingress:
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
podSelector:
matchLabels:
app.kubernetes.io/name: traefik
ports:
- protocol: TCP
port: 9090
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: object-storage-allow-host-nginx-keycloak
namespace: object-storage
labels:
app.kubernetes.io/part-of: platform
app.kubernetes.io/component: oidc-client
spec:
podSelector:
matchLabels:
aistor.min.io/objectStore: minio-aistor
policyTypes:
- Egress
egress:
- to:
- ipBlock:
cidr: 192.168.0.107/32
ports:
- protocol: TCP
port: 443
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: object-storage-allow-blackbox-console-health
namespace: object-storage
labels:
app.kubernetes.io/part-of: platform
app.kubernetes.io/component: admin-console
spec:
podSelector:
matchLabels:
aistor.min.io/objectStore: minio-aistor
policyTypes:
- Ingress
ingress:
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: observability
podSelector:
matchLabels:
app.kubernetes.io/name: prometheus-blackbox-exporter
app.kubernetes.io/instance: blackbox-exporter
ports:
- protocol: TCP
port: 9090
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: object-storage-allow-prometheus-metrics
namespace: object-storage
labels:
app.kubernetes.io/part-of: platform
app.kubernetes.io/component: object-storage-metrics
spec:
podSelector:
matchLabels:
aistor.min.io/objectStore: minio-aistor
policyTypes:
- Ingress
ingress:
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: observability
podSelector:
matchLabels:
app.kubernetes.io/name: prometheus
app.kubernetes.io/instance: observability-core-kube-pr-prometheus
ports:
- protocol: TCP
port: 9000
@@ -0,0 +1,32 @@
apiVersion: aistor.min.io/v1
kind: ObjectStore
metadata:
name: minio-aistor
spec:
env:
- name: MINIO_PROMETHEUS_AUTH_TYPE
value: public
- name: MINIO_BROWSER_REDIRECT_URL
value: https://storage-admin.learn.hyeonworks.com
- name: MINIO_IDENTITY_OPENID_CONFIG_URL
value: https://id.learn.hyeonworks.com/realms/hyeonworks/.well-known/openid-configuration
- name: MINIO_IDENTITY_OPENID_CLIENT_ID
valueFrom:
secretKeyRef:
name: aistor-keycloak-oidc
key: client-id
- name: MINIO_IDENTITY_OPENID_CLIENT_SECRET
valueFrom:
secretKeyRef:
name: aistor-keycloak-oidc
key: client-secret
- name: MINIO_IDENTITY_OPENID_SCOPES
value: openid,profile,email
- name: MINIO_IDENTITY_OPENID_DISPLAY_NAME
value: Hyeonworks ID
- name: MINIO_IDENTITY_OPENID_CLAIM_NAME
value: policy
- name: MINIO_IDENTITY_OPENID_REDIRECT_URI_DYNAMIC
value: "on"
- name: MINIO_BROWSER_SESSION_DURATION
value: 8h
+67
View File
@@ -0,0 +1,67 @@
secrets:
name: aistor-root-configuration
existingSecret: true
disableRoot: false
objectStore:
name: minio-aistor
configuration:
name: aistor-root-configuration
pools:
- name: pool-0
servers: 1
volumesPerServer: 1
storageNamePrefix: data
size: 900Gi
storageClassName: aistor-local-xfs-retain
nodeSelector:
kubernetes.io/hostname: donghyeon-system-product-name
resources:
requests:
cpu: 500m
memory: 2Gi
limits:
cpu: "3"
memory: 8Gi
securityContext:
runAsUser: 1000
runAsGroup: 1000
runAsNonRoot: true
fsGroup: 1000
fsGroupChangePolicy: OnRootMismatch
containerSecurityContext:
runAsUser: 1000
runAsGroup: 1000
runAsNonRoot: true
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
seccompProfile:
type: RuntimeDefault
mountPath: /export
subPath: /data
certificates:
disableAutoCert: true
services:
minio:
serviceType: ClusterIP
console:
serviceType: ClusterIP
pvcProtection: true
prometheusOperator: false
sideCars:
resources:
requests:
cpu: 100m
memory: 128Mi
limits:
cpu: 500m
memory: 512Mi
+18
View File
@@ -0,0 +1,18 @@
# Observability core
This root composes the internal observability workloads deployed after the
namespace, CRD, and Local PV prerequisites:
- kube-prometheus-stack without Grafana or its bundled node-exporter;
- single-binary Loki and Tempo using isolated AIStor buckets;
- Alloy for Pod logs and internal OTLP/gRPC traces;
- a standalone node-exporter;
- the exact core NetworkPolicy matrix, platform rules, scrape targets, verified
dashboards, and source-only Alertmanager Slack routing.
It intentionally does not create public ingress, Grafana, OIDC, or credential
Secrets. The Slack routing source references
`alertmanager-slack-webhook/url`; while that Secret is absent, do not apply the
AlertmanagerConfig, its kube-prometheus-stack reference, its public HTTPS
NetworkPolicy, or alert rules. Namespace, CRD, and Local PV roots remain outside
this aggregate because their protected apply order is different.
@@ -0,0 +1,24 @@
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: observability-allow-alertmanager-public-https
namespace: observability
spec:
podSelector:
matchLabels:
app.kubernetes.io/name: alertmanager
app.kubernetes.io/instance: observability-core-kube-pr-alertmanager
policyTypes:
- Egress
egress:
- to:
- ipBlock:
cidr: 0.0.0.0/0
except:
- 10.0.0.0/8
- 100.64.0.0/10
- 172.16.0.0/12
- 192.168.0.0/16
ports:
- protocol: TCP
port: 443
@@ -0,0 +1,8 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: observability
resources:
- platform-alertmanager.yaml
- alertmanager-public-https-egress.yaml
@@ -0,0 +1,80 @@
apiVersion: monitoring.coreos.com/v1alpha1
kind: AlertmanagerConfig
metadata:
name: platform-alertmanager
namespace: observability
labels:
observability.hyeonworks.com/instance: home
spec:
route:
receiver: platform-slack
groupBy:
- cluster
- namespace
- alertname
- severity
groupWait: 30s
groupInterval: 5m
repeatInterval: 4h
routes:
- receiver: platform-null
matchers:
- name: alertname
matchType: "="
value: InfoInhibitor
inhibitRules:
- sourceMatch:
- name: alertname
matchType: "="
value: InfoInhibitor
targetMatch:
- name: severity
matchType: "="
value: info
equal:
- namespace
receivers:
- name: platform-null
- name: platform-slack
slackConfigs:
- apiURL:
name: alertmanager-slack-webhook
key: url
sendResolved: true
linkNames: false
mrkdwnIn:
- text
- fields
fallback: >-
{{ if eq .Status "firing" }}FIRING{{ else }}RESOLVED{{ end }}: {{ if .CommonLabels.severity }}{{ .CommonLabels.severity | toUpper }}{{ else }}UNKNOWN{{ end }} · {{ .CommonLabels.alertname }}
title: >-
{{ if eq .Status "firing" }}[FIRING:{{ .Alerts.Firing | len }}]{{ else }}[RESOLVED]{{ end }} {{ if .CommonLabels.severity }}{{ .CommonLabels.severity | toUpper }}{{ else }}UNKNOWN{{ end }} · {{ .CommonLabels.alertname }}
titleLink: https://grafana.learn.hyeonworks.com/
color: >-
{{ if eq .Status "resolved" }}good{{ else if or (eq .CommonLabels.severity "emergency") (eq .CommonLabels.severity "critical") }}danger{{ else if eq .CommonLabels.severity "warning" }}warning{{ else }}#439FE0{{ end }}
fields:
- title: Status
value: '{{ .Status | toUpper }}'
short: true
- title: Severity
value: '{{ if .CommonLabels.severity }}{{ .CommonLabels.severity | toUpper }}{{ else }}UNKNOWN{{ end }}'
short: true
- title: Location
value: '{{ if .CommonLabels.cluster }}{{ .CommonLabels.cluster }}{{ else }}unknown-cluster{{ end }} / {{ if .CommonLabels.namespace }}{{ .CommonLabels.namespace }}{{ else }}cluster-scoped{{ end }}'
short: true
- title: Alert count
value: '{{ len .Alerts }}'
short: true
text: |-
{{ range .Alerts }}
*Alert status:* {{ .Status | toUpper }}
*Target:* {{ $target := .Labels.Remove $.GroupLabels.Names }}{{ if $target }}{{ range $target.SortedPairs }}{{ .Name }}={{ .Value }} {{ end }}{{ else }}unknown{{ end }}
*Summary:* {{ with .Annotations.summary }}{{ . }}{{ else }}No summary provided{{ end }}
*Details:* {{ with .Annotations.description }}{{ . }}{{ else }}No description provided{{ end }}
*Started:* {{ .StartsAt.Format "2006-01-02T15:04:05Z07:00" }}
{{ if eq .Status "resolved" }}*Ended:* {{ .EndsAt.Format "2006-01-02T15:04:05Z07:00" }}{{ end }}
{{ with .Annotations.runbook_url }}*Runbook:* <{{ . }}|대응 절차 열기>{{ end }}
{{ end }}
<https://grafana.learn.hyeonworks.com/|Grafana> · <https://grafana.learn.hyeonworks.com/explore|Explore>
footer: hyeonworks observability · Alertmanager
+12
View File
@@ -0,0 +1,12 @@
# Alloy
Grafana Alloy runs as a single-node DaemonSet in `observability-agent`.
It tails only `/var/log/pods`, writes logs to the in-cluster Loki service, and
accepts OTLP/gRPC traces on port `4317` for the in-cluster Tempo service.
The deployment deliberately disables host namespaces, automatic service
account token mounts, CRD creation, the config reloader, public exposure, and
persistent OpenTelemetry queues. The 2 GiB PVC stores only Alloy file
positions and local state. Alloy keeps primary UID/GID `473:473`; supplemental
group `0` permits read-only traversal of k3s' `root:root 0750` Pod log tree
without changing host permissions or running the container as root.
+214
View File
@@ -0,0 +1,214 @@
discovery.kubernetes "pods" {
role = "pod"
selectors {
role = "pod"
field = "spec.nodeName=" + sys.env("HOSTNAME")
}
}
discovery.relabel "pod_logs" {
targets = discovery.kubernetes.pods.targets
rule {
source_labels = ["__meta_kubernetes_namespace"]
target_label = "namespace"
}
rule {
source_labels = ["__meta_kubernetes_pod_name"]
target_label = "pod"
}
rule {
source_labels = ["__meta_kubernetes_pod_container_name"]
target_label = "container"
}
rule {
source_labels = ["__meta_kubernetes_pod_label_app_kubernetes_io_name"]
regex = "(.+)"
target_label = "app"
}
rule {
source_labels = [
"__meta_kubernetes_pod_label_app_kubernetes_io_name",
"__meta_kubernetes_pod_label_app",
]
separator = ";"
regex = "^;(.+)$"
replacement = "$1"
target_label = "app"
}
rule {
source_labels = [
"__meta_kubernetes_pod_uid",
"__meta_kubernetes_pod_container_name",
]
separator = "/"
replacement = "/var/log/pods/*$1/*.log"
target_label = "__path__"
}
rule {
source_labels = ["__meta_kubernetes_pod_container_id"]
regex = `^(\S+):\/\/.+$`
replacement = "$1"
target_label = "container_runtime"
}
}
local.file_match "pod_logs" {
path_targets = discovery.relabel.pod_logs.output
}
loki.source.file "pod_logs" {
targets = local.file_match.pod_logs.targets
forward_to = [loki.process.pod_logs.receiver]
}
loki.process "pod_logs" {
stage.match {
selector = `{container_runtime=~"containerd|cri-o"}`
stage.cri {}
}
stage.match {
selector = `{container_runtime="docker"}`
stage.docker {}
}
stage.replace {
expression = `(?i)"(?:authorization|bearer|jwt|password|passwd|client[_-]?secret)"\s*:\s*"([^"\\]*(?:\\.[^"\\]*)*)"`
replace = "[REDACTED]"
}
stage.replace {
expression = `(?i)(?:authorization\s*[:=]\s*)(?:bearer\s+)?([A-Za-z0-9._~+/=-]+)`
replace = "[REDACTED]"
}
stage.replace {
expression = `(?i)\bbearer\s+([A-Za-z0-9_-]+(?:\.[A-Za-z0-9_-]+){0,2})`
replace = "[REDACTED]"
}
stage.replace {
expression = `(?i)(?:password|passwd|client[_-]?secret)\s*[:=]\s*([^\s"'{},]+)`
replace = "[REDACTED]"
}
stage.json {
expressions = {
level = "level",
}
}
stage.labels {
values = {
level = "",
}
}
stage.match {
selector = `{level!~"(?i)^(trace|debug|info|warn|warning|error|fatal|panic)$"}`
stage.label_drop {
values = ["level"]
}
}
stage.static_labels {
values = {
cluster = "home",
}
}
stage.label_keep {
values = ["cluster", "namespace", "app", "pod", "container", "level"]
}
forward_to = [loki.write.local.receiver]
}
loki.write "local" {
endpoint {
url = "http://loki.observability.svc.cluster.local:3100/loki/api/v1/push"
batch_size = "256KiB"
batch_wait = "1s"
remote_timeout = "10s"
min_backoff_period = "500ms"
max_backoff_period = "30s"
max_backoff_retries = 10
}
}
otelcol.receiver.otlp "traces" {
grpc {
endpoint = "0.0.0.0:4317"
}
output {
traces = [otelcol.processor.k8sattributes.traces.input]
}
}
otelcol.processor.k8sattributes "traces" {
auth_type = "serviceAccount"
extract {
deployment_name_from_replicaset = false
metadata = [
"k8s.namespace.name",
"k8s.deployment.name",
"k8s.node.name",
"k8s.pod.name",
"k8s.pod.uid",
"k8s.pod.start_time",
"k8s.container.name",
]
}
pod_association {
source {
from = "resource_attribute"
name = "k8s.pod.uid"
}
}
pod_association {
source {
from = "resource_attribute"
name = "k8s.pod.ip"
}
}
pod_association {
source {
from = "connection"
}
}
output {
traces = [otelcol.processor.batch.traces.input]
}
}
otelcol.processor.batch "traces" {
timeout = "1s"
send_batch_size = 512
send_batch_max_size = 1024
output {
traces = [otelcol.exporter.otlp.tempo.input]
}
}
otelcol.exporter.otlp "tempo" {
client {
endpoint = "tempo.observability.svc.cluster.local:4317"
tls {
insecure = true
}
}
retry_on_failure {
enabled = true
initial_interval = "1s"
max_interval = "15s"
max_elapsed_time = "5m"
}
sending_queue {
enabled = true
queue_size = 256
num_consumers = 2
sizer = "requests"
block_on_overflow = true
}
}
@@ -0,0 +1,51 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: observability-agent
resources:
- pvc.yaml
configMapGenerator:
- name: alloy-config
files:
- config.alloy=config.alloy
helmGlobals:
chartHome: .helm/charts
helmCharts:
- name: alloy
repo: https://grafana.github.io/helm-charts
version: 1.11.0
releaseName: alloy
namespace: observability-agent
valuesFile: values/home.yaml
includeCRDs: false
skipTests: true
images:
- name: docker.io/grafana/alloy
newName: docker.io/grafana/alloy
digest: sha256:eb21f4c0858edffcdd1b385910ddeef26f692fc2c282f61baa724fc09d274a17
patches:
- target:
group: apps
version: v1
kind: DaemonSet
name: alloy
patch: |-
- op: add
path: /spec/template/spec/automountServiceAccountToken
value: false
- op: add
path: /spec/template/spec/hostNetwork
value: false
- op: add
path: /spec/template/spec/hostPID
value: false
- op: add
path: /spec/template/spec/hostIPC
value: false
+17
View File
@@ -0,0 +1,17 @@
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: alloy-storage
namespace: observability-agent
labels:
app.kubernetes.io/name: alloy
observability.hyeonworks.com/instance: home
spec:
accessModes:
- ReadWriteOnce
storageClassName: ssd-local-observability-alloy-retain
volumeName: observability-alloy-local-pv
resources:
requests:
storage: 2Gi
@@ -0,0 +1,155 @@
crds:
create: false
global:
podSecurityContext:
runAsNonRoot: true
runAsUser: 473
runAsGroup: 473
fsGroup: 473
fsGroupChangePolicy: OnRootMismatch
# k3s keeps /var/log/pods root:root 0750. Retain Alloy's non-root primary
# identity while granting read-only traversal of that single hostPath.
supplementalGroups:
- 0
seccompProfile:
type: RuntimeDefault
alloy:
configMap:
create: false
name: alloy-config
key: config.alloy
clustering:
enabled: false
stabilityLevel: generally-available
storagePath: /var/lib/alloy
enableHttpServerPort: true
listenAddr: 0.0.0.0
listenPort: 12345
enableReporting: false
extraPorts:
- name: otlp-grpc
port: 4317
targetPort: 4317
protocol: TCP
mounts:
varlog: false
dockercontainers: false
extra:
- name: pod-logs
mountPath: /var/log/pods
readOnly: true
- name: alloy-storage
mountPath: /var/lib/alloy
- name: tmp
mountPath: /tmp
- name: kubernetes-api-access
mountPath: /var/run/secrets/kubernetes.io/serviceaccount
readOnly: true
securityContext:
runAsNonRoot: true
runAsUser: 473
runAsGroup: 473
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop:
- ALL
seccompProfile:
type: RuntimeDefault
resources:
requests:
cpu: 200m
memory: 384Mi
limits:
cpu: "1"
memory: 768Mi
image:
registry: docker.io
repository: grafana/alloy
tag: v1.18.0
digest: sha256:eb21f4c0858edffcdd1b385910ddeef26f692fc2c282f61baa724fc09d274a17
pullPolicy: IfNotPresent
rbac:
create: true
namespaces: []
rules:
- apiGroups:
- ""
resources:
- pods
- namespaces
verbs:
- get
- list
- watch
clusterRules:
- apiGroups:
- apps
resources:
- replicasets
verbs:
- get
- list
- watch
serviceAccount:
create: true
name: alloy
automountServiceAccountToken: false
configReloader:
enabled: false
controller:
type: daemonset
hostNetwork: false
hostPID: false
volumes:
extra:
- name: pod-logs
hostPath:
path: /var/log/pods
type: Directory
- name: alloy-storage
persistentVolumeClaim:
claimName: alloy-storage
- name: tmp
emptyDir: {}
- name: kubernetes-api-access
projected:
defaultMode: 420
sources:
- serviceAccountToken:
expirationSeconds: 3600
path: token
- configMap:
name: kube-root-ca.crt
items:
- key: ca.crt
path: ca.crt
- downwardAPI:
items:
- path: namespace
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
networkPolicy:
enabled: false
service:
enabled: true
type: ClusterIP
serviceMonitor:
enabled: true
interval: 30s
additionalLabels:
observability.hyeonworks.com/instance: home
ingress:
enabled: false
@@ -0,0 +1,22 @@
# Blackbox exporter
This component deploys the pinned Prometheus blackbox exporter as an internal
`ClusterIP` service in the `observability` namespace. The `/probe` handler is
reachable only from the Prometheus Pod; it has no Ingress, NodePort, host port,
or service-account token.
Only three fixed `Probe` groups are declared:
- `platform-public-edge`: Gitea health and Keycloak discovery through Host Nginx
- `platform-private-edge`: Grafana, pgAdmin, and AIStor Console must return 403
- `platform-private-internal`: the same private services must return 200 through
their ClusterIP Services
The internal Grafana and pgAdmin targets use the blackbox exporter's `hostname`
parameter to set their required HTTP Host header. Target URLs and modules are
declarative; arbitrary callers are not granted access to port 9115.
The private-edge source boundary was proven separately and recorded as a
root-only `platform-blackbox-source-v1` evidence file. Applying this component
must remain gated on that evidence matching the active deny-only Nginx config.
@@ -0,0 +1,45 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: observability
resources:
- probes-public.yaml
- probes-private-edge.yaml
- probes-private-internal.yaml
- network-policy.yaml
helmGlobals:
chartHome: .helm/charts
helmCharts:
- name: prometheus-blackbox-exporter
repo: https://prometheus-community.github.io/helm-charts
version: 11.16.0
releaseName: blackbox-exporter
namespace: observability
valuesFile: values/home.yaml
includeCRDs: false
skipTests: true
patches:
- target:
version: v1
kind: ServiceAccount
name: blackbox-exporter
patch: |-
- op: add
path: /automountServiceAccountToken
value: false
- target:
group: apps
version: v1
kind: Deployment
name: blackbox-exporter
patch: |-
- op: add
path: /spec/template/spec/hostPID
value: false
- op: add
path: /spec/template/spec/hostIPC
value: false
@@ -0,0 +1,88 @@
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: observability-allow-prometheus-to-blackbox
namespace: observability
labels:
app.kubernetes.io/name: prometheus-blackbox-exporter
app.kubernetes.io/part-of: platform-observability
spec:
podSelector:
matchLabels:
app.kubernetes.io/name: prometheus-blackbox-exporter
app.kubernetes.io/instance: blackbox-exporter
policyTypes:
- Ingress
ingress:
- from:
- podSelector:
matchLabels:
app.kubernetes.io/name: prometheus
app.kubernetes.io/instance: observability-core-kube-pr-prometheus
ports:
- protocol: TCP
port: 9115
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: observability-allow-blackbox-egress
namespace: observability
labels:
app.kubernetes.io/name: prometheus-blackbox-exporter
app.kubernetes.io/part-of: platform-observability
spec:
podSelector:
matchLabels:
app.kubernetes.io/name: prometheus-blackbox-exporter
app.kubernetes.io/instance: blackbox-exporter
policyTypes:
- Egress
egress:
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
podSelector:
matchLabels:
k8s-app: kube-dns
ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53
- to:
- ipBlock:
cidr: 192.168.0.107/32
ports:
- protocol: TCP
port: 443
- to:
- podSelector:
matchLabels:
app.kubernetes.io/name: grafana
app.kubernetes.io/instance: grafana
ports:
- protocol: TCP
port: 3000
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: platform-admin
podSelector:
matchLabels:
app.kubernetes.io/name: pgadmin4
app.kubernetes.io/instance: pgadmin
ports:
- protocol: TCP
port: 5050
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: object-storage
podSelector:
matchLabels:
aistor.min.io/objectStore: minio-aistor
ports:
- protocol: TCP
port: 9090
@@ -0,0 +1,28 @@
apiVersion: monitoring.coreos.com/v1
kind: Probe
metadata:
name: platform-private-edge
namespace: observability
labels:
app.kubernetes.io/name: prometheus-blackbox-exporter
app.kubernetes.io/instance: blackbox-exporter
app.kubernetes.io/part-of: platform-observability
observability.hyeonworks.com/instance: home
spec:
jobName: blackbox-private-edge
interval: 30s
scrapeTimeout: 15s
module: http_private_edge_403
prober:
url: blackbox-exporter.observability.svc.cluster.local:9115
scheme: http
path: /probe
proxyFromEnvironment: false
targets:
staticConfig:
labels:
observability.hyeonworks.com/probe-group: private-edge
static:
- https://grafana.learn.hyeonworks.com/
- https://storage-admin.learn.hyeonworks.com/
- https://db-admin.learn.hyeonworks.com/
@@ -0,0 +1,44 @@
apiVersion: monitoring.coreos.com/v1
kind: Probe
metadata:
name: platform-private-internal
namespace: observability
labels:
app.kubernetes.io/name: prometheus-blackbox-exporter
app.kubernetes.io/instance: blackbox-exporter
app.kubernetes.io/part-of: platform-observability
observability.hyeonworks.com/instance: home
spec:
jobName: blackbox-private-internal
interval: 30s
scrapeTimeout: 15s
module: http_private_internal_200
prober:
url: blackbox-exporter.observability.svc.cluster.local:9115
scheme: http
path: /probe
proxyFromEnvironment: false
targets:
staticConfig:
labels:
observability.hyeonworks.com/probe-group: private-internal
static:
- http://grafana.observability.svc.cluster.local/api/health
- http://pgadmin.platform-admin.svc.cluster.local/misc/ping
- http://minio-aistor-console.object-storage.svc.cluster.local:9090/
relabelingConfigs:
- sourceLabels:
- __param_target
regex: http://grafana[.]observability[.]svc[.]cluster[.]local/api/health
targetLabel: __param_hostname
replacement: grafana.learn.hyeonworks.com
- sourceLabels:
- __param_target
regex: http://pgadmin[.]platform-admin[.]svc[.]cluster[.]local/misc/ping
targetLabel: __param_hostname
replacement: db-admin.learn.hyeonworks.com
- sourceLabels:
- __param_target
regex: http://minio-aistor-console[.]object-storage[.]svc[.]cluster[.]local:9090/
targetLabel: __param_hostname
replacement: storage-admin.learn.hyeonworks.com
@@ -0,0 +1,27 @@
apiVersion: monitoring.coreos.com/v1
kind: Probe
metadata:
name: platform-public-edge
namespace: observability
labels:
app.kubernetes.io/name: prometheus-blackbox-exporter
app.kubernetes.io/instance: blackbox-exporter
app.kubernetes.io/part-of: platform-observability
observability.hyeonworks.com/instance: home
spec:
jobName: blackbox-public-edge
interval: 30s
scrapeTimeout: 15s
module: http_2xx
prober:
url: blackbox-exporter.observability.svc.cluster.local:9115
scheme: http
path: /probe
proxyFromEnvironment: false
targets:
staticConfig:
labels:
observability.hyeonworks.com/probe-group: public-edge
static:
- https://git.learn.hyeonworks.com/api/healthz
- https://id.learn.hyeonworks.com/realms/hyeonworks/.well-known/openid-configuration
@@ -0,0 +1,128 @@
fullnameOverride: blackbox-exporter
commonLabels:
observability.hyeonworks.com/instance: home
kind: Deployment
replicas: 1
revisionHistoryLimit: 3
strategy:
type: Recreate
rollingUpdate: null
automountServiceAccountToken: false
hostNetwork: false
hostPort: 0
image:
registry: quay.io
repository: prometheus/blackbox-exporter
tag: v0.28.0
digest: sha256:43027b43fb785b7c5adc53bd3b5dbc1a258270a2e8aff24f477b45c4e38dac68
pullPolicy: IfNotPresent
podSecurityContext:
runAsNonRoot: true
runAsUser: 1000
runAsGroup: 1000
seccompProfile:
type: RuntimeDefault
securityContext:
runAsNonRoot: true
runAsUser: 1000
runAsGroup: 1000
readOnlyRootFilesystem: true
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
seccompProfile:
type: RuntimeDefault
resources:
requests:
cpu: 25m
memory: 32Mi
limits:
cpu: 200m
memory: 128Mi
serviceAccount:
create: true
name: blackbox-exporter
service:
type: ClusterIP
port: 9115
ingress:
enabled: false
route:
main:
enabled: false
networkPolicy:
enabled: false
configReloader:
enabled: false
serviceMonitor:
enabled: false
selfMonitor:
enabled: true
port: http
interval: 30s
scrapeTimeout: 10s
config:
modules:
http_2xx:
prober: http
timeout: 10s
http:
method: GET
valid_status_codes:
- 200
valid_http_versions:
- HTTP/1.1
- HTTP/2.0
follow_redirects: true
enable_http2: true
preferred_ip_protocol: ip4
ip_protocol_fallback: false
tls_config:
insecure_skip_verify: false
http_private_edge_403:
prober: http
timeout: 10s
http:
method: GET
valid_status_codes:
- 403
valid_http_versions:
- HTTP/1.1
- HTTP/2.0
follow_redirects: true
enable_http2: true
preferred_ip_protocol: ip4
ip_protocol_fallback: false
tls_config:
insecure_skip_verify: false
http_private_internal_200:
prober: http
timeout: 10s
http:
method: GET
valid_status_codes:
- 200
valid_http_versions:
- HTTP/1.1
- HTTP/2.0
follow_redirects: true
enable_http2: true
preferred_ip_protocol: ip4
ip_protocol_fallback: false
@@ -0,0 +1,208 @@
{
"annotations": {
"list": []
},
"editable": false,
"fiscalYearStartMonth": 0,
"graphTooltip": 1,
"id": null,
"links": [],
"panels": [
{
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"fieldConfig": {
"defaults": {
"max": 1,
"min": 0,
"unit": "short"
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 0
},
"id": 1,
"options": {
"legend": {
"displayMode": "table",
"placement": "bottom"
},
"tooltip": {
"mode": "multi"
}
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"editorMode": "code",
"expr": "min by (job, instance) (probe_success{})",
"legendFormat": "{{job}} / {{instance}}",
"range": true,
"refId": "A"
}
],
"title": "HTTPS Probe Success",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"fieldConfig": {
"defaults": {
"min": 0,
"unit": "s"
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 0
},
"id": 2,
"options": {
"legend": {
"displayMode": "table",
"placement": "bottom"
},
"tooltip": {
"mode": "multi"
}
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"editorMode": "code",
"expr": "max by (job, instance) (probe_duration_seconds{})",
"legendFormat": "{{job}} / {{instance}}",
"range": true,
"refId": "A"
}
],
"title": "HTTPS Probe Duration",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"fieldConfig": {
"defaults": {
"min": 0,
"unit": "short"
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 8
},
"id": 3,
"options": {
"legend": {
"displayMode": "table",
"placement": "bottom"
},
"tooltip": {
"mode": "multi"
}
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"editorMode": "code",
"expr": "max by (instance) (probe_http_status_code{job=\"blackbox-private-edge\"})",
"legendFormat": "{{instance}}",
"range": true,
"refId": "A"
}
],
"title": "Observed HTTP Status (403 Is Expected at the Private Edge)",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"fieldConfig": {
"defaults": {
"min": 0,
"unit": "d"
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 8
},
"id": 4,
"options": {
"legend": {
"displayMode": "table",
"placement": "bottom"
},
"tooltip": {
"mode": "multi"
}
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"editorMode": "code",
"expr": "(probe_ssl_earliest_cert_expiry{} - time()) / 86400",
"legendFormat": "{{job}} / {{instance}}",
"range": true,
"refId": "A"
}
],
"title": "TLS Certificate Days Remaining",
"type": "timeseries"
}
],
"refresh": "30s",
"schemaVersion": 42,
"tags": [
"platform",
"observability",
"https"
],
"templating": {
"list": []
},
"time": {
"from": "now-6h",
"to": "now"
},
"timepicker": {},
"timezone": "browser",
"title": "Platform / HTTPS Endpoints",
"uid": "platform-https-endpoints",
"version": 1
}
@@ -0,0 +1,266 @@
{
"annotations": {
"list": []
},
"editable": false,
"fiscalYearStartMonth": 0,
"graphTooltip": 1,
"id": null,
"links": [],
"panels": [
{
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"fieldConfig": {
"defaults": {
"max": 100,
"min": 0,
"unit": "percent"
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 0
},
"id": 1,
"options": {
"legend": {
"displayMode": "table",
"placement": "bottom"
},
"tooltip": {
"mode": "multi"
}
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"editorMode": "code",
"expr": "100 * (1 - avg by (instance) (rate(node_cpu_seconds_total{mode=\"idle\"}[5m])))",
"legendFormat": "{{instance}}",
"range": true,
"refId": "A"
}
],
"title": "Node CPU Used",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"fieldConfig": {
"defaults": {
"max": 100,
"min": 0,
"unit": "percent"
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 0
},
"id": 2,
"options": {
"legend": {
"displayMode": "table",
"placement": "bottom"
},
"tooltip": {
"mode": "multi"
}
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"editorMode": "code",
"expr": "100 * (1 - node_memory_MemAvailable_bytes{} / node_memory_MemTotal_bytes{})",
"legendFormat": "{{instance}}",
"range": true,
"refId": "A"
}
],
"title": "Node Memory Used",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"fieldConfig": {
"defaults": {
"max": 100,
"min": 0,
"unit": "percent"
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 8
},
"id": 3,
"options": {
"legend": {
"displayMode": "table",
"placement": "bottom"
},
"tooltip": {
"mode": "multi"
}
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"editorMode": "code",
"expr": "100 * (1 - node_filesystem_avail_bytes{mountpoint=\"/\"} / node_filesystem_size_bytes{mountpoint=\"/\"})",
"legendFormat": "{{instance}} {{device}}",
"range": true,
"refId": "A"
}
],
"title": "Root Filesystem Used",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"fieldConfig": {
"defaults": {
"max": 100,
"min": 0,
"unit": "percent"
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 8
},
"id": 4,
"options": {
"legend": {
"displayMode": "table",
"placement": "bottom"
},
"tooltip": {
"mode": "multi"
}
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"editorMode": "code",
"expr": "100 * (1 - node_filesystem_files_free{mountpoint=\"/\"} / node_filesystem_files{mountpoint=\"/\"})",
"legendFormat": "{{instance}} {{device}}",
"range": true,
"refId": "A"
}
],
"title": "Root Filesystem Inodes Used",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"fieldConfig": {
"defaults": {
"min": 0,
"unit": "Bps"
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 24,
"x": 0,
"y": 16
},
"id": 5,
"options": {
"legend": {
"displayMode": "table",
"placement": "bottom"
},
"tooltip": {
"mode": "multi"
}
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"editorMode": "code",
"expr": "sum by (instance) (rate(node_network_receive_bytes_total{device!=\"lo\"}[5m]))",
"legendFormat": "{{instance}} receive",
"range": true,
"refId": "A"
},
{
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"editorMode": "code",
"expr": "sum by (instance) (rate(node_network_transmit_bytes_total{device!=\"lo\"}[5m]))",
"legendFormat": "{{instance}} transmit",
"range": true,
"refId": "B"
}
],
"title": "Node Network Throughput",
"type": "timeseries"
}
],
"refresh": "30s",
"schemaVersion": 42,
"tags": [
"platform",
"observability",
"node"
],
"templating": {
"list": []
},
"time": {
"from": "now-6h",
"to": "now"
},
"timepicker": {},
"timezone": "browser",
"title": "Platform / Kubernetes Node",
"uid": "platform-kubernetes-node",
"version": 1
}
@@ -0,0 +1,28 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: observability
generatorOptions:
disableNameSuffixHash: true
labels:
grafana_dashboard: "1"
observability.hyeonworks.com/instance: home
observability.hyeonworks.com/owner: platform-observability
configMapGenerator:
- name: grafana-dashboard-kubernetes-node
files:
- kubernetes-node.json
- name: grafana-dashboard-workload-health
files:
- workload-health.json
- name: grafana-dashboard-platform-services
files:
- platform-services.json
- name: grafana-dashboard-observability-backends
files:
- observability-backends.json
- name: grafana-dashboard-https-endpoints
files:
- https-endpoints.json
@@ -0,0 +1,384 @@
{
"annotations": {
"list": []
},
"editable": false,
"fiscalYearStartMonth": 0,
"graphTooltip": 1,
"id": null,
"links": [],
"panels": [
{
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"fieldConfig": {
"defaults": {
"min": 0,
"unit": "short"
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 0
},
"id": 1,
"options": {
"legend": {
"displayMode": "table",
"placement": "bottom"
},
"tooltip": {
"mode": "multi"
}
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"editorMode": "code",
"expr": "max(prometheus_tsdb_head_series{})",
"legendFormat": "head series",
"range": true,
"refId": "A"
},
{
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"editorMode": "code",
"expr": "sum(prometheus_engine_queries{})",
"legendFormat": "active queries",
"range": true,
"refId": "B"
}
],
"title": "Prometheus Head Series and Active Queries",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"fieldConfig": {
"defaults": {
"min": 0,
"unit": "ops"
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 0
},
"id": 2,
"options": {
"legend": {
"displayMode": "table",
"placement": "bottom"
},
"tooltip": {
"mode": "multi"
}
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"editorMode": "code",
"expr": "sum(rate(prometheus_notifications_errors_total{}[5m]))",
"legendFormat": "errors / second",
"range": true,
"refId": "A"
}
],
"title": "Prometheus Notification Error Rate",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"fieldConfig": {
"defaults": {
"min": 0,
"unit": "short"
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 8
},
"id": 3,
"options": {
"legend": {
"displayMode": "table",
"placement": "bottom"
},
"tooltip": {
"mode": "multi"
}
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"editorMode": "code",
"expr": "sum(loki_ingester_memory_chunks{})",
"legendFormat": "memory chunks",
"range": true,
"refId": "A"
},
{
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"editorMode": "code",
"expr": "sum(loki_ingester_flush_queue_length{})",
"legendFormat": "flush queue",
"range": true,
"refId": "B"
}
],
"title": "Loki Ingester State",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"fieldConfig": {
"defaults": {
"min": 0,
"unit": "s"
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 8
},
"id": 4,
"options": {
"legend": {
"displayMode": "table",
"placement": "bottom"
},
"tooltip": {
"mode": "multi"
}
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"editorMode": "code",
"expr": "time() - max(loki_objstore_bucket_last_successful_upload_time{})",
"legendFormat": "seconds since upload",
"range": true,
"refId": "A"
}
],
"title": "Loki Object Store Upload Age",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"fieldConfig": {
"defaults": {
"min": 0,
"unit": "short"
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 16
},
"id": 5,
"options": {
"legend": {
"displayMode": "table",
"placement": "bottom"
},
"tooltip": {
"mode": "multi"
}
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"editorMode": "code",
"expr": "sum(tempo_ingester_live_traces{})",
"legendFormat": "live traces",
"range": true,
"refId": "A"
},
{
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"editorMode": "code",
"expr": "sum(tempo_ingester_flush_queue_length{})",
"legendFormat": "flush queue",
"range": true,
"refId": "B"
}
],
"title": "Tempo Ingester State",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"fieldConfig": {
"defaults": {
"max": 1,
"min": 0,
"unit": "short"
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 16
},
"id": 6,
"options": {
"legend": {
"displayMode": "table",
"placement": "bottom"
},
"tooltip": {
"mode": "multi"
}
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"editorMode": "code",
"expr": "min(alloy_config_last_load_successful{})",
"legendFormat": "last load successful",
"range": true,
"refId": "A"
}
],
"title": "Alloy Configuration State",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"fieldConfig": {
"defaults": {
"min": 0,
"unit": "ops"
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 24,
"x": 0,
"y": 24
},
"id": 7,
"options": {
"legend": {
"displayMode": "table",
"placement": "bottom"
},
"tooltip": {
"mode": "multi"
}
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"editorMode": "code",
"expr": "sum(rate(loki_write_batch_retries_total{}[5m]))",
"legendFormat": "retry / second",
"range": true,
"refId": "A"
},
{
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"editorMode": "code",
"expr": "sum(rate(loki_write_dropped_entries_total{}[5m]))",
"legendFormat": "dropped / second",
"range": true,
"refId": "B"
}
],
"title": "Alloy Delivery Retry and Drop Rate",
"type": "timeseries"
}
],
"refresh": "30s",
"schemaVersion": 42,
"tags": [
"platform",
"observability",
"backends"
],
"templating": {
"list": []
},
"time": {
"from": "now-6h",
"to": "now"
},
"timepicker": {},
"timezone": "browser",
"title": "Platform / Observability Backends",
"uid": "platform-observability-backends",
"version": 1
}
@@ -0,0 +1,419 @@
{
"annotations": {
"list": []
},
"editable": false,
"fiscalYearStartMonth": 0,
"graphTooltip": 1,
"id": null,
"links": [],
"panels": [
{
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"fieldConfig": {
"defaults": {
"min": 0,
"unit": "short"
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 8,
"x": 0,
"y": 0
},
"id": 1,
"options": {
"legend": {
"displayMode": "table",
"placement": "bottom"
},
"tooltip": {
"mode": "multi"
}
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"editorMode": "code",
"expr": "max(gitea_repositories{})",
"legendFormat": "repositories",
"range": true,
"refId": "A"
},
{
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"editorMode": "code",
"expr": "sum(gitea_users{})",
"legendFormat": "users",
"range": true,
"refId": "B"
}
],
"title": "Gitea Inventory",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"fieldConfig": {
"defaults": {
"min": 0,
"unit": "short"
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 8,
"x": 8,
"y": 0
},
"id": 2,
"options": {
"legend": {
"displayMode": "table",
"placement": "bottom"
},
"tooltip": {
"mode": "multi"
}
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"editorMode": "code",
"expr": "sum(agroal_active_count{})",
"legendFormat": "active",
"range": true,
"refId": "A"
},
{
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"editorMode": "code",
"expr": "sum(agroal_available_count{})",
"legendFormat": "available",
"range": true,
"refId": "B"
}
],
"title": "Keycloak Database Pool",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"fieldConfig": {
"defaults": {
"min": 0,
"unit": "short"
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 8,
"x": 16,
"y": 0
},
"id": 3,
"options": {
"legend": {
"displayMode": "table",
"placement": "bottom"
},
"tooltip": {
"mode": "multi"
}
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"editorMode": "code",
"expr": "min(cnpg_collector_up{})",
"legendFormat": "collector up",
"range": true,
"refId": "A"
},
{
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"editorMode": "code",
"expr": "sum(cnpg_backends_total{})",
"legendFormat": "backends",
"range": true,
"refId": "B"
}
],
"title": "CloudNativePG Collector and Connections",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"fieldConfig": {
"defaults": {
"min": 0,
"unit": "bytes"
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 8
},
"id": 4,
"options": {
"legend": {
"displayMode": "table",
"placement": "bottom"
},
"tooltip": {
"mode": "multi"
}
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"editorMode": "code",
"expr": "sum by (datname) (cnpg_pg_database_size_bytes{})",
"legendFormat": "{{datname}}",
"range": true,
"refId": "A"
}
],
"title": "CloudNativePG Database Size",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"fieldConfig": {
"defaults": {
"max": 100,
"min": 0,
"unit": "percent"
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 8
},
"id": 5,
"options": {
"legend": {
"displayMode": "table",
"placement": "bottom"
},
"tooltip": {
"mode": "multi"
}
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"editorMode": "code",
"expr": "100 * max by (bucket) (minio_cluster_usage_buckets_total_bytes{bucket=~\"loki|tempo\"}) / max by (bucket) (minio_cluster_usage_buckets_quota_total_bytes{bucket=~\"loki|tempo\"} > 0)",
"legendFormat": "{{bucket}}",
"range": true,
"refId": "A"
}
],
"title": "AIStor Bucket Quota Used",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"fieldConfig": {
"defaults": {
"min": 0,
"unit": "reqps"
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 16
},
"id": 6,
"options": {
"legend": {
"displayMode": "table",
"placement": "bottom"
},
"tooltip": {
"mode": "multi"
}
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"editorMode": "code",
"expr": "sum by (entrypoint) (rate(traefik_entrypoint_requests_total{}[5m]))",
"legendFormat": "{{entrypoint}}",
"range": true,
"refId": "A"
}
],
"title": "Traefik Request Rate",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"fieldConfig": {
"defaults": {
"max": 100,
"min": 0,
"unit": "percent"
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 16
},
"id": 7,
"options": {
"legend": {
"displayMode": "table",
"placement": "bottom"
},
"tooltip": {
"mode": "multi"
}
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"editorMode": "code",
"expr": "100 * sum(rate(traefik_entrypoint_requests_total{code=~\"5..\"}[5m])) / clamp_min(sum(rate(traefik_entrypoint_requests_total{}[5m])), 1e-9)",
"legendFormat": "5xx ratio",
"range": true,
"refId": "A"
}
],
"title": "Traefik 5xx Ratio",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"fieldConfig": {
"defaults": {
"max": 1,
"min": 0,
"unit": "short"
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 24,
"x": 0,
"y": 24
},
"id": 8,
"options": {
"legend": {
"displayMode": "table",
"placement": "bottom"
},
"tooltip": {
"mode": "multi"
}
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"editorMode": "code",
"expr": "min by (job) (probe_success{})",
"legendFormat": "{{job}}",
"range": true,
"refId": "A"
}
],
"title": "Platform HTTPS Boundary Health",
"type": "timeseries"
}
],
"refresh": "30s",
"schemaVersion": 42,
"tags": [
"platform",
"observability",
"services"
],
"templating": {
"list": []
},
"time": {
"from": "now-6h",
"to": "now"
},
"timepicker": {},
"timezone": "browser",
"title": "Platform / Services",
"uid": "platform-services",
"version": 1
}
@@ -0,0 +1,218 @@
{
"annotations": {
"list": []
},
"editable": false,
"fiscalYearStartMonth": 0,
"graphTooltip": 1,
"id": null,
"links": [],
"panels": [
{
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"fieldConfig": {
"defaults": {
"min": 0,
"unit": "short"
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 0
},
"id": 1,
"options": {
"legend": {
"displayMode": "table",
"placement": "bottom"
},
"tooltip": {
"mode": "multi"
}
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"editorMode": "code",
"expr": "sum by (namespace, deployment) (kube_deployment_status_replicas_unavailable{})",
"legendFormat": "{{namespace}} / {{deployment}}",
"range": true,
"refId": "A"
}
],
"title": "Deployment Unavailable Replicas",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"fieldConfig": {
"defaults": {
"min": 0,
"unit": "short"
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 0
},
"id": 2,
"options": {
"legend": {
"displayMode": "table",
"placement": "bottom"
},
"tooltip": {
"mode": "multi"
}
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"editorMode": "code",
"expr": "clamp_min(sum by (namespace, statefulset) (kube_statefulset_replicas{}) - sum by (namespace, statefulset) (kube_statefulset_status_replicas_ready{}), 0)",
"legendFormat": "{{namespace}} / {{statefulset}}",
"range": true,
"refId": "A"
}
],
"title": "StatefulSet Replicas Not Ready",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"fieldConfig": {
"defaults": {
"min": 0,
"unit": "short"
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 8
},
"id": 3,
"options": {
"legend": {
"displayMode": "table",
"placement": "bottom"
},
"tooltip": {
"mode": "multi"
}
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"editorMode": "code",
"expr": "sum by (namespace, pod) (increase(kube_pod_container_status_restarts_total{}[15m]))",
"legendFormat": "{{namespace}} / {{pod}}",
"range": true,
"refId": "A"
}
],
"title": "Container Restarts in 15 Minutes",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"fieldConfig": {
"defaults": {
"min": 0,
"unit": "short"
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 8
},
"id": 4,
"options": {
"legend": {
"displayMode": "table",
"placement": "bottom"
},
"tooltip": {
"mode": "multi"
}
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"editorMode": "code",
"expr": "sum(kube_persistentvolumeclaim_status_phase{phase=\"Pending\"})",
"legendFormat": "Pending",
"range": true,
"refId": "A"
},
{
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"editorMode": "code",
"expr": "sum(kube_persistentvolumeclaim_status_phase{phase=\"Lost\"})",
"legendFormat": "Lost",
"range": true,
"refId": "B"
}
],
"title": "PersistentVolumeClaim Health",
"type": "timeseries"
}
],
"refresh": "30s",
"schemaVersion": 42,
"tags": [
"platform",
"observability",
"workloads"
],
"templating": {
"list": []
},
"time": {
"from": "now-6h",
"to": "now"
},
"timepicker": {},
"timezone": "browser",
"title": "Platform / Workload Health",
"uid": "platform-workload-health",
"version": 1
}
+24
View File
@@ -0,0 +1,24 @@
# Grafana
Private Grafana for the home observability stack. The pinned Grafana chart is
rendered as the independent `grafana` release in `observability`; the
kube-prometheus-stack embedded Grafana remains disabled.
The workload consumes, but never renders, these existing Secret contracts:
- `grafana-admin`: `admin-user`, `admin-password`
- `grafana-keycloak-oidc`: `client-id`, `client-secret`
The `grafana` PVC is fixed to `observability-grafana-local-pv` (2Gi, Retain).
Host Nginx terminates the exact-SAN certificate and forwards plain HTTP through
Traefik; the Kubernetes Ingress therefore contains no TLS material.
The Pod does not automount a service-account token. A 3600-second projected
token, CA, and namespace are mounted only into `grafana-sc-dashboard`, whose
namespaced Role can read ConfigMaps but not Secrets. Dashboard ConfigMaps must
carry `grafana_dashboard: "1"` in `observability`.
Provisioned datasource UIDs are `prometheus`, `loki`, and `tempo`. Tempo service
map configuration is present for future use, but the metrics-generator and
Prometheus remote-write path are intentionally not enabled, so service-graph
availability is not an initial acceptance condition.
@@ -0,0 +1,27 @@
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: grafana-dashboard-sidecar
namespace: observability
rules:
- apiGroups:
- ""
resources:
- configmaps
verbs:
- list
- watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: grafana-dashboard-sidecar
namespace: observability
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: grafana-dashboard-sidecar
subjects:
- kind: ServiceAccount
name: grafana
namespace: observability
@@ -0,0 +1,37 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: grafana
namespace: observability
spec:
template:
spec:
automountServiceAccountToken: false
hostNetwork: false
hostPID: false
hostIPC: false
containers:
- name: grafana-sc-dashboard
volumeMounts:
- name: dashboard-sidecar-api-access
mountPath: /var/run/secrets/kubernetes.io/serviceaccount
readOnly: true
volumes:
- name: dashboard-sidecar-api-access
projected:
defaultMode: 420
sources:
- serviceAccountToken:
expirationSeconds: 3600
path: token
- configMap:
name: kube-root-ca.crt
items:
- key: ca.crt
path: ca.crt
- downwardAPI:
items:
- path: namespace
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
@@ -0,0 +1,7 @@
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: grafana
namespace: observability
spec:
ingressClassName: traefik
@@ -0,0 +1,24 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: observability
resources:
- dashboard-sidecar-rbac.yaml
helmGlobals:
chartHome: .helm/charts
helmCharts:
- name: grafana
repo: https://grafana-community.github.io/helm-charts
version: 12.10.1
releaseName: grafana
namespace: observability
valuesFile: values/home.yaml
includeCRDs: false
skipTests: true
patches:
- path: ingress-class-patch.yaml
- path: dashboard-sidecar-token-patch.yaml
@@ -0,0 +1,262 @@
replicas: 1
extraLabels:
observability.hyeonworks.com/instance: home
deploymentStrategy:
type: Recreate
revisionHistoryLimit: 3
automountServiceAccountToken: false
enableServiceLinks: false
rbac:
create: false
serviceAccount:
create: true
name: grafana
automountServiceAccountToken: false
image:
registry: docker.io
repository: grafana/grafana
tag: 13.1.1
sha: f33c692ba1a5ee15724cf6b22db65e9de39dde14d80f7d73a9546e3fc917270b
pullPolicy: IfNotPresent
testFramework:
enabled: false
securityContext:
runAsNonRoot: true
runAsUser: 472
runAsGroup: 472
fsGroup: 472
fsGroupChangePolicy: OnRootMismatch
seccompProfile:
type: RuntimeDefault
containerSecurityContext:
allowPrivilegeEscalation: false
privileged: false
capabilities:
drop:
- ALL
seccompProfile:
type: RuntimeDefault
resources:
requests:
cpu: 100m
memory: 256Mi
limits:
cpu: 500m
memory: 512Mi
initChownData:
enabled: false
persistence:
type: pvc
enabled: true
storageClassName: ssd-local-observability-grafana-retain
volumeName: observability-grafana-local-pv
lookupVolumeName: false
accessModes:
- ReadWriteOnce
size: 2Gi
finalizers:
- kubernetes.io/pvc-protection
admin:
existingSecret: grafana-admin
userKey: admin-user
passwordKey: admin-password
envValueFrom:
GF_AUTH_GENERIC_OAUTH_CLIENT_ID:
secretKeyRef:
name: grafana-keycloak-oidc
key: client-id
GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET:
secretKeyRef:
name: grafana-keycloak-oidc
key: client-secret
service:
enabled: true
type: ClusterIP
port: 80
targetPort: 3000
portName: service
serviceMonitor:
enabled: true
interval: 30s
path: /metrics
ingress:
enabled: true
annotations: {}
path: /
pathType: Prefix
hosts:
- grafana.learn.hyeonworks.com
tls: []
grafana.ini:
paths:
data: /var/lib/grafana/
logs: /var/log/grafana
plugins: /var/lib/grafana/plugins
provisioning: /etc/grafana/provisioning
analytics:
check_for_updates: false
check_for_plugin_updates: false
reporting_enabled: false
plugins:
preinstall_disabled: true
server:
domain: grafana.learn.hyeonworks.com
root_url: https://grafana.learn.hyeonworks.com/
enforce_domain: true
security:
cookie_secure: true
cookie_samesite: lax
auth:
disable_login_form: false
oauth_auto_login: false
login_maximum_lifetime_duration: 8h
login_maximum_inactive_lifetime_duration: 30m
auth.anonymous:
enabled: false
auth.basic:
enabled: true
auth.generic_oauth:
enabled: true
name: Keycloak
allow_sign_up: true
use_pkce: true
scopes: openid profile email
groups_attribute_path: groups
allowed_groups: /platform-observability-admins /platform-observability-viewers
role_attribute_strict: true
allow_assign_grafana_admin: false
skip_org_role_sync: false
validate_id_token: true
use_refresh_token: true
role_attribute_path: "contains(groups[*], '/platform-observability-admins') && 'Admin' || contains(groups[*], '/platform-observability-viewers') && 'Viewer' || null"
auth_url: https://id.learn.hyeonworks.com/realms/hyeonworks/protocol/openid-connect/auth
token_url: https://id.learn.hyeonworks.com/realms/hyeonworks/protocol/openid-connect/token
api_url: https://id.learn.hyeonworks.com/realms/hyeonworks/protocol/openid-connect/userinfo
signout_redirect_url: https://id.learn.hyeonworks.com/realms/hyeonworks/protocol/openid-connect/logout
datasources:
datasources.yaml:
apiVersion: 1
deleteDatasources:
- name: Prometheus
orgId: 1
- name: Loki
orgId: 1
- name: Tempo
orgId: 1
datasources:
- name: Prometheus
uid: prometheus
type: prometheus
access: proxy
url: http://observability-core-kube-pr-prometheus.observability.svc.cluster.local:9090
isDefault: true
editable: false
jsonData:
httpMethod: POST
timeInterval: 30s
- name: Loki
uid: loki
type: loki
access: proxy
url: http://loki.observability.svc.cluster.local:3100
isDefault: false
editable: false
jsonData:
derivedFields:
- name: trace_id
matcherRegex: '"trace_id"[[:space:]]*:[[:space:]]*"([0-9a-f]{32})"'
datasourceUid: tempo
url: '$${__value.raw}'
- name: Tempo
uid: tempo
type: tempo
access: proxy
url: http://tempo.observability.svc.cluster.local:3200
isDefault: false
editable: false
jsonData:
httpMethod: GET
nodeGraph:
enabled: true
serviceMap:
datasourceUid: prometheus
tracesToLogsV2:
datasourceUid: loki
spanStartTimeShift: -1m
spanEndTimeShift: 1m
tags:
- key: k8s.namespace.name
value: namespace
- key: k8s.pod.name
value: pod
filterByTraceID: true
filterBySpanID: false
sidecar:
image:
registry: quay.io
repository: kiwigrid/k8s-sidecar
tag: 2.10.0
sha: 129877c81acf2bc8c3fa000e89a62e020eb89d41ceb94767c657aef5bb0cc0d3
imagePullPolicy: IfNotPresent
resources:
requests:
cpu: 25m
memory: 64Mi
limits:
cpu: 100m
memory: 128Mi
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop:
- ALL
seccompProfile:
type: RuntimeDefault
dashboards:
enabled: true
label: grafana_dashboard
labelValue: "1"
searchNamespace:
- observability
resource: configmap
watchMethod: WATCH
skipReload: true
provider:
name: sidecarProvider
orgid: 1
folder: ""
folderUid: ""
type: file
disableDelete: false
allowUiUpdates: false
foldersFromFilesStructure: false
imageRenderer:
enabled: false
networkPolicy:
enabled: false
assertNoLeakedSecrets: true
@@ -0,0 +1,15 @@
# kube-prometheus-stack
고정 chart `88.0.1``observability` namespace에 렌더한다. Grafana와
node-exporter, control-plane에 존재하지 않는 scheduler/controller-manager/etcd/
kube-proxy monitor는 끈다. Prometheus와 Alertmanager 데이터는 각각의 `Retain`
Local PV에 보존하며 CRD는 별도 root에서 먼저 적용한다.
Prometheus discovery는 namespace label
`observability.hyeonworks.com/enabled=true`와 resource label
`observability.hyeonworks.com/instance=home`의 교집합만 선택한다.
Alertmanager source는 selector discovery 대신 global
`alertmanagerConfiguration.name=platform-alertmanager`를 참조한다. 참조 대상의
Slack Secret이 준비되기 전에는 이 변경을 live에 적용하지 않고 기존 null 구성을
유지한다.
@@ -0,0 +1,37 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: observability
helmGlobals:
chartHome: .helm/charts
helmCharts:
- name: kube-prometheus-stack
repo: https://prometheus-community.github.io/helm-charts
version: 88.0.1
releaseName: observability-core
namespace: observability
valuesFile: values/home.yaml
includeCRDs: false
skipTests: true
images:
- name: quay.io/prometheus-operator/prometheus-operator
newName: quay.io/prometheus-operator/prometheus-operator
digest: sha256:64eb7914e4705dbb64438e3b3193da1226ad2ea4db2924983693999888cda9b2
- name: registry.k8s.io/kube-state-metrics/kube-state-metrics
newName: registry.k8s.io/kube-state-metrics/kube-state-metrics
digest: sha256:7661da8c99b733d43117e4cba12bd9865d335e5777191d0af3d789807aded9f4
patches:
- target:
group: monitoring.coreos.com
version: v1
kind: Alertmanager
name: observability-core-kube-pr-alertmanager
patch: |-
- op: remove
path: /spec/alertmanagerConfigSelector
- op: remove
path: /spec/alertmanagerConfigNamespaceSelector
@@ -0,0 +1,282 @@
commonLabels:
observability.hyeonworks.com/instance: home
crds:
enabled: false
grafana:
enabled: false
nodeExporter:
enabled: false
kubeApiServer:
enabled: true
kubelet:
enabled: true
coreDns:
enabled: true
kubeControllerManager:
enabled: false
kubeScheduler:
enabled: false
kubeEtcd:
enabled: false
kubeProxy:
enabled: false
defaultRules:
disabled:
Watchdog: true
rules:
etcd: false
kubeControllerManager: false
kubeProxy: false
kubeSchedulerAlerting: false
kubeSchedulerRecording: false
windows: false
prometheusOperator:
admissionWebhooks:
enabled: false
patch:
enabled: false
tls:
enabled: false
image:
registry: quay.io
repository: prometheus-operator/prometheus-operator
tag: v0.93.0
sha: 64eb7914e4705dbb64438e3b3193da1226ad2ea4db2924983693999888cda9b2
resources:
requests:
cpu: 100m
memory: 128Mi
limits:
cpu: 500m
memory: 256Mi
securityContext:
runAsNonRoot: true
runAsUser: 65534
runAsGroup: 65534
fsGroup: 65534
seccompProfile:
type: RuntimeDefault
containerSecurityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
readOnlyRootFilesystem: true
serviceAccount:
automountServiceAccountToken: true
prometheusConfigReloader:
image:
registry: quay.io
repository: prometheus-operator/prometheus-config-reloader
tag: v0.93.0
sha: 65b90f44d5535b106015ac60bafb57803f65dc928c187874de6cd7a9ec6c8905
resources:
requests:
cpu: 25m
memory: 32Mi
limits:
cpu: 100m
memory: 64Mi
kubeStateMetrics:
enabled: true
kube-state-metrics:
image:
registry: registry.k8s.io
repository: kube-state-metrics/kube-state-metrics
tag: v2.19.1
sha: sha256:7661da8c99b733d43117e4cba12bd9865d335e5777191d0af3d789807aded9f4
resources:
requests:
cpu: 100m
memory: 128Mi
limits:
cpu: 500m
memory: 256Mi
securityContext:
enabled: true
runAsNonRoot: true
runAsUser: 65534
runAsGroup: 65534
fsGroup: 65534
seccompProfile:
type: RuntimeDefault
containerSecurityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
readOnlyRootFilesystem: true
automountServiceAccountToken: true
collectors:
- certificatesigningrequests
- configmaps
- cronjobs
- daemonsets
- deployments
- endpointslices
- horizontalpodautoscalers
- ingresses
- jobs
- leases
- limitranges
- mutatingwebhookconfigurations
- namespaces
- networkpolicies
- nodes
- persistentvolumeclaims
- persistentvolumes
- poddisruptionbudgets
- pods
- replicasets
- replicationcontrollers
- resourcequotas
- services
- statefulsets
- storageclasses
- validatingwebhookconfigurations
- volumeattachments
prometheus:
monitor:
additionalLabels:
observability.hyeonworks.com/instance: home
alertmanager:
enabled: true
config:
global:
resolve_timeout: 5m
route:
group_by:
- namespace
group_wait: 30s
group_interval: 5m
repeat_interval: 12h
receiver: "null"
routes: []
receivers:
- name: "null"
serviceAccount:
automountServiceAccountToken: false
alertmanagerSpec:
replicas: 1
retention: 120h
automountServiceAccountToken: false
alertmanagerConfiguration:
name: platform-alertmanager
image:
registry: quay.io
repository: prometheus/alertmanager
tag: v0.33.1
sha: a89f8d4520954079275441eecdb71444328bd90633dd4eddfc33b9ed657f349b
resources:
requests:
cpu: 50m
memory: 128Mi
limits:
cpu: 250m
memory: 256Mi
securityContext:
runAsNonRoot: true
runAsUser: 1000
runAsGroup: 2000
fsGroup: 2000
fsGroupChangePolicy: OnRootMismatch
seccompProfile:
type: RuntimeDefault
persistentVolumeClaimRetentionPolicy:
whenDeleted: Retain
whenScaled: Retain
storage:
volumeClaimTemplate:
spec:
storageClassName: ssd-local-observability-alertmanager-retain
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
prometheus:
enabled: true
prometheusSpec:
replicas: 1
retention: 15d
retentionSize: 16GB
scrapeInterval: 30s
evaluationInterval: 30s
walCompression: true
automountServiceAccountToken: true
image:
registry: quay.io
repository: prometheus/prometheus
tag: v3.13.2-distroless
sha: ce95cfa77eff5aad28bd7a65aff19868cf78d9e17e4c254da7dfe22ade78318b
resources:
requests:
cpu: 750m
memory: 2Gi
limits:
cpu: "2"
memory: 3Gi
securityContext:
runAsNonRoot: true
runAsUser: 1000
runAsGroup: 2000
fsGroup: 2000
fsGroupChangePolicy: OnRootMismatch
seccompProfile:
type: RuntimeDefault
persistentVolumeClaimRetentionPolicy:
whenDeleted: Retain
whenScaled: Retain
storageSpec:
volumeClaimTemplate:
spec:
storageClassName: ssd-local-observability-prometheus-retain
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 20Gi
serviceMonitorSelectorNilUsesHelmValues: false
podMonitorSelectorNilUsesHelmValues: false
probeSelectorNilUsesHelmValues: false
ruleSelectorNilUsesHelmValues: false
serviceMonitorNamespaceSelector:
matchLabels:
observability.hyeonworks.com/enabled: "true"
podMonitorNamespaceSelector:
matchLabels:
observability.hyeonworks.com/enabled: "true"
probeNamespaceSelector:
matchLabels:
observability.hyeonworks.com/enabled: "true"
ruleNamespaceSelector:
matchLabels:
observability.hyeonworks.com/enabled: "true"
serviceMonitorSelector:
matchLabels:
observability.hyeonworks.com/instance: home
podMonitorSelector:
matchLabels:
observability.hyeonworks.com/instance: home
probeSelector:
matchLabels:
observability.hyeonworks.com/instance: home
ruleSelector:
matchLabels:
observability.hyeonworks.com/instance: home
+16
View File
@@ -0,0 +1,16 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- kube-prometheus-stack
- loki
- tempo
- alloy
- node-exporter
- grafana
- blackbox-exporter
- network-policies
- targets
- rules/platform
- alerting
- dashboards/platform
+16
View File
@@ -0,0 +1,16 @@
# Loki
This root renders `grafana-community/loki` chart `18.7.1` as one Loki
`3.7.4` monolithic StatefulSet in `observability`.
- The image is pinned to the reviewed amd64 digest.
- Logs use the `loki` AIStor bucket through `loki-object-storage`; no root
credential is referenced.
- Retention is 30 days. The WAL and working data use the dedicated 5Gi
Retain Local PV.
- Gateway, canary, caches, embedded MinIO, tests, and rule sidecars are
disabled.
- The only client-facing service is ClusterIP port 3100.
Do not delete the PVC or bucket during rollback. Scale the StatefulSet down,
preserve its Retain volume, and investigate S3/WAL state before reapplying.
@@ -0,0 +1,17 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: observability
helmGlobals:
chartHome: .helm/charts
helmCharts:
- name: loki
repo: https://grafana-community.github.io/helm-charts
version: 18.7.1
releaseName: loki
namespace: observability
valuesFile: values/home.yaml
includeCRDs: false
skipTests: true
@@ -0,0 +1,146 @@
deploymentMode: Monolithic
commonLabels:
observability.hyeonworks.com/instance: home
defaults:
automountServiceAccountToken: false
serviceAccount:
automountServiceAccountToken: false
loki:
auth_enabled: false
image:
registry: docker.io
repository: grafana/loki
tag: "3.7.4"
digest: sha256:d80be589cf74b6d92747a1b155bec6f07bdd65e261f797fe93a78fb4e833032e
podSecurityContext:
runAsNonRoot: true
runAsUser: 10001
runAsGroup: 10001
fsGroup: 10001
fsGroupChangePolicy: OnRootMismatch
seccompProfile:
type: RuntimeDefault
containerSecurityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop:
- ALL
seccompProfile:
type: RuntimeDefault
commonConfig:
path_prefix: /var/loki
replication_factor: 1
schemaConfig:
configs:
- from: "2026-07-01"
store: tsdb
object_store: s3
schema: v13
index:
prefix: index_
period: 24h
storage:
type: s3
bucketNames:
chunks: loki
ruler: loki
admin: loki
s3:
endpoint: http://minio.object-storage.svc.cluster.local
s3ForcePathStyle: true
insecure: true
accessKeyId: ${AWS_ACCESS_KEY_ID}
secretAccessKey: ${AWS_SECRET_ACCESS_KEY}
useTestSchema: false
limits_config:
retention_period: 720h
ingestion_rate_mb: 4
ingestion_burst_size_mb: 6
max_line_size: 256KB
discover_log_levels: false
discover_service_name: []
ingester:
chunk_idle_period: 30m
max_chunk_age: 1h
wal:
checkpoint_duration: 5m
replay_memory_ceiling: 512MB
disk_full_threshold: 0.90
flush_on_shutdown: true
compactor:
retention_enabled: true
delete_request_store: s3
retention_delete_delay: 2h
retention_delete_worker_count: 10
max_compaction_parallelism: 1
singleBinary:
replicas: 1
sidecar: false
podLabels:
platform.hyeonworks.com/aistor-client: "true"
service:
type: ClusterIP
resources:
requests:
cpu: 350m
memory: 768Mi
limits:
cpu: 1500m
memory: 1536Mi
extraEnv:
- name: AWS_ACCESS_KEY_ID
valueFrom:
secretKeyRef:
name: loki-object-storage
key: access-key-id
- name: AWS_SECRET_ACCESS_KEY
valueFrom:
secretKeyRef:
name: loki-object-storage
key: secret-access-key
persistence:
enabled: true
size: 5Gi
storageClass: ssd-local-observability-loki-retain
enableStatefulSetAutoDeletePVC: true
whenScaled: Retain
whenDeleted: Retain
read:
replicas: 0
write:
replicas: 0
backend:
replicas: 0
ruler:
sidecar: false
gateway:
enabled: false
lokiCanary:
enabled: false
chunksCache:
enabled: false
resultsCache:
enabled: false
memcached:
enabled: false
minio:
enabled: false
test:
enabled: false
sidecar:
rules:
enabled: false
monitoring:
serviceMonitor:
enabled: true
interval: 30s
@@ -0,0 +1,8 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- observability.yaml
- observability-agent.yaml
- object-storage-counterpart.yaml
@@ -0,0 +1,31 @@
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: object-storage-allow-observability-s3
namespace: object-storage
labels:
app.kubernetes.io/component: observability-object-storage
app.kubernetes.io/part-of: platform
spec:
podSelector:
matchLabels:
aistor.min.io/objectStore: minio-aistor
policyTypes:
- Ingress
ingress:
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: observability
podSelector:
matchLabels:
platform.hyeonworks.com/aistor-client: "true"
matchExpressions:
- key: app.kubernetes.io/name
operator: In
values:
- loki
- tempo
ports:
- protocol: TCP
port: 9000
@@ -0,0 +1,140 @@
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: observability-agent-allow-alloy-dns
namespace: observability-agent
spec:
podSelector:
matchLabels:
app.kubernetes.io/name: alloy
policyTypes:
- Egress
egress:
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
podSelector:
matchLabels:
k8s-app: kube-dns
ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: observability-agent-allow-alloy-api
namespace: observability-agent
spec:
podSelector:
matchLabels:
app.kubernetes.io/name: alloy
policyTypes:
- Egress
egress:
- to:
- ipBlock:
cidr: 10.43.0.1/32
ports:
- protocol: TCP
port: 443
- to:
- ipBlock:
cidr: 192.168.0.107/32
ports:
- protocol: TCP
port: 6443
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: observability-agent-allow-alloy-backends
namespace: observability-agent
spec:
podSelector:
matchLabels:
app.kubernetes.io/name: alloy
policyTypes:
- Egress
egress:
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: observability
podSelector:
matchLabels:
app.kubernetes.io/name: loki
ports:
- protocol: TCP
port: 3100
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: observability
podSelector:
matchLabels:
app.kubernetes.io/name: tempo
ports:
- protocol: TCP
port: 4317
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: observability-agent-allow-prometheus-to-alloy
namespace: observability-agent
spec:
podSelector:
matchLabels:
app.kubernetes.io/name: alloy
policyTypes:
- Ingress
ingress:
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: observability
podSelector:
matchLabels:
app.kubernetes.io/name: prometheus
ports:
- protocol: TCP
port: 12345
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: observability-agent-allow-prometheus-to-node-exporter
namespace: observability-agent
spec:
podSelector:
matchLabels:
app.kubernetes.io/name: prometheus-node-exporter
policyTypes:
- Ingress
ingress:
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: observability
podSelector:
matchLabels:
app.kubernetes.io/name: prometheus
ports:
- protocol: TCP
port: 9100
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: observability-agent-default-deny
namespace: observability-agent
spec:
podSelector: {}
policyTypes:
- Ingress
- Egress
@@ -0,0 +1,548 @@
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: observability-allow-dns
namespace: observability
spec:
podSelector: {}
policyTypes:
- Egress
egress:
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
podSelector:
matchLabels:
k8s-app: kube-dns
ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: observability-allow-kubernetes-api
namespace: observability
spec:
podSelector:
matchExpressions:
- key: app.kubernetes.io/name
operator: In
values:
- kube-prometheus-stack-prometheus-operator
- kube-state-metrics
- prometheus
policyTypes:
- Egress
egress:
- to:
- ipBlock:
cidr: 10.43.0.1/32
ports:
- protocol: TCP
port: 443
- to:
- ipBlock:
cidr: 192.168.0.107/32
ports:
- protocol: TCP
port: 6443
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: observability-allow-prometheus-egress
namespace: observability
spec:
podSelector:
matchLabels:
app.kubernetes.io/name: prometheus
policyTypes:
- Egress
egress:
- to:
- podSelector:
matchLabels:
app.kubernetes.io/name: kube-prometheus-stack-prometheus-operator
ports:
- protocol: TCP
port: 8080
- to:
- podSelector:
matchLabels:
app.kubernetes.io/name: kube-state-metrics
ports:
- protocol: TCP
port: 8080
- protocol: TCP
port: 8081
- to:
- podSelector:
matchLabels:
app.kubernetes.io/name: prometheus
ports:
- protocol: TCP
port: 9090
- protocol: TCP
port: 8080
- to:
- podSelector:
matchLabels:
app.kubernetes.io/name: alertmanager
ports:
- protocol: TCP
port: 9093
- protocol: TCP
port: 8080
- to:
- podSelector:
matchLabels:
app.kubernetes.io/name: loki
ports:
- protocol: TCP
port: 3100
- to:
- podSelector:
matchLabels:
app.kubernetes.io/name: tempo
ports:
- protocol: TCP
port: 3200
- to:
- podSelector:
matchLabels:
app.kubernetes.io/name: grafana
app.kubernetes.io/instance: grafana
ports:
- protocol: TCP
port: 3000
- to:
- podSelector:
matchLabels:
app.kubernetes.io/name: prometheus-blackbox-exporter
app.kubernetes.io/instance: blackbox-exporter
ports:
- protocol: TCP
port: 9115
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: observability-agent
podSelector:
matchLabels:
app.kubernetes.io/name: alloy
ports:
- protocol: TCP
port: 12345
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: observability-agent
podSelector:
matchLabels:
app.kubernetes.io/name: prometheus-node-exporter
ports:
- protocol: TCP
port: 9100
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
podSelector:
matchLabels:
k8s-app: kube-dns
ports:
- protocol: TCP
port: 9153
- to:
- ipBlock:
cidr: 192.168.0.107/32
ports:
- protocol: TCP
port: 10250
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: observability-allow-prometheus-to-operator
namespace: observability
spec:
podSelector:
matchLabels:
app.kubernetes.io/name: kube-prometheus-stack-prometheus-operator
policyTypes:
- Ingress
ingress:
- from:
- podSelector:
matchLabels:
app.kubernetes.io/name: prometheus
ports:
- protocol: TCP
port: 8080
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: observability-allow-prometheus-to-ksm
namespace: observability
spec:
podSelector:
matchLabels:
app.kubernetes.io/name: kube-state-metrics
policyTypes:
- Ingress
ingress:
- from:
- podSelector:
matchLabels:
app.kubernetes.io/name: prometheus
ports:
- protocol: TCP
port: 8080
- protocol: TCP
port: 8081
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: observability-allow-prometheus-self
namespace: observability
spec:
podSelector:
matchLabels:
app.kubernetes.io/name: prometheus
policyTypes:
- Ingress
ingress:
- from:
- podSelector:
matchLabels:
app.kubernetes.io/name: prometheus
ports:
- protocol: TCP
port: 9090
- protocol: TCP
port: 8080
- from:
- podSelector:
matchLabels:
app.kubernetes.io/name: grafana
app.kubernetes.io/instance: grafana
ports:
- protocol: TCP
port: 9090
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: observability-allow-alertmanager
namespace: observability
spec:
podSelector:
matchLabels:
app.kubernetes.io/name: alertmanager
policyTypes:
- Ingress
- Egress
ingress:
- from:
- podSelector:
matchLabels:
app.kubernetes.io/name: prometheus
ports:
- protocol: TCP
port: 9093
- protocol: TCP
port: 8080
- from:
- podSelector:
matchLabels:
app.kubernetes.io/name: alertmanager
ports:
- protocol: TCP
port: 9094
- protocol: UDP
port: 9094
egress:
- to:
- podSelector:
matchLabels:
app.kubernetes.io/name: alertmanager
ports:
- protocol: TCP
port: 9094
- protocol: UDP
port: 9094
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: observability-allow-loki
namespace: observability
spec:
podSelector:
matchLabels:
app.kubernetes.io/name: loki
policyTypes:
- Ingress
- Egress
ingress:
- from:
- podSelector:
matchLabels:
app.kubernetes.io/name: prometheus
ports:
- protocol: TCP
port: 3100
- from:
- podSelector:
matchLabels:
app.kubernetes.io/name: grafana
app.kubernetes.io/instance: grafana
ports:
- protocol: TCP
port: 3100
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: observability-agent
podSelector:
matchLabels:
app.kubernetes.io/name: alloy
ports:
- protocol: TCP
port: 3100
- from:
- podSelector:
matchLabels:
app.kubernetes.io/name: loki
ports:
- protocol: TCP
port: 7946
- protocol: UDP
port: 7946
egress:
- to:
- podSelector:
matchLabels:
app.kubernetes.io/name: loki
ports:
- protocol: TCP
port: 7946
- protocol: UDP
port: 7946
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: observability-allow-tempo
namespace: observability
spec:
podSelector:
matchLabels:
app.kubernetes.io/name: tempo
policyTypes:
- Ingress
- Egress
ingress:
- from:
- podSelector:
matchLabels:
app.kubernetes.io/name: prometheus
ports:
- protocol: TCP
port: 3200
- from:
- podSelector:
matchLabels:
app.kubernetes.io/name: grafana
app.kubernetes.io/instance: grafana
ports:
- protocol: TCP
port: 3200
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: observability-agent
podSelector:
matchLabels:
app.kubernetes.io/name: alloy
ports:
- protocol: TCP
port: 4317
- from:
- podSelector:
matchLabels:
app.kubernetes.io/name: tempo
ports:
- protocol: TCP
port: 7946
- protocol: UDP
port: 7946
egress:
- to:
- podSelector:
matchLabels:
app.kubernetes.io/name: tempo
ports:
- protocol: TCP
port: 7946
- protocol: UDP
port: 7946
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: observability-allow-aistor-egress
namespace: observability
spec:
podSelector:
matchLabels:
platform.hyeonworks.com/aistor-client: "true"
matchExpressions:
- key: app.kubernetes.io/name
operator: In
values:
- loki
- tempo
policyTypes:
- Egress
egress:
- to:
- ipBlock:
cidr: 10.43.124.248/32
ports:
- protocol: TCP
port: 80
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: object-storage
podSelector:
matchLabels:
aistor.min.io/objectStore: minio-aistor
ports:
- protocol: TCP
port: 9000
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: observability-allow-grafana-ingress
namespace: observability
spec:
podSelector:
matchLabels:
app.kubernetes.io/name: grafana
app.kubernetes.io/instance: grafana
policyTypes:
- Ingress
ingress:
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
podSelector:
matchLabels:
app.kubernetes.io/name: traefik
app.kubernetes.io/instance: traefik-kube-system
ports:
- protocol: TCP
port: 3000
- from:
- podSelector:
matchLabels:
app.kubernetes.io/name: prometheus
ports:
- protocol: TCP
port: 3000
- from:
- podSelector:
matchLabels:
app.kubernetes.io/name: prometheus-blackbox-exporter
app.kubernetes.io/instance: blackbox-exporter
ports:
- protocol: TCP
port: 3000
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: observability-allow-grafana-datasources
namespace: observability
spec:
podSelector:
matchLabels:
app.kubernetes.io/name: grafana
app.kubernetes.io/instance: grafana
policyTypes:
- Egress
egress:
- to:
- podSelector:
matchLabels:
app.kubernetes.io/name: prometheus
ports:
- protocol: TCP
port: 9090
- to:
- podSelector:
matchLabels:
app.kubernetes.io/name: loki
ports:
- protocol: TCP
port: 3100
- to:
- podSelector:
matchLabels:
app.kubernetes.io/name: tempo
ports:
- protocol: TCP
port: 3200
- to:
- ipBlock:
cidr: 192.168.0.107/32
ports:
- protocol: TCP
port: 443
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: observability-allow-grafana-dashboard-api
namespace: observability
spec:
podSelector:
matchLabels:
app.kubernetes.io/name: grafana
app.kubernetes.io/instance: grafana
policyTypes:
- Egress
egress:
- to:
- ipBlock:
cidr: 10.43.0.1/32
ports:
- protocol: TCP
port: 443
- to:
- ipBlock:
cidr: 192.168.0.107/32
ports:
- protocol: TCP
port: 6443
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: observability-default-deny
namespace: observability
spec:
podSelector: {}
policyTypes:
- Ingress
- Egress
@@ -0,0 +1,11 @@
# node-exporter
The standalone node-exporter runs in `observability-agent` with the embedded
kube-prometheus-stack copy and kube-rbac-proxy disabled. It mounts only
`/proc`, `/sys`, and `/` from the node, all read-only, and exposes an internal
ClusterIP metrics Service on port `9100`.
Host network, PID, and IPC namespaces remain disabled by the platform security
contract. Consequently, network-namespace metrics are treated as a known
single-node visibility limitation rather than authoritative host-network data.
@@ -0,0 +1,23 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: observability-agent
helmGlobals:
chartHome: .helm/charts
helmCharts:
- name: prometheus-node-exporter
repo: https://prometheus-community.github.io/helm-charts
version: 4.56.1
releaseName: node-exporter
namespace: observability-agent
valuesFile: values/home.yaml
includeCRDs: false
skipTests: true
images:
- name: quay.io/prometheus/node-exporter
newName: quay.io/prometheus/node-exporter
digest: sha256:da83fae85603c4e47e6c68369a7d746e2dda683dc35ea2e234b4f171e0d92798
@@ -0,0 +1,78 @@
fullnameOverride: node-exporter
namespaceOverride: observability-agent
image:
registry: quay.io
repository: prometheus/node-exporter
tag: v1.12.1
pullPolicy: IfNotPresent
kubeRBACProxy:
enabled: false
service:
enabled: true
type: ClusterIP
port: 9100
targetPort: 9100
portName: metrics
listenOnAllInterfaces: true
prometheus:
monitor:
enabled: true
additionalLabels:
observability.hyeonworks.com/instance: home
interval: 30s
podMonitor:
enabled: false
resources:
requests:
cpu: 100m
memory: 128Mi
limits:
cpu: 250m
memory: 256Mi
serviceAccount:
create: true
automountServiceAccountToken: false
rbac:
create: true
securityContext:
fsGroup: 65534
runAsGroup: 65534
runAsNonRoot: true
runAsUser: 65534
seccompProfile:
type: RuntimeDefault
containerSecurityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
privileged: false
readOnlyRootFilesystem: true
hostNetwork: false
hostPID: false
hostIPC: false
hostRootFsMount:
enabled: true
mountPropagation: None
hostProcFsMount:
mountPropagation: None
hostSysFsMount:
mountPropagation: None
nodeSelector:
kubernetes.io/os: linux
tolerations:
- effect: NoSchedule
operator: Exists
@@ -0,0 +1,10 @@
# Observability object storage
Loki와 Tempo는 AIStor에서 서로 다른 bucket, policy, user와 Kubernetes Secret을
사용한다. Loki quota는 200Gi, Tempo quota는 100Gi다. 정책은 자기 bucket의 위치·목록과
자기 object의 읽기·쓰기·삭제·multipart 처리만 허용한다.
운영 적용 전에는 K3s Secret 암호화와 격리 복구 증거, AIStor root credential의 암호화된
외부 복구 가능성을 각각 다시 확인한다. 기존 bucket·object·policy·user·quota는 자동
삭제하거나 credential을 자동 회전하지 않는다. 한쪽 Secret만 존재하는 partial 상태에서는
추가 mutation을 중단한다.
@@ -0,0 +1,29 @@
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:GetBucketLocation",
"s3:ListBucket",
"s3:ListBucketMultipartUploads"
],
"Resource": [
"arn:aws:s3:::loki"
]
},
{
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:PutObject",
"s3:DeleteObject",
"s3:AbortMultipartUpload",
"s3:ListMultipartUploadParts"
],
"Resource": [
"arn:aws:s3:::loki/*"
]
}
]
}
@@ -0,0 +1,29 @@
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:GetBucketLocation",
"s3:ListBucket",
"s3:ListBucketMultipartUploads"
],
"Resource": [
"arn:aws:s3:::tempo"
]
},
{
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:PutObject",
"s3:DeleteObject",
"s3:AbortMultipartUpload",
"s3:ListMultipartUploadParts"
],
"Resource": [
"arn:aws:s3:::tempo/*"
]
}
]
}
@@ -0,0 +1,87 @@
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: platform-certificate-probes
namespace: observability
labels:
app.kubernetes.io/name: platform-observability-rules
app.kubernetes.io/part-of: observability
observability.hyeonworks.com/instance: home
spec:
groups:
- name: platform.blackbox
rules:
- alert: PlatformPublicEdgeProbeFailed
expr: >-
probe_success{
namespace="observability",
"observability.hyeonworks.com/probe-group"="public-edge"
} == 0
for: 5m
labels:
severity: critical
annotations:
summary: 외부 엔드포인트 프로브 실패
description: '{{ $labels.instance }} 외부 엔드포인트 프로브가 5분간 실패했습니다.'
runbook_url: https://git.learn.hyeonworks.com/donghyeon.kang/project-infra/src/branch/main/docs/runbooks/2026-07-31-observability-access-cutover.md
- alert: PlatformPrivateEdgeProbeFailed
expr: >-
probe_success{
namespace="observability",
"observability.hyeonworks.com/probe-group"="private-edge"
} == 0
for: 5m
labels:
severity: critical
annotations:
summary: 비공개 경계 프로브 실패
description: '{{ $labels.instance }} 비공개 경계의 예상 응답이 5분간 실패했습니다. HTTP 403이면서 probe_success=1이면 정상입니다.'
runbook_url: https://git.learn.hyeonworks.com/donghyeon.kang/project-infra/src/branch/main/docs/runbooks/2026-07-31-observability-access-cutover.md
- alert: PlatformInternalHealthProbeFailed
expr: >-
probe_success{
namespace="observability",
"observability.hyeonworks.com/probe-group"="private-internal"
} == 0
for: 5m
labels:
severity: critical
annotations:
summary: 내부 상태 확인 프로브 실패
description: '{{ $labels.instance }} 내부 상태 확인 프로브가 5분간 실패했습니다.'
runbook_url: https://git.learn.hyeonworks.com/donghyeon.kang/project-infra/src/branch/main/docs/runbooks/2026-07-31-observability-access-cutover.md
- name: platform.certificates
rules:
- alert: PlatformCertificateExpiry
expr: >-
(
probe_ssl_earliest_cert_expiry{
namespace="observability",
"observability.hyeonworks.com/probe-group"=~"public-edge|private-edge"
} - time()
) >= 14 * 24 * 60 * 60
and
(
probe_ssl_earliest_cert_expiry{
namespace="observability",
"observability.hyeonworks.com/probe-group"=~"public-edge|private-edge"
} - time()
) < 30 * 24 * 60 * 60
labels:
severity: warning
annotations:
summary: TLS 인증서가 30일 이내 만료됨
description: '{{ $labels.instance }} 인증서 중 가장 이른 만료 시점이 14일 이상 30일 미만 남았습니다.'
runbook_url: https://git.learn.hyeonworks.com/donghyeon.kang/project-infra/src/branch/main/docs/runbooks/2026-07-31-observability-access-cutover.md
- alert: PlatformCertificateExpiry
expr: >-
probe_ssl_earliest_cert_expiry{
namespace="observability",
"observability.hyeonworks.com/probe-group"=~"public-edge|private-edge"
} - time() < 14 * 24 * 60 * 60
labels:
severity: critical
annotations:
summary: TLS 인증서가 14일 이내 만료됨
description: '{{ $labels.instance }} 인증서 중 가장 이른 만료 시점이 14일 미만 남았습니다.'
runbook_url: https://git.learn.hyeonworks.com/donghyeon.kang/project-infra/src/branch/main/docs/runbooks/2026-07-31-observability-access-cutover.md
@@ -0,0 +1,117 @@
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: platform-observability-core
namespace: observability
labels:
app.kubernetes.io/name: platform-observability-rules
app.kubernetes.io/part-of: observability
observability.hyeonworks.com/instance: home
spec:
groups:
- name: platform.storage
rules:
- alert: PlatformPersistentVolumeClaimNotBound
expr: kube_persistentvolumeclaim_status_phase{phase=~"Pending|Lost"} == 1
for: 10m
labels:
severity: warning
annotations:
summary: PersistentVolumeClaim이 바인딩되지 않음
description: 플랫폼 PersistentVolumeClaim이 Pending 또는 Lost 상태로 10분간 유지됐습니다.
runbook_url: https://git.learn.hyeonworks.com/donghyeon.kang/project-infra/src/branch/main/docs/runbooks/2026-07-31-observability-access-cutover.md
- alert: PlatformRootFilesystemUsage
expr: >-
100 * (1 -
node_filesystem_avail_bytes{job="prometheus-node-exporter",mountpoint="/",fstype!~"rootfs|tmpfs|ramfs|squashfs|overlay|nsfs|tracefs|debugfs|securityfs|proc|sysfs|cgroup2?"}
/
node_filesystem_size_bytes{job="prometheus-node-exporter",mountpoint="/",fstype!~"rootfs|tmpfs|ramfs|squashfs|overlay|nsfs|tracefs|debugfs|securityfs|proc|sysfs|cgroup2?"}
) > 75
for: 30m
labels:
severity: info
threshold: "75"
annotations:
summary: 루트 파일시스템 사용률이 75%를 초과함
description: 노드 루트 파일시스템 사용률이 정보 임계치인 75%를 30분간 초과했습니다.
runbook_url: https://git.learn.hyeonworks.com/donghyeon.kang/project-infra/src/branch/main/docs/runbooks/2026-07-31-observability-access-cutover.md
- alert: PlatformRootFilesystemUsage
expr: >-
100 * (1 -
node_filesystem_avail_bytes{job="prometheus-node-exporter",mountpoint="/",fstype!~"rootfs|tmpfs|ramfs|squashfs|overlay|nsfs|tracefs|debugfs|securityfs|proc|sysfs|cgroup2?"}
/
node_filesystem_size_bytes{job="prometheus-node-exporter",mountpoint="/",fstype!~"rootfs|tmpfs|ramfs|squashfs|overlay|nsfs|tracefs|debugfs|securityfs|proc|sysfs|cgroup2?"}
) > 85
for: 15m
labels:
severity: warning
threshold: "85"
annotations:
summary: 루트 파일시스템 사용률이 85%를 초과함
description: 노드 루트 파일시스템 사용률이 경고 임계치인 85%를 15분간 초과했습니다.
runbook_url: https://git.learn.hyeonworks.com/donghyeon.kang/project-infra/src/branch/main/docs/runbooks/2026-07-31-observability-access-cutover.md
- alert: PlatformRootFilesystemUsage
expr: >-
100 * (1 -
node_filesystem_avail_bytes{job="prometheus-node-exporter",mountpoint="/",fstype!~"rootfs|tmpfs|ramfs|squashfs|overlay|nsfs|tracefs|debugfs|securityfs|proc|sysfs|cgroup2?"}
/
node_filesystem_size_bytes{job="prometheus-node-exporter",mountpoint="/",fstype!~"rootfs|tmpfs|ramfs|squashfs|overlay|nsfs|tracefs|debugfs|securityfs|proc|sysfs|cgroup2?"}
) > 90
for: 5m
labels:
severity: critical
threshold: "90"
annotations:
summary: 루트 파일시스템 사용률이 90%를 초과함
description: 노드 루트 파일시스템 사용률이 심각 임계치인 90%를 5분간 초과했습니다.
runbook_url: https://git.learn.hyeonworks.com/donghyeon.kang/project-infra/src/branch/main/docs/runbooks/2026-07-31-observability-access-cutover.md
- alert: PlatformRootFilesystemInodes
expr: >-
100 * (1 -
node_filesystem_files_free{job="prometheus-node-exporter",mountpoint="/",fstype!~"rootfs|tmpfs|ramfs|squashfs|overlay|nsfs|tracefs|debugfs|securityfs|proc|sysfs|cgroup2?"}
/
node_filesystem_files{job="prometheus-node-exporter",mountpoint="/",fstype!~"rootfs|tmpfs|ramfs|squashfs|overlay|nsfs|tracefs|debugfs|securityfs|proc|sysfs|cgroup2?"}
) > 80
for: 30m
labels:
severity: warning
threshold: "80"
annotations:
summary: 루트 파일시스템 inode 사용률이 80%를 초과함
description: 노드 루트 파일시스템 inode 사용률이 경고 임계치인 80%를 30분간 초과했습니다.
runbook_url: https://git.learn.hyeonworks.com/donghyeon.kang/project-infra/src/branch/main/docs/runbooks/2026-07-31-observability-access-cutover.md
- alert: PlatformRootFilesystemInodes
expr: >-
100 * (1 -
node_filesystem_files_free{job="prometheus-node-exporter",mountpoint="/",fstype!~"rootfs|tmpfs|ramfs|squashfs|overlay|nsfs|tracefs|debugfs|securityfs|proc|sysfs|cgroup2?"}
/
node_filesystem_files{job="prometheus-node-exporter",mountpoint="/",fstype!~"rootfs|tmpfs|ramfs|squashfs|overlay|nsfs|tracefs|debugfs|securityfs|proc|sysfs|cgroup2?"}
) > 90
for: 15m
labels:
severity: critical
threshold: "90"
annotations:
summary: 루트 파일시스템 inode 사용률이 90%를 초과함
description: 노드 루트 파일시스템 inode 사용률이 심각 임계치인 90%를 15분간 초과했습니다.
runbook_url: https://git.learn.hyeonworks.com/donghyeon.kang/project-infra/src/branch/main/docs/runbooks/2026-07-31-observability-access-cutover.md
- name: platform.workloads
rules:
- alert: PlatformPodRestartBurst
expr: increase(kube_pod_container_status_restarts_total[15m]) > 3
for: 5m
labels:
severity: warning
annotations:
summary: 컨테이너 재시작 급증 감지
description: 한 컨테이너가 최근 15분 동안 3회를 초과해 재시작했습니다.
runbook_url: https://git.learn.hyeonworks.com/donghyeon.kang/project-infra/src/branch/main/docs/runbooks/2026-07-31-observability-access-cutover.md
- alert: PlatformPodOOMKilled
expr: kube_pod_container_status_last_terminated_reason{reason="OOMKilled"} == 1
for: 5m
labels:
severity: critical
annotations:
summary: 컨테이너가 OOM Killer에 의해 종료됨
description: 컨테이너의 최근 종료 사유가 OOMKilled인 상태로 5분간 유지됐습니다.
runbook_url: https://git.learn.hyeonworks.com/donghyeon.kang/project-infra/src/branch/main/docs/runbooks/2026-07-31-observability-access-cutover.md
@@ -0,0 +1,10 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: observability
resources:
- core-rules.yaml
- storage-quota-rules.yaml
- certificate-rules.yaml
- verified-service-rules.yaml
@@ -0,0 +1,55 @@
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: platform-aistor-storage-quota
namespace: observability
labels:
app.kubernetes.io/name: platform-observability-rules
app.kubernetes.io/part-of: observability
observability.hyeonworks.com/instance: home
spec:
groups:
- name: platform.aistor-storage-quota
rules:
- record: platform:aistor_bucket_quota_usage_percent
expr: >-
100
* max by (bucket) (
minio_cluster_usage_buckets_total_bytes{bucket=~"loki|tempo"}
)
/ max by (bucket) (
minio_cluster_usage_buckets_quota_total_bytes{bucket=~"loki|tempo"} > 0
)
- alert: PlatformAIStorBucketQuotaUsage
expr: >-
platform:aistor_bucket_quota_usage_percent >= 70
and
platform:aistor_bucket_quota_usage_percent < 85
for: 15m
labels:
severity: warning
annotations:
summary: AIStor 버킷 할당량 사용률 경고
description: '{{ $labels.bucket }} 버킷 사용률이 70% 이상 85% 미만으로 15분간 유지됐습니다.'
runbook_url: https://git.learn.hyeonworks.com/donghyeon.kang/project-infra/src/branch/main/docs/runbooks/2026-07-31-observability-access-cutover.md
- alert: PlatformAIStorBucketQuotaUsage
expr: >-
platform:aistor_bucket_quota_usage_percent >= 85
and
platform:aistor_bucket_quota_usage_percent < 95
for: 10m
labels:
severity: critical
annotations:
summary: AIStor 버킷 할당량 사용률 심각
description: '{{ $labels.bucket }} 버킷 사용률이 85% 이상 95% 미만으로 10분간 유지됐습니다.'
runbook_url: https://git.learn.hyeonworks.com/donghyeon.kang/project-infra/src/branch/main/docs/runbooks/2026-07-31-observability-access-cutover.md
- alert: PlatformAIStorBucketQuotaUsage
expr: platform:aistor_bucket_quota_usage_percent >= 95
for: 5m
labels:
severity: emergency
annotations:
summary: AIStor 버킷 할당량 사용률 긴급
description: '대응 절차에 따라 {{ $labels.bucket }} 버킷의 수집 경로를 중지하고 객체는 자동으로 삭제하지 마세요.'
runbook_url: https://git.learn.hyeonworks.com/donghyeon.kang/project-infra/src/branch/main/docs/runbooks/2026-07-31-observability-access-cutover.md
@@ -0,0 +1,88 @@
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: platform-verified-services
namespace: observability
labels:
app.kubernetes.io/name: platform-observability-rules
app.kubernetes.io/part-of: observability
observability.hyeonworks.com/instance: home
spec:
groups:
- name: platform.verified-services
rules:
- alert: PlatformCNPGCollectorDown
expr: >-
min by (cluster, namespace) (
cnpg_collector_up{
namespace="platform-data",
cluster="platform-postgres"
}
) == 0
for: 5m
labels:
severity: critical
annotations:
summary: CloudNativePG 수집기가 PostgreSQL 중단을 보고함
description: '{{ $labels.cluster }} CloudNativePG 수집기가 PostgreSQL 중단 상태를 5분간 보고했습니다.'
runbook_url: https://git.learn.hyeonworks.com/donghyeon.kang/project-infra/src/branch/main/docs/runbooks/2026-07-31-observability-access-cutover.md
- alert: PlatformAlloyConfigLoadFailed
expr: >-
min by (namespace) (
alloy_config_last_load_successful{namespace="observability-agent"}
) == 0
for: 5m
labels:
severity: critical
annotations:
summary: Alloy 설정 로드 실패
description: Alloy의 마지막 설정 로드 실패 상태가 5분간 유지됐습니다.
runbook_url: https://git.learn.hyeonworks.com/donghyeon.kang/project-infra/src/branch/main/docs/runbooks/2026-07-31-observability-access-cutover.md
- alert: PlatformAlloyLogDeliveryRetries
expr: >-
sum by (namespace) (
increase(loki_write_batch_retries_total{namespace="observability-agent"}[15m])
) > 0
for: 5m
labels:
severity: warning
annotations:
summary: Alloy 로그 배치 전송 재시도 발생
description: Alloy가 최근 15분 동안 Loki 쓰기 배치를 한 번 이상 재시도했습니다.
runbook_url: https://git.learn.hyeonworks.com/donghyeon.kang/project-infra/src/branch/main/docs/runbooks/2026-07-31-observability-access-cutover.md
- alert: PlatformAlloyLogEntriesDropped
expr: >-
sum by (namespace) (
increase(loki_write_dropped_entries_total{namespace="observability-agent"}[15m])
) > 0
for: 5m
labels:
severity: critical
annotations:
summary: Alloy 로그 항목 유실 발생
description: Alloy가 최근 15분 동안 재시도를 모두 소진한 뒤 로그 항목을 유실했습니다.
runbook_url: https://git.learn.hyeonworks.com/donghyeon.kang/project-infra/src/branch/main/docs/runbooks/2026-07-31-observability-access-cutover.md
- alert: PlatformLokiRuntimeConfigReloadFailed
expr: >-
min by (namespace) (
loki_runtime_config_last_reload_successful{namespace="observability"}
) == 0
for: 5m
labels:
severity: critical
annotations:
summary: Loki 런타임 설정 재로드 실패
description: Loki의 마지막 런타임 설정 재로드 실패 상태가 5분간 유지됐습니다.
runbook_url: https://git.learn.hyeonworks.com/donghyeon.kang/project-infra/src/branch/main/docs/runbooks/2026-07-31-observability-access-cutover.md
- alert: PlatformLokiWALDiskUsageHigh
expr: >-
max by (namespace) (
loki_ingester_wal_disk_usage_percent{namespace="observability"}
) >= 0.8
for: 15m
labels:
severity: warning
annotations:
summary: Loki WAL 디스크 사용률 높음
description: Loki WAL 디스크 사용률이 80% 이상으로 15분간 유지됐습니다.
runbook_url: https://git.learn.hyeonworks.com/donghyeon.kang/project-infra/src/branch/main/docs/runbooks/2026-07-31-observability-access-cutover.md
+24
View File
@@ -0,0 +1,24 @@
# Platform metric targets
This component declares the manually owned monitor resources and counterpart
NetworkPolicy needed to onboard existing platform services into the home
Prometheus instance.
The owners remain authoritative for their endpoint activation:
- Gitea chart: `/metrics` and its generated ServiceMonitor
- Keycloak Operator: management metrics and its generated ServiceMonitor
- CloudNativePG: the declared `metrics` Pod port through a manual PodMonitor
- AIStor: the v3 per-bucket usage endpoint through a manual ServiceMonitor
- packaged Traefik chart: its dedicated metrics Service and ServiceMonitor
The AIStor endpoint uses `MINIO_PROMETHEUS_AUTH_TYPE=public`, but no public
route is created. Port 9000 remains protected by the namespace default-deny and
the exact Prometheus/client allow policies. Because S3 and metrics share that
L4 port, an already-authorized S3 client can also reach the credential-less
metrics path; HTTP-path isolation is not expressible with NetworkPolicy.
The first inventory is captured before blackbox is deployed. Grafana already
exists and must be recorded explicitly as a preexisting, verify-only substrate
target; it is not owned or rolled back by this component.
@@ -0,0 +1,28 @@
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: aistor-bucket-usage
namespace: object-storage
labels:
app.kubernetes.io/name: minio-aistor
app.kubernetes.io/component: bucket-usage-metrics
app.kubernetes.io/part-of: platform
observability.hyeonworks.com/instance: home
spec:
jobLabel: aistor.min.io/objectStore
selector:
matchLabels:
aistor.min.io/objectStore: minio-aistor
namespaceSelector:
matchNames:
- object-storage
endpoints:
- port: http-minio
path: /minio/metrics/v3/cluster/usage/buckets
interval: 60s
scrapeTimeout: 10s
relabelings:
- action: keep
sourceLabels:
- __meta_kubernetes_service_name
regex: minio
@@ -0,0 +1,24 @@
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
name: platform-postgres
namespace: platform-data
labels:
app.kubernetes.io/name: platform-postgres
app.kubernetes.io/component: database-metrics
app.kubernetes.io/part-of: platform
observability.hyeonworks.com/instance: home
spec:
jobLabel: cnpg.io/cluster
selector:
matchLabels:
cnpg.io/cluster: platform-postgres
namespaceSelector:
matchNames:
- platform-data
podMetricsEndpoints:
- port: metrics
path: /metrics
interval: 30s
scrapeTimeout: 10s
@@ -0,0 +1,108 @@
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: traefik-preserve-ingress-and-allow-prometheus-metrics
namespace: kube-system
labels:
app.kubernetes.io/name: traefik
app.kubernetes.io/part-of: platform
spec:
podSelector:
matchLabels:
app.kubernetes.io/name: traefik
app.kubernetes.io/instance: traefik-kube-system
policyTypes:
- Ingress
ingress:
# Preserve the pre-policy HTTP, HTTPS, and health reachability. The new
# policy isolates the selected Pod, so omitting these ports would break the
# existing NodePort and controller health paths.
- ports:
- protocol: TCP
port: 8000
- protocol: TCP
port: 8443
- protocol: TCP
port: 8080
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: observability
podSelector:
matchLabels:
app.kubernetes.io/name: prometheus
app.kubernetes.io/instance: observability-core-kube-pr-prometheus
ports:
- protocol: TCP
port: 9100
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: observability-allow-prometheus-platform-targets
namespace: observability
labels:
app.kubernetes.io/name: prometheus
app.kubernetes.io/part-of: platform-observability
spec:
podSelector:
matchLabels:
app.kubernetes.io/name: prometheus
app.kubernetes.io/instance: observability-core-kube-pr-prometheus
policyTypes:
- Egress
egress:
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: gitea
podSelector:
matchLabels:
app.kubernetes.io/name: gitea
app.kubernetes.io/instance: gitea
ports:
- protocol: TCP
port: 3000
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: keycloak
podSelector:
matchLabels:
app: keycloak
app.kubernetes.io/instance: keycloak
app.kubernetes.io/managed-by: keycloak-operator
ports:
- protocol: TCP
port: 9000
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: platform-data
podSelector:
matchLabels:
cnpg.io/cluster: platform-postgres
ports:
- protocol: TCP
port: 9187
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: object-storage
podSelector:
matchLabels:
aistor.min.io/objectStore: minio-aistor
ports:
- protocol: TCP
port: 9000
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
podSelector:
matchLabels:
app.kubernetes.io/name: traefik
app.kubernetes.io/instance: traefik-kube-system
ports:
- protocol: TCP
port: 9100
@@ -0,0 +1,8 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- cloudnative-pg/pod-monitor.yaml
- aistor/service-monitor.yaml
- counterpart-network-policies.yaml
+15
View File
@@ -0,0 +1,15 @@
# Tempo
This root renders `grafana-community/tempo` chart `2.2.3` as one Tempo
`2.10.7` StatefulSet in `observability`.
- The image is rewritten to the reviewed amd64 digest.
- Traces use the `tempo` AIStor bucket through `tempo-object-storage`; no root
credential is referenced.
- Retention is seven days. WAL and local working data use the dedicated 5Gi
Retain Local PV.
- Only the query/metrics port 3200 and OTLP/gRPC port 4317 are exposed.
- OTLP/HTTP, Jaeger, Zipkin, metrics-generator, and tempo-query are disabled.
Do not delete the PVC or bucket during rollback. Stop the StatefulSet and
preserve its Retain volume while S3/WAL state is investigated.
@@ -0,0 +1,56 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: observability
helmGlobals:
chartHome: .helm/charts
helmCharts:
- name: tempo
repo: https://grafana-community.github.io/helm-charts
version: 2.2.3
releaseName: tempo
namespace: observability
valuesFile: values/home.yaml
includeCRDs: false
skipTests: true
images:
- name: docker.io/grafana/tempo
newName: docker.io/grafana/tempo
digest: sha256:6616b00287a4d7001951b5de117828ad5c6f93744935c1b7a5e044736373352c
patches:
- target:
group: ""
version: v1
kind: Service
name: tempo
patch: |-
- op: replace
path: /spec/ports
value:
- name: tempo-prom-metrics
port: 3200
protocol: TCP
targetPort: 3200
- name: grpc-tempo-otlp
port: 4317
protocol: TCP
targetPort: 4317
- target:
group: apps
version: v1
kind: StatefulSet
name: tempo
patch: |-
- op: replace
path: /spec/template/spec/containers/0/ports
value:
- containerPort: 3200
name: prom-metrics
protocol: TCP
- containerPort: 4317
name: otlp-grpc
protocol: TCP
@@ -0,0 +1,125 @@
global:
commonLabels:
observability.hyeonworks.com/instance: home
replicas: 1
config: |
stream_over_http_enabled: false
memberlist:
cluster_label: "{{ .Release.Name }}.{{ .Release.Namespace }}"
multitenancy_enabled: false
usage_report:
reporting_enabled: false
compactor:
compaction:
block_retention: 168h
distributor:
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
ingester:
trace_idle_period: 10s
max_block_duration: 5m
max_block_bytes: 100000000
complete_block_timeout: 15m
flush_all_on_shutdown: true
server:
http_listen_port: 3200
storage:
trace:
backend: s3
s3:
bucket: tempo
endpoint: minio.object-storage.svc.cluster.local:80
insecure: true
forcepathstyle: true
access_key: ${AWS_ACCESS_KEY_ID}
secret_key: ${AWS_SECRET_ACCESS_KEY}
wal:
path: /var/tempo/wal
querier: {}
query_frontend: {}
overrides:
defaults:
ingestion:
rate_limit_bytes: 5000000
burst_size_bytes: 10000000
max_traces_per_user: 5000
global:
max_bytes_per_trace: 5000000
tempo:
registry: docker.io
repository: grafana/tempo
tag: 2.10.7
resources:
requests:
cpu: 250m
memory: 512Mi
limits:
cpu: "1"
memory: 1Gi
memBallastSizeMbs: 0
reportingEnabled: false
metricsGenerator:
enabled: false
extraArgs:
config.expand-env: true
extraEnv:
- name: AWS_ACCESS_KEY_ID
valueFrom:
secretKeyRef:
name: tempo-object-storage
key: access-key-id
- name: AWS_SECRET_ACCESS_KEY
valueFrom:
secretKeyRef:
name: tempo-object-storage
key: secret-access-key
securityContext:
runAsNonRoot: true
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop:
- ALL
seccompProfile:
type: RuntimeDefault
tempoQuery:
enabled: false
securityContext:
runAsUser: 10001
runAsGroup: 10001
runAsNonRoot: true
fsGroup: 10001
fsGroupChangePolicy: OnRootMismatch
seccompProfile:
type: RuntimeDefault
serviceAccount:
automountServiceAccountToken: false
service:
type: ClusterIP
serviceMonitor:
enabled: true
interval: 30s
additionalLabels:
observability.hyeonworks.com/instance: home
persistence:
enabled: true
enableStatefulSetAutoDeletePVC: false
storageClassName: ssd-local-observability-tempo-retain
accessModes:
- ReadWriteOnce
size: 5Gi
podLabels:
platform.hyeonworks.com/aistor-client: "true"
+19
View File
@@ -0,0 +1,19 @@
# pgAdmin
PostgreSQL 관리 화면을 platform-admin 네임스페이스에 배포합니다.
- 주소: db-admin.learn.hyeonworks.com
- 실제 접근 범위: 호스트 Nginx에서 LAN·Tailscale만 허용
- 차트: dpage/pgadmin4-helm:9.16.0
- 이미지: dpage/pgadmin4:9.16의 amd64 digest 고정
- 저장소: 2Gi SSD Local PV, Retain
- 배포 전략: 단일 replica, Recreate
Keycloak 그룹 /platform-db-admins가 있는 사용자만 OIDC 로그인이
가능합니다. 내부 관리자는 비상용으로 유지합니다. 공유 서버 정의에는
비밀번호가 없으며, DB 비밀번호 저장도 비활성화합니다.
필요한 Secret은 다음 두 개뿐이며 Git에 저장하지 않습니다.
- pgadmin-keycloak-oidc: client-id, client-secret
- pgadmin-bootstrap: password
+42
View File
@@ -0,0 +1,42 @@
apiVersion: v1
kind: ConfigMap
metadata:
name: pgadmin-config
namespace: platform-admin
labels:
app.kubernetes.io/name: pgadmin4
app.kubernetes.io/instance: pgadmin
app.kubernetes.io/part-of: platform
data:
config_local.py: |
import os
AUTHENTICATION_SOURCES = ['oauth2', 'internal']
OAUTH2_AUTO_CREATE_USER = True
OAUTH2_CONFIG = [{
'OAUTH2_NAME': 'keycloak',
'OAUTH2_DISPLAY_NAME': 'Hyeonworks ID',
'OAUTH2_CLIENT_ID': os.environ['PGADMIN_OIDC_CLIENT_ID'],
'OAUTH2_CLIENT_SECRET': os.environ['PGADMIN_OIDC_CLIENT_SECRET'],
'OAUTH2_SERVER_METADATA_URL': (
'https://id.learn.hyeonworks.com/realms/hyeonworks/'
'.well-known/openid-configuration'
),
'OAUTH2_SCOPE': 'openid email profile',
'OAUTH2_USERNAME_CLAIM': 'preferred_username',
'OAUTH2_ADDITIONAL_CLAIMS': {
'groups': ['/platform-db-admins'],
},
}]
ALLOW_SAVE_PASSWORD = False
ENHANCED_COOKIE_PROTECTION = False
SESSION_COOKIE_SECURE = True
SESSION_COOKIE_HTTPONLY = True
SESSION_COOKIE_SAMESITE = 'Lax'
ALLOWED_HOSTS = ['db-admin.learn.hyeonworks.com']
UPGRADE_CHECK_ENABLED = False
SHOW_GRAVATAR_IMAGE = False
LLM_ENABLED = False
ENABLE_PSQL = False
ENABLE_SERVER_PASS_EXEC_CMD = False
@@ -0,0 +1,5 @@
- op: remove
path: /metadata/annotations/kubernetes.io~1ingress.class
- op: add
path: /spec/ingressClassName
value: traefik
+38
View File
@@ -0,0 +1,38 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
helmGlobals:
chartHome: .helm/charts
helmCharts:
- name: pgadmin4-helm
repo: oci://docker.io/dpage
version: 9.16.0
releaseName: pgadmin
namespace: platform-admin
valuesFile: values/home.yaml
includeCRDs: false
skipTests: true
resources:
- config.yaml
- network-policies
patches:
- patch: |-
- op: add
path: /metadata/namespace
value: platform-admin
target:
labelSelector: app.kubernetes.io/instance=pgadmin
- path: ingress-class-patch.yaml
target:
group: networking.k8s.io
version: v1
kind: Ingress
name: pgadmin
images:
- name: docker.io/dpage/pgadmin4
newName: docker.io/dpage/pgadmin4
digest: sha256:66a300a7ecdcc1f325af0c430315329bca46cd4a7067227d6899802238167c6e
@@ -0,0 +1,6 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- pgadmin.yaml
- platform-postgres.yaml

Some files were not shown because too many files have changed in this diff Show More