Add platform infrastructure configuration
This commit is contained in:
@@ -0,0 +1,229 @@
|
||||
# Gitea 서비스
|
||||
|
||||
Gitea는 같은 공식 Helm Chart와 공통 values를 사용하는 두 Kustomize 빌드 루트로
|
||||
관리한다. 업스트림 Chart 템플릿은 복제하거나 수정하지 않는다.
|
||||
|
||||
| 용도 | 빌드 루트 | 검증 산출물 | 적용 진입점 |
|
||||
| --- | --- | --- | --- |
|
||||
| 신규 설치 baseline | `services/gitea` | `gitea.yaml` | `apply-phase1-gitea.sh` |
|
||||
| OIDC·브랜딩 목표 상태 | `services/gitea/profiles/oidc` | `gitea-oidc.yaml` | `apply-gitea-oidc.sh` |
|
||||
|
||||
baseline 루트는 OIDC Secret, `id` host alias, Keycloak 전용 egress와 브랜딩을
|
||||
포함하지 않는다. OIDC profile만 공통 기반에 이 네 요소를 추가한다.
|
||||
|
||||
상태(2026-07-23): Host Nginx public discovery 전환과 OIDC-enabled Gitea
|
||||
rollout을 완료했다. 적용 manifest SHA-256은 `d25a757...a157`이며, 활성
|
||||
`keycloak` OAuth2 source, 외부 인증 전용 가입 정책, authorization-code redirect와
|
||||
아래 선언형 브랜딩의 자동 검증을 모두 통과했다. 실제 realm 사용자의 브라우저
|
||||
login/callback/logout, 비상 관리자 실제 로그인과 Pod 재시작 뒤 설정 지속성은
|
||||
별도 수용 시험으로 남아 있다.
|
||||
|
||||
## 고정된 업스트림 릴리스
|
||||
|
||||
| 항목 | 값 |
|
||||
| --- | --- |
|
||||
| Chart 저장소 | `https://dl.gitea.com/charts/` |
|
||||
| Chart | `gitea` |
|
||||
| Chart 버전 | `12.7.0` |
|
||||
| Chart 패키지 SHA-256 | `5881ef9c59400bee2d5547e77c4cd0efb925143c2f5d93fb4f38446db76b0167` |
|
||||
| Gitea 이미지 | `docker.gitea.com/gitea:1.27.0-rootless` |
|
||||
|
||||
Chart는 렌더링할 때 저장소 공통 `.gitignore`에서 제외한
|
||||
`profiles/oidc/.helm/charts`에 내려받는다. baseline 루트와 OIDC profile이 이 고정
|
||||
SHA 검증 캐시 하나를 각자의 `chartHome`으로 참조한다. values와 Patch는 모두
|
||||
`services/gitea` 아래에 있으므로 Kustomize의 기본 `RootOnly` 로드 제한을 유지한다.
|
||||
|
||||
## baseline과 OIDC 로그인 정책
|
||||
|
||||
두 프로필이 공유하는 `profiles/oidc/values/baseline.yaml`은 다음 정책을 선언한다.
|
||||
|
||||
- 자체 가입을 닫고(`DISABLE_REGISTRATION=true`) 외부 인증 전용 가입도 끈다.
|
||||
- 가입 버튼을 숨기고 기존 `gitea-admin` 로컬 비상 로그인을 유지한다.
|
||||
- PostgreSQL, PVC, probe, resource와 restricted securityContext를 공통으로 둔다.
|
||||
|
||||
`profiles/oidc/values/oidc.yaml`은 baseline 위에 다음 항목만 덮어쓴다.
|
||||
|
||||
- `APP_NAME=Hyeonworks Git`으로 브라우저 제목과 애플리케이션 이름을 고정한다.
|
||||
- `DISABLE_REGISTRATION=false`와 `ALLOW_ONLY_EXTERNAL_REGISTRATION=true`를 함께 사용해
|
||||
Keycloak 같은 외부 인증을 통한 신규 사용자 생성만 허용한다.
|
||||
- `SHOW_REGISTRATION_BUTTON=false`로 Gitea 자체 공개 가입 버튼을 숨긴다.
|
||||
- `oauth2_client.ENABLE_AUTO_REGISTRATION=true`, `USERNAME=preferred_username`,
|
||||
`OPENID_CONNECT_SCOPES="profile email"`로 OIDC 최초 로그인 계정을 생성한다.
|
||||
- baseline의 `ENABLE_PASSWORD_SIGNIN_FORM=true`를 유지해 Keycloak 장애 시에도 로컬
|
||||
비상 관리자가 로그인할 수 있게 한다.
|
||||
|
||||
`ACCOUNT_LINKING=login`은 이메일이 같은 기존 계정을 자동 병합하지 않는다. 사용자가
|
||||
기존 Gitea 자격 증명으로 한 번 더 확인해야 연결되므로, 이메일 일치만으로 계정을
|
||||
탈취하는 위험을 줄인다.
|
||||
|
||||
## 의존성과 트래픽
|
||||
|
||||
- `gitea`와 `platform-data` namespace를 먼저 적용한다.
|
||||
- 이 서비스를 적용하기 전에 `/srv/k3s/ssd/gitea`를 준비하고 `gitea-local-pv`
|
||||
기반 구성을 적용한다. `gitea-data` Claim은 `ssd-local-gitea-retain`을 통해 해당
|
||||
PV에 바인딩되며 `50Gi`를 요청한다.
|
||||
- 외부 Database endpoint는
|
||||
`platform-postgres-rw.platform-data.svc.cluster.local:5432`다.
|
||||
- Host Nginx에서 공개 TLS를 종료한다. Traefik은 HTTP를 받아 `gitea-http:3000`으로
|
||||
전달하며, 이 Ingress에는 의도적으로 TLS 섹션이 없다.
|
||||
- OIDC profile의 Gitea 서버가 discovery와 token endpoint에 접근할 때도 공개 issuer
|
||||
`https://id.learn.hyeonworks.com/realms/hyeonworks`를 사용한다. 홈 라우터의 NAT
|
||||
loopback에 의존하지 않도록 Pod의 `hostAliases`는 이 이름만 노드
|
||||
`192.168.0.107`로 해석하고, 전용 NetworkPolicy가 해당 `/32`의 TCP 443만
|
||||
허용한다. 연결은 Host Nginx TLS와 Traefik을 그대로 통과한다.
|
||||
- 첫 배포에서는 Gitea SSH, Actions, Git LFS 및 Package Registry를 비활성화한다.
|
||||
Chart `12.7.0`은 항상 SSH Service를 렌더링하므로 baseline 루트는 inline patch,
|
||||
OIDC profile은 `patches/delete-ssh-service.yaml`로 Helm 렌더링 뒤
|
||||
`gitea-ssh`를 삭제한다.
|
||||
- 공통 egress는 클러스터 DNS, 플랫폼 PostgreSQL Pod 및 공개 IPv4 HTTPS 대상으로
|
||||
제한한다. OIDC profile만 Host Nginx 노드 `/32`의 TCP 443을 추가한다. 내부
|
||||
HTTPS, SMTP, SSH remote 및 사설 네트워크 webhook을 사용하려면 향후
|
||||
NetworkPolicy를 명시적으로 변경해야 한다.
|
||||
|
||||
## 선언형 브랜딩
|
||||
|
||||
브랜딩은 Gitea 공식 custom path와 Chart의 volume 확장 필드만 사용한다.
|
||||
Chart 템플릿과 Gitea 전체 페이지 템플릿은 복제하거나 수정하지 않는다.
|
||||
|
||||
```text
|
||||
branding/
|
||||
├── public/assets/
|
||||
│ ├── css/hyeonworks.css
|
||||
│ └── img/
|
||||
│ ├── logo.svg
|
||||
│ └── favicon.svg
|
||||
└── templates/custom/
|
||||
├── header.tmpl
|
||||
└── extra_links.tmpl
|
||||
```
|
||||
|
||||
`branding/kustomization.yaml`의 `configMapGenerator`가 자산과 템플릿 ConfigMap을
|
||||
각각 생성하고 `profiles/oidc/kustomization.yaml`만 이 구성요소를 포함한다. 이름
|
||||
suffix hash를 끄지 않았으므로 파일 내용이 바뀌면 이름도 바뀌고, Kustomize가
|
||||
Deployment의 ConfigMap 참조를 같은 해시 이름으로 치환한다.
|
||||
|
||||
`profiles/oidc/values/oidc.yaml`은 공식 Chart 확장 지점을 다음과 같이 사용한다.
|
||||
|
||||
- 자산 ConfigMap을 `/data/gitea/public/assets`에 마운트한다.
|
||||
- 템플릿 ConfigMap을 `/data/gitea/templates/custom`에 마운트한다.
|
||||
- 두 volume은 rootless Gitea 주 컨테이너에만 `readOnly: true`로 마운트하며,
|
||||
init container에는 추가하지 않는다.
|
||||
|
||||
`header.tmpl`은 같은 origin의 CSS와 모바일 브라우저 색상만 추가한다.
|
||||
`extra_links.tmpl`은 실제 Gitea 내부 경로인 `/explore/organizations`만 연결한다.
|
||||
존재하지 않는 Learning 또는 Identity 서비스 링크는 미리 만들지 않는다.
|
||||
전체 `home.tmpl`은 덮어쓰지 않아 Gitea 업그레이드 때 template drift 범위를 줄인다.
|
||||
Gitea는 custom template을 시작할 때 읽으므로 변경 적용에는 Pod 재시작이 필요하다.
|
||||
|
||||
SVG 원본만 저장소에서 관리한다. Gitea가 참조하는 Open Graph, Apple touch icon 및
|
||||
구형 브라우저용 PNG는 검증된 변환 절차를 추가하기 전까지 업스트림 기본 자산을
|
||||
사용한다.
|
||||
|
||||
- [Gitea 1.27 커스터마이징 공식 문서](https://docs.gitea.com/administration/customizing-gitea)
|
||||
- [Gitea Chart 12.7.0 공식 values](https://gitea.com/gitea/helm-chart/src/tag/v12.7.0/values.yaml)
|
||||
|
||||
## Secret 계약
|
||||
|
||||
credential Secret 매니페스트나 credential 값은 이 저장소에 두지 않는다.
|
||||
`gitea-admin`과 `gitea-db-credentials`는 baseline Deployment 최초 시작 전에,
|
||||
`gitea-keycloak-oidc`는 OIDC-enabled Deployment rollout 전에 `gitea`
|
||||
namespace에 있어야 한다.
|
||||
|
||||
### `gitea-admin`
|
||||
|
||||
| 속성 | 필수 값 |
|
||||
| --- | --- |
|
||||
| Kubernetes 유형 | `kubernetes.io/basic-auth` |
|
||||
| `username` | 최초 로컬 비상 관리자 username |
|
||||
| `password` | 강력한 일회용 관리자 비밀번호 |
|
||||
|
||||
Chart는 `initialOnlyRequireReset`을 사용하므로 최초 로그인 때 초기 비밀번호를
|
||||
변경해야 하며 이후에는 조정하지 않는다.
|
||||
|
||||
### `gitea-db-credentials`
|
||||
|
||||
| 속성 | 필수 값 |
|
||||
| --- | --- |
|
||||
| Kubernetes 유형 | `kubernetes.io/basic-auth` |
|
||||
| `username` | `gitea` |
|
||||
| `password` | PostgreSQL `gitea` Role에 할당한 비밀번호 |
|
||||
|
||||
Kubernetes에서는 namespace를 넘어 Secret을 참조할 수 없다. `platform-data`에 있는
|
||||
같은 이름의 Secret은 CloudNativePG/bootstrap 사본이다. `gitea` namespace의
|
||||
Secret은 동일한 Database credential을 담아야 하며 Chart가 이 사본을 참조한다.
|
||||
|
||||
### `gitea-keycloak-oidc`
|
||||
|
||||
| 속성 | 필수 값 |
|
||||
| --- | --- |
|
||||
| Kubernetes 유형 | `Opaque` |
|
||||
| `key` | `gitea` |
|
||||
| `secret` | Keycloak이 생성한 confidential client secret |
|
||||
|
||||
`scripts/bootstrap/configure-keycloak-gitea-oidc.sh`가 Keycloak Admin REST API로
|
||||
정확한 callback URI를 가진 client를 생성하거나 갱신하고 이 Secret을 만든다.
|
||||
공식 Chart의 `gitea.oauth[].existingSecret`이 `key`와 `secret`을 환경 변수로만
|
||||
초기화 컨테이너에 전달하고, 인증 소스 이름 `keycloak`을 반복 실행 시 갱신한다.
|
||||
Secret payload는 values, 렌더 결과 또는 Git에 포함되지 않는다.
|
||||
|
||||
업스트림 Chart는 credential이 아닌 초기화 스크립트와 inline 설정이 담긴 Opaque
|
||||
Secret도 렌더링한다. 이 생성 리소스들은 고정된 Chart의 일부이며 위 세 credential
|
||||
계약의 값은 포함하지 않는다.
|
||||
|
||||
## 렌더링과 검사
|
||||
|
||||
Helm은 `PATH`에 있어야 하지만 릴리스를 직접 설치하는 데 사용하지 않는다. 플랫폼
|
||||
저장소 루트에서 고정 Chart SHA를 확인하는 검증기로 렌더링한다.
|
||||
|
||||
```sh
|
||||
PLATFORM_HELM_BIN=/home/donghyeon/.local/bin/helm \
|
||||
bash scripts/validate/render-phase1.sh
|
||||
```
|
||||
|
||||
검증기는 `gitea.yaml` baseline과 `gitea-oidc.yaml` 목표 상태를 모두 렌더한다.
|
||||
두 결과에는 각각 Gitea Deployment 하나, `gitea-http`, `gitea-data`, Traefik
|
||||
Ingress가 있어야 하고 `gitea-ssh`, NodePort, LoadBalancer와 내부 TLS는 없어야
|
||||
한다. 두 Deployment 모두 `gitea-admin`, `gitea-db-credentials`를 참조만 하고
|
||||
생성하지 않는다.
|
||||
|
||||
baseline은 `gitea-keycloak-oidc`, ID host alias, Keycloak 전용 NetworkPolicy,
|
||||
브랜딩 ConfigMap과 외부 인증 전용 가입 설정이 없어야 한다. 반대로 OIDC 산출물은
|
||||
`gitea-keycloak-oidc`를 값 노출 없이 참조하고 다음을 모두 만족해야 한다.
|
||||
|
||||
- 두 브랜딩 ConfigMap 이름에 content hash가 있고 Deployment 참조와 일치한다.
|
||||
- `/data/gitea/public/assets`와 `/data/gitea/templates/custom` 마운트는 read-only다.
|
||||
- inline 설정에 외부 인증 전용 가입, OIDC 자동 계정 생성, `ACCOUNT_LINKING=login`,
|
||||
로컬 비밀번호 로그인 유지가 모두 렌더링된다.
|
||||
- OAuth 인증 소스 이름은 callback URI의 경로와 같은 `keycloak`이고 discovery
|
||||
issuer는 `https://id.learn.hyeonworks.com/realms/hyeonworks`다.
|
||||
|
||||
## Keycloak OIDC 안전 적용
|
||||
|
||||
Host Nginx의 Keycloak discovery가 실제 JSON issuer를 반환하고 위 세 Secret 계약이
|
||||
준비된 뒤 다음 전용 진입점으로 적용한다.
|
||||
|
||||
1. Host Nginx의 `id.learn.hyeonworks.com` 정적 hold를 Keycloak proxy 후보로
|
||||
전환한다.
|
||||
2. public discovery JSON의 `issuer`가
|
||||
`https://id.learn.hyeonworks.com/realms/hyeonworks`와 정확히 같은지 확인한다.
|
||||
3. 다음 전용 스크립트로 OIDC 목표 상태와 브랜딩을 함께 적용한다.
|
||||
|
||||
```sh
|
||||
PLATFORM_HELM_BIN=/home/donghyeon/.local/bin/helm \
|
||||
bash scripts/bootstrap/apply-gitea-oidc.sh --execute
|
||||
```
|
||||
|
||||
스크립트는 public discovery와 Secret의 type·key 이름만 먼저 확인하고, 고정 Chart
|
||||
SHA 검증 결과 중 `gitea-oidc.yaml` 하나만 적용한다. 이후 Deployment·EndpointSlice,
|
||||
활성 `keycloak` OAuth2 source, 외부 인증 전용 가입 정책, OIDC authorization-code
|
||||
redirect와 브랜딩 자산 SHA-256을 검증한다. Secret 값, bearer token과 OIDC state가
|
||||
포함된 전체 Location은 출력하지 않는다.
|
||||
|
||||
2026-07-23 실제 적용에서는 manifest SHA-256 `d25a757...a157`의 rollout과 위 자동
|
||||
검사를 모두 통과했다. 이 redirect 검사는 realm 사용자의 실제 브라우저
|
||||
login/callback/logout 시험을 대신하지 않는다. self-registration은 비활성이므로
|
||||
named 관리자와 MFA를 준비한 뒤 시험 사용자를 별도로 생성한다. 로컬 비상 관리자
|
||||
실제 로그인과 Pod 재시작 뒤 OAuth source·정책·브랜딩 지속성도 별도로 확인한다.
|
||||
|
||||
기존 Gitea에 `apply-phase1-gitea.sh`를 다시 실행해 OIDC 상태를 적용하거나
|
||||
복구하지 않는다.
|
||||
@@ -0,0 +1,25 @@
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: Ingress
|
||||
metadata:
|
||||
name: gitea-http
|
||||
namespace: gitea
|
||||
labels:
|
||||
app.kubernetes.io/name: gitea
|
||||
app.kubernetes.io/instance: gitea
|
||||
app.kubernetes.io/component: source-control
|
||||
app.kubernetes.io/part-of: platform
|
||||
annotations:
|
||||
traefik.ingress.kubernetes.io/router.entrypoints: web
|
||||
spec:
|
||||
ingressClassName: traefik
|
||||
rules:
|
||||
- host: git.learn.hyeonworks.com
|
||||
http:
|
||||
paths:
|
||||
- path: /
|
||||
pathType: Prefix
|
||||
backend:
|
||||
service:
|
||||
name: gitea-http
|
||||
port:
|
||||
number: 3000
|
||||
@@ -0,0 +1,7 @@
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
|
||||
resources:
|
||||
- pvc.yaml
|
||||
- ingress.yaml
|
||||
- network-policies.yaml
|
||||
@@ -0,0 +1,161 @@
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: gitea-default-deny
|
||||
namespace: gitea
|
||||
labels:
|
||||
app.kubernetes.io/name: gitea
|
||||
app.kubernetes.io/instance: gitea
|
||||
app.kubernetes.io/part-of: platform
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: gitea
|
||||
app.kubernetes.io/instance: gitea
|
||||
policyTypes:
|
||||
- Ingress
|
||||
- Egress
|
||||
---
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: gitea-allow-traefik-http
|
||||
namespace: gitea
|
||||
labels:
|
||||
app.kubernetes.io/name: gitea
|
||||
app.kubernetes.io/instance: gitea
|
||||
app.kubernetes.io/part-of: platform
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: gitea
|
||||
app.kubernetes.io/instance: gitea
|
||||
policyTypes:
|
||||
- Ingress
|
||||
ingress:
|
||||
- from:
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
kubernetes.io/metadata.name: kube-system
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: traefik
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 3000
|
||||
---
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: gitea-allow-dns
|
||||
namespace: gitea
|
||||
labels:
|
||||
app.kubernetes.io/name: gitea
|
||||
app.kubernetes.io/instance: gitea
|
||||
app.kubernetes.io/part-of: platform
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: gitea
|
||||
app.kubernetes.io/instance: gitea
|
||||
policyTypes:
|
||||
- Egress
|
||||
egress:
|
||||
- to:
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
kubernetes.io/metadata.name: kube-system
|
||||
podSelector:
|
||||
matchLabels:
|
||||
k8s-app: kube-dns
|
||||
ports:
|
||||
- protocol: UDP
|
||||
port: 53
|
||||
- protocol: TCP
|
||||
port: 53
|
||||
---
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: gitea-allow-platform-postgres
|
||||
namespace: gitea
|
||||
labels:
|
||||
app.kubernetes.io/name: gitea
|
||||
app.kubernetes.io/instance: gitea
|
||||
app.kubernetes.io/part-of: platform
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: gitea
|
||||
app.kubernetes.io/instance: gitea
|
||||
policyTypes:
|
||||
- Egress
|
||||
egress:
|
||||
- to:
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
kubernetes.io/metadata.name: platform-data
|
||||
podSelector:
|
||||
matchLabels:
|
||||
cnpg.io/cluster: platform-postgres
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 5432
|
||||
---
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: gitea-allow-public-git-https
|
||||
namespace: gitea
|
||||
labels:
|
||||
app.kubernetes.io/name: gitea
|
||||
app.kubernetes.io/instance: gitea
|
||||
app.kubernetes.io/part-of: platform
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: gitea
|
||||
app.kubernetes.io/instance: gitea
|
||||
policyTypes:
|
||||
- Egress
|
||||
egress:
|
||||
- to:
|
||||
- ipBlock:
|
||||
cidr: 0.0.0.0/0
|
||||
except:
|
||||
- 10.0.0.0/8
|
||||
- 172.16.0.0/12
|
||||
- 192.168.0.0/16
|
||||
- 169.254.0.0/16
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 443
|
||||
---
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: gitea-allow-prometheus-metrics
|
||||
namespace: gitea
|
||||
labels:
|
||||
app.kubernetes.io/name: gitea
|
||||
app.kubernetes.io/instance: gitea
|
||||
app.kubernetes.io/part-of: platform
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: gitea
|
||||
app.kubernetes.io/instance: gitea
|
||||
policyTypes:
|
||||
- Ingress
|
||||
ingress:
|
||||
- from:
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
kubernetes.io/metadata.name: observability
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: prometheus
|
||||
app.kubernetes.io/instance: observability-core-kube-pr-prometheus
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 3000
|
||||
@@ -0,0 +1,19 @@
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: gitea-data
|
||||
namespace: gitea
|
||||
labels:
|
||||
app.kubernetes.io/name: gitea
|
||||
app.kubernetes.io/instance: gitea
|
||||
app.kubernetes.io/component: source-control
|
||||
app.kubernetes.io/part-of: platform
|
||||
spec:
|
||||
accessModes:
|
||||
- ReadWriteOnce
|
||||
volumeMode: Filesystem
|
||||
storageClassName: ssd-local-gitea-retain
|
||||
volumeName: gitea-local-pv
|
||||
resources:
|
||||
requests:
|
||||
storage: 50Gi
|
||||
@@ -0,0 +1,15 @@
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
|
||||
namespace: gitea
|
||||
|
||||
configMapGenerator:
|
||||
- name: gitea-branding-assets
|
||||
files:
|
||||
- hyeonworks.css=public/assets/css/hyeonworks.css
|
||||
- logo.svg=public/assets/img/logo.svg
|
||||
- favicon.svg=public/assets/img/favicon.svg
|
||||
- name: gitea-branding-templates
|
||||
files:
|
||||
- header.tmpl=templates/custom/header.tmpl
|
||||
- extra_links.tmpl=templates/custom/extra_links.tmpl
|
||||
@@ -0,0 +1,59 @@
|
||||
:root {
|
||||
--hyeonworks-accent: #2563eb;
|
||||
--hyeonworks-accent-hover: #1d4ed8;
|
||||
--hyeonworks-focus: rgba(37, 99, 235, 0.35);
|
||||
}
|
||||
|
||||
#navbar-logo {
|
||||
gap: 0.5rem;
|
||||
}
|
||||
|
||||
#navbar-logo img {
|
||||
width: 2rem;
|
||||
height: 2rem;
|
||||
}
|
||||
|
||||
#navbar-logo::after {
|
||||
content: "Hyeonworks Git";
|
||||
font-weight: 700;
|
||||
letter-spacing: -0.015em;
|
||||
white-space: nowrap;
|
||||
}
|
||||
|
||||
#navbar .hw-brand-link {
|
||||
font-weight: 600;
|
||||
}
|
||||
|
||||
#navbar .hw-brand-link:hover,
|
||||
#navbar .hw-brand-link:focus-visible {
|
||||
color: var(--hyeonworks-accent);
|
||||
}
|
||||
|
||||
#navbar .hw-brand-link:focus-visible {
|
||||
border-radius: 0.375rem;
|
||||
outline: 3px solid var(--hyeonworks-focus);
|
||||
outline-offset: -3px;
|
||||
}
|
||||
|
||||
.ui.primary.button,
|
||||
.ui.primary.buttons .button {
|
||||
background-color: var(--hyeonworks-accent);
|
||||
}
|
||||
|
||||
.ui.primary.button:hover,
|
||||
.ui.primary.buttons .button:hover {
|
||||
background-color: var(--hyeonworks-accent-hover);
|
||||
}
|
||||
|
||||
@media (max-width: 767.98px) {
|
||||
#navbar-logo::after {
|
||||
display: none;
|
||||
}
|
||||
}
|
||||
|
||||
@media (prefers-reduced-motion: reduce) {
|
||||
#navbar .hw-brand-link,
|
||||
.ui.button {
|
||||
transition: none;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,12 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 64 64" role="img" aria-labelledby="title">
|
||||
<title id="title">Hyeonworks Git</title>
|
||||
<defs>
|
||||
<linearGradient id="brand-gradient" x1="6" y1="6" x2="58" y2="58" gradientUnits="userSpaceOnUse">
|
||||
<stop stop-color="#38BDF8"/>
|
||||
<stop offset="1" stop-color="#4F46E5"/>
|
||||
</linearGradient>
|
||||
</defs>
|
||||
<rect x="3" y="3" width="58" height="58" rx="16" fill="url(#brand-gradient)"/>
|
||||
<path d="M19 17v30M45 17v30M19 32h26" fill="none" stroke="#fff" stroke-width="6" stroke-linecap="round"/>
|
||||
<path d="m14 25-6 7 6 7M50 25l6 7-6 7" fill="none" stroke="#BAE6FD" stroke-width="3" stroke-linecap="round" stroke-linejoin="round"/>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 695 B |
@@ -0,0 +1,13 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 64 64" role="img" aria-labelledby="title description">
|
||||
<title id="title">Hyeonworks Git</title>
|
||||
<desc id="description">겹쳐진 코드 괄호와 H 문자를 사용한 Hyeonworks Git 로고</desc>
|
||||
<defs>
|
||||
<linearGradient id="brand-gradient" x1="6" y1="6" x2="58" y2="58" gradientUnits="userSpaceOnUse">
|
||||
<stop stop-color="#38BDF8"/>
|
||||
<stop offset="1" stop-color="#4F46E5"/>
|
||||
</linearGradient>
|
||||
</defs>
|
||||
<rect x="3" y="3" width="58" height="58" rx="16" fill="url(#brand-gradient)"/>
|
||||
<path d="M19 17v30M45 17v30M19 32h26" fill="none" stroke="#fff" stroke-width="6" stroke-linecap="round"/>
|
||||
<path d="m14 25-6 7 6 7M50 25l6 7-6 7" fill="none" stroke="#BAE6FD" stroke-width="3" stroke-linecap="round" stroke-linejoin="round"/>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 810 B |
@@ -0,0 +1 @@
|
||||
<a class="item hw-brand-link" href="{{AppSubUrl}}/explore/organizations">조직</a>
|
||||
@@ -0,0 +1,2 @@
|
||||
<meta name="theme-color" content="#0f172a">
|
||||
<link rel="stylesheet" href="{{AssetUrlPrefix}}/css/hyeonworks.css">
|
||||
@@ -0,0 +1,40 @@
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
|
||||
namespace: gitea
|
||||
|
||||
helmGlobals:
|
||||
chartHome: profiles/oidc/.helm/charts
|
||||
|
||||
helmCharts:
|
||||
- name: gitea
|
||||
repo: https://dl.gitea.com/charts/
|
||||
version: 12.7.0
|
||||
releaseName: gitea
|
||||
namespace: gitea
|
||||
valuesFile: profiles/oidc/values/baseline.yaml
|
||||
includeCRDs: false
|
||||
skipTests: true
|
||||
|
||||
resources:
|
||||
- base
|
||||
|
||||
patches:
|
||||
- path: profiles/oidc/patches/servicemonitor-job-label-patch.yaml
|
||||
target:
|
||||
group: monitoring.coreos.com
|
||||
version: v1
|
||||
kind: ServiceMonitor
|
||||
name: gitea
|
||||
namespace: gitea
|
||||
- target:
|
||||
group: ""
|
||||
version: v1
|
||||
kind: Service
|
||||
name: gitea-ssh
|
||||
patch: |-
|
||||
$patch: delete
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: gitea-ssh
|
||||
@@ -0,0 +1,39 @@
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
|
||||
namespace: gitea
|
||||
|
||||
helmGlobals:
|
||||
chartHome: .helm/charts
|
||||
|
||||
helmCharts:
|
||||
- name: gitea
|
||||
repo: https://dl.gitea.com/charts/
|
||||
version: 12.7.0
|
||||
releaseName: gitea
|
||||
namespace: gitea
|
||||
valuesFile: values/baseline.yaml
|
||||
additionalValuesFiles:
|
||||
- values/oidc.yaml
|
||||
includeCRDs: false
|
||||
skipTests: true
|
||||
|
||||
resources:
|
||||
- ../../base
|
||||
- ../../branding
|
||||
- network-policy.yaml
|
||||
|
||||
patches:
|
||||
- path: patches/servicemonitor-job-label-patch.yaml
|
||||
target:
|
||||
group: monitoring.coreos.com
|
||||
version: v1
|
||||
kind: ServiceMonitor
|
||||
name: gitea
|
||||
namespace: gitea
|
||||
- path: patches/delete-ssh-service.yaml
|
||||
target:
|
||||
group: ""
|
||||
version: v1
|
||||
kind: Service
|
||||
name: gitea-ssh
|
||||
@@ -0,0 +1,24 @@
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: gitea-allow-host-nginx-keycloak
|
||||
namespace: gitea
|
||||
labels:
|
||||
app.kubernetes.io/name: gitea
|
||||
app.kubernetes.io/instance: gitea
|
||||
app.kubernetes.io/part-of: platform
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: gitea
|
||||
app.kubernetes.io/instance: gitea
|
||||
policyTypes:
|
||||
- Egress
|
||||
egress:
|
||||
- to:
|
||||
- ipBlock:
|
||||
# CoreDNS split DNS resolves the public issuer name to Host Nginx.
|
||||
cidr: 192.168.0.107/32
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 443
|
||||
@@ -0,0 +1,6 @@
|
||||
$patch: delete
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: gitea-ssh
|
||||
namespace: gitea
|
||||
@@ -0,0 +1,7 @@
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: ServiceMonitor
|
||||
metadata:
|
||||
name: gitea
|
||||
namespace: gitea
|
||||
spec:
|
||||
jobLabel: app.kubernetes.io/name
|
||||
@@ -0,0 +1,189 @@
|
||||
namespace: gitea
|
||||
|
||||
replicaCount: 1
|
||||
|
||||
image:
|
||||
registry: docker.gitea.com
|
||||
repository: gitea
|
||||
tag: "1.27.0"
|
||||
pullPolicy: IfNotPresent
|
||||
rootless: true
|
||||
|
||||
# The official chart applies this context to its built-in init containers,
|
||||
# configure-gitea, and the main container.
|
||||
containerSecurityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
capabilities:
|
||||
drop:
|
||||
- ALL
|
||||
runAsNonRoot: true
|
||||
seccompProfile:
|
||||
type: RuntimeDefault
|
||||
|
||||
# A Local RWO volume cannot be attached to an old and a replacement Pod at the
|
||||
# same time. Terminate the old Pod before creating its replacement.
|
||||
strategy:
|
||||
type: RollingUpdate
|
||||
rollingUpdate:
|
||||
maxSurge: 0
|
||||
maxUnavailable: 1
|
||||
|
||||
service:
|
||||
http:
|
||||
type: ClusterIP
|
||||
port: 3000
|
||||
clusterIP: ""
|
||||
nodePort: null
|
||||
ssh:
|
||||
type: ClusterIP
|
||||
port: 22
|
||||
clusterIP: ""
|
||||
nodePort: null
|
||||
hostPort: null
|
||||
|
||||
# Routing is owned by ../../base/ingress.yaml. Host Nginx terminates TLS before
|
||||
# forwarding requests to Traefik's HTTP entrypoint.
|
||||
ingress:
|
||||
enabled: false
|
||||
|
||||
route:
|
||||
enabled: false
|
||||
|
||||
gatewayAPI:
|
||||
enabled: false
|
||||
|
||||
persistence:
|
||||
enabled: true
|
||||
create: false
|
||||
mount: true
|
||||
claimName: gitea-data
|
||||
|
||||
gitea:
|
||||
admin:
|
||||
existingSecret: gitea-admin
|
||||
email: admin@hyeonworks.com
|
||||
passwordMode: initialOnlyRequireReset
|
||||
|
||||
config:
|
||||
server:
|
||||
PROTOCOL: http
|
||||
DOMAIN: git.learn.hyeonworks.com
|
||||
ROOT_URL: https://git.learn.hyeonworks.com/
|
||||
HTTP_PORT: 3000
|
||||
DISABLE_SSH: true
|
||||
START_SSH_SERVER: false
|
||||
LFS_START_SERVER: false
|
||||
|
||||
# Baseline allows only the pre-provisioned local emergency administrator.
|
||||
# The guarded OIDC profile overrides the first two settings.
|
||||
service:
|
||||
DISABLE_REGISTRATION: true
|
||||
ALLOW_ONLY_EXTERNAL_REGISTRATION: false
|
||||
SHOW_REGISTRATION_BUTTON: false
|
||||
ENABLE_PASSWORD_SIGNIN_FORM: true
|
||||
|
||||
database:
|
||||
DB_TYPE: postgres
|
||||
HOST: platform-postgres-rw.platform-data.svc.cluster.local:5432
|
||||
NAME: gitea
|
||||
SSL_MODE: disable
|
||||
|
||||
session:
|
||||
PROVIDER: db
|
||||
cache:
|
||||
ADAPTER: memory
|
||||
queue:
|
||||
TYPE: level
|
||||
|
||||
actions:
|
||||
ENABLED: false
|
||||
packages:
|
||||
ENABLED: false
|
||||
repository:
|
||||
DISABLED_REPO_UNITS: "repo.packages,repo.actions"
|
||||
|
||||
additionalConfigFromEnvs:
|
||||
- name: GITEA__DATABASE__USER
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: gitea-db-credentials
|
||||
key: username
|
||||
- name: GITEA__DATABASE__PASSWD
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: gitea-db-credentials
|
||||
key: password
|
||||
|
||||
metrics:
|
||||
enabled: true
|
||||
serviceMonitor:
|
||||
enabled: true
|
||||
additionalLabels:
|
||||
observability.hyeonworks.com/instance: home
|
||||
interval: 30s
|
||||
scrapeTimeout: 10s
|
||||
|
||||
livenessProbe:
|
||||
enabled: true
|
||||
httpGet:
|
||||
path: /api/healthz
|
||||
port: http
|
||||
initialDelaySeconds: 200
|
||||
timeoutSeconds: 5
|
||||
periodSeconds: 10
|
||||
successThreshold: 1
|
||||
failureThreshold: 10
|
||||
|
||||
readinessProbe:
|
||||
enabled: true
|
||||
httpGet:
|
||||
path: /api/healthz
|
||||
port: http
|
||||
initialDelaySeconds: 5
|
||||
timeoutSeconds: 5
|
||||
periodSeconds: 10
|
||||
successThreshold: 1
|
||||
failureThreshold: 3
|
||||
|
||||
startupProbe:
|
||||
enabled: true
|
||||
httpGet:
|
||||
path: /api/healthz
|
||||
port: http
|
||||
initialDelaySeconds: 10
|
||||
timeoutSeconds: 5
|
||||
periodSeconds: 10
|
||||
successThreshold: 1
|
||||
failureThreshold: 60
|
||||
|
||||
postgresql:
|
||||
enabled: false
|
||||
|
||||
postgresql-ha:
|
||||
enabled: false
|
||||
|
||||
valkey:
|
||||
enabled: false
|
||||
|
||||
valkey-cluster:
|
||||
enabled: false
|
||||
|
||||
resources:
|
||||
requests:
|
||||
cpu: 250m
|
||||
memory: 512Mi
|
||||
limits:
|
||||
cpu: "2"
|
||||
memory: 2Gi
|
||||
|
||||
initContainers:
|
||||
resources:
|
||||
requests:
|
||||
cpu: 100m
|
||||
memory: 128Mi
|
||||
limits:
|
||||
cpu: 500m
|
||||
memory: 512Mi
|
||||
|
||||
test:
|
||||
enabled: false
|
||||
@@ -0,0 +1,49 @@
|
||||
extraVolumes:
|
||||
- name: branding-assets
|
||||
configMap:
|
||||
name: gitea-branding-assets
|
||||
defaultMode: 292
|
||||
items:
|
||||
- key: hyeonworks.css
|
||||
path: css/hyeonworks.css
|
||||
- key: logo.svg
|
||||
path: img/logo.svg
|
||||
- key: favicon.svg
|
||||
path: img/favicon.svg
|
||||
- name: branding-templates
|
||||
configMap:
|
||||
name: gitea-branding-templates
|
||||
defaultMode: 292
|
||||
items:
|
||||
- key: header.tmpl
|
||||
path: header.tmpl
|
||||
- key: extra_links.tmpl
|
||||
path: extra_links.tmpl
|
||||
|
||||
extraContainerVolumeMounts:
|
||||
- name: branding-assets
|
||||
mountPath: /data/gitea/public/assets
|
||||
readOnly: true
|
||||
- name: branding-templates
|
||||
mountPath: /data/gitea/templates/custom
|
||||
readOnly: true
|
||||
|
||||
gitea:
|
||||
oauth:
|
||||
- name: keycloak
|
||||
provider: openidConnect
|
||||
existingSecret: gitea-keycloak-oidc
|
||||
autoDiscoverUrl: https://id.learn.hyeonworks.com/realms/hyeonworks/.well-known/openid-configuration
|
||||
|
||||
config:
|
||||
APP_NAME: Hyeonworks Git
|
||||
|
||||
service:
|
||||
DISABLE_REGISTRATION: false
|
||||
ALLOW_ONLY_EXTERNAL_REGISTRATION: true
|
||||
|
||||
oauth2_client:
|
||||
ENABLE_AUTO_REGISTRATION: true
|
||||
USERNAME: preferred_username
|
||||
ACCOUNT_LINKING: login
|
||||
OPENID_CONNECT_SCOPES: "profile email"
|
||||
@@ -0,0 +1,111 @@
|
||||
# Keycloak 서비스
|
||||
|
||||
이 Kustomization은 `keycloak` namespace에 Keycloak 26.7.0 인스턴스 하나를
|
||||
선언한다. 이 인스턴스는 독립적인 OIDC Provider이며 인증 proxy나 Traefik
|
||||
ForwardAuth middleware가 아니다.
|
||||
|
||||
외부 요청 경로는 다음과 같다.
|
||||
|
||||
```text
|
||||
Client -> Host Nginx (TLS) -> Traefik web/HTTP -> keycloak-service:8080
|
||||
```
|
||||
|
||||
상태(2026-07-23): Host Nginx의 Keycloak 공개 전환과 Gitea OIDC·브랜딩 rollout을
|
||||
완료했다. Nginx 후보 SHA-256은
|
||||
`5c5cd74b4992f537fd27c50cf2209573a80a9904e0b19b58c3154717be6ff4a5`, 백업은
|
||||
`/etc/nginx/sites-available/learn-services.before-keycloak-20260723160519`이며
|
||||
discovery issuer와 Gitea OAuth source·정책·redirect·브랜딩 자동 검증을 통과했다.
|
||||
실제 realm 사용자의 브라우저 login/callback/logout은 별도 수용 시험으로 남아 있다.
|
||||
|
||||
Keycloak이 생성하는 Ingress는 비활성화한다. 저장소에서 관리하는 Ingress는
|
||||
Traefik의 내부 HTTP entrypoint를 통해 `id.learn.hyeonworks.com`만 노출한다. 관리
|
||||
포트 `9000`에는 Ingress나 NodePort가 없으며, NetworkPolicy의 출발지는 namespace
|
||||
범위 Operator로 제한한다. TLS는 Host Nginx에서만 종료하므로 Keycloak은 Traefik이
|
||||
선택한 HTTP 경로의 `xforwarded` header만 수락한다.
|
||||
|
||||
## 필수 Secret 계약
|
||||
|
||||
credential Secret 매니페스트나 값은 이 저장소에 보관하지 않는다. Keycloak
|
||||
리소스를 생성하기 전에 동일하게 생성한 비밀번호를 사용하는
|
||||
`keycloak-db-credentials`를 다음 두 namespace에 생성한다.
|
||||
|
||||
- `platform-data`: CloudNativePG의 `DatabaseRole`에서 사용
|
||||
- `keycloak`: Keycloak 서버에서 사용
|
||||
|
||||
각 Secret의 유형은 `kubernetes.io/basic-auth`, username은 정확히 `keycloak`이어야
|
||||
하며 `username`과 `password` 키가 있어야 한다. `platform-data` 사본에는
|
||||
`cnpg.io/reload: "true"` label도 있어야 한다.
|
||||
|
||||
인스턴스를 처음 조정할 때 Operator가 `keycloak` namespace에
|
||||
`keycloak-initial-admin`을 생성한다. 이 생성된 Secret은 소스 관리하지 않는다.
|
||||
서비스를 운영 준비 완료 상태로 판단하기 전에 bootstrap credential을 교체하고
|
||||
MFA를 활성화한다.
|
||||
|
||||
## 의존성과 렌더링 순서
|
||||
|
||||
1. `keycloak`, `platform-data`, `cnpg-system` namespace가 존재한다.
|
||||
2. CloudNativePG와 `platform-postgres`가 Ready 상태다.
|
||||
3. `keycloak-db-credentials`의 두 사본이 모두 존재한다.
|
||||
4. Keycloak Operator가 설치되어 Ready 상태다.
|
||||
5. 이 서비스 Kustomization을 적용한다.
|
||||
6. Ready 상태와 내부 Traefik routing을 확인한 후에만 Host Nginx에
|
||||
`id.learn.hyeonworks.com`을 설정한다.
|
||||
|
||||
Helm 없이 렌더링한다.
|
||||
|
||||
```bash
|
||||
kubectl kustomize services/keycloak
|
||||
```
|
||||
|
||||
배포 후 다음을 검증한다.
|
||||
|
||||
```bash
|
||||
kubectl -n keycloak wait --for=condition=Ready \
|
||||
keycloak/keycloak --timeout=15m
|
||||
kubectl -n keycloak get keycloak,pod,service,ingress,networkpolicy
|
||||
curl --fail --silent --show-error \
|
||||
https://id.learn.hyeonworks.com/realms/hyeonworks/.well-known/openid-configuration \
|
||||
| jq --exit-status \
|
||||
--arg issuer 'https://id.learn.hyeonworks.com/realms/hyeonworks' \
|
||||
'.issuer == $issuer'
|
||||
```
|
||||
|
||||
마지막 검사는 HTTP 성공 여부만 보지 않고 JSON의 `issuer`가 공개 issuer와 정확히
|
||||
같은지 확인한다. 따라서 전환 전 Host Nginx의 정적 hold 응답이 HTTP 200을
|
||||
반환하더라도 성공으로 오인하지 않는다.
|
||||
|
||||
TCP 9000을 대상으로 하는 Ingress, NodePort 또는 LoadBalancer가 없는지도 별도로
|
||||
확인한다.
|
||||
|
||||
## 수동 bootstrap 진입점
|
||||
|
||||
Keycloak-only 적용은 AIStor와 분리한다. 저장소 루트에서 다음 순서를 사용하며
|
||||
Secret payload는 명령 인자나 출력에 넣지 않는다.
|
||||
|
||||
```sh
|
||||
kubectl apply --filename=infrastructure/namespaces/phase2/keycloak.yaml
|
||||
bash scripts/bootstrap/create-keycloak-secrets.sh --generate --execute
|
||||
bash scripts/bootstrap/apply-keycloak.sh --execute
|
||||
bash scripts/bootstrap/configure-keycloak-gitea-oidc.sh --execute
|
||||
```
|
||||
|
||||
각 스크립트는 context와 적용 범위를 다시 확인한다. 마지막 스크립트는
|
||||
`hyeonworks` realm, confidential Gitea client와 Gitea OIDC Secret을 구성한다.
|
||||
|
||||
public discovery 전환은 클러스터 적용과 분리된 root 작업이며 실제 전환을
|
||||
완료했다.
|
||||
|
||||
```sh
|
||||
sudo bash scripts/bootstrap/apply-host-nginx-keycloak.sh --execute
|
||||
```
|
||||
|
||||
상세 명령·출력·실패 경계는
|
||||
[중앙 OIDC 실행 원장](../../../docs/platform/runbooks/2026-07-23-keycloak-gitea-oidc-cutover.md)에
|
||||
기록한다.
|
||||
|
||||
## 공식 참고 문서
|
||||
|
||||
- <https://www.keycloak.org/operator/basic-deployment>
|
||||
- <https://www.keycloak.org/operator/advanced-configuration>
|
||||
- <https://www.keycloak.org/server/reverseproxy>
|
||||
- <https://www.keycloak.org/server/hostname>
|
||||
@@ -0,0 +1,23 @@
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: Ingress
|
||||
metadata:
|
||||
name: keycloak-http
|
||||
labels:
|
||||
app.kubernetes.io/name: keycloak
|
||||
app.kubernetes.io/instance: keycloak
|
||||
app.kubernetes.io/component: identity-provider
|
||||
annotations:
|
||||
traefik.ingress.kubernetes.io/router.entrypoints: web
|
||||
spec:
|
||||
ingressClassName: traefik
|
||||
rules:
|
||||
- host: id.learn.hyeonworks.com
|
||||
http:
|
||||
paths:
|
||||
- path: /
|
||||
pathType: Prefix
|
||||
backend:
|
||||
service:
|
||||
name: keycloak-service
|
||||
port:
|
||||
number: 8080
|
||||
@@ -0,0 +1,94 @@
|
||||
apiVersion: k8s.keycloak.org/v2beta1
|
||||
kind: Keycloak
|
||||
metadata:
|
||||
name: keycloak
|
||||
labels:
|
||||
app.kubernetes.io/name: keycloak
|
||||
app.kubernetes.io/instance: keycloak
|
||||
app.kubernetes.io/component: identity-provider
|
||||
spec:
|
||||
instances: 1
|
||||
|
||||
db:
|
||||
vendor: postgres
|
||||
host: platform-postgres-rw.platform-data.svc.cluster.local
|
||||
port: 5432
|
||||
database: keycloak
|
||||
schema: public
|
||||
usernameSecret:
|
||||
name: keycloak-db-credentials
|
||||
key: username
|
||||
passwordSecret:
|
||||
name: keycloak-db-credentials
|
||||
key: password
|
||||
poolInitialSize: 5
|
||||
poolMinSize: 5
|
||||
poolMaxSize: 20
|
||||
|
||||
hostname:
|
||||
hostname: https://id.learn.hyeonworks.com
|
||||
strict: true
|
||||
|
||||
http:
|
||||
httpEnabled: true
|
||||
httpPort: 8080
|
||||
serviceHttpPort: 8080
|
||||
|
||||
proxy:
|
||||
headers: xforwarded
|
||||
|
||||
ingress:
|
||||
enabled: false
|
||||
|
||||
additionalOptions:
|
||||
- name: metrics-enabled
|
||||
value: "true"
|
||||
|
||||
serviceMonitor:
|
||||
enabled: true
|
||||
interval: 30s
|
||||
scrapeTimeout: 10s
|
||||
labels:
|
||||
observability.hyeonworks.com/instance: home
|
||||
|
||||
networkPolicy:
|
||||
enabled: true
|
||||
http:
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
kubernetes.io/metadata.name: kube-system
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: traefik
|
||||
https:
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
kubernetes.io/metadata.name: kube-system
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: traefik
|
||||
management:
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
kubernetes.io/metadata.name: keycloak
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: keycloak-operator
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
kubernetes.io/metadata.name: observability
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: prometheus
|
||||
app.kubernetes.io/instance: observability-core-kube-pr-prometheus
|
||||
|
||||
resources:
|
||||
requests:
|
||||
cpu: 500m
|
||||
memory: 1Gi
|
||||
limits:
|
||||
cpu: "2"
|
||||
memory: 2Gi
|
||||
|
||||
transaction:
|
||||
xaEnabled: false
|
||||
@@ -0,0 +1,6 @@
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
|
||||
resources:
|
||||
- keycloak.yaml
|
||||
- ingress.yaml
|
||||
@@ -0,0 +1,13 @@
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
|
||||
namespace: keycloak
|
||||
|
||||
resources:
|
||||
- base
|
||||
|
||||
labels:
|
||||
- pairs:
|
||||
app.kubernetes.io/part-of: platform
|
||||
app.kubernetes.io/managed-by: kustomize
|
||||
includeSelectors: false
|
||||
@@ -0,0 +1,6 @@
|
||||
# 홈 환경 Overlay
|
||||
|
||||
현재 홈 클러스터 구성은 `../base`의 안정적인 리소스로 직접 표현한다.
|
||||
`services/keycloak`을 유일한 서비스 빌드 루트로 유지한다. 향후 홈 환경 전용
|
||||
Patch는 이곳에 추가하고, 독립적으로 렌더링하는 두 번째 애플리케이션을 만들지 말고
|
||||
서비스 루트 Kustomization에서 참조한다.
|
||||
@@ -0,0 +1,81 @@
|
||||
# MinIO AIStor ObjectStore
|
||||
|
||||
상태(2026-07-23):
|
||||
|
||||
- baseline ObjectStore와 인증 S3 스모크: 실제 적용 완료
|
||||
- admin-oidc profile과 비공개 Console Ingress: 선언형 구현·dry-run 완료, live 적용 대기
|
||||
|
||||
공식 Chart를 수정하지 않고 Kustomize가 Helm을 렌더한다.
|
||||
|
||||
## 빌드 루트
|
||||
|
||||
- services/minio-aistor: 현재 live baseline 진입점
|
||||
- services/minio-aistor/base/deployment: 공통 Helm baseline과 values
|
||||
- services/minio-aistor/profiles/admin-oidc: baseline에 OIDC, Console Ingress와
|
||||
추가 NetworkPolicy를 결합한 진입점
|
||||
|
||||
root와 profile이 같은 형제 baseline을 참조하므로 profile이 조상 root를 다시
|
||||
포함하는 Kustomize 순환이 없다.
|
||||
|
||||
## 고정 원본
|
||||
|
||||
| 항목 | 값 |
|
||||
|---|---|
|
||||
| Chart 저장소 | https://helm.min.io/ |
|
||||
| Chart | aistor-objectstore |
|
||||
| 버전 | 1.0.16 |
|
||||
| archive SHA-256 | 50ffa6a4e014cdc48566b237593baab41d039f103cdc50c1b1ac173b8d8bf71e |
|
||||
|
||||
검증기가 만드는 임시 Chart cache는
|
||||
base/deployment/.helm/charts/aistor-objectstore-1.0.16 아래에만 존재하고 종료 시
|
||||
제거한다.
|
||||
|
||||
## baseline 계약
|
||||
|
||||
- namespace: object-storage
|
||||
- ObjectStore: minio-aistor
|
||||
- pool/server/volume: 각각 1
|
||||
- 저장소: aistor-local-xfs-retain, 900Gi RWO
|
||||
- S3와 Console Service: ClusterIP
|
||||
- baseline Ingress, NodePort, LoadBalancer: 없음
|
||||
- pvcProtection과 Retain Local PV
|
||||
- 기존 root 계정: 비상용으로 유지
|
||||
|
||||
Standalone이므로 erasure-code 이중화와 노드 failover는 없다.
|
||||
|
||||
## Secret 계약
|
||||
|
||||
baseline:
|
||||
|
||||
- object-storage/aistor-root-configuration: config.env
|
||||
|
||||
admin-oidc profile:
|
||||
|
||||
- object-storage/aistor-keycloak-oidc: client-id, client-secret
|
||||
|
||||
평문, Base64, checksum을 Git과 문서에 기록하지 않는다.
|
||||
|
||||
## 렌더와 적용
|
||||
|
||||
baseline 정적 렌더:
|
||||
|
||||
kubectl kustomize --enable-helm services/minio-aistor
|
||||
|
||||
admin profile 전체 정적 검증:
|
||||
|
||||
PLATFORM_HELM_BIN=/home/donghyeon/.local/bin/helm \
|
||||
bash scripts/validate/render-admin-services.sh
|
||||
|
||||
admin profile 실제 적용은 private DNS, admin 인증서와 Keycloak client/group 준비
|
||||
뒤 apply-admin-services.sh가 수행한다. S3 API에는 Ingress를 만들지 않는다.
|
||||
|
||||
현재 baseline은 Initialized/green이고 data0-minio-aistor-pool-0-0 PVC가
|
||||
aistor-data-local-pv에 Bound다. S3 회귀 검사는 다음과 같다.
|
||||
|
||||
bash scripts/validate/aistor-s3-smoke.sh --execute
|
||||
|
||||
## 공식 문서
|
||||
|
||||
- https://docs.min.io/aistor/reference/kubernetes/object-store-helm-chart/
|
||||
- https://docs.min.io/aistor/reference/kubernetes/pvc-protection/
|
||||
- https://docs.min.io/aistor/administration/iam/identity/keycloak-identity/
|
||||
@@ -0,0 +1,7 @@
|
||||
# 기본 확장 지점
|
||||
|
||||
공통 ObjectStore Helm baseline은 base/deployment에 있다. 저장소, resource,
|
||||
ClusterIP와 root 비상 계정 경계만 이 계층이 소유한다.
|
||||
|
||||
OIDC 환경 변수와 Console Ingress는 profiles/admin-oidc에서만 추가한다.
|
||||
Chart 템플릿을 저장소에 복제하거나 직접 수정하지 않는다.
|
||||
@@ -0,0 +1,6 @@
|
||||
# AIStor ObjectStore baseline
|
||||
|
||||
AIStor ObjectStore의 공통 Helm baseline입니다. 저장소, root 비상 계정,
|
||||
리소스 제한과 ClusterIP 경계만 정의합니다.
|
||||
|
||||
Console Ingress와 Keycloak OIDC는 profiles/admin-oidc에서만 추가합니다.
|
||||
@@ -0,0 +1,17 @@
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
|
||||
namespace: object-storage
|
||||
|
||||
helmGlobals:
|
||||
chartHome: .helm/charts
|
||||
|
||||
helmCharts:
|
||||
- name: aistor-objectstore
|
||||
repo: https://helm.min.io/
|
||||
version: 1.0.16
|
||||
releaseName: minio-aistor
|
||||
namespace: object-storage
|
||||
valuesFile: values/home.yaml
|
||||
includeCRDs: false
|
||||
skipTests: true
|
||||
@@ -0,0 +1,67 @@
|
||||
secrets:
|
||||
name: aistor-root-configuration
|
||||
existingSecret: true
|
||||
disableRoot: false
|
||||
|
||||
objectStore:
|
||||
name: minio-aistor
|
||||
|
||||
configuration:
|
||||
name: aistor-root-configuration
|
||||
|
||||
pools:
|
||||
- name: pool-0
|
||||
servers: 1
|
||||
volumesPerServer: 1
|
||||
storageNamePrefix: data
|
||||
size: 900Gi
|
||||
storageClassName: aistor-local-xfs-retain
|
||||
nodeSelector:
|
||||
kubernetes.io/hostname: donghyeon-system-product-name
|
||||
resources:
|
||||
requests:
|
||||
cpu: 500m
|
||||
memory: 2Gi
|
||||
limits:
|
||||
cpu: "3"
|
||||
memory: 8Gi
|
||||
securityContext:
|
||||
runAsUser: 1000
|
||||
runAsGroup: 1000
|
||||
runAsNonRoot: true
|
||||
fsGroup: 1000
|
||||
fsGroupChangePolicy: OnRootMismatch
|
||||
containerSecurityContext:
|
||||
runAsUser: 1000
|
||||
runAsGroup: 1000
|
||||
runAsNonRoot: true
|
||||
allowPrivilegeEscalation: false
|
||||
capabilities:
|
||||
drop:
|
||||
- ALL
|
||||
seccompProfile:
|
||||
type: RuntimeDefault
|
||||
|
||||
mountPath: /export
|
||||
subPath: /data
|
||||
|
||||
certificates:
|
||||
disableAutoCert: true
|
||||
|
||||
services:
|
||||
minio:
|
||||
serviceType: ClusterIP
|
||||
console:
|
||||
serviceType: ClusterIP
|
||||
|
||||
pvcProtection: true
|
||||
prometheusOperator: false
|
||||
|
||||
sideCars:
|
||||
resources:
|
||||
requests:
|
||||
cpu: 100m
|
||||
memory: 128Mi
|
||||
limits:
|
||||
cpu: 500m
|
||||
memory: 512Mi
|
||||
@@ -0,0 +1,5 @@
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
|
||||
resources:
|
||||
- base/deployment
|
||||
@@ -0,0 +1,8 @@
|
||||
# 홈 환경 Overlay 예약 지점
|
||||
|
||||
현재 홈 환경의 공통 Helm values는
|
||||
../../base/deployment/values/home.yaml에 있다. 실제 배포 진입점은 저장소 root와
|
||||
profiles/admin-oidc다.
|
||||
|
||||
이 디렉터리는 이후 홈 환경 전용 patch가 필요할 때만 사용하며 현재 활성
|
||||
Kustomization 리소스는 없다.
|
||||
@@ -0,0 +1,16 @@
|
||||
# AIStor Console OIDC 프로필
|
||||
|
||||
기존 ObjectStore baseline은 그대로 두고 다음 항목만 추가합니다.
|
||||
|
||||
- storage-admin.learn.hyeonworks.com Console Ingress
|
||||
- Keycloak 표준 OIDC 환경 변수와 policy claim
|
||||
- Traefik의 Console 접근과 Keycloak HTTPS egress
|
||||
|
||||
S3 API Service에는 Ingress를 만들지 않습니다. aistor-keycloak-oidc
|
||||
Secret의 client-id, client-secret 값은 적용 스크립트가 만들며 Git에
|
||||
저장하지 않습니다. 기존 root 계정은 비상용으로 유지합니다.
|
||||
|
||||
`MINIO_IDENTITY_OPENID_VENDOR=keycloak`은 사용하지 않습니다. 이 옵션은
|
||||
AIStor가 Keycloak Admin API로 사용자를 추가 검증할 때 admin URL, realm과
|
||||
service-account 권한을 요구합니다. 이 프로필은 Keycloak이 발급한 `policy`
|
||||
claim을 표준 OIDC로 소비하므로 불필요한 Admin API 권한을 부여하지 않습니다.
|
||||
@@ -0,0 +1,24 @@
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: Ingress
|
||||
metadata:
|
||||
name: minio-aistor-console
|
||||
namespace: object-storage
|
||||
labels:
|
||||
app.kubernetes.io/name: minio-aistor
|
||||
app.kubernetes.io/component: admin-console
|
||||
app.kubernetes.io/part-of: platform
|
||||
annotations:
|
||||
traefik.ingress.kubernetes.io/router.entrypoints: web
|
||||
spec:
|
||||
ingressClassName: traefik
|
||||
rules:
|
||||
- host: storage-admin.learn.hyeonworks.com
|
||||
http:
|
||||
paths:
|
||||
- path: /
|
||||
pathType: Prefix
|
||||
backend:
|
||||
service:
|
||||
name: minio-aistor-console
|
||||
port:
|
||||
number: 9090
|
||||
@@ -0,0 +1,15 @@
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
|
||||
resources:
|
||||
- ../../base/deployment
|
||||
- console-ingress.yaml
|
||||
- network-policies.yaml
|
||||
|
||||
patches:
|
||||
- path: objectstore-oidc-patch.yaml
|
||||
target:
|
||||
group: aistor.min.io
|
||||
version: v1
|
||||
kind: ObjectStore
|
||||
name: minio-aistor
|
||||
@@ -0,0 +1,101 @@
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: object-storage-allow-traefik-console
|
||||
namespace: object-storage
|
||||
labels:
|
||||
app.kubernetes.io/part-of: platform
|
||||
app.kubernetes.io/component: admin-console
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
aistor.min.io/objectStore: minio-aistor
|
||||
policyTypes:
|
||||
- Ingress
|
||||
ingress:
|
||||
- from:
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
kubernetes.io/metadata.name: kube-system
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: traefik
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 9090
|
||||
---
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: object-storage-allow-host-nginx-keycloak
|
||||
namespace: object-storage
|
||||
labels:
|
||||
app.kubernetes.io/part-of: platform
|
||||
app.kubernetes.io/component: oidc-client
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
aistor.min.io/objectStore: minio-aistor
|
||||
policyTypes:
|
||||
- Egress
|
||||
egress:
|
||||
- to:
|
||||
- ipBlock:
|
||||
cidr: 192.168.0.107/32
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 443
|
||||
---
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: object-storage-allow-blackbox-console-health
|
||||
namespace: object-storage
|
||||
labels:
|
||||
app.kubernetes.io/part-of: platform
|
||||
app.kubernetes.io/component: admin-console
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
aistor.min.io/objectStore: minio-aistor
|
||||
policyTypes:
|
||||
- Ingress
|
||||
ingress:
|
||||
- from:
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
kubernetes.io/metadata.name: observability
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: prometheus-blackbox-exporter
|
||||
app.kubernetes.io/instance: blackbox-exporter
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 9090
|
||||
---
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: object-storage-allow-prometheus-metrics
|
||||
namespace: object-storage
|
||||
labels:
|
||||
app.kubernetes.io/part-of: platform
|
||||
app.kubernetes.io/component: object-storage-metrics
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
aistor.min.io/objectStore: minio-aistor
|
||||
policyTypes:
|
||||
- Ingress
|
||||
ingress:
|
||||
- from:
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
kubernetes.io/metadata.name: observability
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: prometheus
|
||||
app.kubernetes.io/instance: observability-core-kube-pr-prometheus
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 9000
|
||||
@@ -0,0 +1,32 @@
|
||||
apiVersion: aistor.min.io/v1
|
||||
kind: ObjectStore
|
||||
metadata:
|
||||
name: minio-aistor
|
||||
spec:
|
||||
env:
|
||||
- name: MINIO_PROMETHEUS_AUTH_TYPE
|
||||
value: public
|
||||
- name: MINIO_BROWSER_REDIRECT_URL
|
||||
value: https://storage-admin.learn.hyeonworks.com
|
||||
- name: MINIO_IDENTITY_OPENID_CONFIG_URL
|
||||
value: https://id.learn.hyeonworks.com/realms/hyeonworks/.well-known/openid-configuration
|
||||
- name: MINIO_IDENTITY_OPENID_CLIENT_ID
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: aistor-keycloak-oidc
|
||||
key: client-id
|
||||
- name: MINIO_IDENTITY_OPENID_CLIENT_SECRET
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: aistor-keycloak-oidc
|
||||
key: client-secret
|
||||
- name: MINIO_IDENTITY_OPENID_SCOPES
|
||||
value: openid,profile,email
|
||||
- name: MINIO_IDENTITY_OPENID_DISPLAY_NAME
|
||||
value: Hyeonworks ID
|
||||
- name: MINIO_IDENTITY_OPENID_CLAIM_NAME
|
||||
value: policy
|
||||
- name: MINIO_IDENTITY_OPENID_REDIRECT_URI_DYNAMIC
|
||||
value: "on"
|
||||
- name: MINIO_BROWSER_SESSION_DURATION
|
||||
value: 8h
|
||||
@@ -0,0 +1,67 @@
|
||||
secrets:
|
||||
name: aistor-root-configuration
|
||||
existingSecret: true
|
||||
disableRoot: false
|
||||
|
||||
objectStore:
|
||||
name: minio-aistor
|
||||
|
||||
configuration:
|
||||
name: aistor-root-configuration
|
||||
|
||||
pools:
|
||||
- name: pool-0
|
||||
servers: 1
|
||||
volumesPerServer: 1
|
||||
storageNamePrefix: data
|
||||
size: 900Gi
|
||||
storageClassName: aistor-local-xfs-retain
|
||||
nodeSelector:
|
||||
kubernetes.io/hostname: donghyeon-system-product-name
|
||||
resources:
|
||||
requests:
|
||||
cpu: 500m
|
||||
memory: 2Gi
|
||||
limits:
|
||||
cpu: "3"
|
||||
memory: 8Gi
|
||||
securityContext:
|
||||
runAsUser: 1000
|
||||
runAsGroup: 1000
|
||||
runAsNonRoot: true
|
||||
fsGroup: 1000
|
||||
fsGroupChangePolicy: OnRootMismatch
|
||||
containerSecurityContext:
|
||||
runAsUser: 1000
|
||||
runAsGroup: 1000
|
||||
runAsNonRoot: true
|
||||
allowPrivilegeEscalation: false
|
||||
capabilities:
|
||||
drop:
|
||||
- ALL
|
||||
seccompProfile:
|
||||
type: RuntimeDefault
|
||||
|
||||
mountPath: /export
|
||||
subPath: /data
|
||||
|
||||
certificates:
|
||||
disableAutoCert: true
|
||||
|
||||
services:
|
||||
minio:
|
||||
serviceType: ClusterIP
|
||||
console:
|
||||
serviceType: ClusterIP
|
||||
|
||||
pvcProtection: true
|
||||
prometheusOperator: false
|
||||
|
||||
sideCars:
|
||||
resources:
|
||||
requests:
|
||||
cpu: 100m
|
||||
memory: 128Mi
|
||||
limits:
|
||||
cpu: 500m
|
||||
memory: 512Mi
|
||||
@@ -0,0 +1,18 @@
|
||||
# Observability core
|
||||
|
||||
This root composes the internal observability workloads deployed after the
|
||||
namespace, CRD, and Local PV prerequisites:
|
||||
|
||||
- kube-prometheus-stack without Grafana or its bundled node-exporter;
|
||||
- single-binary Loki and Tempo using isolated AIStor buckets;
|
||||
- Alloy for Pod logs and internal OTLP/gRPC traces;
|
||||
- a standalone node-exporter;
|
||||
- the exact core NetworkPolicy matrix, platform rules, scrape targets, verified
|
||||
dashboards, and source-only Alertmanager Slack routing.
|
||||
|
||||
It intentionally does not create public ingress, Grafana, OIDC, or credential
|
||||
Secrets. The Slack routing source references
|
||||
`alertmanager-slack-webhook/url`; while that Secret is absent, do not apply the
|
||||
AlertmanagerConfig, its kube-prometheus-stack reference, its public HTTPS
|
||||
NetworkPolicy, or alert rules. Namespace, CRD, and Local PV roots remain outside
|
||||
this aggregate because their protected apply order is different.
|
||||
@@ -0,0 +1,24 @@
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: observability-allow-alertmanager-public-https
|
||||
namespace: observability
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: alertmanager
|
||||
app.kubernetes.io/instance: observability-core-kube-pr-alertmanager
|
||||
policyTypes:
|
||||
- Egress
|
||||
egress:
|
||||
- to:
|
||||
- ipBlock:
|
||||
cidr: 0.0.0.0/0
|
||||
except:
|
||||
- 10.0.0.0/8
|
||||
- 100.64.0.0/10
|
||||
- 172.16.0.0/12
|
||||
- 192.168.0.0/16
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 443
|
||||
@@ -0,0 +1,8 @@
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
|
||||
namespace: observability
|
||||
|
||||
resources:
|
||||
- platform-alertmanager.yaml
|
||||
- alertmanager-public-https-egress.yaml
|
||||
@@ -0,0 +1,80 @@
|
||||
apiVersion: monitoring.coreos.com/v1alpha1
|
||||
kind: AlertmanagerConfig
|
||||
metadata:
|
||||
name: platform-alertmanager
|
||||
namespace: observability
|
||||
labels:
|
||||
observability.hyeonworks.com/instance: home
|
||||
spec:
|
||||
route:
|
||||
receiver: platform-slack
|
||||
groupBy:
|
||||
- cluster
|
||||
- namespace
|
||||
- alertname
|
||||
- severity
|
||||
groupWait: 30s
|
||||
groupInterval: 5m
|
||||
repeatInterval: 4h
|
||||
routes:
|
||||
- receiver: platform-null
|
||||
matchers:
|
||||
- name: alertname
|
||||
matchType: "="
|
||||
value: InfoInhibitor
|
||||
inhibitRules:
|
||||
- sourceMatch:
|
||||
- name: alertname
|
||||
matchType: "="
|
||||
value: InfoInhibitor
|
||||
targetMatch:
|
||||
- name: severity
|
||||
matchType: "="
|
||||
value: info
|
||||
equal:
|
||||
- namespace
|
||||
receivers:
|
||||
- name: platform-null
|
||||
- name: platform-slack
|
||||
slackConfigs:
|
||||
- apiURL:
|
||||
name: alertmanager-slack-webhook
|
||||
key: url
|
||||
sendResolved: true
|
||||
linkNames: false
|
||||
mrkdwnIn:
|
||||
- text
|
||||
- fields
|
||||
fallback: >-
|
||||
{{ if eq .Status "firing" }}FIRING{{ else }}RESOLVED{{ end }}: {{ if .CommonLabels.severity }}{{ .CommonLabels.severity | toUpper }}{{ else }}UNKNOWN{{ end }} · {{ .CommonLabels.alertname }}
|
||||
title: >-
|
||||
{{ if eq .Status "firing" }}[FIRING:{{ .Alerts.Firing | len }}]{{ else }}[RESOLVED]{{ end }} {{ if .CommonLabels.severity }}{{ .CommonLabels.severity | toUpper }}{{ else }}UNKNOWN{{ end }} · {{ .CommonLabels.alertname }}
|
||||
titleLink: https://grafana.learn.hyeonworks.com/
|
||||
color: >-
|
||||
{{ if eq .Status "resolved" }}good{{ else if or (eq .CommonLabels.severity "emergency") (eq .CommonLabels.severity "critical") }}danger{{ else if eq .CommonLabels.severity "warning" }}warning{{ else }}#439FE0{{ end }}
|
||||
fields:
|
||||
- title: Status
|
||||
value: '{{ .Status | toUpper }}'
|
||||
short: true
|
||||
- title: Severity
|
||||
value: '{{ if .CommonLabels.severity }}{{ .CommonLabels.severity | toUpper }}{{ else }}UNKNOWN{{ end }}'
|
||||
short: true
|
||||
- title: Location
|
||||
value: '{{ if .CommonLabels.cluster }}{{ .CommonLabels.cluster }}{{ else }}unknown-cluster{{ end }} / {{ if .CommonLabels.namespace }}{{ .CommonLabels.namespace }}{{ else }}cluster-scoped{{ end }}'
|
||||
short: true
|
||||
- title: Alert count
|
||||
value: '{{ len .Alerts }}'
|
||||
short: true
|
||||
text: |-
|
||||
{{ range .Alerts }}
|
||||
*Alert status:* {{ .Status | toUpper }}
|
||||
*Target:* {{ $target := .Labels.Remove $.GroupLabels.Names }}{{ if $target }}{{ range $target.SortedPairs }}{{ .Name }}={{ .Value }} {{ end }}{{ else }}unknown{{ end }}
|
||||
*Summary:* {{ with .Annotations.summary }}{{ . }}{{ else }}No summary provided{{ end }}
|
||||
*Details:* {{ with .Annotations.description }}{{ . }}{{ else }}No description provided{{ end }}
|
||||
*Started:* {{ .StartsAt.Format "2006-01-02T15:04:05Z07:00" }}
|
||||
{{ if eq .Status "resolved" }}*Ended:* {{ .EndsAt.Format "2006-01-02T15:04:05Z07:00" }}{{ end }}
|
||||
{{ with .Annotations.runbook_url }}*Runbook:* <{{ . }}|대응 절차 열기>{{ end }}
|
||||
|
||||
{{ end }}
|
||||
<https://grafana.learn.hyeonworks.com/|Grafana> · <https://grafana.learn.hyeonworks.com/explore|Explore>
|
||||
footer: hyeonworks observability · Alertmanager
|
||||
@@ -0,0 +1,12 @@
|
||||
# Alloy
|
||||
|
||||
Grafana Alloy runs as a single-node DaemonSet in `observability-agent`.
|
||||
It tails only `/var/log/pods`, writes logs to the in-cluster Loki service, and
|
||||
accepts OTLP/gRPC traces on port `4317` for the in-cluster Tempo service.
|
||||
|
||||
The deployment deliberately disables host namespaces, automatic service
|
||||
account token mounts, CRD creation, the config reloader, public exposure, and
|
||||
persistent OpenTelemetry queues. The 2 GiB PVC stores only Alloy file
|
||||
positions and local state. Alloy keeps primary UID/GID `473:473`; supplemental
|
||||
group `0` permits read-only traversal of k3s' `root:root 0750` Pod log tree
|
||||
without changing host permissions or running the container as root.
|
||||
@@ -0,0 +1,214 @@
|
||||
discovery.kubernetes "pods" {
|
||||
role = "pod"
|
||||
|
||||
selectors {
|
||||
role = "pod"
|
||||
field = "spec.nodeName=" + sys.env("HOSTNAME")
|
||||
}
|
||||
}
|
||||
|
||||
discovery.relabel "pod_logs" {
|
||||
targets = discovery.kubernetes.pods.targets
|
||||
|
||||
rule {
|
||||
source_labels = ["__meta_kubernetes_namespace"]
|
||||
target_label = "namespace"
|
||||
}
|
||||
rule {
|
||||
source_labels = ["__meta_kubernetes_pod_name"]
|
||||
target_label = "pod"
|
||||
}
|
||||
rule {
|
||||
source_labels = ["__meta_kubernetes_pod_container_name"]
|
||||
target_label = "container"
|
||||
}
|
||||
rule {
|
||||
source_labels = ["__meta_kubernetes_pod_label_app_kubernetes_io_name"]
|
||||
regex = "(.+)"
|
||||
target_label = "app"
|
||||
}
|
||||
rule {
|
||||
source_labels = [
|
||||
"__meta_kubernetes_pod_label_app_kubernetes_io_name",
|
||||
"__meta_kubernetes_pod_label_app",
|
||||
]
|
||||
separator = ";"
|
||||
regex = "^;(.+)$"
|
||||
replacement = "$1"
|
||||
target_label = "app"
|
||||
}
|
||||
rule {
|
||||
source_labels = [
|
||||
"__meta_kubernetes_pod_uid",
|
||||
"__meta_kubernetes_pod_container_name",
|
||||
]
|
||||
separator = "/"
|
||||
replacement = "/var/log/pods/*$1/*.log"
|
||||
target_label = "__path__"
|
||||
}
|
||||
rule {
|
||||
source_labels = ["__meta_kubernetes_pod_container_id"]
|
||||
regex = `^(\S+):\/\/.+$`
|
||||
replacement = "$1"
|
||||
target_label = "container_runtime"
|
||||
}
|
||||
}
|
||||
|
||||
local.file_match "pod_logs" {
|
||||
path_targets = discovery.relabel.pod_logs.output
|
||||
}
|
||||
|
||||
loki.source.file "pod_logs" {
|
||||
targets = local.file_match.pod_logs.targets
|
||||
forward_to = [loki.process.pod_logs.receiver]
|
||||
}
|
||||
|
||||
loki.process "pod_logs" {
|
||||
stage.match {
|
||||
selector = `{container_runtime=~"containerd|cri-o"}`
|
||||
stage.cri {}
|
||||
}
|
||||
stage.match {
|
||||
selector = `{container_runtime="docker"}`
|
||||
stage.docker {}
|
||||
}
|
||||
|
||||
stage.replace {
|
||||
expression = `(?i)"(?:authorization|bearer|jwt|password|passwd|client[_-]?secret)"\s*:\s*"([^"\\]*(?:\\.[^"\\]*)*)"`
|
||||
replace = "[REDACTED]"
|
||||
}
|
||||
stage.replace {
|
||||
expression = `(?i)(?:authorization\s*[:=]\s*)(?:bearer\s+)?([A-Za-z0-9._~+/=-]+)`
|
||||
replace = "[REDACTED]"
|
||||
}
|
||||
stage.replace {
|
||||
expression = `(?i)\bbearer\s+([A-Za-z0-9_-]+(?:\.[A-Za-z0-9_-]+){0,2})`
|
||||
replace = "[REDACTED]"
|
||||
}
|
||||
stage.replace {
|
||||
expression = `(?i)(?:password|passwd|client[_-]?secret)\s*[:=]\s*([^\s"'{},]+)`
|
||||
replace = "[REDACTED]"
|
||||
}
|
||||
|
||||
stage.json {
|
||||
expressions = {
|
||||
level = "level",
|
||||
}
|
||||
}
|
||||
stage.labels {
|
||||
values = {
|
||||
level = "",
|
||||
}
|
||||
}
|
||||
stage.match {
|
||||
selector = `{level!~"(?i)^(trace|debug|info|warn|warning|error|fatal|panic)$"}`
|
||||
stage.label_drop {
|
||||
values = ["level"]
|
||||
}
|
||||
}
|
||||
stage.static_labels {
|
||||
values = {
|
||||
cluster = "home",
|
||||
}
|
||||
}
|
||||
stage.label_keep {
|
||||
values = ["cluster", "namespace", "app", "pod", "container", "level"]
|
||||
}
|
||||
|
||||
forward_to = [loki.write.local.receiver]
|
||||
}
|
||||
|
||||
loki.write "local" {
|
||||
endpoint {
|
||||
url = "http://loki.observability.svc.cluster.local:3100/loki/api/v1/push"
|
||||
batch_size = "256KiB"
|
||||
batch_wait = "1s"
|
||||
remote_timeout = "10s"
|
||||
min_backoff_period = "500ms"
|
||||
max_backoff_period = "30s"
|
||||
max_backoff_retries = 10
|
||||
}
|
||||
}
|
||||
|
||||
otelcol.receiver.otlp "traces" {
|
||||
grpc {
|
||||
endpoint = "0.0.0.0:4317"
|
||||
}
|
||||
|
||||
output {
|
||||
traces = [otelcol.processor.k8sattributes.traces.input]
|
||||
}
|
||||
}
|
||||
|
||||
otelcol.processor.k8sattributes "traces" {
|
||||
auth_type = "serviceAccount"
|
||||
|
||||
extract {
|
||||
deployment_name_from_replicaset = false
|
||||
metadata = [
|
||||
"k8s.namespace.name",
|
||||
"k8s.deployment.name",
|
||||
"k8s.node.name",
|
||||
"k8s.pod.name",
|
||||
"k8s.pod.uid",
|
||||
"k8s.pod.start_time",
|
||||
"k8s.container.name",
|
||||
]
|
||||
}
|
||||
|
||||
pod_association {
|
||||
source {
|
||||
from = "resource_attribute"
|
||||
name = "k8s.pod.uid"
|
||||
}
|
||||
}
|
||||
pod_association {
|
||||
source {
|
||||
from = "resource_attribute"
|
||||
name = "k8s.pod.ip"
|
||||
}
|
||||
}
|
||||
pod_association {
|
||||
source {
|
||||
from = "connection"
|
||||
}
|
||||
}
|
||||
|
||||
output {
|
||||
traces = [otelcol.processor.batch.traces.input]
|
||||
}
|
||||
}
|
||||
|
||||
otelcol.processor.batch "traces" {
|
||||
timeout = "1s"
|
||||
send_batch_size = 512
|
||||
send_batch_max_size = 1024
|
||||
|
||||
output {
|
||||
traces = [otelcol.exporter.otlp.tempo.input]
|
||||
}
|
||||
}
|
||||
|
||||
otelcol.exporter.otlp "tempo" {
|
||||
client {
|
||||
endpoint = "tempo.observability.svc.cluster.local:4317"
|
||||
tls {
|
||||
insecure = true
|
||||
}
|
||||
}
|
||||
|
||||
retry_on_failure {
|
||||
enabled = true
|
||||
initial_interval = "1s"
|
||||
max_interval = "15s"
|
||||
max_elapsed_time = "5m"
|
||||
}
|
||||
|
||||
sending_queue {
|
||||
enabled = true
|
||||
queue_size = 256
|
||||
num_consumers = 2
|
||||
sizer = "requests"
|
||||
block_on_overflow = true
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,51 @@
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
|
||||
namespace: observability-agent
|
||||
|
||||
resources:
|
||||
- pvc.yaml
|
||||
|
||||
configMapGenerator:
|
||||
- name: alloy-config
|
||||
files:
|
||||
- config.alloy=config.alloy
|
||||
|
||||
helmGlobals:
|
||||
chartHome: .helm/charts
|
||||
|
||||
helmCharts:
|
||||
- name: alloy
|
||||
repo: https://grafana.github.io/helm-charts
|
||||
version: 1.11.0
|
||||
releaseName: alloy
|
||||
namespace: observability-agent
|
||||
valuesFile: values/home.yaml
|
||||
includeCRDs: false
|
||||
skipTests: true
|
||||
|
||||
images:
|
||||
- name: docker.io/grafana/alloy
|
||||
newName: docker.io/grafana/alloy
|
||||
digest: sha256:eb21f4c0858edffcdd1b385910ddeef26f692fc2c282f61baa724fc09d274a17
|
||||
|
||||
patches:
|
||||
- target:
|
||||
group: apps
|
||||
version: v1
|
||||
kind: DaemonSet
|
||||
name: alloy
|
||||
patch: |-
|
||||
- op: add
|
||||
path: /spec/template/spec/automountServiceAccountToken
|
||||
value: false
|
||||
- op: add
|
||||
path: /spec/template/spec/hostNetwork
|
||||
value: false
|
||||
- op: add
|
||||
path: /spec/template/spec/hostPID
|
||||
value: false
|
||||
- op: add
|
||||
path: /spec/template/spec/hostIPC
|
||||
value: false
|
||||
|
||||
@@ -0,0 +1,17 @@
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: alloy-storage
|
||||
namespace: observability-agent
|
||||
labels:
|
||||
app.kubernetes.io/name: alloy
|
||||
observability.hyeonworks.com/instance: home
|
||||
spec:
|
||||
accessModes:
|
||||
- ReadWriteOnce
|
||||
storageClassName: ssd-local-observability-alloy-retain
|
||||
volumeName: observability-alloy-local-pv
|
||||
resources:
|
||||
requests:
|
||||
storage: 2Gi
|
||||
|
||||
@@ -0,0 +1,155 @@
|
||||
crds:
|
||||
create: false
|
||||
|
||||
global:
|
||||
podSecurityContext:
|
||||
runAsNonRoot: true
|
||||
runAsUser: 473
|
||||
runAsGroup: 473
|
||||
fsGroup: 473
|
||||
fsGroupChangePolicy: OnRootMismatch
|
||||
# k3s keeps /var/log/pods root:root 0750. Retain Alloy's non-root primary
|
||||
# identity while granting read-only traversal of that single hostPath.
|
||||
supplementalGroups:
|
||||
- 0
|
||||
seccompProfile:
|
||||
type: RuntimeDefault
|
||||
|
||||
alloy:
|
||||
configMap:
|
||||
create: false
|
||||
name: alloy-config
|
||||
key: config.alloy
|
||||
clustering:
|
||||
enabled: false
|
||||
stabilityLevel: generally-available
|
||||
storagePath: /var/lib/alloy
|
||||
enableHttpServerPort: true
|
||||
listenAddr: 0.0.0.0
|
||||
listenPort: 12345
|
||||
enableReporting: false
|
||||
extraPorts:
|
||||
- name: otlp-grpc
|
||||
port: 4317
|
||||
targetPort: 4317
|
||||
protocol: TCP
|
||||
mounts:
|
||||
varlog: false
|
||||
dockercontainers: false
|
||||
extra:
|
||||
- name: pod-logs
|
||||
mountPath: /var/log/pods
|
||||
readOnly: true
|
||||
- name: alloy-storage
|
||||
mountPath: /var/lib/alloy
|
||||
- name: tmp
|
||||
mountPath: /tmp
|
||||
- name: kubernetes-api-access
|
||||
mountPath: /var/run/secrets/kubernetes.io/serviceaccount
|
||||
readOnly: true
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
runAsUser: 473
|
||||
runAsGroup: 473
|
||||
allowPrivilegeEscalation: false
|
||||
readOnlyRootFilesystem: true
|
||||
capabilities:
|
||||
drop:
|
||||
- ALL
|
||||
seccompProfile:
|
||||
type: RuntimeDefault
|
||||
resources:
|
||||
requests:
|
||||
cpu: 200m
|
||||
memory: 384Mi
|
||||
limits:
|
||||
cpu: "1"
|
||||
memory: 768Mi
|
||||
|
||||
image:
|
||||
registry: docker.io
|
||||
repository: grafana/alloy
|
||||
tag: v1.18.0
|
||||
digest: sha256:eb21f4c0858edffcdd1b385910ddeef26f692fc2c282f61baa724fc09d274a17
|
||||
pullPolicy: IfNotPresent
|
||||
|
||||
rbac:
|
||||
create: true
|
||||
namespaces: []
|
||||
rules:
|
||||
- apiGroups:
|
||||
- ""
|
||||
resources:
|
||||
- pods
|
||||
- namespaces
|
||||
verbs:
|
||||
- get
|
||||
- list
|
||||
- watch
|
||||
clusterRules:
|
||||
- apiGroups:
|
||||
- apps
|
||||
resources:
|
||||
- replicasets
|
||||
verbs:
|
||||
- get
|
||||
- list
|
||||
- watch
|
||||
|
||||
serviceAccount:
|
||||
create: true
|
||||
name: alloy
|
||||
automountServiceAccountToken: false
|
||||
|
||||
configReloader:
|
||||
enabled: false
|
||||
|
||||
controller:
|
||||
type: daemonset
|
||||
hostNetwork: false
|
||||
hostPID: false
|
||||
volumes:
|
||||
extra:
|
||||
- name: pod-logs
|
||||
hostPath:
|
||||
path: /var/log/pods
|
||||
type: Directory
|
||||
- name: alloy-storage
|
||||
persistentVolumeClaim:
|
||||
claimName: alloy-storage
|
||||
- name: tmp
|
||||
emptyDir: {}
|
||||
- name: kubernetes-api-access
|
||||
projected:
|
||||
defaultMode: 420
|
||||
sources:
|
||||
- serviceAccountToken:
|
||||
expirationSeconds: 3600
|
||||
path: token
|
||||
- configMap:
|
||||
name: kube-root-ca.crt
|
||||
items:
|
||||
- key: ca.crt
|
||||
path: ca.crt
|
||||
- downwardAPI:
|
||||
items:
|
||||
- path: namespace
|
||||
fieldRef:
|
||||
apiVersion: v1
|
||||
fieldPath: metadata.namespace
|
||||
|
||||
networkPolicy:
|
||||
enabled: false
|
||||
|
||||
service:
|
||||
enabled: true
|
||||
type: ClusterIP
|
||||
|
||||
serviceMonitor:
|
||||
enabled: true
|
||||
interval: 30s
|
||||
additionalLabels:
|
||||
observability.hyeonworks.com/instance: home
|
||||
|
||||
ingress:
|
||||
enabled: false
|
||||
@@ -0,0 +1,22 @@
|
||||
# Blackbox exporter
|
||||
|
||||
This component deploys the pinned Prometheus blackbox exporter as an internal
|
||||
`ClusterIP` service in the `observability` namespace. The `/probe` handler is
|
||||
reachable only from the Prometheus Pod; it has no Ingress, NodePort, host port,
|
||||
or service-account token.
|
||||
|
||||
Only three fixed `Probe` groups are declared:
|
||||
|
||||
- `platform-public-edge`: Gitea health and Keycloak discovery through Host Nginx
|
||||
- `platform-private-edge`: Grafana, pgAdmin, and AIStor Console must return 403
|
||||
- `platform-private-internal`: the same private services must return 200 through
|
||||
their ClusterIP Services
|
||||
|
||||
The internal Grafana and pgAdmin targets use the blackbox exporter's `hostname`
|
||||
parameter to set their required HTTP Host header. Target URLs and modules are
|
||||
declarative; arbitrary callers are not granted access to port 9115.
|
||||
|
||||
The private-edge source boundary was proven separately and recorded as a
|
||||
root-only `platform-blackbox-source-v1` evidence file. Applying this component
|
||||
must remain gated on that evidence matching the active deny-only Nginx config.
|
||||
|
||||
@@ -0,0 +1,45 @@
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
|
||||
namespace: observability
|
||||
|
||||
resources:
|
||||
- probes-public.yaml
|
||||
- probes-private-edge.yaml
|
||||
- probes-private-internal.yaml
|
||||
- network-policy.yaml
|
||||
|
||||
helmGlobals:
|
||||
chartHome: .helm/charts
|
||||
|
||||
helmCharts:
|
||||
- name: prometheus-blackbox-exporter
|
||||
repo: https://prometheus-community.github.io/helm-charts
|
||||
version: 11.16.0
|
||||
releaseName: blackbox-exporter
|
||||
namespace: observability
|
||||
valuesFile: values/home.yaml
|
||||
includeCRDs: false
|
||||
skipTests: true
|
||||
|
||||
patches:
|
||||
- target:
|
||||
version: v1
|
||||
kind: ServiceAccount
|
||||
name: blackbox-exporter
|
||||
patch: |-
|
||||
- op: add
|
||||
path: /automountServiceAccountToken
|
||||
value: false
|
||||
- target:
|
||||
group: apps
|
||||
version: v1
|
||||
kind: Deployment
|
||||
name: blackbox-exporter
|
||||
patch: |-
|
||||
- op: add
|
||||
path: /spec/template/spec/hostPID
|
||||
value: false
|
||||
- op: add
|
||||
path: /spec/template/spec/hostIPC
|
||||
value: false
|
||||
@@ -0,0 +1,88 @@
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: observability-allow-prometheus-to-blackbox
|
||||
namespace: observability
|
||||
labels:
|
||||
app.kubernetes.io/name: prometheus-blackbox-exporter
|
||||
app.kubernetes.io/part-of: platform-observability
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: prometheus-blackbox-exporter
|
||||
app.kubernetes.io/instance: blackbox-exporter
|
||||
policyTypes:
|
||||
- Ingress
|
||||
ingress:
|
||||
- from:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: prometheus
|
||||
app.kubernetes.io/instance: observability-core-kube-pr-prometheus
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 9115
|
||||
---
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: observability-allow-blackbox-egress
|
||||
namespace: observability
|
||||
labels:
|
||||
app.kubernetes.io/name: prometheus-blackbox-exporter
|
||||
app.kubernetes.io/part-of: platform-observability
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: prometheus-blackbox-exporter
|
||||
app.kubernetes.io/instance: blackbox-exporter
|
||||
policyTypes:
|
||||
- Egress
|
||||
egress:
|
||||
- to:
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
kubernetes.io/metadata.name: kube-system
|
||||
podSelector:
|
||||
matchLabels:
|
||||
k8s-app: kube-dns
|
||||
ports:
|
||||
- protocol: UDP
|
||||
port: 53
|
||||
- protocol: TCP
|
||||
port: 53
|
||||
- to:
|
||||
- ipBlock:
|
||||
cidr: 192.168.0.107/32
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 443
|
||||
- to:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: grafana
|
||||
app.kubernetes.io/instance: grafana
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 3000
|
||||
- to:
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
kubernetes.io/metadata.name: platform-admin
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: pgadmin4
|
||||
app.kubernetes.io/instance: pgadmin
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 5050
|
||||
- to:
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
kubernetes.io/metadata.name: object-storage
|
||||
podSelector:
|
||||
matchLabels:
|
||||
aistor.min.io/objectStore: minio-aistor
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 9090
|
||||
@@ -0,0 +1,28 @@
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: Probe
|
||||
metadata:
|
||||
name: platform-private-edge
|
||||
namespace: observability
|
||||
labels:
|
||||
app.kubernetes.io/name: prometheus-blackbox-exporter
|
||||
app.kubernetes.io/instance: blackbox-exporter
|
||||
app.kubernetes.io/part-of: platform-observability
|
||||
observability.hyeonworks.com/instance: home
|
||||
spec:
|
||||
jobName: blackbox-private-edge
|
||||
interval: 30s
|
||||
scrapeTimeout: 15s
|
||||
module: http_private_edge_403
|
||||
prober:
|
||||
url: blackbox-exporter.observability.svc.cluster.local:9115
|
||||
scheme: http
|
||||
path: /probe
|
||||
proxyFromEnvironment: false
|
||||
targets:
|
||||
staticConfig:
|
||||
labels:
|
||||
observability.hyeonworks.com/probe-group: private-edge
|
||||
static:
|
||||
- https://grafana.learn.hyeonworks.com/
|
||||
- https://storage-admin.learn.hyeonworks.com/
|
||||
- https://db-admin.learn.hyeonworks.com/
|
||||
@@ -0,0 +1,44 @@
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: Probe
|
||||
metadata:
|
||||
name: platform-private-internal
|
||||
namespace: observability
|
||||
labels:
|
||||
app.kubernetes.io/name: prometheus-blackbox-exporter
|
||||
app.kubernetes.io/instance: blackbox-exporter
|
||||
app.kubernetes.io/part-of: platform-observability
|
||||
observability.hyeonworks.com/instance: home
|
||||
spec:
|
||||
jobName: blackbox-private-internal
|
||||
interval: 30s
|
||||
scrapeTimeout: 15s
|
||||
module: http_private_internal_200
|
||||
prober:
|
||||
url: blackbox-exporter.observability.svc.cluster.local:9115
|
||||
scheme: http
|
||||
path: /probe
|
||||
proxyFromEnvironment: false
|
||||
targets:
|
||||
staticConfig:
|
||||
labels:
|
||||
observability.hyeonworks.com/probe-group: private-internal
|
||||
static:
|
||||
- http://grafana.observability.svc.cluster.local/api/health
|
||||
- http://pgadmin.platform-admin.svc.cluster.local/misc/ping
|
||||
- http://minio-aistor-console.object-storage.svc.cluster.local:9090/
|
||||
relabelingConfigs:
|
||||
- sourceLabels:
|
||||
- __param_target
|
||||
regex: http://grafana[.]observability[.]svc[.]cluster[.]local/api/health
|
||||
targetLabel: __param_hostname
|
||||
replacement: grafana.learn.hyeonworks.com
|
||||
- sourceLabels:
|
||||
- __param_target
|
||||
regex: http://pgadmin[.]platform-admin[.]svc[.]cluster[.]local/misc/ping
|
||||
targetLabel: __param_hostname
|
||||
replacement: db-admin.learn.hyeonworks.com
|
||||
- sourceLabels:
|
||||
- __param_target
|
||||
regex: http://minio-aistor-console[.]object-storage[.]svc[.]cluster[.]local:9090/
|
||||
targetLabel: __param_hostname
|
||||
replacement: storage-admin.learn.hyeonworks.com
|
||||
@@ -0,0 +1,27 @@
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: Probe
|
||||
metadata:
|
||||
name: platform-public-edge
|
||||
namespace: observability
|
||||
labels:
|
||||
app.kubernetes.io/name: prometheus-blackbox-exporter
|
||||
app.kubernetes.io/instance: blackbox-exporter
|
||||
app.kubernetes.io/part-of: platform-observability
|
||||
observability.hyeonworks.com/instance: home
|
||||
spec:
|
||||
jobName: blackbox-public-edge
|
||||
interval: 30s
|
||||
scrapeTimeout: 15s
|
||||
module: http_2xx
|
||||
prober:
|
||||
url: blackbox-exporter.observability.svc.cluster.local:9115
|
||||
scheme: http
|
||||
path: /probe
|
||||
proxyFromEnvironment: false
|
||||
targets:
|
||||
staticConfig:
|
||||
labels:
|
||||
observability.hyeonworks.com/probe-group: public-edge
|
||||
static:
|
||||
- https://git.learn.hyeonworks.com/api/healthz
|
||||
- https://id.learn.hyeonworks.com/realms/hyeonworks/.well-known/openid-configuration
|
||||
@@ -0,0 +1,128 @@
|
||||
fullnameOverride: blackbox-exporter
|
||||
|
||||
commonLabels:
|
||||
observability.hyeonworks.com/instance: home
|
||||
|
||||
kind: Deployment
|
||||
replicas: 1
|
||||
revisionHistoryLimit: 3
|
||||
|
||||
strategy:
|
||||
type: Recreate
|
||||
rollingUpdate: null
|
||||
|
||||
automountServiceAccountToken: false
|
||||
hostNetwork: false
|
||||
hostPort: 0
|
||||
|
||||
image:
|
||||
registry: quay.io
|
||||
repository: prometheus/blackbox-exporter
|
||||
tag: v0.28.0
|
||||
digest: sha256:43027b43fb785b7c5adc53bd3b5dbc1a258270a2e8aff24f477b45c4e38dac68
|
||||
pullPolicy: IfNotPresent
|
||||
|
||||
podSecurityContext:
|
||||
runAsNonRoot: true
|
||||
runAsUser: 1000
|
||||
runAsGroup: 1000
|
||||
seccompProfile:
|
||||
type: RuntimeDefault
|
||||
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
runAsUser: 1000
|
||||
runAsGroup: 1000
|
||||
readOnlyRootFilesystem: true
|
||||
allowPrivilegeEscalation: false
|
||||
capabilities:
|
||||
drop:
|
||||
- ALL
|
||||
seccompProfile:
|
||||
type: RuntimeDefault
|
||||
|
||||
resources:
|
||||
requests:
|
||||
cpu: 25m
|
||||
memory: 32Mi
|
||||
limits:
|
||||
cpu: 200m
|
||||
memory: 128Mi
|
||||
|
||||
serviceAccount:
|
||||
create: true
|
||||
name: blackbox-exporter
|
||||
|
||||
service:
|
||||
type: ClusterIP
|
||||
port: 9115
|
||||
|
||||
ingress:
|
||||
enabled: false
|
||||
|
||||
route:
|
||||
main:
|
||||
enabled: false
|
||||
|
||||
networkPolicy:
|
||||
enabled: false
|
||||
|
||||
configReloader:
|
||||
enabled: false
|
||||
|
||||
serviceMonitor:
|
||||
enabled: false
|
||||
selfMonitor:
|
||||
enabled: true
|
||||
port: http
|
||||
interval: 30s
|
||||
scrapeTimeout: 10s
|
||||
|
||||
config:
|
||||
modules:
|
||||
http_2xx:
|
||||
prober: http
|
||||
timeout: 10s
|
||||
http:
|
||||
method: GET
|
||||
valid_status_codes:
|
||||
- 200
|
||||
valid_http_versions:
|
||||
- HTTP/1.1
|
||||
- HTTP/2.0
|
||||
follow_redirects: true
|
||||
enable_http2: true
|
||||
preferred_ip_protocol: ip4
|
||||
ip_protocol_fallback: false
|
||||
tls_config:
|
||||
insecure_skip_verify: false
|
||||
http_private_edge_403:
|
||||
prober: http
|
||||
timeout: 10s
|
||||
http:
|
||||
method: GET
|
||||
valid_status_codes:
|
||||
- 403
|
||||
valid_http_versions:
|
||||
- HTTP/1.1
|
||||
- HTTP/2.0
|
||||
follow_redirects: true
|
||||
enable_http2: true
|
||||
preferred_ip_protocol: ip4
|
||||
ip_protocol_fallback: false
|
||||
tls_config:
|
||||
insecure_skip_verify: false
|
||||
http_private_internal_200:
|
||||
prober: http
|
||||
timeout: 10s
|
||||
http:
|
||||
method: GET
|
||||
valid_status_codes:
|
||||
- 200
|
||||
valid_http_versions:
|
||||
- HTTP/1.1
|
||||
- HTTP/2.0
|
||||
follow_redirects: true
|
||||
enable_http2: true
|
||||
preferred_ip_protocol: ip4
|
||||
ip_protocol_fallback: false
|
||||
@@ -0,0 +1,208 @@
|
||||
{
|
||||
"annotations": {
|
||||
"list": []
|
||||
},
|
||||
"editable": false,
|
||||
"fiscalYearStartMonth": 0,
|
||||
"graphTooltip": 1,
|
||||
"id": null,
|
||||
"links": [],
|
||||
"panels": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"max": 1,
|
||||
"min": 0,
|
||||
"unit": "short"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 0
|
||||
},
|
||||
"id": 1,
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "table",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi"
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"editorMode": "code",
|
||||
"expr": "min by (job, instance) (probe_success{})",
|
||||
"legendFormat": "{{job}} / {{instance}}",
|
||||
"range": true,
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "HTTPS Probe Success",
|
||||
"type": "timeseries"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"min": 0,
|
||||
"unit": "s"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 0
|
||||
},
|
||||
"id": 2,
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "table",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi"
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"editorMode": "code",
|
||||
"expr": "max by (job, instance) (probe_duration_seconds{})",
|
||||
"legendFormat": "{{job}} / {{instance}}",
|
||||
"range": true,
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "HTTPS Probe Duration",
|
||||
"type": "timeseries"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"min": 0,
|
||||
"unit": "short"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 8
|
||||
},
|
||||
"id": 3,
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "table",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi"
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"editorMode": "code",
|
||||
"expr": "max by (instance) (probe_http_status_code{job=\"blackbox-private-edge\"})",
|
||||
"legendFormat": "{{instance}}",
|
||||
"range": true,
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Observed HTTP Status (403 Is Expected at the Private Edge)",
|
||||
"type": "timeseries"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"min": 0,
|
||||
"unit": "d"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 8
|
||||
},
|
||||
"id": 4,
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "table",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi"
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"editorMode": "code",
|
||||
"expr": "(probe_ssl_earliest_cert_expiry{} - time()) / 86400",
|
||||
"legendFormat": "{{job}} / {{instance}}",
|
||||
"range": true,
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "TLS Certificate Days Remaining",
|
||||
"type": "timeseries"
|
||||
}
|
||||
],
|
||||
"refresh": "30s",
|
||||
"schemaVersion": 42,
|
||||
"tags": [
|
||||
"platform",
|
||||
"observability",
|
||||
"https"
|
||||
],
|
||||
"templating": {
|
||||
"list": []
|
||||
},
|
||||
"time": {
|
||||
"from": "now-6h",
|
||||
"to": "now"
|
||||
},
|
||||
"timepicker": {},
|
||||
"timezone": "browser",
|
||||
"title": "Platform / HTTPS Endpoints",
|
||||
"uid": "platform-https-endpoints",
|
||||
"version": 1
|
||||
}
|
||||
@@ -0,0 +1,266 @@
|
||||
{
|
||||
"annotations": {
|
||||
"list": []
|
||||
},
|
||||
"editable": false,
|
||||
"fiscalYearStartMonth": 0,
|
||||
"graphTooltip": 1,
|
||||
"id": null,
|
||||
"links": [],
|
||||
"panels": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"max": 100,
|
||||
"min": 0,
|
||||
"unit": "percent"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 0
|
||||
},
|
||||
"id": 1,
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "table",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi"
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"editorMode": "code",
|
||||
"expr": "100 * (1 - avg by (instance) (rate(node_cpu_seconds_total{mode=\"idle\"}[5m])))",
|
||||
"legendFormat": "{{instance}}",
|
||||
"range": true,
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Node CPU Used",
|
||||
"type": "timeseries"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"max": 100,
|
||||
"min": 0,
|
||||
"unit": "percent"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 0
|
||||
},
|
||||
"id": 2,
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "table",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi"
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"editorMode": "code",
|
||||
"expr": "100 * (1 - node_memory_MemAvailable_bytes{} / node_memory_MemTotal_bytes{})",
|
||||
"legendFormat": "{{instance}}",
|
||||
"range": true,
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Node Memory Used",
|
||||
"type": "timeseries"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"max": 100,
|
||||
"min": 0,
|
||||
"unit": "percent"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 8
|
||||
},
|
||||
"id": 3,
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "table",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi"
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"editorMode": "code",
|
||||
"expr": "100 * (1 - node_filesystem_avail_bytes{mountpoint=\"/\"} / node_filesystem_size_bytes{mountpoint=\"/\"})",
|
||||
"legendFormat": "{{instance}} {{device}}",
|
||||
"range": true,
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Root Filesystem Used",
|
||||
"type": "timeseries"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"max": 100,
|
||||
"min": 0,
|
||||
"unit": "percent"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 8
|
||||
},
|
||||
"id": 4,
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "table",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi"
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"editorMode": "code",
|
||||
"expr": "100 * (1 - node_filesystem_files_free{mountpoint=\"/\"} / node_filesystem_files{mountpoint=\"/\"})",
|
||||
"legendFormat": "{{instance}} {{device}}",
|
||||
"range": true,
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Root Filesystem Inodes Used",
|
||||
"type": "timeseries"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"min": 0,
|
||||
"unit": "Bps"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 16
|
||||
},
|
||||
"id": 5,
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "table",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi"
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"editorMode": "code",
|
||||
"expr": "sum by (instance) (rate(node_network_receive_bytes_total{device!=\"lo\"}[5m]))",
|
||||
"legendFormat": "{{instance}} receive",
|
||||
"range": true,
|
||||
"refId": "A"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"editorMode": "code",
|
||||
"expr": "sum by (instance) (rate(node_network_transmit_bytes_total{device!=\"lo\"}[5m]))",
|
||||
"legendFormat": "{{instance}} transmit",
|
||||
"range": true,
|
||||
"refId": "B"
|
||||
}
|
||||
],
|
||||
"title": "Node Network Throughput",
|
||||
"type": "timeseries"
|
||||
}
|
||||
],
|
||||
"refresh": "30s",
|
||||
"schemaVersion": 42,
|
||||
"tags": [
|
||||
"platform",
|
||||
"observability",
|
||||
"node"
|
||||
],
|
||||
"templating": {
|
||||
"list": []
|
||||
},
|
||||
"time": {
|
||||
"from": "now-6h",
|
||||
"to": "now"
|
||||
},
|
||||
"timepicker": {},
|
||||
"timezone": "browser",
|
||||
"title": "Platform / Kubernetes Node",
|
||||
"uid": "platform-kubernetes-node",
|
||||
"version": 1
|
||||
}
|
||||
@@ -0,0 +1,28 @@
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
|
||||
namespace: observability
|
||||
|
||||
generatorOptions:
|
||||
disableNameSuffixHash: true
|
||||
labels:
|
||||
grafana_dashboard: "1"
|
||||
observability.hyeonworks.com/instance: home
|
||||
observability.hyeonworks.com/owner: platform-observability
|
||||
|
||||
configMapGenerator:
|
||||
- name: grafana-dashboard-kubernetes-node
|
||||
files:
|
||||
- kubernetes-node.json
|
||||
- name: grafana-dashboard-workload-health
|
||||
files:
|
||||
- workload-health.json
|
||||
- name: grafana-dashboard-platform-services
|
||||
files:
|
||||
- platform-services.json
|
||||
- name: grafana-dashboard-observability-backends
|
||||
files:
|
||||
- observability-backends.json
|
||||
- name: grafana-dashboard-https-endpoints
|
||||
files:
|
||||
- https-endpoints.json
|
||||
@@ -0,0 +1,384 @@
|
||||
{
|
||||
"annotations": {
|
||||
"list": []
|
||||
},
|
||||
"editable": false,
|
||||
"fiscalYearStartMonth": 0,
|
||||
"graphTooltip": 1,
|
||||
"id": null,
|
||||
"links": [],
|
||||
"panels": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"min": 0,
|
||||
"unit": "short"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 0
|
||||
},
|
||||
"id": 1,
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "table",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi"
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"editorMode": "code",
|
||||
"expr": "max(prometheus_tsdb_head_series{})",
|
||||
"legendFormat": "head series",
|
||||
"range": true,
|
||||
"refId": "A"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"editorMode": "code",
|
||||
"expr": "sum(prometheus_engine_queries{})",
|
||||
"legendFormat": "active queries",
|
||||
"range": true,
|
||||
"refId": "B"
|
||||
}
|
||||
],
|
||||
"title": "Prometheus Head Series and Active Queries",
|
||||
"type": "timeseries"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"min": 0,
|
||||
"unit": "ops"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 0
|
||||
},
|
||||
"id": 2,
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "table",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi"
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"editorMode": "code",
|
||||
"expr": "sum(rate(prometheus_notifications_errors_total{}[5m]))",
|
||||
"legendFormat": "errors / second",
|
||||
"range": true,
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Prometheus Notification Error Rate",
|
||||
"type": "timeseries"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"min": 0,
|
||||
"unit": "short"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 8
|
||||
},
|
||||
"id": 3,
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "table",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi"
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"editorMode": "code",
|
||||
"expr": "sum(loki_ingester_memory_chunks{})",
|
||||
"legendFormat": "memory chunks",
|
||||
"range": true,
|
||||
"refId": "A"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"editorMode": "code",
|
||||
"expr": "sum(loki_ingester_flush_queue_length{})",
|
||||
"legendFormat": "flush queue",
|
||||
"range": true,
|
||||
"refId": "B"
|
||||
}
|
||||
],
|
||||
"title": "Loki Ingester State",
|
||||
"type": "timeseries"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"min": 0,
|
||||
"unit": "s"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 8
|
||||
},
|
||||
"id": 4,
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "table",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi"
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"editorMode": "code",
|
||||
"expr": "time() - max(loki_objstore_bucket_last_successful_upload_time{})",
|
||||
"legendFormat": "seconds since upload",
|
||||
"range": true,
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Loki Object Store Upload Age",
|
||||
"type": "timeseries"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"min": 0,
|
||||
"unit": "short"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 16
|
||||
},
|
||||
"id": 5,
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "table",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi"
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"editorMode": "code",
|
||||
"expr": "sum(tempo_ingester_live_traces{})",
|
||||
"legendFormat": "live traces",
|
||||
"range": true,
|
||||
"refId": "A"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"editorMode": "code",
|
||||
"expr": "sum(tempo_ingester_flush_queue_length{})",
|
||||
"legendFormat": "flush queue",
|
||||
"range": true,
|
||||
"refId": "B"
|
||||
}
|
||||
],
|
||||
"title": "Tempo Ingester State",
|
||||
"type": "timeseries"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"max": 1,
|
||||
"min": 0,
|
||||
"unit": "short"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 16
|
||||
},
|
||||
"id": 6,
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "table",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi"
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"editorMode": "code",
|
||||
"expr": "min(alloy_config_last_load_successful{})",
|
||||
"legendFormat": "last load successful",
|
||||
"range": true,
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Alloy Configuration State",
|
||||
"type": "timeseries"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"min": 0,
|
||||
"unit": "ops"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 24
|
||||
},
|
||||
"id": 7,
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "table",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi"
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"editorMode": "code",
|
||||
"expr": "sum(rate(loki_write_batch_retries_total{}[5m]))",
|
||||
"legendFormat": "retry / second",
|
||||
"range": true,
|
||||
"refId": "A"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"editorMode": "code",
|
||||
"expr": "sum(rate(loki_write_dropped_entries_total{}[5m]))",
|
||||
"legendFormat": "dropped / second",
|
||||
"range": true,
|
||||
"refId": "B"
|
||||
}
|
||||
],
|
||||
"title": "Alloy Delivery Retry and Drop Rate",
|
||||
"type": "timeseries"
|
||||
}
|
||||
],
|
||||
"refresh": "30s",
|
||||
"schemaVersion": 42,
|
||||
"tags": [
|
||||
"platform",
|
||||
"observability",
|
||||
"backends"
|
||||
],
|
||||
"templating": {
|
||||
"list": []
|
||||
},
|
||||
"time": {
|
||||
"from": "now-6h",
|
||||
"to": "now"
|
||||
},
|
||||
"timepicker": {},
|
||||
"timezone": "browser",
|
||||
"title": "Platform / Observability Backends",
|
||||
"uid": "platform-observability-backends",
|
||||
"version": 1
|
||||
}
|
||||
@@ -0,0 +1,419 @@
|
||||
{
|
||||
"annotations": {
|
||||
"list": []
|
||||
},
|
||||
"editable": false,
|
||||
"fiscalYearStartMonth": 0,
|
||||
"graphTooltip": 1,
|
||||
"id": null,
|
||||
"links": [],
|
||||
"panels": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"min": 0,
|
||||
"unit": "short"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 8,
|
||||
"x": 0,
|
||||
"y": 0
|
||||
},
|
||||
"id": 1,
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "table",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi"
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"editorMode": "code",
|
||||
"expr": "max(gitea_repositories{})",
|
||||
"legendFormat": "repositories",
|
||||
"range": true,
|
||||
"refId": "A"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"editorMode": "code",
|
||||
"expr": "sum(gitea_users{})",
|
||||
"legendFormat": "users",
|
||||
"range": true,
|
||||
"refId": "B"
|
||||
}
|
||||
],
|
||||
"title": "Gitea Inventory",
|
||||
"type": "timeseries"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"min": 0,
|
||||
"unit": "short"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 8,
|
||||
"x": 8,
|
||||
"y": 0
|
||||
},
|
||||
"id": 2,
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "table",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi"
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"editorMode": "code",
|
||||
"expr": "sum(agroal_active_count{})",
|
||||
"legendFormat": "active",
|
||||
"range": true,
|
||||
"refId": "A"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"editorMode": "code",
|
||||
"expr": "sum(agroal_available_count{})",
|
||||
"legendFormat": "available",
|
||||
"range": true,
|
||||
"refId": "B"
|
||||
}
|
||||
],
|
||||
"title": "Keycloak Database Pool",
|
||||
"type": "timeseries"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"min": 0,
|
||||
"unit": "short"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 8,
|
||||
"x": 16,
|
||||
"y": 0
|
||||
},
|
||||
"id": 3,
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "table",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi"
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"editorMode": "code",
|
||||
"expr": "min(cnpg_collector_up{})",
|
||||
"legendFormat": "collector up",
|
||||
"range": true,
|
||||
"refId": "A"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"editorMode": "code",
|
||||
"expr": "sum(cnpg_backends_total{})",
|
||||
"legendFormat": "backends",
|
||||
"range": true,
|
||||
"refId": "B"
|
||||
}
|
||||
],
|
||||
"title": "CloudNativePG Collector and Connections",
|
||||
"type": "timeseries"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"min": 0,
|
||||
"unit": "bytes"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 8
|
||||
},
|
||||
"id": 4,
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "table",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi"
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"editorMode": "code",
|
||||
"expr": "sum by (datname) (cnpg_pg_database_size_bytes{})",
|
||||
"legendFormat": "{{datname}}",
|
||||
"range": true,
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "CloudNativePG Database Size",
|
||||
"type": "timeseries"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"max": 100,
|
||||
"min": 0,
|
||||
"unit": "percent"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 8
|
||||
},
|
||||
"id": 5,
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "table",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi"
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"editorMode": "code",
|
||||
"expr": "100 * max by (bucket) (minio_cluster_usage_buckets_total_bytes{bucket=~\"loki|tempo\"}) / max by (bucket) (minio_cluster_usage_buckets_quota_total_bytes{bucket=~\"loki|tempo\"} > 0)",
|
||||
"legendFormat": "{{bucket}}",
|
||||
"range": true,
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "AIStor Bucket Quota Used",
|
||||
"type": "timeseries"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"min": 0,
|
||||
"unit": "reqps"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 16
|
||||
},
|
||||
"id": 6,
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "table",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi"
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"editorMode": "code",
|
||||
"expr": "sum by (entrypoint) (rate(traefik_entrypoint_requests_total{}[5m]))",
|
||||
"legendFormat": "{{entrypoint}}",
|
||||
"range": true,
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Traefik Request Rate",
|
||||
"type": "timeseries"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"max": 100,
|
||||
"min": 0,
|
||||
"unit": "percent"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 16
|
||||
},
|
||||
"id": 7,
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "table",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi"
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"editorMode": "code",
|
||||
"expr": "100 * sum(rate(traefik_entrypoint_requests_total{code=~\"5..\"}[5m])) / clamp_min(sum(rate(traefik_entrypoint_requests_total{}[5m])), 1e-9)",
|
||||
"legendFormat": "5xx ratio",
|
||||
"range": true,
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Traefik 5xx Ratio",
|
||||
"type": "timeseries"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"max": 1,
|
||||
"min": 0,
|
||||
"unit": "short"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 24
|
||||
},
|
||||
"id": 8,
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "table",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi"
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"editorMode": "code",
|
||||
"expr": "min by (job) (probe_success{})",
|
||||
"legendFormat": "{{job}}",
|
||||
"range": true,
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Platform HTTPS Boundary Health",
|
||||
"type": "timeseries"
|
||||
}
|
||||
],
|
||||
"refresh": "30s",
|
||||
"schemaVersion": 42,
|
||||
"tags": [
|
||||
"platform",
|
||||
"observability",
|
||||
"services"
|
||||
],
|
||||
"templating": {
|
||||
"list": []
|
||||
},
|
||||
"time": {
|
||||
"from": "now-6h",
|
||||
"to": "now"
|
||||
},
|
||||
"timepicker": {},
|
||||
"timezone": "browser",
|
||||
"title": "Platform / Services",
|
||||
"uid": "platform-services",
|
||||
"version": 1
|
||||
}
|
||||
@@ -0,0 +1,218 @@
|
||||
{
|
||||
"annotations": {
|
||||
"list": []
|
||||
},
|
||||
"editable": false,
|
||||
"fiscalYearStartMonth": 0,
|
||||
"graphTooltip": 1,
|
||||
"id": null,
|
||||
"links": [],
|
||||
"panels": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"min": 0,
|
||||
"unit": "short"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 0
|
||||
},
|
||||
"id": 1,
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "table",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi"
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"editorMode": "code",
|
||||
"expr": "sum by (namespace, deployment) (kube_deployment_status_replicas_unavailable{})",
|
||||
"legendFormat": "{{namespace}} / {{deployment}}",
|
||||
"range": true,
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Deployment Unavailable Replicas",
|
||||
"type": "timeseries"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"min": 0,
|
||||
"unit": "short"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 0
|
||||
},
|
||||
"id": 2,
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "table",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi"
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"editorMode": "code",
|
||||
"expr": "clamp_min(sum by (namespace, statefulset) (kube_statefulset_replicas{}) - sum by (namespace, statefulset) (kube_statefulset_status_replicas_ready{}), 0)",
|
||||
"legendFormat": "{{namespace}} / {{statefulset}}",
|
||||
"range": true,
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "StatefulSet Replicas Not Ready",
|
||||
"type": "timeseries"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"min": 0,
|
||||
"unit": "short"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 8
|
||||
},
|
||||
"id": 3,
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "table",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi"
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"editorMode": "code",
|
||||
"expr": "sum by (namespace, pod) (increase(kube_pod_container_status_restarts_total{}[15m]))",
|
||||
"legendFormat": "{{namespace}} / {{pod}}",
|
||||
"range": true,
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Container Restarts in 15 Minutes",
|
||||
"type": "timeseries"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"min": 0,
|
||||
"unit": "short"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 8
|
||||
},
|
||||
"id": 4,
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "table",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi"
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"editorMode": "code",
|
||||
"expr": "sum(kube_persistentvolumeclaim_status_phase{phase=\"Pending\"})",
|
||||
"legendFormat": "Pending",
|
||||
"range": true,
|
||||
"refId": "A"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"editorMode": "code",
|
||||
"expr": "sum(kube_persistentvolumeclaim_status_phase{phase=\"Lost\"})",
|
||||
"legendFormat": "Lost",
|
||||
"range": true,
|
||||
"refId": "B"
|
||||
}
|
||||
],
|
||||
"title": "PersistentVolumeClaim Health",
|
||||
"type": "timeseries"
|
||||
}
|
||||
],
|
||||
"refresh": "30s",
|
||||
"schemaVersion": 42,
|
||||
"tags": [
|
||||
"platform",
|
||||
"observability",
|
||||
"workloads"
|
||||
],
|
||||
"templating": {
|
||||
"list": []
|
||||
},
|
||||
"time": {
|
||||
"from": "now-6h",
|
||||
"to": "now"
|
||||
},
|
||||
"timepicker": {},
|
||||
"timezone": "browser",
|
||||
"title": "Platform / Workload Health",
|
||||
"uid": "platform-workload-health",
|
||||
"version": 1
|
||||
}
|
||||
@@ -0,0 +1,24 @@
|
||||
# Grafana
|
||||
|
||||
Private Grafana for the home observability stack. The pinned Grafana chart is
|
||||
rendered as the independent `grafana` release in `observability`; the
|
||||
kube-prometheus-stack embedded Grafana remains disabled.
|
||||
|
||||
The workload consumes, but never renders, these existing Secret contracts:
|
||||
|
||||
- `grafana-admin`: `admin-user`, `admin-password`
|
||||
- `grafana-keycloak-oidc`: `client-id`, `client-secret`
|
||||
|
||||
The `grafana` PVC is fixed to `observability-grafana-local-pv` (2Gi, Retain).
|
||||
Host Nginx terminates the exact-SAN certificate and forwards plain HTTP through
|
||||
Traefik; the Kubernetes Ingress therefore contains no TLS material.
|
||||
|
||||
The Pod does not automount a service-account token. A 3600-second projected
|
||||
token, CA, and namespace are mounted only into `grafana-sc-dashboard`, whose
|
||||
namespaced Role can read ConfigMaps but not Secrets. Dashboard ConfigMaps must
|
||||
carry `grafana_dashboard: "1"` in `observability`.
|
||||
|
||||
Provisioned datasource UIDs are `prometheus`, `loki`, and `tempo`. Tempo service
|
||||
map configuration is present for future use, but the metrics-generator and
|
||||
Prometheus remote-write path are intentionally not enabled, so service-graph
|
||||
availability is not an initial acceptance condition.
|
||||
@@ -0,0 +1,27 @@
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: Role
|
||||
metadata:
|
||||
name: grafana-dashboard-sidecar
|
||||
namespace: observability
|
||||
rules:
|
||||
- apiGroups:
|
||||
- ""
|
||||
resources:
|
||||
- configmaps
|
||||
verbs:
|
||||
- list
|
||||
- watch
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: RoleBinding
|
||||
metadata:
|
||||
name: grafana-dashboard-sidecar
|
||||
namespace: observability
|
||||
roleRef:
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
kind: Role
|
||||
name: grafana-dashboard-sidecar
|
||||
subjects:
|
||||
- kind: ServiceAccount
|
||||
name: grafana
|
||||
namespace: observability
|
||||
@@ -0,0 +1,37 @@
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: grafana
|
||||
namespace: observability
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
automountServiceAccountToken: false
|
||||
hostNetwork: false
|
||||
hostPID: false
|
||||
hostIPC: false
|
||||
containers:
|
||||
- name: grafana-sc-dashboard
|
||||
volumeMounts:
|
||||
- name: dashboard-sidecar-api-access
|
||||
mountPath: /var/run/secrets/kubernetes.io/serviceaccount
|
||||
readOnly: true
|
||||
volumes:
|
||||
- name: dashboard-sidecar-api-access
|
||||
projected:
|
||||
defaultMode: 420
|
||||
sources:
|
||||
- serviceAccountToken:
|
||||
expirationSeconds: 3600
|
||||
path: token
|
||||
- configMap:
|
||||
name: kube-root-ca.crt
|
||||
items:
|
||||
- key: ca.crt
|
||||
path: ca.crt
|
||||
- downwardAPI:
|
||||
items:
|
||||
- path: namespace
|
||||
fieldRef:
|
||||
apiVersion: v1
|
||||
fieldPath: metadata.namespace
|
||||
@@ -0,0 +1,7 @@
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: Ingress
|
||||
metadata:
|
||||
name: grafana
|
||||
namespace: observability
|
||||
spec:
|
||||
ingressClassName: traefik
|
||||
@@ -0,0 +1,24 @@
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
|
||||
namespace: observability
|
||||
|
||||
resources:
|
||||
- dashboard-sidecar-rbac.yaml
|
||||
|
||||
helmGlobals:
|
||||
chartHome: .helm/charts
|
||||
|
||||
helmCharts:
|
||||
- name: grafana
|
||||
repo: https://grafana-community.github.io/helm-charts
|
||||
version: 12.10.1
|
||||
releaseName: grafana
|
||||
namespace: observability
|
||||
valuesFile: values/home.yaml
|
||||
includeCRDs: false
|
||||
skipTests: true
|
||||
|
||||
patches:
|
||||
- path: ingress-class-patch.yaml
|
||||
- path: dashboard-sidecar-token-patch.yaml
|
||||
@@ -0,0 +1,262 @@
|
||||
replicas: 1
|
||||
|
||||
extraLabels:
|
||||
observability.hyeonworks.com/instance: home
|
||||
|
||||
deploymentStrategy:
|
||||
type: Recreate
|
||||
|
||||
revisionHistoryLimit: 3
|
||||
|
||||
automountServiceAccountToken: false
|
||||
enableServiceLinks: false
|
||||
|
||||
rbac:
|
||||
create: false
|
||||
|
||||
serviceAccount:
|
||||
create: true
|
||||
name: grafana
|
||||
automountServiceAccountToken: false
|
||||
|
||||
image:
|
||||
registry: docker.io
|
||||
repository: grafana/grafana
|
||||
tag: 13.1.1
|
||||
sha: f33c692ba1a5ee15724cf6b22db65e9de39dde14d80f7d73a9546e3fc917270b
|
||||
pullPolicy: IfNotPresent
|
||||
|
||||
testFramework:
|
||||
enabled: false
|
||||
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
runAsUser: 472
|
||||
runAsGroup: 472
|
||||
fsGroup: 472
|
||||
fsGroupChangePolicy: OnRootMismatch
|
||||
seccompProfile:
|
||||
type: RuntimeDefault
|
||||
|
||||
containerSecurityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
privileged: false
|
||||
capabilities:
|
||||
drop:
|
||||
- ALL
|
||||
seccompProfile:
|
||||
type: RuntimeDefault
|
||||
|
||||
resources:
|
||||
requests:
|
||||
cpu: 100m
|
||||
memory: 256Mi
|
||||
limits:
|
||||
cpu: 500m
|
||||
memory: 512Mi
|
||||
|
||||
initChownData:
|
||||
enabled: false
|
||||
|
||||
persistence:
|
||||
type: pvc
|
||||
enabled: true
|
||||
storageClassName: ssd-local-observability-grafana-retain
|
||||
volumeName: observability-grafana-local-pv
|
||||
lookupVolumeName: false
|
||||
accessModes:
|
||||
- ReadWriteOnce
|
||||
size: 2Gi
|
||||
finalizers:
|
||||
- kubernetes.io/pvc-protection
|
||||
|
||||
admin:
|
||||
existingSecret: grafana-admin
|
||||
userKey: admin-user
|
||||
passwordKey: admin-password
|
||||
|
||||
envValueFrom:
|
||||
GF_AUTH_GENERIC_OAUTH_CLIENT_ID:
|
||||
secretKeyRef:
|
||||
name: grafana-keycloak-oidc
|
||||
key: client-id
|
||||
GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET:
|
||||
secretKeyRef:
|
||||
name: grafana-keycloak-oidc
|
||||
key: client-secret
|
||||
|
||||
service:
|
||||
enabled: true
|
||||
type: ClusterIP
|
||||
port: 80
|
||||
targetPort: 3000
|
||||
portName: service
|
||||
|
||||
serviceMonitor:
|
||||
enabled: true
|
||||
interval: 30s
|
||||
path: /metrics
|
||||
|
||||
ingress:
|
||||
enabled: true
|
||||
annotations: {}
|
||||
path: /
|
||||
pathType: Prefix
|
||||
hosts:
|
||||
- grafana.learn.hyeonworks.com
|
||||
tls: []
|
||||
|
||||
grafana.ini:
|
||||
paths:
|
||||
data: /var/lib/grafana/
|
||||
logs: /var/log/grafana
|
||||
plugins: /var/lib/grafana/plugins
|
||||
provisioning: /etc/grafana/provisioning
|
||||
analytics:
|
||||
check_for_updates: false
|
||||
check_for_plugin_updates: false
|
||||
reporting_enabled: false
|
||||
plugins:
|
||||
preinstall_disabled: true
|
||||
server:
|
||||
domain: grafana.learn.hyeonworks.com
|
||||
root_url: https://grafana.learn.hyeonworks.com/
|
||||
enforce_domain: true
|
||||
security:
|
||||
cookie_secure: true
|
||||
cookie_samesite: lax
|
||||
auth:
|
||||
disable_login_form: false
|
||||
oauth_auto_login: false
|
||||
login_maximum_lifetime_duration: 8h
|
||||
login_maximum_inactive_lifetime_duration: 30m
|
||||
auth.anonymous:
|
||||
enabled: false
|
||||
auth.basic:
|
||||
enabled: true
|
||||
auth.generic_oauth:
|
||||
enabled: true
|
||||
name: Keycloak
|
||||
allow_sign_up: true
|
||||
use_pkce: true
|
||||
scopes: openid profile email
|
||||
groups_attribute_path: groups
|
||||
allowed_groups: /platform-observability-admins /platform-observability-viewers
|
||||
role_attribute_strict: true
|
||||
allow_assign_grafana_admin: false
|
||||
skip_org_role_sync: false
|
||||
validate_id_token: true
|
||||
use_refresh_token: true
|
||||
role_attribute_path: "contains(groups[*], '/platform-observability-admins') && 'Admin' || contains(groups[*], '/platform-observability-viewers') && 'Viewer' || null"
|
||||
auth_url: https://id.learn.hyeonworks.com/realms/hyeonworks/protocol/openid-connect/auth
|
||||
token_url: https://id.learn.hyeonworks.com/realms/hyeonworks/protocol/openid-connect/token
|
||||
api_url: https://id.learn.hyeonworks.com/realms/hyeonworks/protocol/openid-connect/userinfo
|
||||
signout_redirect_url: https://id.learn.hyeonworks.com/realms/hyeonworks/protocol/openid-connect/logout
|
||||
|
||||
datasources:
|
||||
datasources.yaml:
|
||||
apiVersion: 1
|
||||
deleteDatasources:
|
||||
- name: Prometheus
|
||||
orgId: 1
|
||||
- name: Loki
|
||||
orgId: 1
|
||||
- name: Tempo
|
||||
orgId: 1
|
||||
datasources:
|
||||
- name: Prometheus
|
||||
uid: prometheus
|
||||
type: prometheus
|
||||
access: proxy
|
||||
url: http://observability-core-kube-pr-prometheus.observability.svc.cluster.local:9090
|
||||
isDefault: true
|
||||
editable: false
|
||||
jsonData:
|
||||
httpMethod: POST
|
||||
timeInterval: 30s
|
||||
- name: Loki
|
||||
uid: loki
|
||||
type: loki
|
||||
access: proxy
|
||||
url: http://loki.observability.svc.cluster.local:3100
|
||||
isDefault: false
|
||||
editable: false
|
||||
jsonData:
|
||||
derivedFields:
|
||||
- name: trace_id
|
||||
matcherRegex: '"trace_id"[[:space:]]*:[[:space:]]*"([0-9a-f]{32})"'
|
||||
datasourceUid: tempo
|
||||
url: '$${__value.raw}'
|
||||
- name: Tempo
|
||||
uid: tempo
|
||||
type: tempo
|
||||
access: proxy
|
||||
url: http://tempo.observability.svc.cluster.local:3200
|
||||
isDefault: false
|
||||
editable: false
|
||||
jsonData:
|
||||
httpMethod: GET
|
||||
nodeGraph:
|
||||
enabled: true
|
||||
serviceMap:
|
||||
datasourceUid: prometheus
|
||||
tracesToLogsV2:
|
||||
datasourceUid: loki
|
||||
spanStartTimeShift: -1m
|
||||
spanEndTimeShift: 1m
|
||||
tags:
|
||||
- key: k8s.namespace.name
|
||||
value: namespace
|
||||
- key: k8s.pod.name
|
||||
value: pod
|
||||
filterByTraceID: true
|
||||
filterBySpanID: false
|
||||
|
||||
sidecar:
|
||||
image:
|
||||
registry: quay.io
|
||||
repository: kiwigrid/k8s-sidecar
|
||||
tag: 2.10.0
|
||||
sha: 129877c81acf2bc8c3fa000e89a62e020eb89d41ceb94767c657aef5bb0cc0d3
|
||||
imagePullPolicy: IfNotPresent
|
||||
resources:
|
||||
requests:
|
||||
cpu: 25m
|
||||
memory: 64Mi
|
||||
limits:
|
||||
cpu: 100m
|
||||
memory: 128Mi
|
||||
securityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
readOnlyRootFilesystem: true
|
||||
capabilities:
|
||||
drop:
|
||||
- ALL
|
||||
seccompProfile:
|
||||
type: RuntimeDefault
|
||||
dashboards:
|
||||
enabled: true
|
||||
label: grafana_dashboard
|
||||
labelValue: "1"
|
||||
searchNamespace:
|
||||
- observability
|
||||
resource: configmap
|
||||
watchMethod: WATCH
|
||||
skipReload: true
|
||||
provider:
|
||||
name: sidecarProvider
|
||||
orgid: 1
|
||||
folder: ""
|
||||
folderUid: ""
|
||||
type: file
|
||||
disableDelete: false
|
||||
allowUiUpdates: false
|
||||
foldersFromFilesStructure: false
|
||||
|
||||
imageRenderer:
|
||||
enabled: false
|
||||
|
||||
networkPolicy:
|
||||
enabled: false
|
||||
|
||||
assertNoLeakedSecrets: true
|
||||
@@ -0,0 +1,15 @@
|
||||
# kube-prometheus-stack
|
||||
|
||||
고정 chart `88.0.1`을 `observability` namespace에 렌더한다. Grafana와
|
||||
node-exporter, control-plane에 존재하지 않는 scheduler/controller-manager/etcd/
|
||||
kube-proxy monitor는 끈다. Prometheus와 Alertmanager 데이터는 각각의 `Retain`
|
||||
Local PV에 보존하며 CRD는 별도 root에서 먼저 적용한다.
|
||||
|
||||
Prometheus discovery는 namespace label
|
||||
`observability.hyeonworks.com/enabled=true`와 resource label
|
||||
`observability.hyeonworks.com/instance=home`의 교집합만 선택한다.
|
||||
|
||||
Alertmanager source는 selector discovery 대신 global
|
||||
`alertmanagerConfiguration.name=platform-alertmanager`를 참조한다. 참조 대상의
|
||||
Slack Secret이 준비되기 전에는 이 변경을 live에 적용하지 않고 기존 null 구성을
|
||||
유지한다.
|
||||
@@ -0,0 +1,37 @@
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
|
||||
namespace: observability
|
||||
|
||||
helmGlobals:
|
||||
chartHome: .helm/charts
|
||||
|
||||
helmCharts:
|
||||
- name: kube-prometheus-stack
|
||||
repo: https://prometheus-community.github.io/helm-charts
|
||||
version: 88.0.1
|
||||
releaseName: observability-core
|
||||
namespace: observability
|
||||
valuesFile: values/home.yaml
|
||||
includeCRDs: false
|
||||
skipTests: true
|
||||
|
||||
images:
|
||||
- name: quay.io/prometheus-operator/prometheus-operator
|
||||
newName: quay.io/prometheus-operator/prometheus-operator
|
||||
digest: sha256:64eb7914e4705dbb64438e3b3193da1226ad2ea4db2924983693999888cda9b2
|
||||
- name: registry.k8s.io/kube-state-metrics/kube-state-metrics
|
||||
newName: registry.k8s.io/kube-state-metrics/kube-state-metrics
|
||||
digest: sha256:7661da8c99b733d43117e4cba12bd9865d335e5777191d0af3d789807aded9f4
|
||||
|
||||
patches:
|
||||
- target:
|
||||
group: monitoring.coreos.com
|
||||
version: v1
|
||||
kind: Alertmanager
|
||||
name: observability-core-kube-pr-alertmanager
|
||||
patch: |-
|
||||
- op: remove
|
||||
path: /spec/alertmanagerConfigSelector
|
||||
- op: remove
|
||||
path: /spec/alertmanagerConfigNamespaceSelector
|
||||
@@ -0,0 +1,282 @@
|
||||
commonLabels:
|
||||
observability.hyeonworks.com/instance: home
|
||||
|
||||
crds:
|
||||
enabled: false
|
||||
|
||||
grafana:
|
||||
enabled: false
|
||||
|
||||
nodeExporter:
|
||||
enabled: false
|
||||
|
||||
kubeApiServer:
|
||||
enabled: true
|
||||
|
||||
kubelet:
|
||||
enabled: true
|
||||
|
||||
coreDns:
|
||||
enabled: true
|
||||
|
||||
kubeControllerManager:
|
||||
enabled: false
|
||||
|
||||
kubeScheduler:
|
||||
enabled: false
|
||||
|
||||
kubeEtcd:
|
||||
enabled: false
|
||||
|
||||
kubeProxy:
|
||||
enabled: false
|
||||
|
||||
defaultRules:
|
||||
disabled:
|
||||
Watchdog: true
|
||||
rules:
|
||||
etcd: false
|
||||
kubeControllerManager: false
|
||||
kubeProxy: false
|
||||
kubeSchedulerAlerting: false
|
||||
kubeSchedulerRecording: false
|
||||
windows: false
|
||||
|
||||
prometheusOperator:
|
||||
admissionWebhooks:
|
||||
enabled: false
|
||||
patch:
|
||||
enabled: false
|
||||
tls:
|
||||
enabled: false
|
||||
image:
|
||||
registry: quay.io
|
||||
repository: prometheus-operator/prometheus-operator
|
||||
tag: v0.93.0
|
||||
sha: 64eb7914e4705dbb64438e3b3193da1226ad2ea4db2924983693999888cda9b2
|
||||
resources:
|
||||
requests:
|
||||
cpu: 100m
|
||||
memory: 128Mi
|
||||
limits:
|
||||
cpu: 500m
|
||||
memory: 256Mi
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
runAsUser: 65534
|
||||
runAsGroup: 65534
|
||||
fsGroup: 65534
|
||||
seccompProfile:
|
||||
type: RuntimeDefault
|
||||
containerSecurityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
capabilities:
|
||||
drop:
|
||||
- ALL
|
||||
readOnlyRootFilesystem: true
|
||||
serviceAccount:
|
||||
automountServiceAccountToken: true
|
||||
prometheusConfigReloader:
|
||||
image:
|
||||
registry: quay.io
|
||||
repository: prometheus-operator/prometheus-config-reloader
|
||||
tag: v0.93.0
|
||||
sha: 65b90f44d5535b106015ac60bafb57803f65dc928c187874de6cd7a9ec6c8905
|
||||
resources:
|
||||
requests:
|
||||
cpu: 25m
|
||||
memory: 32Mi
|
||||
limits:
|
||||
cpu: 100m
|
||||
memory: 64Mi
|
||||
|
||||
kubeStateMetrics:
|
||||
enabled: true
|
||||
|
||||
kube-state-metrics:
|
||||
image:
|
||||
registry: registry.k8s.io
|
||||
repository: kube-state-metrics/kube-state-metrics
|
||||
tag: v2.19.1
|
||||
sha: sha256:7661da8c99b733d43117e4cba12bd9865d335e5777191d0af3d789807aded9f4
|
||||
resources:
|
||||
requests:
|
||||
cpu: 100m
|
||||
memory: 128Mi
|
||||
limits:
|
||||
cpu: 500m
|
||||
memory: 256Mi
|
||||
securityContext:
|
||||
enabled: true
|
||||
runAsNonRoot: true
|
||||
runAsUser: 65534
|
||||
runAsGroup: 65534
|
||||
fsGroup: 65534
|
||||
seccompProfile:
|
||||
type: RuntimeDefault
|
||||
containerSecurityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
capabilities:
|
||||
drop:
|
||||
- ALL
|
||||
readOnlyRootFilesystem: true
|
||||
automountServiceAccountToken: true
|
||||
collectors:
|
||||
- certificatesigningrequests
|
||||
- configmaps
|
||||
- cronjobs
|
||||
- daemonsets
|
||||
- deployments
|
||||
- endpointslices
|
||||
- horizontalpodautoscalers
|
||||
- ingresses
|
||||
- jobs
|
||||
- leases
|
||||
- limitranges
|
||||
- mutatingwebhookconfigurations
|
||||
- namespaces
|
||||
- networkpolicies
|
||||
- nodes
|
||||
- persistentvolumeclaims
|
||||
- persistentvolumes
|
||||
- poddisruptionbudgets
|
||||
- pods
|
||||
- replicasets
|
||||
- replicationcontrollers
|
||||
- resourcequotas
|
||||
- services
|
||||
- statefulsets
|
||||
- storageclasses
|
||||
- validatingwebhookconfigurations
|
||||
- volumeattachments
|
||||
prometheus:
|
||||
monitor:
|
||||
additionalLabels:
|
||||
observability.hyeonworks.com/instance: home
|
||||
|
||||
alertmanager:
|
||||
enabled: true
|
||||
config:
|
||||
global:
|
||||
resolve_timeout: 5m
|
||||
route:
|
||||
group_by:
|
||||
- namespace
|
||||
group_wait: 30s
|
||||
group_interval: 5m
|
||||
repeat_interval: 12h
|
||||
receiver: "null"
|
||||
routes: []
|
||||
receivers:
|
||||
- name: "null"
|
||||
serviceAccount:
|
||||
automountServiceAccountToken: false
|
||||
alertmanagerSpec:
|
||||
replicas: 1
|
||||
retention: 120h
|
||||
automountServiceAccountToken: false
|
||||
alertmanagerConfiguration:
|
||||
name: platform-alertmanager
|
||||
image:
|
||||
registry: quay.io
|
||||
repository: prometheus/alertmanager
|
||||
tag: v0.33.1
|
||||
sha: a89f8d4520954079275441eecdb71444328bd90633dd4eddfc33b9ed657f349b
|
||||
resources:
|
||||
requests:
|
||||
cpu: 50m
|
||||
memory: 128Mi
|
||||
limits:
|
||||
cpu: 250m
|
||||
memory: 256Mi
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
runAsUser: 1000
|
||||
runAsGroup: 2000
|
||||
fsGroup: 2000
|
||||
fsGroupChangePolicy: OnRootMismatch
|
||||
seccompProfile:
|
||||
type: RuntimeDefault
|
||||
persistentVolumeClaimRetentionPolicy:
|
||||
whenDeleted: Retain
|
||||
whenScaled: Retain
|
||||
storage:
|
||||
volumeClaimTemplate:
|
||||
spec:
|
||||
storageClassName: ssd-local-observability-alertmanager-retain
|
||||
accessModes:
|
||||
- ReadWriteOnce
|
||||
resources:
|
||||
requests:
|
||||
storage: 1Gi
|
||||
|
||||
prometheus:
|
||||
enabled: true
|
||||
prometheusSpec:
|
||||
replicas: 1
|
||||
retention: 15d
|
||||
retentionSize: 16GB
|
||||
scrapeInterval: 30s
|
||||
evaluationInterval: 30s
|
||||
walCompression: true
|
||||
automountServiceAccountToken: true
|
||||
image:
|
||||
registry: quay.io
|
||||
repository: prometheus/prometheus
|
||||
tag: v3.13.2-distroless
|
||||
sha: ce95cfa77eff5aad28bd7a65aff19868cf78d9e17e4c254da7dfe22ade78318b
|
||||
resources:
|
||||
requests:
|
||||
cpu: 750m
|
||||
memory: 2Gi
|
||||
limits:
|
||||
cpu: "2"
|
||||
memory: 3Gi
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
runAsUser: 1000
|
||||
runAsGroup: 2000
|
||||
fsGroup: 2000
|
||||
fsGroupChangePolicy: OnRootMismatch
|
||||
seccompProfile:
|
||||
type: RuntimeDefault
|
||||
persistentVolumeClaimRetentionPolicy:
|
||||
whenDeleted: Retain
|
||||
whenScaled: Retain
|
||||
storageSpec:
|
||||
volumeClaimTemplate:
|
||||
spec:
|
||||
storageClassName: ssd-local-observability-prometheus-retain
|
||||
accessModes:
|
||||
- ReadWriteOnce
|
||||
resources:
|
||||
requests:
|
||||
storage: 20Gi
|
||||
serviceMonitorSelectorNilUsesHelmValues: false
|
||||
podMonitorSelectorNilUsesHelmValues: false
|
||||
probeSelectorNilUsesHelmValues: false
|
||||
ruleSelectorNilUsesHelmValues: false
|
||||
serviceMonitorNamespaceSelector:
|
||||
matchLabels:
|
||||
observability.hyeonworks.com/enabled: "true"
|
||||
podMonitorNamespaceSelector:
|
||||
matchLabels:
|
||||
observability.hyeonworks.com/enabled: "true"
|
||||
probeNamespaceSelector:
|
||||
matchLabels:
|
||||
observability.hyeonworks.com/enabled: "true"
|
||||
ruleNamespaceSelector:
|
||||
matchLabels:
|
||||
observability.hyeonworks.com/enabled: "true"
|
||||
serviceMonitorSelector:
|
||||
matchLabels:
|
||||
observability.hyeonworks.com/instance: home
|
||||
podMonitorSelector:
|
||||
matchLabels:
|
||||
observability.hyeonworks.com/instance: home
|
||||
probeSelector:
|
||||
matchLabels:
|
||||
observability.hyeonworks.com/instance: home
|
||||
ruleSelector:
|
||||
matchLabels:
|
||||
observability.hyeonworks.com/instance: home
|
||||
@@ -0,0 +1,16 @@
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
|
||||
resources:
|
||||
- kube-prometheus-stack
|
||||
- loki
|
||||
- tempo
|
||||
- alloy
|
||||
- node-exporter
|
||||
- grafana
|
||||
- blackbox-exporter
|
||||
- network-policies
|
||||
- targets
|
||||
- rules/platform
|
||||
- alerting
|
||||
- dashboards/platform
|
||||
@@ -0,0 +1,16 @@
|
||||
# Loki
|
||||
|
||||
This root renders `grafana-community/loki` chart `18.7.1` as one Loki
|
||||
`3.7.4` monolithic StatefulSet in `observability`.
|
||||
|
||||
- The image is pinned to the reviewed amd64 digest.
|
||||
- Logs use the `loki` AIStor bucket through `loki-object-storage`; no root
|
||||
credential is referenced.
|
||||
- Retention is 30 days. The WAL and working data use the dedicated 5Gi
|
||||
Retain Local PV.
|
||||
- Gateway, canary, caches, embedded MinIO, tests, and rule sidecars are
|
||||
disabled.
|
||||
- The only client-facing service is ClusterIP port 3100.
|
||||
|
||||
Do not delete the PVC or bucket during rollback. Scale the StatefulSet down,
|
||||
preserve its Retain volume, and investigate S3/WAL state before reapplying.
|
||||
@@ -0,0 +1,17 @@
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
|
||||
namespace: observability
|
||||
|
||||
helmGlobals:
|
||||
chartHome: .helm/charts
|
||||
|
||||
helmCharts:
|
||||
- name: loki
|
||||
repo: https://grafana-community.github.io/helm-charts
|
||||
version: 18.7.1
|
||||
releaseName: loki
|
||||
namespace: observability
|
||||
valuesFile: values/home.yaml
|
||||
includeCRDs: false
|
||||
skipTests: true
|
||||
@@ -0,0 +1,146 @@
|
||||
deploymentMode: Monolithic
|
||||
|
||||
commonLabels:
|
||||
observability.hyeonworks.com/instance: home
|
||||
|
||||
defaults:
|
||||
automountServiceAccountToken: false
|
||||
|
||||
serviceAccount:
|
||||
automountServiceAccountToken: false
|
||||
|
||||
loki:
|
||||
auth_enabled: false
|
||||
image:
|
||||
registry: docker.io
|
||||
repository: grafana/loki
|
||||
tag: "3.7.4"
|
||||
digest: sha256:d80be589cf74b6d92747a1b155bec6f07bdd65e261f797fe93a78fb4e833032e
|
||||
podSecurityContext:
|
||||
runAsNonRoot: true
|
||||
runAsUser: 10001
|
||||
runAsGroup: 10001
|
||||
fsGroup: 10001
|
||||
fsGroupChangePolicy: OnRootMismatch
|
||||
seccompProfile:
|
||||
type: RuntimeDefault
|
||||
containerSecurityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
readOnlyRootFilesystem: true
|
||||
capabilities:
|
||||
drop:
|
||||
- ALL
|
||||
seccompProfile:
|
||||
type: RuntimeDefault
|
||||
commonConfig:
|
||||
path_prefix: /var/loki
|
||||
replication_factor: 1
|
||||
schemaConfig:
|
||||
configs:
|
||||
- from: "2026-07-01"
|
||||
store: tsdb
|
||||
object_store: s3
|
||||
schema: v13
|
||||
index:
|
||||
prefix: index_
|
||||
period: 24h
|
||||
storage:
|
||||
type: s3
|
||||
bucketNames:
|
||||
chunks: loki
|
||||
ruler: loki
|
||||
admin: loki
|
||||
s3:
|
||||
endpoint: http://minio.object-storage.svc.cluster.local
|
||||
s3ForcePathStyle: true
|
||||
insecure: true
|
||||
accessKeyId: ${AWS_ACCESS_KEY_ID}
|
||||
secretAccessKey: ${AWS_SECRET_ACCESS_KEY}
|
||||
useTestSchema: false
|
||||
limits_config:
|
||||
retention_period: 720h
|
||||
ingestion_rate_mb: 4
|
||||
ingestion_burst_size_mb: 6
|
||||
max_line_size: 256KB
|
||||
discover_log_levels: false
|
||||
discover_service_name: []
|
||||
ingester:
|
||||
chunk_idle_period: 30m
|
||||
max_chunk_age: 1h
|
||||
wal:
|
||||
checkpoint_duration: 5m
|
||||
replay_memory_ceiling: 512MB
|
||||
disk_full_threshold: 0.90
|
||||
flush_on_shutdown: true
|
||||
compactor:
|
||||
retention_enabled: true
|
||||
delete_request_store: s3
|
||||
retention_delete_delay: 2h
|
||||
retention_delete_worker_count: 10
|
||||
max_compaction_parallelism: 1
|
||||
|
||||
singleBinary:
|
||||
replicas: 1
|
||||
sidecar: false
|
||||
podLabels:
|
||||
platform.hyeonworks.com/aistor-client: "true"
|
||||
service:
|
||||
type: ClusterIP
|
||||
resources:
|
||||
requests:
|
||||
cpu: 350m
|
||||
memory: 768Mi
|
||||
limits:
|
||||
cpu: 1500m
|
||||
memory: 1536Mi
|
||||
extraEnv:
|
||||
- name: AWS_ACCESS_KEY_ID
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: loki-object-storage
|
||||
key: access-key-id
|
||||
- name: AWS_SECRET_ACCESS_KEY
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: loki-object-storage
|
||||
key: secret-access-key
|
||||
persistence:
|
||||
enabled: true
|
||||
size: 5Gi
|
||||
storageClass: ssd-local-observability-loki-retain
|
||||
enableStatefulSetAutoDeletePVC: true
|
||||
whenScaled: Retain
|
||||
whenDeleted: Retain
|
||||
|
||||
read:
|
||||
replicas: 0
|
||||
write:
|
||||
replicas: 0
|
||||
backend:
|
||||
replicas: 0
|
||||
|
||||
ruler:
|
||||
sidecar: false
|
||||
|
||||
gateway:
|
||||
enabled: false
|
||||
lokiCanary:
|
||||
enabled: false
|
||||
chunksCache:
|
||||
enabled: false
|
||||
resultsCache:
|
||||
enabled: false
|
||||
memcached:
|
||||
enabled: false
|
||||
minio:
|
||||
enabled: false
|
||||
test:
|
||||
enabled: false
|
||||
sidecar:
|
||||
rules:
|
||||
enabled: false
|
||||
|
||||
monitoring:
|
||||
serviceMonitor:
|
||||
enabled: true
|
||||
interval: 30s
|
||||
@@ -0,0 +1,8 @@
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
|
||||
resources:
|
||||
- observability.yaml
|
||||
- observability-agent.yaml
|
||||
- object-storage-counterpart.yaml
|
||||
|
||||
@@ -0,0 +1,31 @@
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: object-storage-allow-observability-s3
|
||||
namespace: object-storage
|
||||
labels:
|
||||
app.kubernetes.io/component: observability-object-storage
|
||||
app.kubernetes.io/part-of: platform
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
aistor.min.io/objectStore: minio-aistor
|
||||
policyTypes:
|
||||
- Ingress
|
||||
ingress:
|
||||
- from:
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
kubernetes.io/metadata.name: observability
|
||||
podSelector:
|
||||
matchLabels:
|
||||
platform.hyeonworks.com/aistor-client: "true"
|
||||
matchExpressions:
|
||||
- key: app.kubernetes.io/name
|
||||
operator: In
|
||||
values:
|
||||
- loki
|
||||
- tempo
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 9000
|
||||
@@ -0,0 +1,140 @@
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: observability-agent-allow-alloy-dns
|
||||
namespace: observability-agent
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: alloy
|
||||
policyTypes:
|
||||
- Egress
|
||||
egress:
|
||||
- to:
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
kubernetes.io/metadata.name: kube-system
|
||||
podSelector:
|
||||
matchLabels:
|
||||
k8s-app: kube-dns
|
||||
ports:
|
||||
- protocol: UDP
|
||||
port: 53
|
||||
- protocol: TCP
|
||||
port: 53
|
||||
---
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: observability-agent-allow-alloy-api
|
||||
namespace: observability-agent
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: alloy
|
||||
policyTypes:
|
||||
- Egress
|
||||
egress:
|
||||
- to:
|
||||
- ipBlock:
|
||||
cidr: 10.43.0.1/32
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 443
|
||||
- to:
|
||||
- ipBlock:
|
||||
cidr: 192.168.0.107/32
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 6443
|
||||
---
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: observability-agent-allow-alloy-backends
|
||||
namespace: observability-agent
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: alloy
|
||||
policyTypes:
|
||||
- Egress
|
||||
egress:
|
||||
- to:
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
kubernetes.io/metadata.name: observability
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: loki
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 3100
|
||||
- to:
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
kubernetes.io/metadata.name: observability
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: tempo
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 4317
|
||||
---
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: observability-agent-allow-prometheus-to-alloy
|
||||
namespace: observability-agent
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: alloy
|
||||
policyTypes:
|
||||
- Ingress
|
||||
ingress:
|
||||
- from:
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
kubernetes.io/metadata.name: observability
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: prometheus
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 12345
|
||||
---
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: observability-agent-allow-prometheus-to-node-exporter
|
||||
namespace: observability-agent
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: prometheus-node-exporter
|
||||
policyTypes:
|
||||
- Ingress
|
||||
ingress:
|
||||
- from:
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
kubernetes.io/metadata.name: observability
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: prometheus
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 9100
|
||||
---
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: observability-agent-default-deny
|
||||
namespace: observability-agent
|
||||
spec:
|
||||
podSelector: {}
|
||||
policyTypes:
|
||||
- Ingress
|
||||
- Egress
|
||||
|
||||
@@ -0,0 +1,548 @@
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: observability-allow-dns
|
||||
namespace: observability
|
||||
spec:
|
||||
podSelector: {}
|
||||
policyTypes:
|
||||
- Egress
|
||||
egress:
|
||||
- to:
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
kubernetes.io/metadata.name: kube-system
|
||||
podSelector:
|
||||
matchLabels:
|
||||
k8s-app: kube-dns
|
||||
ports:
|
||||
- protocol: UDP
|
||||
port: 53
|
||||
- protocol: TCP
|
||||
port: 53
|
||||
---
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: observability-allow-kubernetes-api
|
||||
namespace: observability
|
||||
spec:
|
||||
podSelector:
|
||||
matchExpressions:
|
||||
- key: app.kubernetes.io/name
|
||||
operator: In
|
||||
values:
|
||||
- kube-prometheus-stack-prometheus-operator
|
||||
- kube-state-metrics
|
||||
- prometheus
|
||||
policyTypes:
|
||||
- Egress
|
||||
egress:
|
||||
- to:
|
||||
- ipBlock:
|
||||
cidr: 10.43.0.1/32
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 443
|
||||
- to:
|
||||
- ipBlock:
|
||||
cidr: 192.168.0.107/32
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 6443
|
||||
---
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: observability-allow-prometheus-egress
|
||||
namespace: observability
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: prometheus
|
||||
policyTypes:
|
||||
- Egress
|
||||
egress:
|
||||
- to:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: kube-prometheus-stack-prometheus-operator
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 8080
|
||||
- to:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: kube-state-metrics
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 8080
|
||||
- protocol: TCP
|
||||
port: 8081
|
||||
- to:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: prometheus
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 9090
|
||||
- protocol: TCP
|
||||
port: 8080
|
||||
- to:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: alertmanager
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 9093
|
||||
- protocol: TCP
|
||||
port: 8080
|
||||
- to:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: loki
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 3100
|
||||
- to:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: tempo
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 3200
|
||||
- to:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: grafana
|
||||
app.kubernetes.io/instance: grafana
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 3000
|
||||
- to:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: prometheus-blackbox-exporter
|
||||
app.kubernetes.io/instance: blackbox-exporter
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 9115
|
||||
- to:
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
kubernetes.io/metadata.name: observability-agent
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: alloy
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 12345
|
||||
- to:
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
kubernetes.io/metadata.name: observability-agent
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: prometheus-node-exporter
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 9100
|
||||
- to:
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
kubernetes.io/metadata.name: kube-system
|
||||
podSelector:
|
||||
matchLabels:
|
||||
k8s-app: kube-dns
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 9153
|
||||
- to:
|
||||
- ipBlock:
|
||||
cidr: 192.168.0.107/32
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 10250
|
||||
---
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: observability-allow-prometheus-to-operator
|
||||
namespace: observability
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: kube-prometheus-stack-prometheus-operator
|
||||
policyTypes:
|
||||
- Ingress
|
||||
ingress:
|
||||
- from:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: prometheus
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 8080
|
||||
---
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: observability-allow-prometheus-to-ksm
|
||||
namespace: observability
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: kube-state-metrics
|
||||
policyTypes:
|
||||
- Ingress
|
||||
ingress:
|
||||
- from:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: prometheus
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 8080
|
||||
- protocol: TCP
|
||||
port: 8081
|
||||
---
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: observability-allow-prometheus-self
|
||||
namespace: observability
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: prometheus
|
||||
policyTypes:
|
||||
- Ingress
|
||||
ingress:
|
||||
- from:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: prometheus
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 9090
|
||||
- protocol: TCP
|
||||
port: 8080
|
||||
- from:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: grafana
|
||||
app.kubernetes.io/instance: grafana
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 9090
|
||||
---
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: observability-allow-alertmanager
|
||||
namespace: observability
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: alertmanager
|
||||
policyTypes:
|
||||
- Ingress
|
||||
- Egress
|
||||
ingress:
|
||||
- from:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: prometheus
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 9093
|
||||
- protocol: TCP
|
||||
port: 8080
|
||||
- from:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: alertmanager
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 9094
|
||||
- protocol: UDP
|
||||
port: 9094
|
||||
egress:
|
||||
- to:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: alertmanager
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 9094
|
||||
- protocol: UDP
|
||||
port: 9094
|
||||
---
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: observability-allow-loki
|
||||
namespace: observability
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: loki
|
||||
policyTypes:
|
||||
- Ingress
|
||||
- Egress
|
||||
ingress:
|
||||
- from:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: prometheus
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 3100
|
||||
- from:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: grafana
|
||||
app.kubernetes.io/instance: grafana
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 3100
|
||||
- from:
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
kubernetes.io/metadata.name: observability-agent
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: alloy
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 3100
|
||||
- from:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: loki
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 7946
|
||||
- protocol: UDP
|
||||
port: 7946
|
||||
egress:
|
||||
- to:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: loki
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 7946
|
||||
- protocol: UDP
|
||||
port: 7946
|
||||
---
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: observability-allow-tempo
|
||||
namespace: observability
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: tempo
|
||||
policyTypes:
|
||||
- Ingress
|
||||
- Egress
|
||||
ingress:
|
||||
- from:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: prometheus
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 3200
|
||||
- from:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: grafana
|
||||
app.kubernetes.io/instance: grafana
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 3200
|
||||
- from:
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
kubernetes.io/metadata.name: observability-agent
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: alloy
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 4317
|
||||
- from:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: tempo
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 7946
|
||||
- protocol: UDP
|
||||
port: 7946
|
||||
egress:
|
||||
- to:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: tempo
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 7946
|
||||
- protocol: UDP
|
||||
port: 7946
|
||||
---
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: observability-allow-aistor-egress
|
||||
namespace: observability
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
platform.hyeonworks.com/aistor-client: "true"
|
||||
matchExpressions:
|
||||
- key: app.kubernetes.io/name
|
||||
operator: In
|
||||
values:
|
||||
- loki
|
||||
- tempo
|
||||
policyTypes:
|
||||
- Egress
|
||||
egress:
|
||||
- to:
|
||||
- ipBlock:
|
||||
cidr: 10.43.124.248/32
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 80
|
||||
- to:
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
kubernetes.io/metadata.name: object-storage
|
||||
podSelector:
|
||||
matchLabels:
|
||||
aistor.min.io/objectStore: minio-aistor
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 9000
|
||||
---
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: observability-allow-grafana-ingress
|
||||
namespace: observability
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: grafana
|
||||
app.kubernetes.io/instance: grafana
|
||||
policyTypes:
|
||||
- Ingress
|
||||
ingress:
|
||||
- from:
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
kubernetes.io/metadata.name: kube-system
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: traefik
|
||||
app.kubernetes.io/instance: traefik-kube-system
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 3000
|
||||
- from:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: prometheus
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 3000
|
||||
- from:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: prometheus-blackbox-exporter
|
||||
app.kubernetes.io/instance: blackbox-exporter
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 3000
|
||||
---
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: observability-allow-grafana-datasources
|
||||
namespace: observability
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: grafana
|
||||
app.kubernetes.io/instance: grafana
|
||||
policyTypes:
|
||||
- Egress
|
||||
egress:
|
||||
- to:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: prometheus
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 9090
|
||||
- to:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: loki
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 3100
|
||||
- to:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: tempo
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 3200
|
||||
- to:
|
||||
- ipBlock:
|
||||
cidr: 192.168.0.107/32
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 443
|
||||
---
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: observability-allow-grafana-dashboard-api
|
||||
namespace: observability
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: grafana
|
||||
app.kubernetes.io/instance: grafana
|
||||
policyTypes:
|
||||
- Egress
|
||||
egress:
|
||||
- to:
|
||||
- ipBlock:
|
||||
cidr: 10.43.0.1/32
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 443
|
||||
- to:
|
||||
- ipBlock:
|
||||
cidr: 192.168.0.107/32
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 6443
|
||||
---
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: observability-default-deny
|
||||
namespace: observability
|
||||
spec:
|
||||
podSelector: {}
|
||||
policyTypes:
|
||||
- Ingress
|
||||
- Egress
|
||||
@@ -0,0 +1,11 @@
|
||||
# node-exporter
|
||||
|
||||
The standalone node-exporter runs in `observability-agent` with the embedded
|
||||
kube-prometheus-stack copy and kube-rbac-proxy disabled. It mounts only
|
||||
`/proc`, `/sys`, and `/` from the node, all read-only, and exposes an internal
|
||||
ClusterIP metrics Service on port `9100`.
|
||||
|
||||
Host network, PID, and IPC namespaces remain disabled by the platform security
|
||||
contract. Consequently, network-namespace metrics are treated as a known
|
||||
single-node visibility limitation rather than authoritative host-network data.
|
||||
|
||||
@@ -0,0 +1,23 @@
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
|
||||
namespace: observability-agent
|
||||
|
||||
helmGlobals:
|
||||
chartHome: .helm/charts
|
||||
|
||||
helmCharts:
|
||||
- name: prometheus-node-exporter
|
||||
repo: https://prometheus-community.github.io/helm-charts
|
||||
version: 4.56.1
|
||||
releaseName: node-exporter
|
||||
namespace: observability-agent
|
||||
valuesFile: values/home.yaml
|
||||
includeCRDs: false
|
||||
skipTests: true
|
||||
|
||||
images:
|
||||
- name: quay.io/prometheus/node-exporter
|
||||
newName: quay.io/prometheus/node-exporter
|
||||
digest: sha256:da83fae85603c4e47e6c68369a7d746e2dda683dc35ea2e234b4f171e0d92798
|
||||
|
||||
@@ -0,0 +1,78 @@
|
||||
fullnameOverride: node-exporter
|
||||
namespaceOverride: observability-agent
|
||||
|
||||
image:
|
||||
registry: quay.io
|
||||
repository: prometheus/node-exporter
|
||||
tag: v1.12.1
|
||||
pullPolicy: IfNotPresent
|
||||
|
||||
kubeRBACProxy:
|
||||
enabled: false
|
||||
|
||||
service:
|
||||
enabled: true
|
||||
type: ClusterIP
|
||||
port: 9100
|
||||
targetPort: 9100
|
||||
portName: metrics
|
||||
listenOnAllInterfaces: true
|
||||
|
||||
prometheus:
|
||||
monitor:
|
||||
enabled: true
|
||||
additionalLabels:
|
||||
observability.hyeonworks.com/instance: home
|
||||
interval: 30s
|
||||
podMonitor:
|
||||
enabled: false
|
||||
|
||||
resources:
|
||||
requests:
|
||||
cpu: 100m
|
||||
memory: 128Mi
|
||||
limits:
|
||||
cpu: 250m
|
||||
memory: 256Mi
|
||||
|
||||
serviceAccount:
|
||||
create: true
|
||||
automountServiceAccountToken: false
|
||||
|
||||
rbac:
|
||||
create: true
|
||||
|
||||
securityContext:
|
||||
fsGroup: 65534
|
||||
runAsGroup: 65534
|
||||
runAsNonRoot: true
|
||||
runAsUser: 65534
|
||||
seccompProfile:
|
||||
type: RuntimeDefault
|
||||
|
||||
containerSecurityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
capabilities:
|
||||
drop:
|
||||
- ALL
|
||||
privileged: false
|
||||
readOnlyRootFilesystem: true
|
||||
|
||||
hostNetwork: false
|
||||
hostPID: false
|
||||
hostIPC: false
|
||||
|
||||
hostRootFsMount:
|
||||
enabled: true
|
||||
mountPropagation: None
|
||||
hostProcFsMount:
|
||||
mountPropagation: None
|
||||
hostSysFsMount:
|
||||
mountPropagation: None
|
||||
|
||||
nodeSelector:
|
||||
kubernetes.io/os: linux
|
||||
|
||||
tolerations:
|
||||
- effect: NoSchedule
|
||||
operator: Exists
|
||||
@@ -0,0 +1,10 @@
|
||||
# Observability object storage
|
||||
|
||||
Loki와 Tempo는 AIStor에서 서로 다른 bucket, policy, user와 Kubernetes Secret을
|
||||
사용한다. Loki quota는 200Gi, Tempo quota는 100Gi다. 정책은 자기 bucket의 위치·목록과
|
||||
자기 object의 읽기·쓰기·삭제·multipart 처리만 허용한다.
|
||||
|
||||
운영 적용 전에는 K3s Secret 암호화와 격리 복구 증거, AIStor root credential의 암호화된
|
||||
외부 복구 가능성을 각각 다시 확인한다. 기존 bucket·object·policy·user·quota는 자동
|
||||
삭제하거나 credential을 자동 회전하지 않는다. 한쪽 Secret만 존재하는 partial 상태에서는
|
||||
추가 mutation을 중단한다.
|
||||
@@ -0,0 +1,29 @@
|
||||
{
|
||||
"Version": "2012-10-17",
|
||||
"Statement": [
|
||||
{
|
||||
"Effect": "Allow",
|
||||
"Action": [
|
||||
"s3:GetBucketLocation",
|
||||
"s3:ListBucket",
|
||||
"s3:ListBucketMultipartUploads"
|
||||
],
|
||||
"Resource": [
|
||||
"arn:aws:s3:::loki"
|
||||
]
|
||||
},
|
||||
{
|
||||
"Effect": "Allow",
|
||||
"Action": [
|
||||
"s3:GetObject",
|
||||
"s3:PutObject",
|
||||
"s3:DeleteObject",
|
||||
"s3:AbortMultipartUpload",
|
||||
"s3:ListMultipartUploadParts"
|
||||
],
|
||||
"Resource": [
|
||||
"arn:aws:s3:::loki/*"
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,29 @@
|
||||
{
|
||||
"Version": "2012-10-17",
|
||||
"Statement": [
|
||||
{
|
||||
"Effect": "Allow",
|
||||
"Action": [
|
||||
"s3:GetBucketLocation",
|
||||
"s3:ListBucket",
|
||||
"s3:ListBucketMultipartUploads"
|
||||
],
|
||||
"Resource": [
|
||||
"arn:aws:s3:::tempo"
|
||||
]
|
||||
},
|
||||
{
|
||||
"Effect": "Allow",
|
||||
"Action": [
|
||||
"s3:GetObject",
|
||||
"s3:PutObject",
|
||||
"s3:DeleteObject",
|
||||
"s3:AbortMultipartUpload",
|
||||
"s3:ListMultipartUploadParts"
|
||||
],
|
||||
"Resource": [
|
||||
"arn:aws:s3:::tempo/*"
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,87 @@
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: PrometheusRule
|
||||
metadata:
|
||||
name: platform-certificate-probes
|
||||
namespace: observability
|
||||
labels:
|
||||
app.kubernetes.io/name: platform-observability-rules
|
||||
app.kubernetes.io/part-of: observability
|
||||
observability.hyeonworks.com/instance: home
|
||||
spec:
|
||||
groups:
|
||||
- name: platform.blackbox
|
||||
rules:
|
||||
- alert: PlatformPublicEdgeProbeFailed
|
||||
expr: >-
|
||||
probe_success{
|
||||
namespace="observability",
|
||||
"observability.hyeonworks.com/probe-group"="public-edge"
|
||||
} == 0
|
||||
for: 5m
|
||||
labels:
|
||||
severity: critical
|
||||
annotations:
|
||||
summary: 외부 엔드포인트 프로브 실패
|
||||
description: '{{ $labels.instance }} 외부 엔드포인트 프로브가 5분간 실패했습니다.'
|
||||
runbook_url: https://git.learn.hyeonworks.com/donghyeon.kang/project-infra/src/branch/main/docs/runbooks/2026-07-31-observability-access-cutover.md
|
||||
- alert: PlatformPrivateEdgeProbeFailed
|
||||
expr: >-
|
||||
probe_success{
|
||||
namespace="observability",
|
||||
"observability.hyeonworks.com/probe-group"="private-edge"
|
||||
} == 0
|
||||
for: 5m
|
||||
labels:
|
||||
severity: critical
|
||||
annotations:
|
||||
summary: 비공개 경계 프로브 실패
|
||||
description: '{{ $labels.instance }} 비공개 경계의 예상 응답이 5분간 실패했습니다. HTTP 403이면서 probe_success=1이면 정상입니다.'
|
||||
runbook_url: https://git.learn.hyeonworks.com/donghyeon.kang/project-infra/src/branch/main/docs/runbooks/2026-07-31-observability-access-cutover.md
|
||||
- alert: PlatformInternalHealthProbeFailed
|
||||
expr: >-
|
||||
probe_success{
|
||||
namespace="observability",
|
||||
"observability.hyeonworks.com/probe-group"="private-internal"
|
||||
} == 0
|
||||
for: 5m
|
||||
labels:
|
||||
severity: critical
|
||||
annotations:
|
||||
summary: 내부 상태 확인 프로브 실패
|
||||
description: '{{ $labels.instance }} 내부 상태 확인 프로브가 5분간 실패했습니다.'
|
||||
runbook_url: https://git.learn.hyeonworks.com/donghyeon.kang/project-infra/src/branch/main/docs/runbooks/2026-07-31-observability-access-cutover.md
|
||||
- name: platform.certificates
|
||||
rules:
|
||||
- alert: PlatformCertificateExpiry
|
||||
expr: >-
|
||||
(
|
||||
probe_ssl_earliest_cert_expiry{
|
||||
namespace="observability",
|
||||
"observability.hyeonworks.com/probe-group"=~"public-edge|private-edge"
|
||||
} - time()
|
||||
) >= 14 * 24 * 60 * 60
|
||||
and
|
||||
(
|
||||
probe_ssl_earliest_cert_expiry{
|
||||
namespace="observability",
|
||||
"observability.hyeonworks.com/probe-group"=~"public-edge|private-edge"
|
||||
} - time()
|
||||
) < 30 * 24 * 60 * 60
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: TLS 인증서가 30일 이내 만료됨
|
||||
description: '{{ $labels.instance }} 인증서 중 가장 이른 만료 시점이 14일 이상 30일 미만 남았습니다.'
|
||||
runbook_url: https://git.learn.hyeonworks.com/donghyeon.kang/project-infra/src/branch/main/docs/runbooks/2026-07-31-observability-access-cutover.md
|
||||
- alert: PlatformCertificateExpiry
|
||||
expr: >-
|
||||
probe_ssl_earliest_cert_expiry{
|
||||
namespace="observability",
|
||||
"observability.hyeonworks.com/probe-group"=~"public-edge|private-edge"
|
||||
} - time() < 14 * 24 * 60 * 60
|
||||
labels:
|
||||
severity: critical
|
||||
annotations:
|
||||
summary: TLS 인증서가 14일 이내 만료됨
|
||||
description: '{{ $labels.instance }} 인증서 중 가장 이른 만료 시점이 14일 미만 남았습니다.'
|
||||
runbook_url: https://git.learn.hyeonworks.com/donghyeon.kang/project-infra/src/branch/main/docs/runbooks/2026-07-31-observability-access-cutover.md
|
||||
@@ -0,0 +1,117 @@
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: PrometheusRule
|
||||
metadata:
|
||||
name: platform-observability-core
|
||||
namespace: observability
|
||||
labels:
|
||||
app.kubernetes.io/name: platform-observability-rules
|
||||
app.kubernetes.io/part-of: observability
|
||||
observability.hyeonworks.com/instance: home
|
||||
spec:
|
||||
groups:
|
||||
- name: platform.storage
|
||||
rules:
|
||||
- alert: PlatformPersistentVolumeClaimNotBound
|
||||
expr: kube_persistentvolumeclaim_status_phase{phase=~"Pending|Lost"} == 1
|
||||
for: 10m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: PersistentVolumeClaim이 바인딩되지 않음
|
||||
description: 플랫폼 PersistentVolumeClaim이 Pending 또는 Lost 상태로 10분간 유지됐습니다.
|
||||
runbook_url: https://git.learn.hyeonworks.com/donghyeon.kang/project-infra/src/branch/main/docs/runbooks/2026-07-31-observability-access-cutover.md
|
||||
- alert: PlatformRootFilesystemUsage
|
||||
expr: >-
|
||||
100 * (1 -
|
||||
node_filesystem_avail_bytes{job="prometheus-node-exporter",mountpoint="/",fstype!~"rootfs|tmpfs|ramfs|squashfs|overlay|nsfs|tracefs|debugfs|securityfs|proc|sysfs|cgroup2?"}
|
||||
/
|
||||
node_filesystem_size_bytes{job="prometheus-node-exporter",mountpoint="/",fstype!~"rootfs|tmpfs|ramfs|squashfs|overlay|nsfs|tracefs|debugfs|securityfs|proc|sysfs|cgroup2?"}
|
||||
) > 75
|
||||
for: 30m
|
||||
labels:
|
||||
severity: info
|
||||
threshold: "75"
|
||||
annotations:
|
||||
summary: 루트 파일시스템 사용률이 75%를 초과함
|
||||
description: 노드 루트 파일시스템 사용률이 정보 임계치인 75%를 30분간 초과했습니다.
|
||||
runbook_url: https://git.learn.hyeonworks.com/donghyeon.kang/project-infra/src/branch/main/docs/runbooks/2026-07-31-observability-access-cutover.md
|
||||
- alert: PlatformRootFilesystemUsage
|
||||
expr: >-
|
||||
100 * (1 -
|
||||
node_filesystem_avail_bytes{job="prometheus-node-exporter",mountpoint="/",fstype!~"rootfs|tmpfs|ramfs|squashfs|overlay|nsfs|tracefs|debugfs|securityfs|proc|sysfs|cgroup2?"}
|
||||
/
|
||||
node_filesystem_size_bytes{job="prometheus-node-exporter",mountpoint="/",fstype!~"rootfs|tmpfs|ramfs|squashfs|overlay|nsfs|tracefs|debugfs|securityfs|proc|sysfs|cgroup2?"}
|
||||
) > 85
|
||||
for: 15m
|
||||
labels:
|
||||
severity: warning
|
||||
threshold: "85"
|
||||
annotations:
|
||||
summary: 루트 파일시스템 사용률이 85%를 초과함
|
||||
description: 노드 루트 파일시스템 사용률이 경고 임계치인 85%를 15분간 초과했습니다.
|
||||
runbook_url: https://git.learn.hyeonworks.com/donghyeon.kang/project-infra/src/branch/main/docs/runbooks/2026-07-31-observability-access-cutover.md
|
||||
- alert: PlatformRootFilesystemUsage
|
||||
expr: >-
|
||||
100 * (1 -
|
||||
node_filesystem_avail_bytes{job="prometheus-node-exporter",mountpoint="/",fstype!~"rootfs|tmpfs|ramfs|squashfs|overlay|nsfs|tracefs|debugfs|securityfs|proc|sysfs|cgroup2?"}
|
||||
/
|
||||
node_filesystem_size_bytes{job="prometheus-node-exporter",mountpoint="/",fstype!~"rootfs|tmpfs|ramfs|squashfs|overlay|nsfs|tracefs|debugfs|securityfs|proc|sysfs|cgroup2?"}
|
||||
) > 90
|
||||
for: 5m
|
||||
labels:
|
||||
severity: critical
|
||||
threshold: "90"
|
||||
annotations:
|
||||
summary: 루트 파일시스템 사용률이 90%를 초과함
|
||||
description: 노드 루트 파일시스템 사용률이 심각 임계치인 90%를 5분간 초과했습니다.
|
||||
runbook_url: https://git.learn.hyeonworks.com/donghyeon.kang/project-infra/src/branch/main/docs/runbooks/2026-07-31-observability-access-cutover.md
|
||||
- alert: PlatformRootFilesystemInodes
|
||||
expr: >-
|
||||
100 * (1 -
|
||||
node_filesystem_files_free{job="prometheus-node-exporter",mountpoint="/",fstype!~"rootfs|tmpfs|ramfs|squashfs|overlay|nsfs|tracefs|debugfs|securityfs|proc|sysfs|cgroup2?"}
|
||||
/
|
||||
node_filesystem_files{job="prometheus-node-exporter",mountpoint="/",fstype!~"rootfs|tmpfs|ramfs|squashfs|overlay|nsfs|tracefs|debugfs|securityfs|proc|sysfs|cgroup2?"}
|
||||
) > 80
|
||||
for: 30m
|
||||
labels:
|
||||
severity: warning
|
||||
threshold: "80"
|
||||
annotations:
|
||||
summary: 루트 파일시스템 inode 사용률이 80%를 초과함
|
||||
description: 노드 루트 파일시스템 inode 사용률이 경고 임계치인 80%를 30분간 초과했습니다.
|
||||
runbook_url: https://git.learn.hyeonworks.com/donghyeon.kang/project-infra/src/branch/main/docs/runbooks/2026-07-31-observability-access-cutover.md
|
||||
- alert: PlatformRootFilesystemInodes
|
||||
expr: >-
|
||||
100 * (1 -
|
||||
node_filesystem_files_free{job="prometheus-node-exporter",mountpoint="/",fstype!~"rootfs|tmpfs|ramfs|squashfs|overlay|nsfs|tracefs|debugfs|securityfs|proc|sysfs|cgroup2?"}
|
||||
/
|
||||
node_filesystem_files{job="prometheus-node-exporter",mountpoint="/",fstype!~"rootfs|tmpfs|ramfs|squashfs|overlay|nsfs|tracefs|debugfs|securityfs|proc|sysfs|cgroup2?"}
|
||||
) > 90
|
||||
for: 15m
|
||||
labels:
|
||||
severity: critical
|
||||
threshold: "90"
|
||||
annotations:
|
||||
summary: 루트 파일시스템 inode 사용률이 90%를 초과함
|
||||
description: 노드 루트 파일시스템 inode 사용률이 심각 임계치인 90%를 15분간 초과했습니다.
|
||||
runbook_url: https://git.learn.hyeonworks.com/donghyeon.kang/project-infra/src/branch/main/docs/runbooks/2026-07-31-observability-access-cutover.md
|
||||
- name: platform.workloads
|
||||
rules:
|
||||
- alert: PlatformPodRestartBurst
|
||||
expr: increase(kube_pod_container_status_restarts_total[15m]) > 3
|
||||
for: 5m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: 컨테이너 재시작 급증 감지
|
||||
description: 한 컨테이너가 최근 15분 동안 3회를 초과해 재시작했습니다.
|
||||
runbook_url: https://git.learn.hyeonworks.com/donghyeon.kang/project-infra/src/branch/main/docs/runbooks/2026-07-31-observability-access-cutover.md
|
||||
- alert: PlatformPodOOMKilled
|
||||
expr: kube_pod_container_status_last_terminated_reason{reason="OOMKilled"} == 1
|
||||
for: 5m
|
||||
labels:
|
||||
severity: critical
|
||||
annotations:
|
||||
summary: 컨테이너가 OOM Killer에 의해 종료됨
|
||||
description: 컨테이너의 최근 종료 사유가 OOMKilled인 상태로 5분간 유지됐습니다.
|
||||
runbook_url: https://git.learn.hyeonworks.com/donghyeon.kang/project-infra/src/branch/main/docs/runbooks/2026-07-31-observability-access-cutover.md
|
||||
@@ -0,0 +1,10 @@
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
|
||||
namespace: observability
|
||||
|
||||
resources:
|
||||
- core-rules.yaml
|
||||
- storage-quota-rules.yaml
|
||||
- certificate-rules.yaml
|
||||
- verified-service-rules.yaml
|
||||
@@ -0,0 +1,55 @@
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: PrometheusRule
|
||||
metadata:
|
||||
name: platform-aistor-storage-quota
|
||||
namespace: observability
|
||||
labels:
|
||||
app.kubernetes.io/name: platform-observability-rules
|
||||
app.kubernetes.io/part-of: observability
|
||||
observability.hyeonworks.com/instance: home
|
||||
spec:
|
||||
groups:
|
||||
- name: platform.aistor-storage-quota
|
||||
rules:
|
||||
- record: platform:aistor_bucket_quota_usage_percent
|
||||
expr: >-
|
||||
100
|
||||
* max by (bucket) (
|
||||
minio_cluster_usage_buckets_total_bytes{bucket=~"loki|tempo"}
|
||||
)
|
||||
/ max by (bucket) (
|
||||
minio_cluster_usage_buckets_quota_total_bytes{bucket=~"loki|tempo"} > 0
|
||||
)
|
||||
- alert: PlatformAIStorBucketQuotaUsage
|
||||
expr: >-
|
||||
platform:aistor_bucket_quota_usage_percent >= 70
|
||||
and
|
||||
platform:aistor_bucket_quota_usage_percent < 85
|
||||
for: 15m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: AIStor 버킷 할당량 사용률 경고
|
||||
description: '{{ $labels.bucket }} 버킷 사용률이 70% 이상 85% 미만으로 15분간 유지됐습니다.'
|
||||
runbook_url: https://git.learn.hyeonworks.com/donghyeon.kang/project-infra/src/branch/main/docs/runbooks/2026-07-31-observability-access-cutover.md
|
||||
- alert: PlatformAIStorBucketQuotaUsage
|
||||
expr: >-
|
||||
platform:aistor_bucket_quota_usage_percent >= 85
|
||||
and
|
||||
platform:aistor_bucket_quota_usage_percent < 95
|
||||
for: 10m
|
||||
labels:
|
||||
severity: critical
|
||||
annotations:
|
||||
summary: AIStor 버킷 할당량 사용률 심각
|
||||
description: '{{ $labels.bucket }} 버킷 사용률이 85% 이상 95% 미만으로 10분간 유지됐습니다.'
|
||||
runbook_url: https://git.learn.hyeonworks.com/donghyeon.kang/project-infra/src/branch/main/docs/runbooks/2026-07-31-observability-access-cutover.md
|
||||
- alert: PlatformAIStorBucketQuotaUsage
|
||||
expr: platform:aistor_bucket_quota_usage_percent >= 95
|
||||
for: 5m
|
||||
labels:
|
||||
severity: emergency
|
||||
annotations:
|
||||
summary: AIStor 버킷 할당량 사용률 긴급
|
||||
description: '대응 절차에 따라 {{ $labels.bucket }} 버킷의 수집 경로를 중지하고 객체는 자동으로 삭제하지 마세요.'
|
||||
runbook_url: https://git.learn.hyeonworks.com/donghyeon.kang/project-infra/src/branch/main/docs/runbooks/2026-07-31-observability-access-cutover.md
|
||||
@@ -0,0 +1,88 @@
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: PrometheusRule
|
||||
metadata:
|
||||
name: platform-verified-services
|
||||
namespace: observability
|
||||
labels:
|
||||
app.kubernetes.io/name: platform-observability-rules
|
||||
app.kubernetes.io/part-of: observability
|
||||
observability.hyeonworks.com/instance: home
|
||||
spec:
|
||||
groups:
|
||||
- name: platform.verified-services
|
||||
rules:
|
||||
- alert: PlatformCNPGCollectorDown
|
||||
expr: >-
|
||||
min by (cluster, namespace) (
|
||||
cnpg_collector_up{
|
||||
namespace="platform-data",
|
||||
cluster="platform-postgres"
|
||||
}
|
||||
) == 0
|
||||
for: 5m
|
||||
labels:
|
||||
severity: critical
|
||||
annotations:
|
||||
summary: CloudNativePG 수집기가 PostgreSQL 중단을 보고함
|
||||
description: '{{ $labels.cluster }} CloudNativePG 수집기가 PostgreSQL 중단 상태를 5분간 보고했습니다.'
|
||||
runbook_url: https://git.learn.hyeonworks.com/donghyeon.kang/project-infra/src/branch/main/docs/runbooks/2026-07-31-observability-access-cutover.md
|
||||
- alert: PlatformAlloyConfigLoadFailed
|
||||
expr: >-
|
||||
min by (namespace) (
|
||||
alloy_config_last_load_successful{namespace="observability-agent"}
|
||||
) == 0
|
||||
for: 5m
|
||||
labels:
|
||||
severity: critical
|
||||
annotations:
|
||||
summary: Alloy 설정 로드 실패
|
||||
description: Alloy의 마지막 설정 로드 실패 상태가 5분간 유지됐습니다.
|
||||
runbook_url: https://git.learn.hyeonworks.com/donghyeon.kang/project-infra/src/branch/main/docs/runbooks/2026-07-31-observability-access-cutover.md
|
||||
- alert: PlatformAlloyLogDeliveryRetries
|
||||
expr: >-
|
||||
sum by (namespace) (
|
||||
increase(loki_write_batch_retries_total{namespace="observability-agent"}[15m])
|
||||
) > 0
|
||||
for: 5m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: Alloy 로그 배치 전송 재시도 발생
|
||||
description: Alloy가 최근 15분 동안 Loki 쓰기 배치를 한 번 이상 재시도했습니다.
|
||||
runbook_url: https://git.learn.hyeonworks.com/donghyeon.kang/project-infra/src/branch/main/docs/runbooks/2026-07-31-observability-access-cutover.md
|
||||
- alert: PlatformAlloyLogEntriesDropped
|
||||
expr: >-
|
||||
sum by (namespace) (
|
||||
increase(loki_write_dropped_entries_total{namespace="observability-agent"}[15m])
|
||||
) > 0
|
||||
for: 5m
|
||||
labels:
|
||||
severity: critical
|
||||
annotations:
|
||||
summary: Alloy 로그 항목 유실 발생
|
||||
description: Alloy가 최근 15분 동안 재시도를 모두 소진한 뒤 로그 항목을 유실했습니다.
|
||||
runbook_url: https://git.learn.hyeonworks.com/donghyeon.kang/project-infra/src/branch/main/docs/runbooks/2026-07-31-observability-access-cutover.md
|
||||
- alert: PlatformLokiRuntimeConfigReloadFailed
|
||||
expr: >-
|
||||
min by (namespace) (
|
||||
loki_runtime_config_last_reload_successful{namespace="observability"}
|
||||
) == 0
|
||||
for: 5m
|
||||
labels:
|
||||
severity: critical
|
||||
annotations:
|
||||
summary: Loki 런타임 설정 재로드 실패
|
||||
description: Loki의 마지막 런타임 설정 재로드 실패 상태가 5분간 유지됐습니다.
|
||||
runbook_url: https://git.learn.hyeonworks.com/donghyeon.kang/project-infra/src/branch/main/docs/runbooks/2026-07-31-observability-access-cutover.md
|
||||
- alert: PlatformLokiWALDiskUsageHigh
|
||||
expr: >-
|
||||
max by (namespace) (
|
||||
loki_ingester_wal_disk_usage_percent{namespace="observability"}
|
||||
) >= 0.8
|
||||
for: 15m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: Loki WAL 디스크 사용률 높음
|
||||
description: Loki WAL 디스크 사용률이 80% 이상으로 15분간 유지됐습니다.
|
||||
runbook_url: https://git.learn.hyeonworks.com/donghyeon.kang/project-infra/src/branch/main/docs/runbooks/2026-07-31-observability-access-cutover.md
|
||||
@@ -0,0 +1,24 @@
|
||||
# Platform metric targets
|
||||
|
||||
This component declares the manually owned monitor resources and counterpart
|
||||
NetworkPolicy needed to onboard existing platform services into the home
|
||||
Prometheus instance.
|
||||
|
||||
The owners remain authoritative for their endpoint activation:
|
||||
|
||||
- Gitea chart: `/metrics` and its generated ServiceMonitor
|
||||
- Keycloak Operator: management metrics and its generated ServiceMonitor
|
||||
- CloudNativePG: the declared `metrics` Pod port through a manual PodMonitor
|
||||
- AIStor: the v3 per-bucket usage endpoint through a manual ServiceMonitor
|
||||
- packaged Traefik chart: its dedicated metrics Service and ServiceMonitor
|
||||
|
||||
The AIStor endpoint uses `MINIO_PROMETHEUS_AUTH_TYPE=public`, but no public
|
||||
route is created. Port 9000 remains protected by the namespace default-deny and
|
||||
the exact Prometheus/client allow policies. Because S3 and metrics share that
|
||||
L4 port, an already-authorized S3 client can also reach the credential-less
|
||||
metrics path; HTTP-path isolation is not expressible with NetworkPolicy.
|
||||
|
||||
The first inventory is captured before blackbox is deployed. Grafana already
|
||||
exists and must be recorded explicitly as a preexisting, verify-only substrate
|
||||
target; it is not owned or rolled back by this component.
|
||||
|
||||
@@ -0,0 +1,28 @@
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: ServiceMonitor
|
||||
metadata:
|
||||
name: aistor-bucket-usage
|
||||
namespace: object-storage
|
||||
labels:
|
||||
app.kubernetes.io/name: minio-aistor
|
||||
app.kubernetes.io/component: bucket-usage-metrics
|
||||
app.kubernetes.io/part-of: platform
|
||||
observability.hyeonworks.com/instance: home
|
||||
spec:
|
||||
jobLabel: aistor.min.io/objectStore
|
||||
selector:
|
||||
matchLabels:
|
||||
aistor.min.io/objectStore: minio-aistor
|
||||
namespaceSelector:
|
||||
matchNames:
|
||||
- object-storage
|
||||
endpoints:
|
||||
- port: http-minio
|
||||
path: /minio/metrics/v3/cluster/usage/buckets
|
||||
interval: 60s
|
||||
scrapeTimeout: 10s
|
||||
relabelings:
|
||||
- action: keep
|
||||
sourceLabels:
|
||||
- __meta_kubernetes_service_name
|
||||
regex: minio
|
||||
@@ -0,0 +1,24 @@
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: PodMonitor
|
||||
metadata:
|
||||
name: platform-postgres
|
||||
namespace: platform-data
|
||||
labels:
|
||||
app.kubernetes.io/name: platform-postgres
|
||||
app.kubernetes.io/component: database-metrics
|
||||
app.kubernetes.io/part-of: platform
|
||||
observability.hyeonworks.com/instance: home
|
||||
spec:
|
||||
jobLabel: cnpg.io/cluster
|
||||
selector:
|
||||
matchLabels:
|
||||
cnpg.io/cluster: platform-postgres
|
||||
namespaceSelector:
|
||||
matchNames:
|
||||
- platform-data
|
||||
podMetricsEndpoints:
|
||||
- port: metrics
|
||||
path: /metrics
|
||||
interval: 30s
|
||||
scrapeTimeout: 10s
|
||||
|
||||
@@ -0,0 +1,108 @@
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: traefik-preserve-ingress-and-allow-prometheus-metrics
|
||||
namespace: kube-system
|
||||
labels:
|
||||
app.kubernetes.io/name: traefik
|
||||
app.kubernetes.io/part-of: platform
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: traefik
|
||||
app.kubernetes.io/instance: traefik-kube-system
|
||||
policyTypes:
|
||||
- Ingress
|
||||
ingress:
|
||||
# Preserve the pre-policy HTTP, HTTPS, and health reachability. The new
|
||||
# policy isolates the selected Pod, so omitting these ports would break the
|
||||
# existing NodePort and controller health paths.
|
||||
- ports:
|
||||
- protocol: TCP
|
||||
port: 8000
|
||||
- protocol: TCP
|
||||
port: 8443
|
||||
- protocol: TCP
|
||||
port: 8080
|
||||
- from:
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
kubernetes.io/metadata.name: observability
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: prometheus
|
||||
app.kubernetes.io/instance: observability-core-kube-pr-prometheus
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 9100
|
||||
---
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: observability-allow-prometheus-platform-targets
|
||||
namespace: observability
|
||||
labels:
|
||||
app.kubernetes.io/name: prometheus
|
||||
app.kubernetes.io/part-of: platform-observability
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: prometheus
|
||||
app.kubernetes.io/instance: observability-core-kube-pr-prometheus
|
||||
policyTypes:
|
||||
- Egress
|
||||
egress:
|
||||
- to:
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
kubernetes.io/metadata.name: gitea
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: gitea
|
||||
app.kubernetes.io/instance: gitea
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 3000
|
||||
- to:
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
kubernetes.io/metadata.name: keycloak
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app: keycloak
|
||||
app.kubernetes.io/instance: keycloak
|
||||
app.kubernetes.io/managed-by: keycloak-operator
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 9000
|
||||
- to:
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
kubernetes.io/metadata.name: platform-data
|
||||
podSelector:
|
||||
matchLabels:
|
||||
cnpg.io/cluster: platform-postgres
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 9187
|
||||
- to:
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
kubernetes.io/metadata.name: object-storage
|
||||
podSelector:
|
||||
matchLabels:
|
||||
aistor.min.io/objectStore: minio-aistor
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 9000
|
||||
- to:
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
kubernetes.io/metadata.name: kube-system
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: traefik
|
||||
app.kubernetes.io/instance: traefik-kube-system
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 9100
|
||||
@@ -0,0 +1,8 @@
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
|
||||
resources:
|
||||
- cloudnative-pg/pod-monitor.yaml
|
||||
- aistor/service-monitor.yaml
|
||||
- counterpart-network-policies.yaml
|
||||
|
||||
@@ -0,0 +1,15 @@
|
||||
# Tempo
|
||||
|
||||
This root renders `grafana-community/tempo` chart `2.2.3` as one Tempo
|
||||
`2.10.7` StatefulSet in `observability`.
|
||||
|
||||
- The image is rewritten to the reviewed amd64 digest.
|
||||
- Traces use the `tempo` AIStor bucket through `tempo-object-storage`; no root
|
||||
credential is referenced.
|
||||
- Retention is seven days. WAL and local working data use the dedicated 5Gi
|
||||
Retain Local PV.
|
||||
- Only the query/metrics port 3200 and OTLP/gRPC port 4317 are exposed.
|
||||
- OTLP/HTTP, Jaeger, Zipkin, metrics-generator, and tempo-query are disabled.
|
||||
|
||||
Do not delete the PVC or bucket during rollback. Stop the StatefulSet and
|
||||
preserve its Retain volume while S3/WAL state is investigated.
|
||||
@@ -0,0 +1,56 @@
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
|
||||
namespace: observability
|
||||
|
||||
helmGlobals:
|
||||
chartHome: .helm/charts
|
||||
|
||||
helmCharts:
|
||||
- name: tempo
|
||||
repo: https://grafana-community.github.io/helm-charts
|
||||
version: 2.2.3
|
||||
releaseName: tempo
|
||||
namespace: observability
|
||||
valuesFile: values/home.yaml
|
||||
includeCRDs: false
|
||||
skipTests: true
|
||||
|
||||
images:
|
||||
- name: docker.io/grafana/tempo
|
||||
newName: docker.io/grafana/tempo
|
||||
digest: sha256:6616b00287a4d7001951b5de117828ad5c6f93744935c1b7a5e044736373352c
|
||||
|
||||
patches:
|
||||
- target:
|
||||
group: ""
|
||||
version: v1
|
||||
kind: Service
|
||||
name: tempo
|
||||
patch: |-
|
||||
- op: replace
|
||||
path: /spec/ports
|
||||
value:
|
||||
- name: tempo-prom-metrics
|
||||
port: 3200
|
||||
protocol: TCP
|
||||
targetPort: 3200
|
||||
- name: grpc-tempo-otlp
|
||||
port: 4317
|
||||
protocol: TCP
|
||||
targetPort: 4317
|
||||
- target:
|
||||
group: apps
|
||||
version: v1
|
||||
kind: StatefulSet
|
||||
name: tempo
|
||||
patch: |-
|
||||
- op: replace
|
||||
path: /spec/template/spec/containers/0/ports
|
||||
value:
|
||||
- containerPort: 3200
|
||||
name: prom-metrics
|
||||
protocol: TCP
|
||||
- containerPort: 4317
|
||||
name: otlp-grpc
|
||||
protocol: TCP
|
||||
@@ -0,0 +1,125 @@
|
||||
global:
|
||||
commonLabels:
|
||||
observability.hyeonworks.com/instance: home
|
||||
|
||||
replicas: 1
|
||||
|
||||
config: |
|
||||
stream_over_http_enabled: false
|
||||
memberlist:
|
||||
cluster_label: "{{ .Release.Name }}.{{ .Release.Namespace }}"
|
||||
multitenancy_enabled: false
|
||||
usage_report:
|
||||
reporting_enabled: false
|
||||
compactor:
|
||||
compaction:
|
||||
block_retention: 168h
|
||||
distributor:
|
||||
receivers:
|
||||
otlp:
|
||||
protocols:
|
||||
grpc:
|
||||
endpoint: 0.0.0.0:4317
|
||||
ingester:
|
||||
trace_idle_period: 10s
|
||||
max_block_duration: 5m
|
||||
max_block_bytes: 100000000
|
||||
complete_block_timeout: 15m
|
||||
flush_all_on_shutdown: true
|
||||
server:
|
||||
http_listen_port: 3200
|
||||
storage:
|
||||
trace:
|
||||
backend: s3
|
||||
s3:
|
||||
bucket: tempo
|
||||
endpoint: minio.object-storage.svc.cluster.local:80
|
||||
insecure: true
|
||||
forcepathstyle: true
|
||||
access_key: ${AWS_ACCESS_KEY_ID}
|
||||
secret_key: ${AWS_SECRET_ACCESS_KEY}
|
||||
wal:
|
||||
path: /var/tempo/wal
|
||||
querier: {}
|
||||
query_frontend: {}
|
||||
overrides:
|
||||
defaults:
|
||||
ingestion:
|
||||
rate_limit_bytes: 5000000
|
||||
burst_size_bytes: 10000000
|
||||
max_traces_per_user: 5000
|
||||
global:
|
||||
max_bytes_per_trace: 5000000
|
||||
|
||||
tempo:
|
||||
registry: docker.io
|
||||
repository: grafana/tempo
|
||||
tag: 2.10.7
|
||||
resources:
|
||||
requests:
|
||||
cpu: 250m
|
||||
memory: 512Mi
|
||||
limits:
|
||||
cpu: "1"
|
||||
memory: 1Gi
|
||||
memBallastSizeMbs: 0
|
||||
reportingEnabled: false
|
||||
metricsGenerator:
|
||||
enabled: false
|
||||
extraArgs:
|
||||
config.expand-env: true
|
||||
extraEnv:
|
||||
- name: AWS_ACCESS_KEY_ID
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: tempo-object-storage
|
||||
key: access-key-id
|
||||
- name: AWS_SECRET_ACCESS_KEY
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: tempo-object-storage
|
||||
key: secret-access-key
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
allowPrivilegeEscalation: false
|
||||
readOnlyRootFilesystem: true
|
||||
capabilities:
|
||||
drop:
|
||||
- ALL
|
||||
seccompProfile:
|
||||
type: RuntimeDefault
|
||||
|
||||
tempoQuery:
|
||||
enabled: false
|
||||
|
||||
securityContext:
|
||||
runAsUser: 10001
|
||||
runAsGroup: 10001
|
||||
runAsNonRoot: true
|
||||
fsGroup: 10001
|
||||
fsGroupChangePolicy: OnRootMismatch
|
||||
seccompProfile:
|
||||
type: RuntimeDefault
|
||||
|
||||
serviceAccount:
|
||||
automountServiceAccountToken: false
|
||||
|
||||
service:
|
||||
type: ClusterIP
|
||||
|
||||
serviceMonitor:
|
||||
enabled: true
|
||||
interval: 30s
|
||||
additionalLabels:
|
||||
observability.hyeonworks.com/instance: home
|
||||
|
||||
persistence:
|
||||
enabled: true
|
||||
enableStatefulSetAutoDeletePVC: false
|
||||
storageClassName: ssd-local-observability-tempo-retain
|
||||
accessModes:
|
||||
- ReadWriteOnce
|
||||
size: 5Gi
|
||||
|
||||
podLabels:
|
||||
platform.hyeonworks.com/aistor-client: "true"
|
||||
@@ -0,0 +1,19 @@
|
||||
# pgAdmin
|
||||
|
||||
PostgreSQL 관리 화면을 platform-admin 네임스페이스에 배포합니다.
|
||||
|
||||
- 주소: db-admin.learn.hyeonworks.com
|
||||
- 실제 접근 범위: 호스트 Nginx에서 LAN·Tailscale만 허용
|
||||
- 차트: dpage/pgadmin4-helm:9.16.0
|
||||
- 이미지: dpage/pgadmin4:9.16의 amd64 digest 고정
|
||||
- 저장소: 2Gi SSD Local PV, Retain
|
||||
- 배포 전략: 단일 replica, Recreate
|
||||
|
||||
Keycloak 그룹 /platform-db-admins가 있는 사용자만 OIDC 로그인이
|
||||
가능합니다. 내부 관리자는 비상용으로 유지합니다. 공유 서버 정의에는
|
||||
비밀번호가 없으며, DB 비밀번호 저장도 비활성화합니다.
|
||||
|
||||
필요한 Secret은 다음 두 개뿐이며 Git에 저장하지 않습니다.
|
||||
|
||||
- pgadmin-keycloak-oidc: client-id, client-secret
|
||||
- pgadmin-bootstrap: password
|
||||
@@ -0,0 +1,42 @@
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: pgadmin-config
|
||||
namespace: platform-admin
|
||||
labels:
|
||||
app.kubernetes.io/name: pgadmin4
|
||||
app.kubernetes.io/instance: pgadmin
|
||||
app.kubernetes.io/part-of: platform
|
||||
data:
|
||||
config_local.py: |
|
||||
import os
|
||||
|
||||
AUTHENTICATION_SOURCES = ['oauth2', 'internal']
|
||||
OAUTH2_AUTO_CREATE_USER = True
|
||||
OAUTH2_CONFIG = [{
|
||||
'OAUTH2_NAME': 'keycloak',
|
||||
'OAUTH2_DISPLAY_NAME': 'Hyeonworks ID',
|
||||
'OAUTH2_CLIENT_ID': os.environ['PGADMIN_OIDC_CLIENT_ID'],
|
||||
'OAUTH2_CLIENT_SECRET': os.environ['PGADMIN_OIDC_CLIENT_SECRET'],
|
||||
'OAUTH2_SERVER_METADATA_URL': (
|
||||
'https://id.learn.hyeonworks.com/realms/hyeonworks/'
|
||||
'.well-known/openid-configuration'
|
||||
),
|
||||
'OAUTH2_SCOPE': 'openid email profile',
|
||||
'OAUTH2_USERNAME_CLAIM': 'preferred_username',
|
||||
'OAUTH2_ADDITIONAL_CLAIMS': {
|
||||
'groups': ['/platform-db-admins'],
|
||||
},
|
||||
}]
|
||||
|
||||
ALLOW_SAVE_PASSWORD = False
|
||||
ENHANCED_COOKIE_PROTECTION = False
|
||||
SESSION_COOKIE_SECURE = True
|
||||
SESSION_COOKIE_HTTPONLY = True
|
||||
SESSION_COOKIE_SAMESITE = 'Lax'
|
||||
ALLOWED_HOSTS = ['db-admin.learn.hyeonworks.com']
|
||||
UPGRADE_CHECK_ENABLED = False
|
||||
SHOW_GRAVATAR_IMAGE = False
|
||||
LLM_ENABLED = False
|
||||
ENABLE_PSQL = False
|
||||
ENABLE_SERVER_PASS_EXEC_CMD = False
|
||||
@@ -0,0 +1,5 @@
|
||||
- op: remove
|
||||
path: /metadata/annotations/kubernetes.io~1ingress.class
|
||||
- op: add
|
||||
path: /spec/ingressClassName
|
||||
value: traefik
|
||||
@@ -0,0 +1,38 @@
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
|
||||
helmGlobals:
|
||||
chartHome: .helm/charts
|
||||
|
||||
helmCharts:
|
||||
- name: pgadmin4-helm
|
||||
repo: oci://docker.io/dpage
|
||||
version: 9.16.0
|
||||
releaseName: pgadmin
|
||||
namespace: platform-admin
|
||||
valuesFile: values/home.yaml
|
||||
includeCRDs: false
|
||||
skipTests: true
|
||||
|
||||
resources:
|
||||
- config.yaml
|
||||
- network-policies
|
||||
|
||||
patches:
|
||||
- patch: |-
|
||||
- op: add
|
||||
path: /metadata/namespace
|
||||
value: platform-admin
|
||||
target:
|
||||
labelSelector: app.kubernetes.io/instance=pgadmin
|
||||
- path: ingress-class-patch.yaml
|
||||
target:
|
||||
group: networking.k8s.io
|
||||
version: v1
|
||||
kind: Ingress
|
||||
name: pgadmin
|
||||
|
||||
images:
|
||||
- name: docker.io/dpage/pgadmin4
|
||||
newName: docker.io/dpage/pgadmin4
|
||||
digest: sha256:66a300a7ecdcc1f325af0c430315329bca46cd4a7067227d6899802238167c6e
|
||||
@@ -0,0 +1,6 @@
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
|
||||
resources:
|
||||
- pgadmin.yaml
|
||||
- platform-postgres.yaml
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user