The tech-log tenant needs objects inside platform namespaces: its database and role, ingress to Postgres and AIStor, blackbox probes, alert rules, a dashboard and two observability NetworkPolicies. They were only ever applied with kubectl and no repository holds them, so a deleted object cannot be recreated and a platform re-apply can drop tech-log access without anyone noticing. Declare them in services/tenants/techlog, copied from each object's last-applied configuration. The dashboard follows the repository's dashboard pattern (JSON file + configMapGenerator, same name, same labels). The hyeonworks-prod Namespace moves here as the tenant boundary and gets PSA enforce=baseline, warn/audit=restricted. Rejected: enforce=restricted now. A server dry-run shows all three tech-log pods would violate it; baseline has zero violations. Risk: the render matches the live objects exactly except the added PSA labels. The pg_hba rules and the Secret stay out of this root (Cluster owner and Git policy). Verification (desktop): - render-tenants-and-policies.sh passes. - kubectl diff of the render shows only the Namespace PSA labels. - kubectl label --dry-run=server enforce=baseline: no warnings.
68 lines
2.3 KiB
YAML
68 lines
2.3 KiB
YAML
apiVersion: monitoring.coreos.com/v1
|
|
kind: PrometheusRule
|
|
metadata:
|
|
labels:
|
|
observability.hyeonworks.com/instance: home
|
|
name: techlog
|
|
namespace: observability
|
|
spec:
|
|
groups:
|
|
- name: techlog
|
|
rules:
|
|
- alert: TechLogMetricsDown
|
|
annotations:
|
|
summary: TechLog metrics target is unreachable
|
|
expr: up{namespace="hyeonworks-prod",service="techlog-management"} == 0
|
|
for: 5m
|
|
labels:
|
|
application: techlog
|
|
severity: warning
|
|
- alert: TechLogReadinessDown
|
|
annotations:
|
|
summary: TechLog dependency readiness is DOWN
|
|
expr: probe_success{job="techlog-readiness"} == 0
|
|
for: 3m
|
|
labels:
|
|
application: techlog
|
|
severity: critical
|
|
- alert: TechLogHTTPSDown
|
|
annotations:
|
|
summary: TechLog HTTPS or API health is failing
|
|
expr: probe_success{job="techlog-edge"} == 0
|
|
for: 3m
|
|
labels:
|
|
application: techlog
|
|
severity: critical
|
|
- alert: TechLogHTTP5xx
|
|
annotations:
|
|
summary: TechLog HTTP 5xx exceeds 5%
|
|
expr: (sum(rate(http_server_requests_seconds_count{namespace="hyeonworks-prod",status=~"5.."}[5m])) / clamp_min(sum(rate(http_server_requests_seconds_count{namespace="hyeonworks-prod"}[5m])),0.001)) > 0.05 and sum(increase(http_server_requests_seconds_count{namespace="hyeonworks-prod"}[5m])) > 20
|
|
for: 5m
|
|
labels:
|
|
application: techlog
|
|
severity: warning
|
|
- alert: TechLogLatencyHigh
|
|
annotations:
|
|
summary: TechLog p99 latency exceeds 2 seconds
|
|
expr: histogram_quantile(0.99,sum by(le)(rate(http_server_requests_seconds_bucket{namespace="hyeonworks-prod",uri!~"/actuator.*"}[5m]))) > 2
|
|
for: 10m
|
|
labels:
|
|
application: techlog
|
|
severity: warning
|
|
- alert: TechLogHeapPressure
|
|
annotations:
|
|
summary: TechLog heap usage exceeds 85%
|
|
expr: sum(jvm_memory_used_bytes{namespace="hyeonworks-prod",area="heap"}) / sum(jvm_memory_max_bytes{namespace="hyeonworks-prod",area="heap"}) > 0.85
|
|
for: 10m
|
|
labels:
|
|
application: techlog
|
|
severity: warning
|
|
- alert: TechLogDatabasePoolWaiting
|
|
annotations:
|
|
summary: TechLog database connections are queued
|
|
expr: hikaricp_connections_pending{namespace="hyeonworks-prod"} > 0
|
|
for: 5m
|
|
labels:
|
|
application: techlog
|
|
severity: warning
|