문제: 재리뷰 수정 커밋(bbe944bd)만 다시 본 2차 재리뷰가 Important 7건,
Minor 22건을 찾았다. 결과 불명 503 뒤 같은 키를 다시 보내면 Redis
store가 lease 지난 실행을 넘겨받아 작업을 다시 돌렸다. replay TTL이
lease보다 짧으면 lease 도중 key가 사라져 같은 키가 두 번 실행됐다.
@SystemInvoked 규칙이 매뉴얼 09장의 이벤트 소비 경로까지 막았다. V22가
outbox_event에 가장 강한 잠금을 lock_timeout 없이 걸었다. prod의
fixture- 거부가 HOSTNAME 같은 비밀 아닌 값까지 막았다. 데모 허용 목록의
test·demo 프로필로 sample이 뜨지 않았다. 기동 rule bean의 @Value 필드가
조용히 null이 됐다.
방법: 영역별 4묶음이 test-driven-development 절차로 실패 테스트를
먼저 쓰고 고쳤다. 결정은 재리뷰 README 9.1절이다.
- 두 멱등성 store가 같은 계약을 따른다. lease가 지난 실행 기록은 다시
돌리지 않고 포기로 표시해 500 RECOVERY_REQUIRED로 답한다. 기록 수명은
lease + replay TTL이고, replay-ttl < processing-lease면 기동이 실패한다.
- 사용자 transport만 @SystemInvoked use case에 닿지 못한다. 메시지
handler·scheduler는 새 표시 @SystemCaller(reason)를 달고 use case를
생성자에서 new로 만든다. 매뉴얼 경로를 테스트로 고정했다.
- 아직 어디에도 적용되지 않은 마이그레이션(V20·V22·알림 V11~V13·sample
V8_1·grpc V002·V003)에 lock_timeout 5s를 넣고, 텍스트 검사기와 실 PG
lane을 더했다. 이미 배포된 24개는 동결 목록으로 둔다.
- fixture- 검사는 비밀로 분류된 값만 본다. sample은 local·test·demo에서
object storage를 끈다. 기동 rule bean에 주입 멤버가 있으면 모든
프로필에서 기동이 실패한다. 리뷰어가 제안한 "static @Bean 선언 강제"는
문제없는 등록을 막고 @Value 필드는 통과시켜서 버렸다.
- 이 수정 전에 move된 파일의 쿼터 차이를 찾는 쿼리와 재계산 SQL을
docs/fileserver/operations.md에 적었다.
위험·한계: lease 뒤 늦게 끝난 첫 시도의 결과는 기록되지 않는다(그 요청도
500). @SystemCaller가 없는 메시지 handler, 주입 멤버가 있는 기동 rule,
working-directory 없이 ./gradlew를 부르는 workflow 명령은 파생 서비스에서
새로 실패하거나 세지지 않는다(업그레이드 안내 B-14~16·B-36·B-39).
spotbugsTestFixtures는 게이트 밖이고 기준 커밋에서도 실패하며, 새 검사기의
정규식 경고 8건이 더해졌다. 실제 CI runner 실행은 확인하지 못했다(Gitea
repo의 Actions가 꺼져 있다).
검증: ci, 두 빌드의 verifyStrictLaneStages --dry-run, mainCheck,
:integrationCheck, -p optional-platforms ci integrationCheck
performanceCheck, :performanceCheck, :releaseCheck(-PgitRevision),
verify-compose-profile-contracts(18 lane) 첫 실행 통과. 쿼터 재계산
SQL은 실 PostgreSQL 16에서 차이 2행 → 0행을 확인했다.
Refs: docs/reviews/2026-11-03-rereview/README.md 9·9.1·10절
BREAKING CHANGE: replay-ttl < processing-lease 설정은 기동 실패, ledger
release가 claim token을 받는다, HttpBodyReadException(IOException 하위)이
본문 실패를 싣는다, 기동 rule·비밀 목록 bean의 주입 멤버 금지,
verifyStrictLaneStages가 workflow 명령의 working-directory로 빌드를 정한다.
766 lines
29 KiB
YAML
766 lines
29 KiB
YAML
# Registry: Metrics
|
|
# Schema owner: feature-contract-registry-governance
|
|
# Owner branch: feature-metrics-alerting-contract
|
|
# Last updated: 2026-05-22
|
|
#
|
|
# Notes
|
|
# - Naming: Micrometer dot.case + unit suffix (.seconds | .bytes | .total).
|
|
# - Tag cardinality bounds are SSOT of feature-metrics-alerting-contract "Cardinality Bounds" table.
|
|
# - High-cardinality tags forbidden globally: user_id, request_id, raw_url, raw_query,
|
|
# raw_header_value, ip_address. These MUST NOT appear in any row.
|
|
# - tenant_id label is bounded mapping table id OR cohort bucket only (ULID raw forbidden).
|
|
# - error_code tag cardinality_limit follows error-codes.yaml row count (max 100).
|
|
|
|
metrics:
|
|
|
|
# === HTTP server (inbound) ===
|
|
# source: feature-metrics-alerting-contract — Metric/Alert Defaults
|
|
# "HTTP metric | http.server.requests with method/status/uri-template | raw URL or user id tag"
|
|
- name: http.server.requests
|
|
type: timer
|
|
unit: seconds
|
|
tags:
|
|
- name: method
|
|
cardinality_limit: 8
|
|
allowed_values: [GET, POST, PUT, PATCH, DELETE, HEAD, OPTIONS, OTHER]
|
|
- name: status
|
|
cardinality_limit: 7
|
|
allowed_values: [1xx, 2xx, 3xx, 4xx, 5xx, ok, other]
|
|
- name: uri_template
|
|
cardinality_limit: 200
|
|
validation: must_be_template_not_raw_uri
|
|
percentiles: [0.5, 0.9, 0.95, 0.99]
|
|
histogram_buckets: slo_driven
|
|
alert_severity_thresholds:
|
|
p1: "error_rate > 5% for 5m OR > 10% for 1m"
|
|
p2: "error_rate > 1% for 10m"
|
|
p3: "error_rate > 0.1% for 1h"
|
|
owner_branch: feature-metrics-alerting-contract
|
|
log_field_mapping: [method, status, uri_template]
|
|
compatibility_impact: additive
|
|
required_test: contract-verification:metrics-cardinality
|
|
|
|
# source: feature-metrics-alerting-contract — P1/P2/P3 정량 기준 (HTTP latency p99)
|
|
# "P1: p99 > 5s 5분 / P2: p99 > 1s 10분 / P3: p99 > 500ms 30분"
|
|
- name: http.server.requests.latency
|
|
type: timer
|
|
unit: seconds
|
|
tags:
|
|
- name: method
|
|
cardinality_limit: 8
|
|
allowed_values: [GET, POST, PUT, PATCH, DELETE, HEAD, OPTIONS, OTHER]
|
|
- name: uri_template
|
|
cardinality_limit: 200
|
|
validation: must_be_template_not_raw_uri
|
|
percentiles: [0.5, 0.9, 0.95, 0.99]
|
|
histogram_buckets: slo_driven
|
|
alert_severity_thresholds:
|
|
p1: "p99 > 5s for 5m"
|
|
p2: "p99 > 1s for 10m"
|
|
p3: "p99 > 500ms for 30m"
|
|
owner_branch: feature-metrics-alerting-contract
|
|
log_field_mapping: [method, uri_template, duration_ms]
|
|
compatibility_impact: additive
|
|
required_test: contract-verification:metrics-cardinality
|
|
|
|
# === HTTP client (outbound dependency) ===
|
|
# source: feature-metrics-alerting-contract — Metric/Alert Defaults
|
|
# "dependency metric | dependency.client.requests with dependency.name/type/outcome | endpoint with secret tag"
|
|
# source: feature-outbound-http-client-baseline — "circuit breaker metric은 dependency.name, dependency.type, outcome까지만 tag로 허용"
|
|
- name: dependency.client.requests
|
|
type: timer
|
|
unit: seconds
|
|
tags:
|
|
- name: dependency_name
|
|
cardinality_limit: 50
|
|
- name: dependency_type
|
|
cardinality_limit: 10
|
|
allowed_values: [http, grpc, db, cache, queue, broker, other]
|
|
- name: outcome
|
|
cardinality_limit: 5
|
|
allowed_values: [SUCCESS, FAILURE, CIRCUIT_OPEN, TIMEOUT, REJECTED]
|
|
percentiles: [0.5, 0.9, 0.95, 0.99]
|
|
histogram_buckets: slo_driven
|
|
alert_severity_thresholds:
|
|
p1: "required dep unavailable for 2m"
|
|
p2: "optional dep degraded for 5m"
|
|
p3: "spike alert (10x baseline)"
|
|
owner_branch: feature-metrics-alerting-contract
|
|
log_field_mapping: [dependency_name, dependency_type, outcome, duration_ms]
|
|
compatibility_impact: additive
|
|
required_test: contract-verification:metrics-cardinality
|
|
|
|
# source: feature-outbound-http-client-baseline — decisions
|
|
# "retry/circuit breaker 기본 라이브러리는 Resilience4j"
|
|
# source: feature-metrics-alerting-contract — "retry/CB minimum: resilience4j.retry.calls{outcome}"
|
|
- name: resilience4j.retry.calls
|
|
type: counter
|
|
unit: total
|
|
tags:
|
|
- name: name
|
|
cardinality_limit: 50
|
|
- name: outcome
|
|
cardinality_limit: 5
|
|
allowed_values: [SUCCESS, FAILURE, CIRCUIT_OPEN, TIMEOUT, REJECTED]
|
|
percentiles: null
|
|
histogram_buckets: null
|
|
alert_severity_thresholds:
|
|
p2: "retry exhaustion rate > 1% for 10m"
|
|
owner_branch: feature-outbound-http-client-baseline
|
|
log_field_mapping: [dependency_name, outcome, retry_attempt]
|
|
compatibility_impact: additive
|
|
required_test: contract-verification:metrics-cardinality
|
|
|
|
# source: feature-metrics-alerting-contract — "resilience4j.circuitbreaker.state"
|
|
- name: resilience4j.circuitbreaker.state
|
|
type: gauge
|
|
unit: total
|
|
tags:
|
|
- name: name
|
|
cardinality_limit: 50
|
|
- name: state
|
|
cardinality_limit: 6
|
|
allowed_values: [CLOSED, OPEN, HALF_OPEN, DISABLED, FORCED_OPEN, METRICS_ONLY]
|
|
percentiles: null
|
|
histogram_buckets: null
|
|
alert_severity_thresholds:
|
|
p1: "state == OPEN for required dependency for 2m"
|
|
p2: "state == OPEN for optional dependency for 5m"
|
|
owner_branch: feature-outbound-http-client-baseline
|
|
log_field_mapping: [dependency_name]
|
|
compatibility_impact: additive
|
|
required_test: contract-verification:metrics-cardinality
|
|
|
|
# source: feature-metrics-alerting-contract — "resilience4j.circuitbreaker.calls{outcome}"
|
|
- name: resilience4j.circuitbreaker.calls
|
|
type: timer
|
|
unit: seconds
|
|
tags:
|
|
- name: name
|
|
cardinality_limit: 50
|
|
- name: outcome
|
|
cardinality_limit: 5
|
|
allowed_values: [SUCCESS, FAILURE, CIRCUIT_OPEN, TIMEOUT, REJECTED]
|
|
percentiles: [0.5, 0.9, 0.95, 0.99]
|
|
histogram_buckets: slo_driven
|
|
alert_severity_thresholds:
|
|
p2: "CIRCUIT_OPEN rate > 1% for 10m"
|
|
owner_branch: feature-outbound-http-client-baseline
|
|
log_field_mapping: [dependency_name, outcome, duration_ms]
|
|
compatibility_impact: additive
|
|
required_test: contract-verification:metrics-cardinality
|
|
|
|
# === DB connection pool ===
|
|
# source: feature-persistence-failure-baseline — Hikari Alert Threshold
|
|
# "pool wait p99 > 100ms 5분 지속 → P2 / pool exhaustion (active = max) > 1분 → P1"
|
|
# source: feature-metrics-alerting-contract — "hikaricp.connections.acquire{outcome='timeout'} p99 > 100ms"
|
|
- name: hikaricp.connections.acquire
|
|
type: timer
|
|
unit: seconds
|
|
tags:
|
|
- name: pool
|
|
cardinality_limit: 5
|
|
- name: outcome
|
|
cardinality_limit: 3
|
|
allowed_values: [SUCCESS, TIMEOUT, FAILURE]
|
|
percentiles: [0.5, 0.9, 0.95, 0.99]
|
|
histogram_buckets: slo_driven
|
|
alert_severity_thresholds:
|
|
p1: "pool exhaustion (active == max) for 1m"
|
|
p2: "acquire p99 > 100ms for 5m"
|
|
owner_branch: feature-persistence-failure-baseline
|
|
log_field_mapping: [pool, outcome]
|
|
compatibility_impact: additive
|
|
required_test: contract-verification:metrics-cardinality
|
|
|
|
# source: feature-persistence-failure-baseline — In scope "Hikari metric 노출 기준" + Hikari Alert Threshold
|
|
- name: hikaricp.connections.usage
|
|
type: timer
|
|
unit: seconds
|
|
tags:
|
|
- name: pool
|
|
cardinality_limit: 5
|
|
percentiles: [0.5, 0.9, 0.95, 0.99]
|
|
histogram_buckets: slo_driven
|
|
alert_severity_thresholds:
|
|
p2: "usage p99 elevated > 10m"
|
|
owner_branch: feature-persistence-failure-baseline
|
|
log_field_mapping: [pool]
|
|
compatibility_impact: additive
|
|
required_test: contract-verification:metrics-cardinality
|
|
|
|
# source: feature-persistence-failure-baseline — In scope "Hikari metric 노출 기준"
|
|
- name: hikaricp.connections.active
|
|
type: gauge
|
|
unit: total
|
|
tags:
|
|
- name: pool
|
|
cardinality_limit: 5
|
|
percentiles: null
|
|
histogram_buckets: null
|
|
alert_severity_thresholds:
|
|
p1: "active == max for 1m"
|
|
owner_branch: feature-persistence-failure-baseline
|
|
log_field_mapping: [pool]
|
|
compatibility_impact: additive
|
|
required_test: contract-verification:metrics-cardinality
|
|
|
|
# source: feature-metrics-alerting-contract — Histogram Buckets/Percentile "DB query: same"
|
|
- name: db.query.duration
|
|
type: timer
|
|
unit: seconds
|
|
tags:
|
|
- name: operation
|
|
cardinality_limit: 20
|
|
allowed_values: [select, insert, update, delete, batch, ddl, other]
|
|
- name: outcome
|
|
cardinality_limit: 3
|
|
allowed_values: [SUCCESS, FAILURE, TIMEOUT]
|
|
percentiles: [0.5, 0.9, 0.95, 0.99]
|
|
histogram_buckets: slo_driven
|
|
alert_severity_thresholds:
|
|
p2: "p99 > 1s for 10m"
|
|
owner_branch: feature-metrics-alerting-contract
|
|
log_field_mapping: [operation, outcome, duration_ms]
|
|
compatibility_impact: additive
|
|
required_test: contract-verification:metrics-cardinality
|
|
|
|
# === Background job / async executor ===
|
|
# source: feature-background-job-async-contract — Decisionized Work Items "saturation policy"
|
|
# "AbortPolicy default (core=10, max=50, queue=200)"
|
|
- name: executor.saturation
|
|
type: gauge
|
|
unit: total
|
|
tags:
|
|
- name: executor_name
|
|
cardinality_limit: 10
|
|
percentiles: null
|
|
histogram_buckets: null
|
|
alert_severity_thresholds:
|
|
p2: "queue size > 80% capacity for 5m"
|
|
p1: "rejection rate > 0 for 1m"
|
|
owner_branch: feature-background-job-async-contract
|
|
log_field_mapping: [executor_name]
|
|
compatibility_impact: additive
|
|
required_test: contract-verification:metrics-cardinality
|
|
|
|
# source: feature-background-job-async-contract — Decisionized Work Items "saturation | bounded executor + rejection log"
|
|
- name: executor.rejected.total
|
|
type: counter
|
|
unit: total
|
|
tags:
|
|
- name: executor_name
|
|
cardinality_limit: 10
|
|
- name: policy
|
|
cardinality_limit: 3
|
|
allowed_values: [AbortPolicy, CallerRunsPolicy, DiscardPolicy]
|
|
percentiles: null
|
|
histogram_buckets: null
|
|
alert_severity_thresholds:
|
|
p1: "rejection_count > 0 for 1m"
|
|
owner_branch: feature-background-job-async-contract
|
|
log_field_mapping: [executor_name, policy]
|
|
compatibility_impact: additive
|
|
required_test: contract-verification:metrics-cardinality
|
|
|
|
# source: feature-background-job-async-contract — Decisionized Work Items "retry/DLQ | exp backoff jitter, max 3, DLQ exhausted"
|
|
- name: job.retry.total
|
|
type: counter
|
|
unit: total
|
|
tags:
|
|
- name: job_name
|
|
cardinality_limit: 50
|
|
- name: outcome
|
|
cardinality_limit: 4
|
|
allowed_values: [SUCCESS, RETRY, EXHAUSTED, DLQ]
|
|
percentiles: null
|
|
histogram_buckets: null
|
|
alert_severity_thresholds:
|
|
p2: "EXHAUSTED rate > 1% for 10m"
|
|
owner_branch: feature-background-job-async-contract
|
|
log_field_mapping: [job_name, outcome, retry_attempt]
|
|
compatibility_impact: additive
|
|
required_test: contract-verification:metrics-cardinality
|
|
|
|
# source: feature-background-job-async-contract — "DLQ after exhausted attempts"
|
|
- name: job.dlq.total
|
|
type: counter
|
|
unit: total
|
|
tags:
|
|
- name: job_name
|
|
cardinality_limit: 50
|
|
percentiles: null
|
|
histogram_buckets: null
|
|
alert_severity_thresholds:
|
|
p1: "DLQ rate sustained > 0 for 5m"
|
|
owner_branch: feature-background-job-async-contract
|
|
log_field_mapping: [job_name]
|
|
compatibility_impact: additive
|
|
required_test: contract-verification:metrics-cardinality
|
|
|
|
# === Distributed lock ===
|
|
# source: feature-distributed-lock-contract D7 — "metric lock.acquisition (tag: outcome =
|
|
# acquired/timeout/error) — 신규 제안" / D5 — try-lock + 유한 waitTime + lease(TTL). 분산
|
|
# 상호배제(distributedLockProvider) 획득 시도 결과를 센다. key 는 tag 로 넣지 않는다
|
|
# (무한 cardinality — 위 전역 금지 규칙). timeout outcome 은 LOCK_ACQUISITION_TIMEOUT 발생과 1:1.
|
|
- name: lock.acquisition
|
|
type: counter
|
|
unit: total
|
|
tags:
|
|
- name: outcome
|
|
cardinality_limit: 3
|
|
allowed_values: [acquired, timeout, error]
|
|
percentiles: null
|
|
histogram_buckets: null
|
|
alert_severity_thresholds:
|
|
p2: "timeout rate > 5% for 10m"
|
|
owner_branch: feature-distributed-lock-contract
|
|
log_field_mapping: [outcome]
|
|
compatibility_impact: additive
|
|
required_test: contract-verification:metrics-cardinality
|
|
|
|
# source: feature-distributed-lock-contract §Edge / D5 (SI-LOCK-C5) — "lease 만료 후 unlock →
|
|
# ConcurrentModificationException — 삼킴 금지, 로그+metric 후 정상 흐름 복귀". Counts releases that
|
|
# found the lease already expired (the JdbcLock row was reclaimed by another instance before
|
|
# the holder called close()). A sustained nonzero rate means lease TTL is shorter than real
|
|
# critical-section duration — raise APP/lease TTL or shorten the protected work. Not an
|
|
# acquisition outcome, hence a separate counter from lock.acquisition.
|
|
- name: lock.lease.expired
|
|
type: counter
|
|
unit: total
|
|
tags: []
|
|
percentiles: null
|
|
histogram_buckets: null
|
|
alert_severity_thresholds:
|
|
p2: "lease-expired rate sustained > 0 for 10m"
|
|
owner_branch: feature-distributed-lock-contract
|
|
log_field_mapping: []
|
|
compatibility_impact: additive
|
|
required_test: contract-verification:metrics-cardinality
|
|
|
|
# === Outbox publisher ===
|
|
# source: feature-domain-event-outbox-contract — Outbox Defaults
|
|
# "DB outbox table with eventId, aggregateId, eventType, payload, occurredAt, status, attemptCount, nextAttemptAt"
|
|
- name: outbox.publisher.published.total
|
|
type: counter
|
|
unit: total
|
|
tags:
|
|
- name: event_type
|
|
cardinality_limit: 50
|
|
- name: outcome
|
|
cardinality_limit: 5
|
|
# The relay cycle outcomes (OutboxRelayResult.Outcome), not row statuses: IN_FLIGHT is a status
|
|
# and is never counted. STALE: the relay's claim was taken over before its result could be
|
|
# written (V17 lease fence). RELEASED: a stopping relay gave back a claim it had not used.
|
|
allowed_values: [PUBLISHED, FAILED, DEAD, STALE, RELEASED]
|
|
percentiles: null
|
|
histogram_buckets: null
|
|
alert_severity_thresholds:
|
|
p2: "FAILED rate > 1% for 10m, or STALE > 0 for 10m (in-flight timeout shorter than a relay stall)"
|
|
owner_branch: feature-domain-event-outbox-contract
|
|
log_field_mapping: [event_type, outcome, event_id]
|
|
compatibility_impact: additive
|
|
required_test: contract-verification:metrics-cardinality
|
|
|
|
# source: feature-domain-event-outbox-contract — row status enum PENDING/IN_FLIGHT/PUBLISHED/FAILED/DEAD
|
|
- name: outbox.publisher.lag
|
|
type: gauge
|
|
unit: seconds
|
|
tags:
|
|
- name: event_type
|
|
cardinality_limit: 50
|
|
percentiles: null
|
|
histogram_buckets: null
|
|
alert_severity_thresholds:
|
|
p2: "lag > 60s for 10m"
|
|
p1: "lag > 300s for 5m"
|
|
owner_branch: feature-domain-event-outbox-contract
|
|
log_field_mapping: [event_type]
|
|
compatibility_impact: additive
|
|
required_test: contract-verification:metrics-cardinality
|
|
|
|
# source: feature-domain-event-outbox-contract — row status enum + Outbox Defaults attemptCount
|
|
# PARKED: an operator held a row that was retried without end (re-review 2026-11-03 RR-05-03).
|
|
- name: outbox.pending.size
|
|
type: gauge
|
|
unit: total
|
|
tags:
|
|
- name: status
|
|
cardinality_limit: 6
|
|
allowed_values: [PENDING, IN_FLIGHT, PUBLISHED, FAILED, DEAD, PARKED]
|
|
percentiles: null
|
|
histogram_buckets: null
|
|
# DEAD > 0: the claim moves a row it will not hand out to DEAD without a relay result, so
|
|
# outbox.publisher.published.total{outcome=DEAD} misses it (re-review round 2, RR-02-12).
|
|
alert_severity_thresholds:
|
|
p1: "DEAD > 0 (runbook://outbox/dead-letter)"
|
|
p2: "PENDING size growing for 10m"
|
|
owner_branch: feature-domain-event-outbox-contract
|
|
log_field_mapping: [status]
|
|
compatibility_impact: additive
|
|
required_test: contract-verification:metrics-cardinality
|
|
|
|
# source: re-review 2026-11-03 RR-05-03 — a refusal for now and an unanswered publish are retried
|
|
# without end; these count the FAILED rows deferred at least 10 times (OutboxMetrics
|
|
# LONG_RETRY_DEFERRED_CLAIMS, the publish-failed runbook's §4.1 query) and age the oldest of them.
|
|
- name: outbox.retry.long.size
|
|
type: gauge
|
|
unit: total
|
|
tags: []
|
|
percentiles: null
|
|
histogram_buckets: null
|
|
alert_severity_thresholds:
|
|
p2: "> 0 for 10m (runbook://outbox/publish-failed §4.1)"
|
|
owner_branch: feature-domain-event-outbox-contract
|
|
log_field_mapping: []
|
|
compatibility_impact: additive
|
|
required_test: contract-verification:metrics-cardinality
|
|
|
|
- name: outbox.retry.long.age
|
|
type: gauge
|
|
unit: seconds
|
|
tags: []
|
|
percentiles: null
|
|
histogram_buckets: null
|
|
alert_severity_thresholds:
|
|
p2: "> 3600 (runbook://outbox/publish-failed §4.1)"
|
|
owner_branch: feature-domain-event-outbox-contract
|
|
log_field_mapping: []
|
|
compatibility_impact: additive
|
|
required_test: contract-verification:metrics-cardinality
|
|
|
|
# === Cache ===
|
|
# source: feature-cache-consistency-contract — Decisionized Work Items "cache pattern | cache-aside default"
|
|
- name: cache.gets.total
|
|
type: counter
|
|
unit: total
|
|
tags:
|
|
- name: cache_name
|
|
cardinality_limit: 50
|
|
- name: result
|
|
cardinality_limit: 3
|
|
allowed_values: [hit, miss, error]
|
|
percentiles: null
|
|
histogram_buckets: null
|
|
alert_severity_thresholds:
|
|
p3: "hit_ratio < baseline 0.5x for 1h"
|
|
owner_branch: feature-cache-consistency-contract
|
|
log_field_mapping: [cache_name, result]
|
|
compatibility_impact: additive
|
|
required_test: contract-verification:metrics-cardinality
|
|
|
|
# source: feature-cache-consistency-contract — "invalidation = after-commit only", "invalidation 실패가 조용히 무시되면 실패"
|
|
- name: cache.invalidations.total
|
|
type: counter
|
|
unit: total
|
|
tags:
|
|
- name: cache_name
|
|
cardinality_limit: 50
|
|
- name: outcome
|
|
cardinality_limit: 3
|
|
allowed_values: [SUCCESS, FAILURE, SKIPPED]
|
|
percentiles: null
|
|
histogram_buckets: null
|
|
alert_severity_thresholds:
|
|
p2: "FAILURE rate > 0 for 5m"
|
|
owner_branch: feature-cache-consistency-contract
|
|
log_field_mapping: [cache_name, outcome]
|
|
compatibility_impact: additive
|
|
required_test: contract-verification:metrics-cardinality
|
|
|
|
# source: redis-production-capability — optional bounded cache-only L1
|
|
- name: cache.local.requests.total
|
|
type: counter
|
|
unit: total
|
|
tags:
|
|
- name: cache_name
|
|
cardinality_limit: 50
|
|
- name: result
|
|
cardinality_limit: 4
|
|
allowed_values: [hit, miss, error, bypass]
|
|
percentiles: null
|
|
histogram_buckets: null
|
|
alert_severity_thresholds:
|
|
p3: "bypass or error rate above baseline for 15m"
|
|
owner_branch: redis-production-capability
|
|
log_field_mapping: [cache_name, result]
|
|
compatibility_impact: additive
|
|
required_test: contract-verification:metrics-cardinality
|
|
|
|
# source: redis-production-capability — local stale-age bound
|
|
- name: cache.local.entry.age.seconds
|
|
type: timer
|
|
unit: seconds
|
|
tags:
|
|
- name: cache_name
|
|
cardinality_limit: 50
|
|
percentiles: [0.5, 0.9, 0.95, 0.99]
|
|
histogram_buckets: slo_driven
|
|
alert_severity_thresholds:
|
|
p3: "p99 approaches configured local TTL for 30m"
|
|
owner_branch: redis-production-capability
|
|
log_field_mapping: [cache_name]
|
|
compatibility_impact: additive
|
|
required_test: contract-verification:metrics-cardinality
|
|
|
|
# source: redis-production-capability — bounded invalidation and generation reconciliation
|
|
- name: cache.local.maintenance.total
|
|
type: counter
|
|
unit: total
|
|
tags:
|
|
- name: cache_name
|
|
cardinality_limit: 50
|
|
- name: event
|
|
cardinality_limit: 14
|
|
allowed_values:
|
|
- evict_cardinality
|
|
- evict_weight
|
|
- evict_ttl
|
|
- evict_invalidation
|
|
- flush_invalidation
|
|
- reconcile_generation_changed
|
|
- reconcile_unchanged
|
|
- reconcile_error
|
|
- subscriber_disconnected
|
|
- subscriber_overflow
|
|
- subscriber_malformed
|
|
- subscriber_publish_success
|
|
- subscriber_publish_error
|
|
- other
|
|
percentiles: null
|
|
histogram_buckets: null
|
|
alert_severity_thresholds:
|
|
p2: "reconcile_error, subscriber_overflow, or sustained disconnects for 5m"
|
|
owner_branch: redis-production-capability
|
|
log_field_mapping: [cache_name, event]
|
|
compatibility_impact: additive
|
|
required_test: contract-verification:metrics-cardinality
|
|
|
|
# === Redis connection lanes ===
|
|
# source: review 2026-11-03 RR-04-02 — the cache's conditional writes got their own lane
|
|
# (CACHE_SCRIPT) so that cache fills cannot take the SCRIPT lane's connections from idempotency,
|
|
# rate limiting and leases. These meters show, per lane, whether that holds. Emitted by
|
|
# redis-runtime RedisLaneMetrics for every lane RedisRuntimeOwner pools, only when the context has
|
|
# a MeterRegistry. Every counter series is registered at zero at startup.
|
|
- name: redis.lane.acquire.wait.seconds
|
|
type: timer
|
|
unit: seconds
|
|
tags:
|
|
- name: lane
|
|
# The lane's name: REGULAR, SCRIPT (redis-runtime), CACHE_SCRIPT (cache-redis), BLOCKING,
|
|
# TRANSACTION, ADMIN (redis-sdk), plus any lane a service composes as a RedisLaneContribution.
|
|
cardinality_limit: 12
|
|
- name: outcome
|
|
cardinality_limit: 2
|
|
allowed_values: [admitted, refused]
|
|
# As RedisLaneMetrics registers it: a percentile histogram (publishPercentileHistogram, expected
|
|
# range 1ms..10s) and no client-side percentiles and no SLO boundaries. The p99 below is a
|
|
# histogram_quantile over the buckets (re-review 2026-11-04 RR-03-17).
|
|
percentiles: null
|
|
histogram_buckets: percentile_histogram
|
|
alert_severity_thresholds:
|
|
p2: "SCRIPT admitted p99 > 100ms for 10m (the correctness lane is queueing)"
|
|
p3: "CACHE_SCRIPT admitted p99 > 100ms for 30m"
|
|
owner_branch: redis-production-capability
|
|
log_field_mapping: [lane, outcome]
|
|
compatibility_impact: additive
|
|
required_test: contract-verification:metrics-cardinality
|
|
|
|
- name: redis.lane.acquire.refused.total
|
|
type: counter
|
|
unit: total
|
|
tags:
|
|
- name: lane
|
|
# The lane's name: REGULAR, SCRIPT (redis-runtime), CACHE_SCRIPT (cache-redis), BLOCKING,
|
|
# TRANSACTION, ADMIN (redis-sdk), plus any lane a service composes as a RedisLaneContribution.
|
|
cardinality_limit: 12
|
|
- name: reason
|
|
cardinality_limit: 4
|
|
allowed_values: [saturated, closing, interrupted, connect_failed]
|
|
percentiles: null
|
|
histogram_buckets: null
|
|
alert_severity_thresholds:
|
|
p1: "SCRIPT or REGULAR saturated or connect_failed above zero for 2m while a correctness role is selected"
|
|
p2: "CACHE_SCRIPT saturated above baseline for 10m"
|
|
owner_branch: redis-production-capability
|
|
log_field_mapping: [lane, reason]
|
|
compatibility_impact: additive
|
|
required_test: contract-verification:metrics-cardinality
|
|
|
|
- name: redis.lane.connections.opened.total
|
|
type: counter
|
|
unit: total
|
|
tags:
|
|
- name: lane
|
|
# The lane's name: REGULAR, SCRIPT (redis-runtime), CACHE_SCRIPT (cache-redis), BLOCKING,
|
|
# TRANSACTION, ADMIN (redis-sdk), plus any lane a service composes as a RedisLaneContribution.
|
|
cardinality_limit: 12
|
|
percentiles: null
|
|
histogram_buckets: null
|
|
alert_severity_thresholds:
|
|
p3: "opened rate after warm-up above baseline for 15m (reconnect churn against a slow Redis)"
|
|
owner_branch: redis-production-capability
|
|
log_field_mapping: [lane]
|
|
compatibility_impact: additive
|
|
required_test: contract-verification:metrics-cardinality
|
|
|
|
- name: redis.lane.connections.discarded.total
|
|
type: counter
|
|
unit: total
|
|
tags:
|
|
- name: lane
|
|
# The lane's name: REGULAR, SCRIPT (redis-runtime), CACHE_SCRIPT (cache-redis), BLOCKING,
|
|
# TRANSACTION, ADMIN (redis-sdk), plus any lane a service composes as a RedisLaneContribution.
|
|
cardinality_limit: 12
|
|
- name: reason
|
|
cardinality_limit: 2
|
|
allowed_values: [invalidated, broken]
|
|
percentiles: null
|
|
histogram_buckets: null
|
|
alert_severity_thresholds:
|
|
p3: "invalidated rate above baseline for 15m (timed-out or failed calls discarding connections)"
|
|
owner_branch: redis-production-capability
|
|
log_field_mapping: [lane, reason]
|
|
compatibility_impact: additive
|
|
required_test: contract-verification:metrics-cardinality
|
|
|
|
- name: redis.lane.connections.active
|
|
type: gauge
|
|
unit: total
|
|
tags:
|
|
- name: lane
|
|
# The lane's name: REGULAR, SCRIPT (redis-runtime), CACHE_SCRIPT (cache-redis), BLOCKING,
|
|
# TRANSACTION, ADMIN (redis-sdk), plus any lane a service composes as a RedisLaneContribution.
|
|
cardinality_limit: 12
|
|
percentiles: null
|
|
histogram_buckets: null
|
|
alert_severity_thresholds:
|
|
p2: "SCRIPT active == configured maximum-script-connections for 5m"
|
|
owner_branch: redis-production-capability
|
|
log_field_mapping: [lane]
|
|
compatibility_impact: additive
|
|
required_test: contract-verification:metrics-cardinality
|
|
|
|
- name: redis.lane.connections.pending
|
|
type: gauge
|
|
unit: total
|
|
tags:
|
|
- name: lane
|
|
# The lane's name: REGULAR, SCRIPT (redis-runtime), CACHE_SCRIPT (cache-redis), BLOCKING,
|
|
# TRANSACTION, ADMIN (redis-sdk), plus any lane a service composes as a RedisLaneContribution.
|
|
cardinality_limit: 12
|
|
percentiles: null
|
|
histogram_buckets: null
|
|
alert_severity_thresholds:
|
|
p2: "SCRIPT pending > 0 for 5m"
|
|
owner_branch: redis-production-capability
|
|
log_field_mapping: [lane]
|
|
compatibility_impact: additive
|
|
required_test: contract-verification:metrics-cardinality
|
|
|
|
# === Log appender ===
|
|
# source: feature-log-management-contract — Sampling Policy (final)
|
|
# "async appender overflow default: drop oldest INFO/DEBUG with counter metric (log.appender.dropped.total)"
|
|
- name: log.appender.dropped.total
|
|
type: counter
|
|
unit: total
|
|
tags:
|
|
- name: appender
|
|
cardinality_limit: 5
|
|
- name: level
|
|
cardinality_limit: 2
|
|
allowed_values: [INFO, DEBUG]
|
|
percentiles: null
|
|
histogram_buckets: null
|
|
alert_severity_thresholds:
|
|
p2: "dropped > 0 sustained for 10m"
|
|
owner_branch: feature-log-management-contract
|
|
log_field_mapping: [appender, level]
|
|
compatibility_impact: additive
|
|
required_test: contract-verification:metrics-cardinality
|
|
|
|
# === Distributed tracing ===
|
|
# source: feature-distributed-tracing-contract — decisions
|
|
# "trace sampling rate default = prod 1%, dev/local 100%; shipped runtime profiles are local|dev|prod"
|
|
- name: tracing.sampling.rate
|
|
type: gauge
|
|
unit: total
|
|
tags:
|
|
- name: profile
|
|
cardinality_limit: 3
|
|
allowed_values: [prod, dev, local]
|
|
percentiles: null
|
|
histogram_buckets: null
|
|
alert_severity_thresholds:
|
|
p3: "effective rate deviates from configured for 1h"
|
|
owner_branch: feature-distributed-tracing-contract
|
|
log_field_mapping: [profile]
|
|
compatibility_impact: additive
|
|
required_test: contract-verification:metrics-cardinality
|
|
|
|
# === JVM baseline ===
|
|
# source: feature-metrics-alerting-contract — In scope "JVM/process metric"
|
|
- name: jvm.memory.used
|
|
type: gauge
|
|
unit: bytes
|
|
tags:
|
|
- name: area
|
|
cardinality_limit: 2
|
|
allowed_values: [heap, nonheap]
|
|
- name: id
|
|
cardinality_limit: 10
|
|
percentiles: null
|
|
histogram_buckets: null
|
|
alert_severity_thresholds:
|
|
p2: "heap used / max > 0.85 for 10m"
|
|
owner_branch: feature-metrics-alerting-contract
|
|
log_field_mapping: [area, id]
|
|
compatibility_impact: additive
|
|
required_test: contract-verification:metrics-cardinality
|
|
|
|
# source: feature-metrics-alerting-contract — In scope "JVM/process metric"
|
|
- name: jvm.gc.pause
|
|
type: timer
|
|
unit: seconds
|
|
tags:
|
|
- name: action
|
|
cardinality_limit: 10
|
|
- name: cause
|
|
cardinality_limit: 10
|
|
percentiles: [0.5, 0.9, 0.95, 0.99]
|
|
histogram_buckets: slo_driven
|
|
alert_severity_thresholds:
|
|
p2: "p99 pause > 500ms for 10m"
|
|
owner_branch: feature-metrics-alerting-contract
|
|
log_field_mapping: [action, cause]
|
|
compatibility_impact: additive
|
|
required_test: contract-verification:metrics-cardinality
|
|
|
|
# source: feature-metrics-alerting-contract — In scope "JVM/process metric"
|
|
- name: jvm.threads.live
|
|
type: gauge
|
|
unit: total
|
|
tags: []
|
|
percentiles: null
|
|
histogram_buckets: null
|
|
alert_severity_thresholds:
|
|
p3: "thread count > 2x baseline for 30m"
|
|
owner_branch: feature-metrics-alerting-contract
|
|
log_field_mapping: []
|
|
compatibility_impact: additive
|
|
required_test: contract-verification:metrics-cardinality
|
|
|
|
# source: feature-metrics-alerting-contract — In scope "JVM/process metric" (process uptime)
|
|
- name: process.uptime
|
|
type: gauge
|
|
unit: seconds
|
|
tags: []
|
|
percentiles: null
|
|
histogram_buckets: null
|
|
alert_severity_thresholds:
|
|
p1: "uptime reset unexpectedly < 60s (crash loop signal)"
|
|
owner_branch: feature-metrics-alerting-contract
|
|
log_field_mapping: []
|
|
compatibility_impact: additive
|
|
required_test: contract-verification:metrics-cardinality
|