diff --git a/.claude/skills/be-observability/SKILL.md b/.claude/skills/be-observability/SKILL.md index c40037ea..3bffbbb2 100644 --- a/.claude/skills/be-observability/SKILL.md +++ b/.claude/skills/be-observability/SKILL.md @@ -43,7 +43,7 @@ paths: ## 이 분야에서 실패는 이렇게 보인다 - [ ] 새 구조화 칸은 `docs/observability/log-fields.md`에, 새 MDC 키는 `docs/registries/mdc-keys.yaml`에, 새 지표는 `docs/registries/metrics.yaml`에 먼저 있다. 헤더와 짝이면 `docs/registries/headers.yaml`도. — 각 파일 머리말 -- [ ] 지표 이름은 dot.case에 단위 접미사(`.seconds`·`.bytes`·`.total`)이고, tag마다 cardinality 상한이 있다. `error_code` tag는 registry 행 수를 넘지 않는다. — `docs/registries/metrics.yaml` +- [ ] 지표 이름은 dot.case이고 단위는 registry 행의 `unit`(`seconds`·`bytes`·`total`)이 말한다. tag마다 cardinality 상한이 있다. `error_code` tag는 registry 행 수를 넘지 않는다. — `docs/registries/metrics.yaml` - [ ] 레벨: 클라이언트 탓 WARN, 서버 탓 ERROR + `ServerFailureLog`, 기동 실패는 `StartupFailures`(`startup.phase`, `error.code`). — A8, `docs/observability/log-fields.md` - [ ] 메시지 문장은 사람용, 칸은 기계용이다. 검색·필터·경보는 칸으로 한다. — `docs/observability/log-fields.md` - [ ] 예외는 원인(cause)을 잃지 않게 감싸되, 메시지에 입력값을 넣지 않는다. — 일반 원칙, A8 @@ -58,8 +58,8 @@ paths: Gradle은 desktop에서 돈다(`docs/work/operate.md` 4.1절). -- registry와 코드 대조: app-bootstrap `test`의 `MetricsAlertingContractTest`, `StructuredLogFieldContractTest`, `DistributedTracingContractTest`, `HeaderRegistryRuntimeAlignmentTest`, `RunbookCoverageContractTest`, `OperationalErrorRegistryContractTest` → `scripts/verify.sh leaf :app-bootstrap` -- 새어 나감: `SqlLoggingForbiddenContractTest`, `PiiTokenBodyForbiddenContractTest`(같은 leaf), `:adapter:inbound:web:jpaPersistenceRedactionContractTest`(DB 실패가 로그·span에 닿지 않음) +- registry 파일을 읽는 대조는 `document-contract` 태그라 `test`에서 빠지고 `:app-bootstrap:documentContractTest`(main 단계)에서 돈다: `HeaderRegistryRuntimeAlignmentTest`, `RunbookCoverageContractTest`, `OperationalErrorRegistryContractTest`, `MetricsRegistryCodeAlignmentTest`(지표 행 ↔ 등록 코드 두 방향), `SecretsClassificationRegistryTest`(가림 규칙이 실제로 가리는지), 그리고 `MetricsAlertingContractTest`·`StructuredLogFieldContractTest`·`DistributedTracingContractTest`의 태그 붙은 메서드. 그 세 클래스의 나머지 메서드는 app-bootstrap `test` → `scripts/verify.sh leaf :app-bootstrap` +- 새어 나감: `SqlLoggingForbiddenContractTest`(`repository-contract` → `:app-bootstrap:repositoryContractTest`), `PiiTokenBodyForbiddenContractTest`(app-bootstrap `test`, 그 안의 `requestLoggingHasNoBodyCapturePath`는 `repository-contract`), `:adapter:inbound:web:jpaPersistenceRedactionContractTest`(DB 실패가 로그·span에 닿지 않음) - runtime 배선: `scripts/verify.sh leaf :composition:service-runtime`, span 내보내기는 app-bootstrap `systemTest`의 `TracingExportSystemTest` - runbook 숫자·SQL이 문서와 맞는지: document-contract lane(`:app-bootstrap:documentContractTest`, main 단계)과 runbook SQL을 실 PostgreSQL에서 도는 lane(`:adapter:outbound:persistence-jpa:postgresqlOutboxDispositionRunbookIntegrationTest` 등) diff --git a/docs/httpclient/operations.md b/docs/httpclient/operations.md index f19f8754..879418f0 100644 --- a/docs/httpclient/operations.md +++ b/docs/httpclient/operations.md @@ -13,19 +13,26 @@ | `http.client.timeout` | 단계별 시간 초과 | | `http.client.request.bytes` | 요청 wire 바이트 | | `http.client.response.bytes` | 응답 바이트 | -| `http.client.active` | 진행 중인 시도 | | `http.client.pool.connections` | 빌려 간 connection과 남은 connection | | `http.client.pool.pending` | pool 대기자 | -| `http.client.pool.acquire.duration` | pool 대기 시간 | -| `http.client.dns.duration` | DNS 시간 | -| `http.client.connect.duration` | 연결 시간 | -| `http.client.tls.duration` | TLS 시간 | -| `http.client.circuit.state` | circuit 상태 | +| `http.client.circuit.state` | circuit 상태(1 = open) | +| `http.client.circuit.state.rejected` | circuit이 열려 거부한 호출 | | `http.client.bulkhead.rejected` | bulkhead 거부 | -| `http.client.rate_limit.rejected` | 로컬 속도 제한 거부 | -| `http.client.oauth.refresh` | token 갱신 결과 | +| `http.client.ratelimit.rejected` | 로컬 속도 제한 거부 | | `http.client.ssrf.rejected` | dynamic target 거부 | +이름은 정했지만 아직 아무 코드도 내지 않는 metric이다(`HttpClientObservationNames` 상수, `DeclaredMetricsAreEmittedTest`의 +「아직 안 냄」 목록). 대시보드·경보에 쓰지 않는다. `docs/registries/metrics.yaml`에도 행이 없다. + +| Metric | 내려면 필요한 것 | +|---|---| +| `http.client.active` | 논리 호출 observation 수명에 묶인 진행 중 gauge | +| `http.client.pool.acquire.duration` | engine마다 요청 단계 시간 | +| `http.client.dns.duration` | engine마다 요청 단계 시간 | +| `http.client.connect.duration` | engine마다 요청 단계 시간 | +| `http.client.tls.duration` | engine마다 요청 단계 시간 | +| `http.client.oauth.refresh` | token loader가 `MeterRegistry`를 받는 것 | + `http.client.requests`는 시도를 세고 `http.client.logical.calls`는 사용자 호출을 센다. 둘이 벌어지면 재시도가 실패를 흡수하고 있다는 뜻이다. 장애 때 가장 먼저 볼 곳이다. @@ -35,8 +42,8 @@ |---|---|---| | 논리 호출은 괜찮은데 시도가 치솟는다 | upstream이 나빠졌고 재시도가 흡수하고 있다 | 이유별 `http.client.retry.count` | | `http.client.ambiguous`가 0이 아니다 | 멱등이 아닌 쓰기가 `SENT_NO_RESPONSE`에 닿는다 | upstream과 맞춰 본다. 멱등 key를 검토한다 | -| pool 대기가 오른다 | pool이 작거나 upstream이 느리다 | `http.client.pool.acquire.duration`, `pool.connections` | -| circuit이 열렸다 | 계속된 upstream 실패: transport 오류, 5xx, 408, 429 | `http.client.circuit.state`. 로컬 거부와 그 밖의 4xx는 circuit을 열지 않는다 | +| pool 대기가 오른다 | pool이 작거나 upstream이 느리다 | `http.client.pool.pending`, `pool.connections` | +| circuit이 열렸다 | 계속된 upstream 실패: transport 오류, 5xx, 408, 429 | `http.client.circuit.state`, `http.client.circuit.state.rejected`. 로컬 거부와 그 밖의 4xx는 circuit을 열지 않는다 | | `http.client.ssrf.rejected`가 0이 아니다 | 호출자가 내부 URL을 넣고 있다 | Dynamic Target 정책과 감사 기록 | ## Actuator diff --git a/docs/registries/headers.yaml b/docs/registries/headers.yaml index 5530af1c..8158b55b 100644 --- a/docs/registries/headers.yaml +++ b/docs/registries/headers.yaml @@ -7,13 +7,19 @@ # - W3C 표준 헤더: 소문자 (traceparent, tracestate) # - mdc_key: snake_case (foundation이 정본) # - envelope_meta_field: camelCase (envelope이 정본) +# - 행은 코드가 실제로 읽거나 쓰는 헤더다. `source`는 셋 중 하나다(HeaderRegistryRuntimeAlignmentTest가 대조한다). +# - main 파일 경로: 그 이름 글자를 가지고 실제로 쓰는 파일. 이름이 상수면 그 상수를 쓰는 곳이 있어야 한다. +# - `service: <경로>`: 서비스 controller가 읽으라고 둔 `ApiHeaders` 상수다. 템플릿 main은 그 상수를 쓰지 않는다 +# (파일 서버는 If-Match·If-None-Match를 Spring `HttpHeaders` 상수로 읽는다). 시험이 정한 헤더만 이 출처를 쓴다. +# - `framework: …`: Spring·라이브러리가 읽거나 쓴다. headers: # source: feature-operational-error-observability-foundation.md L97 # "request_id | inbound filter (생성 또는 X-Request-Id 헤더) | response header X-Request-Id" - name: X-Request-Id direction: both - type: ulid + type: uuid-or-caller-token + format: "UUID; 받는 값은 [A-Za-z0-9._-]{1,128}" required: false generated_if_missing: true mdc_key: request_id @@ -22,6 +28,7 @@ headers: case_style: kebab compatibility_impact: none required_test: contract-verification:envelope-headers + source: "src/adapter/inbound/web/src/main/java/dev/caskeleton/adapter/inbound/web/mvc/filter/WebMvcRequestIdFilter.java" # source: feature-api-contract-baseline.md L77 / feature-rate-limit-idempotency-contract.md L66-67 # "idempotency header 이름은 Idempotency-Key" / "기본 scope는 (authenticatedPrincipal, idempotencyKey, useCaseName)" @@ -36,6 +43,7 @@ headers: case_style: kebab compatibility_impact: additive required_test: contract-verification:idempotency-replay + source: "service: src/adapter/inbound/web/src/main/java/dev/caskeleton/adapter/inbound/web/http/ApiHeaders.java" # source: feature-rate-limit-idempotency-contract.md L85 / foundation L85 # "RATE_LIMIT | ... | 429 | true (Retry-After 이후)" / "retry-after 기준 없이 429를 반환하면 실패" @@ -50,6 +58,7 @@ headers: case_style: kebab compatibility_impact: additive required_test: contract-verification:rate-limit-headers + source: "src/adapter/inbound/web/src/main/java/dev/caskeleton/adapter/inbound/web/http/ApiHeaders.java" # source: feature-rate-limit-idempotency-contract.md scope L26 (rate limit response/log 기준) # app.rate-limit.enabled=true인 요청 경로에서만 생성되는 rate-limit 응답 표면 @@ -64,6 +73,7 @@ headers: case_style: kebab compatibility_impact: additive required_test: contract-verification:rate-limit-headers + source: "src/adapter/inbound/web/src/main/java/dev/caskeleton/adapter/inbound/web/http/ApiHeaders.java" # source: feature-rate-limit-idempotency-contract.md scope L26 (rate limit response/log 기준) - name: X-RateLimit-Remaining @@ -77,6 +87,7 @@ headers: case_style: kebab compatibility_impact: additive required_test: contract-verification:rate-limit-headers + source: "src/adapter/inbound/web/src/main/java/dev/caskeleton/adapter/inbound/web/http/ApiHeaders.java" # source: feature-rate-limit-idempotency-contract.md scope L26 (rate limit response/log 기준) - name: X-RateLimit-Reset @@ -90,34 +101,7 @@ headers: case_style: kebab compatibility_impact: additive required_test: contract-verification:rate-limit-headers - - # source: feature-api-compatibility-deprecation-contract.md L87 - # "deprecation marker | OpenAPI deprecated: true + design record | response header optional" - - name: Deprecation - direction: outbound - type: http-date - required: false - generated_if_missing: false - mdc_key: null - envelope_meta_field: null - owner_branch: feature-api-compatibility-deprecation-contract - case_style: kebab - compatibility_impact: additive - required_test: contract-verification:deprecation-marker - - # source: feature-api-compatibility-deprecation-contract.md L87 - # "deprecation marker | OpenAPI deprecated: true + design record | response header optional" (RFC 8594 Sunset) - - name: Sunset - direction: outbound - type: http-date - required: false - generated_if_missing: false - mdc_key: null - envelope_meta_field: null - owner_branch: feature-api-compatibility-deprecation-contract - case_style: kebab - compatibility_impact: additive - required_test: contract-verification:deprecation-marker + source: "src/adapter/inbound/web/src/main/java/dev/caskeleton/adapter/inbound/web/http/ApiHeaders.java" # source: feature-distributed-tracing-contract.md L64, L85 # "propagation header는 W3C traceparent default" / "HTTP | traceparent, tracestate (W3C)" @@ -132,6 +116,7 @@ headers: case_style: kebab compatibility_impact: none required_test: contract-verification:trace-propagation + source: "src/adapter/inbound/web/src/main/java/dev/caskeleton/adapter/inbound/web/mvc/filter/WebMvcRequestIdFilter.java" # source: feature-distributed-tracing-contract.md L66, L85 # "propagation format = W3C traceparent + tracestate only. B3 propagation은 forbidden" @@ -146,12 +131,14 @@ headers: case_style: kebab compatibility_impact: none required_test: contract-verification:trace-propagation + source: "framework: Micrometer Tracing(OTel)의 W3C 전파가 traceparent와 함께 읽고 쓴다" # source: feature-operational-error-observability-foundation.md L100 # "correlation_id | inbound header X-Correlation-Id 또는 생성 | HTTP X-Correlation-Id, message header correlation_id" - name: X-Correlation-Id direction: both - type: ulid + type: uuid-or-caller-token + format: "UUID; 받는 값은 [A-Za-z0-9._-]{1,128}" required: false generated_if_missing: true mdc_key: correlation_id @@ -160,6 +147,7 @@ headers: case_style: kebab compatibility_impact: none required_test: contract-verification:envelope-headers + source: "src/adapter/inbound/web/src/main/java/dev/caskeleton/adapter/inbound/web/mvc/filter/WebMvcRequestIdFilter.java" # source: feature-security-operational-baseline.md L66 # "JWT Resource Server를 baseline security model로 둠" (Authorization 헤더의 Bearer 토큰) @@ -174,6 +162,7 @@ headers: case_style: kebab compatibility_impact: none required_test: contract-verification:jwt-resource-server + source: "framework: Spring Security bearer token resolver가 읽는다" # source: feature-security-operational-baseline.md L83-90 (AuthN/AuthZ Decision Matrix) # 401 응답 시 WWW-Authenticate (Bearer realm/error) — Spring Security JWT Resource Server의 표준 challenge 헤더 @@ -188,3 +177,293 @@ headers: case_style: kebab compatibility_impact: none required_test: contract-verification:jwt-resource-server + source: "framework: HttpHeaders.WWW_AUTHENTICATE(AuthErrorResponseWriter가 401에 쓴다)" + + # === web 응답·조건부 요청 (adapter:inbound:web) === + # 모든 응답에 no-store(CacheControlFilter). 파일 다운로드는 자기 값을 쓴다. + - name: Cache-Control + direction: outbound + type: string + required: false + generated_if_missing: true + mdc_key: null + envelope_meta_field: null + owner_branch: web-platform + case_style: kebab + compatibility_impact: none + required_test: contract-verification:envelope-headers + source: "src/adapter/inbound/web/src/main/java/dev/caskeleton/adapter/inbound/web/http/ApiHeaders.java" + - name: Vary + direction: outbound + type: comma-separated + required: false + generated_if_missing: true + mdc_key: null + envelope_meta_field: null + owner_branch: web-platform + case_style: kebab + compatibility_impact: none + required_test: contract-verification:envelope-headers + source: "src/adapter/inbound/web/src/main/java/dev/caskeleton/adapter/inbound/web/http/ApiHeaders.java" + # 405 응답(GlobalExceptionHandler). + - name: Allow + direction: outbound + type: comma-separated + required: false + generated_if_missing: false + mdc_key: null + envelope_meta_field: null + owner_branch: web-platform + case_style: kebab + compatibility_impact: none + required_test: contract-verification:envelope-headers + source: "framework: Spring HttpHeaders.setAllow — GlobalExceptionHandler가 405 응답에 쓴다(ApiHeaders.ALLOW는 쓰지 않는다)" + # 201 생성 응답(durable operation, 파일 업로드). + - name: Location + direction: outbound + type: uri + required: false + generated_if_missing: false + mdc_key: null + envelope_meta_field: null + owner_branch: web-platform + case_style: kebab + compatibility_impact: none + required_test: contract-verification:envelope-headers + source: "src/adapter/inbound/web/src/main/java/dev/caskeleton/adapter/inbound/web/http/ApiHeaders.java" + - name: ETag + direction: outbound + type: entity-tag + required: false + generated_if_missing: false + mdc_key: null + envelope_meta_field: null + owner_branch: web-platform + case_style: kebab + compatibility_impact: none + required_test: contract-verification:envelope-headers + source: "src/adapter/inbound/web/src/main/java/dev/caskeleton/adapter/inbound/web/http/ApiHeaders.java" + - name: If-Match + direction: inbound + type: entity-tag + required: false + generated_if_missing: false + mdc_key: null + envelope_meta_field: null + owner_branch: web-platform + case_style: kebab + compatibility_impact: none + required_test: contract-verification:envelope-headers + source: "service: src/adapter/inbound/web/src/main/java/dev/caskeleton/adapter/inbound/web/http/ApiHeaders.java" + - name: If-None-Match + direction: inbound + type: entity-tag + required: false + generated_if_missing: false + mdc_key: null + envelope_meta_field: null + owner_branch: web-platform + case_style: kebab + compatibility_impact: none + required_test: contract-verification:envelope-headers + source: "service: src/adapter/inbound/web/src/main/java/dev/caskeleton/adapter/inbound/web/http/ApiHeaders.java" + # Redis 세션(쿠키) 모드의 CSRF 헤더 기본 이름. + - name: X-XSRF-TOKEN + direction: inbound + type: string + required: false + generated_if_missing: false + mdc_key: null + envelope_meta_field: null + owner_branch: web-platform + case_style: kebab + compatibility_impact: none + required_test: contract-verification:envelope-headers + source: "src/adapter/inbound/web/src/main/java/dev/caskeleton/adapter/inbound/web/settings/SecuritySettings.java" + + # === 파일 서버 업로드·다운로드 (composition:fileserver-platform을 켤 때) === + # Range·If-Range·Content-Range·Accept-Ranges·Content-Disposition·Last-Modified·If-Modified-Since·If-Unmodified-Since는 + # HTTP 표준 헤더를 Spring HttpHeaders 상수로 쓴다(MvcDownloadResponseWriter, MvcConditionalRequestFactory). 아래는 이 플랫폼이 + # 이름을 정한 헤더다. + - name: Tus-Resumable + direction: both + type: string + required: false + generated_if_missing: false + mdc_key: null + envelope_meta_field: null + owner_branch: fileserver-platform + case_style: kebab + compatibility_impact: none + required_test: contract-verification:envelope-headers + source: "src/adapter/inbound/web-fileserver/src/main/java/dev/caskeleton/adapter/inbound/web/fileserver/tus/TusHeaders.java" + - name: Tus-Version + direction: outbound + type: string + required: false + generated_if_missing: false + mdc_key: null + envelope_meta_field: null + owner_branch: fileserver-platform + case_style: kebab + compatibility_impact: none + required_test: contract-verification:envelope-headers + source: "src/adapter/inbound/web-fileserver/src/main/java/dev/caskeleton/adapter/inbound/web/fileserver/tus/TusHeaders.java" + - name: Tus-Extension + direction: outbound + type: string + required: false + generated_if_missing: false + mdc_key: null + envelope_meta_field: null + owner_branch: fileserver-platform + case_style: kebab + compatibility_impact: none + required_test: contract-verification:envelope-headers + source: "src/adapter/inbound/web-fileserver/src/main/java/dev/caskeleton/adapter/inbound/web/fileserver/tus/TusHeaders.java" + - name: Tus-Max-Size + direction: outbound + type: string + required: false + generated_if_missing: false + mdc_key: null + envelope_meta_field: null + owner_branch: fileserver-platform + case_style: kebab + compatibility_impact: none + required_test: contract-verification:envelope-headers + source: "src/adapter/inbound/web-fileserver/src/main/java/dev/caskeleton/adapter/inbound/web/fileserver/tus/TusHeaders.java" + - name: Upload-Defer-Length + direction: inbound + type: string + required: false + generated_if_missing: false + mdc_key: null + envelope_meta_field: null + owner_branch: fileserver-platform + case_style: kebab + compatibility_impact: none + required_test: contract-verification:envelope-headers + source: "src/adapter/inbound/web-fileserver/src/main/java/dev/caskeleton/adapter/inbound/web/fileserver/tus/TusHeaders.java" + - name: Upload-Metadata + direction: inbound + type: string + required: false + generated_if_missing: false + mdc_key: null + envelope_meta_field: null + owner_branch: fileserver-platform + case_style: kebab + compatibility_impact: none + required_test: contract-verification:envelope-headers + source: "src/adapter/inbound/web-fileserver/src/main/java/dev/caskeleton/adapter/inbound/web/fileserver/tus/TusHeaders.java" + - name: Upload-Expires + direction: outbound + type: string + required: false + generated_if_missing: false + mdc_key: null + envelope_meta_field: null + owner_branch: fileserver-platform + case_style: kebab + compatibility_impact: none + required_test: contract-verification:envelope-headers + source: "src/adapter/inbound/web-fileserver/src/main/java/dev/caskeleton/adapter/inbound/web/fileserver/tus/TusHeaders.java" + - name: Upload-Checksum + direction: inbound + type: string + required: false + generated_if_missing: false + mdc_key: null + envelope_meta_field: null + owner_branch: fileserver-platform + case_style: kebab + compatibility_impact: none + required_test: contract-verification:envelope-headers + source: "src/adapter/inbound/web-fileserver/src/main/java/dev/caskeleton/adapter/inbound/web/fileserver/tus/TusHeaders.java" + - name: Upload-Offset + direction: both + type: string + required: false + generated_if_missing: false + mdc_key: null + envelope_meta_field: null + owner_branch: fileserver-platform + case_style: kebab + compatibility_impact: none + required_test: contract-verification:envelope-headers + source: "src/adapter/inbound/web-fileserver/src/main/java/dev/caskeleton/adapter/inbound/web/fileserver/draft12/Draft12Headers.java" + - name: Upload-Length + direction: both + type: string + required: false + generated_if_missing: false + mdc_key: null + envelope_meta_field: null + owner_branch: fileserver-platform + case_style: kebab + compatibility_impact: none + required_test: contract-verification:envelope-headers + source: "src/adapter/inbound/web-fileserver/src/main/java/dev/caskeleton/adapter/inbound/web/fileserver/draft12/Draft12Headers.java" + - name: Upload-Complete + direction: both + type: string + required: false + generated_if_missing: false + mdc_key: null + envelope_meta_field: null + owner_branch: fileserver-platform + case_style: kebab + compatibility_impact: none + required_test: contract-verification:envelope-headers + source: "src/adapter/inbound/web-fileserver/src/main/java/dev/caskeleton/adapter/inbound/web/fileserver/draft12/Draft12Headers.java" + - name: Upload-Limit + direction: outbound + type: string + required: false + generated_if_missing: false + mdc_key: null + envelope_meta_field: null + owner_branch: fileserver-platform + case_style: kebab + compatibility_impact: none + required_test: contract-verification:envelope-headers + source: "src/adapter/inbound/web-fileserver/src/main/java/dev/caskeleton/adapter/inbound/web/fileserver/draft12/Draft12Headers.java" + # raw·draft-12 업로드의 원래 파일 이름. + - name: X-Filename + direction: inbound + type: string + required: false + generated_if_missing: false + mdc_key: null + envelope_meta_field: null + owner_branch: fileserver-platform + case_style: kebab + compatibility_impact: none + required_test: contract-verification:envelope-headers + source: "src/adapter/inbound/web-fileserver/src/main/java/dev/caskeleton/adapter/inbound/web/fileserver/draft12/Draft12UploadController.java" + - name: X-Content-Sha256 + direction: inbound + type: string + required: false + generated_if_missing: false + mdc_key: null + envelope_meta_field: null + owner_branch: fileserver-platform + case_style: kebab + compatibility_impact: none + required_test: contract-verification:envelope-headers + source: "src/adapter/inbound/web-fileserver/src/main/java/dev/caskeleton/adapter/inbound/web/fileserver/mapper/RawUploadRequestMapper.java" + # 앞단 nginx에만 주는 내부 헤더다. nginx가 지우고 client에는 가지 않는다. + - name: X-Accel-Redirect + direction: outbound + type: uri + required: false + generated_if_missing: false + mdc_key: null + envelope_meta_field: null + owner_branch: fileserver-platform + case_style: kebab + compatibility_impact: none + required_test: contract-verification:envelope-headers + source: "src/adapter/inbound/web-fileserver/src/main/java/dev/caskeleton/adapter/inbound/web/fileserver/nginx/NginxDownloadStrategy.java" diff --git a/docs/registries/mdc-keys.yaml b/docs/registries/mdc-keys.yaml index fcf391c2..0c866267 100644 --- a/docs/registries/mdc-keys.yaml +++ b/docs/registries/mdc-keys.yaml @@ -1,23 +1,38 @@ # Registry: MDC / Log Keys # Schema owner: feature-contract-registry-governance # MDC SSOT: feature-operational-error-observability-foundation -# Last updated: 2026-05-22 +# Last updated: 2026-11-13 # # 표기 규칙 # - MDC key 이름: snake_case (foundation L93 "snake_case 강제. camelCase / dot.case 금지.") # - cardinality_safe_for_metric=true 인 key만 metric tag로 사용 가능 -# - foundation L93-102 표 "MDC Key Standard (final)"의 key들이 핵심 정본이다 +# - 행은 코드가 실제로 MDC에 넣는 key다. 코드의 정본은 `MdcKeys`(shared-contract)이고, 이 파일의 key 집합과 +# `MdcKeys` 상수 집합이 같은지 `StructuredLogFieldContractTest`가 본다. `written_by`가 넣는 곳이다. +# - 구조화 로그 칸(`error.code`, `capability`, `operation`, `warning.kind` 등, dot.case 포함)은 MDC key가 아니라 +# logstash `kv(...)` 인자다. 그 칸의 정본은 `docs/observability/log-fields.md`다. method·status·duration_ms·uri_template은 +# 요청 로그 줄의 메시지 글자다(`RequestLoggingFilter`). +# +# 전송마다 넣는 key (전송마다 다르다. 같게 맞추지 않고 여기 적는다) +# | 전송 | request_id | correlation_id | trace_id·span_id | +# | --- | --- | --- | --- | +# | web(`WebMvcRequestIdFilter`) | 새 UUID. `app.web.mvc.trust-inbound-request-id=true`일 때만 caller의 `X-Request-Id`(`[A-Za-z0-9._-]{1,128}`) | caller의 `X-Correlation-Id`(같은 문법), 없거나 틀리면 새 UUID | 활성 span, 없으면 caller `traceparent`, 없으면 새로 만든다 | +# | gRPC(`GrpcCorrelationInterceptor`) | 호출마다 새 UUID(caller 값을 받지 않는다) | 넣지 않는다(연결·세션 개념이 없다) | 유효한 `traceparent` metadata가 있을 때만 | +# | STOMP(`WebSocketCorrelationInterceptor`) | frame마다 새 UUID | STOMP session id(한 연결의 frame을 잇는다) | 넣지 않는다 | +# | GraphQL(HTTP) | web filter가 넣은 값을 그대로 쓴다 | 같음 | 같음 | +# - user_principal은 web 요청 로그 줄을 쓰는 동안만 있다(HMAC 가명). 비동기 작업은 `AsyncContextTaskDecorator`가 +# 호출한 쪽 MDC 전체를 옮긴다. mdc_keys: # source: feature-operational-error-observability-foundation.md L97 - # "request_id | inbound filter (생성 또는 X-Request-Id 헤더) | response header X-Request-Id" - key: request_id - type: ulid + type: uuid-or-caller-token + format: "UUID; web은 trust-inbound-request-id=true일 때 [A-Za-z0-9._-]{1,128}" source: inbound_filter + written_by: [WebMvcRequestIdFilter, GrpcCorrelationInterceptor, WebSocketCorrelationInterceptor] required_in: [request, dependency, security, application] http_header_mapping: X-Request-Id envelope_field: meta.requestId - propagation: [http, async, message] + propagation: [http, async] owner_branch: feature-operational-error-observability-foundation cardinality_safe_for_metric: false case_style: snake @@ -25,26 +40,26 @@ mdc_keys: required_test: contract-verification:log-mdc-keys # source: feature-operational-error-observability-foundation.md L98 - # "trace_id | Micrometer Tracing | W3C traceparent header" - key: trace_id type: string + format: "W3C trace-id, 32 lowercase hex" source: observation_context + written_by: [WebMvcRequestIdFilter, GrpcCorrelationInterceptor] required_in: [request, dependency, application] http_header_mapping: traceparent envelope_field: meta.traceId - propagation: [http, async, message] + propagation: [http, async] owner_branch: feature-operational-error-observability-foundation cardinality_safe_for_metric: false case_style: snake compatibility_impact: none required_test: contract-verification:log-mdc-keys - # source: feature-operational-error-observability-foundation.md L99 - # "span_id | Micrometer Tracing | W3C traceparent" - # NOTE: background-job-async-contract L71 "span_id는 Micrometer Observation context에서 자동 전파(MDC explicit copy 불필요)" - key: span_id type: string + format: "W3C parent-id, 16 lowercase hex" source: observation_context + written_by: [WebMvcRequestIdFilter, GrpcCorrelationInterceptor] required_in: [request, dependency] http_header_mapping: traceparent envelope_field: null @@ -56,10 +71,11 @@ mdc_keys: required_test: contract-verification:log-mdc-keys # source: feature-operational-error-observability-foundation.md L100 - # "correlation_id | inbound header X-Correlation-Id 또는 생성 | HTTP X-Correlation-Id, message header correlation_id" - key: correlation_id - type: ulid + type: uuid-or-caller-token + format: "web: caller X-Correlation-Id [A-Za-z0-9._-]{1,128} 또는 UUID; STOMP: session id" source: inbound_filter + written_by: [WebMvcRequestIdFilter, WebSocketCorrelationInterceptor] required_in: [request, dependency, application] http_header_mapping: X-Correlation-Id envelope_field: meta.correlationId @@ -70,28 +86,12 @@ mdc_keys: compatibility_impact: none required_test: contract-verification:log-mdc-keys - # source: feature-operational-error-observability-foundation.md L101 + feature-tenant-context-policy.md L70 - # "tenant_id | tenant context (활성 시) | downstream HTTP X-Tenant-Id (with allowlist)" / - # "tenant ID format = opaque ULID (26 chars Crockford base32)" - # NOTE: tenant L73 "tenant_id ULID 원본은 metric tag에 직접 사용 금지" - - key: tenant_id - type: ulid - source: security_context - required_in: [request, dependency, security, audit] - http_header_mapping: X-Tenant-Id - envelope_field: null - propagation: [http, async, message] - owner_branch: feature-tenant-context-policy - cardinality_safe_for_metric: false - case_style: snake - compatibility_impact: additive - required_test: contract-verification:tenant-leakage - # source: feature-operational-error-observability-foundation.md L102 - # "user_principal | security context (pseudonymized only) | log only, headers forbidden" - key: user_principal type: string + format: "HMAC 가명(PrivacySettings)" source: security_context + written_by: [RequestLoggingFilter] required_in: [security, audit] http_header_mapping: null envelope_field: null @@ -101,193 +101,3 @@ mdc_keys: case_style: snake compatibility_impact: none required_test: contract-verification:log-mdc-keys - - # ── 로그 종류별 확장 (log-management-contract L101-109 "Log Type별 필수 필드") ── - - # source: feature-log-management-contract.md L105 "request | request_id, trace_id, method, uri_template, status, duration_ms" - # 참고: application-port-usecase-contract / 업무 쪽 operation 식별자 (uri_template과 따로 application이 정한다) - - key: operation - type: string - source: application_set - required_in: [application, dependency] - http_header_mapping: null - envelope_field: null - propagation: [none] - owner_branch: feature-log-management-contract - cardinality_safe_for_metric: true - case_style: snake - compatibility_impact: none - required_test: contract-verification:log-fields - - # source: feature-log-management-contract.md L105 "request | request_id, trace_id, method, uri_template, status, duration_ms" - - key: method - type: string - source: inbound_filter - required_in: [request] - http_header_mapping: null - envelope_field: null - propagation: [none] - owner_branch: feature-log-management-contract - cardinality_safe_for_metric: true - case_style: snake - compatibility_impact: none - required_test: contract-verification:log-fields - - # source: feature-log-management-contract.md L105 "request | request_id, trace_id, method, uri_template, status, duration_ms" - # 참고: metrics L86 "status_code | 7 (1xx-5xx + ok/other)" — 값 개수가 정해져 있다 - - key: status - type: numeric - source: inbound_filter - required_in: [request] - http_header_mapping: null - envelope_field: null - propagation: [none] - owner_branch: feature-log-management-contract - cardinality_safe_for_metric: true - case_style: snake - compatibility_impact: none - required_test: contract-verification:log-fields - - # source: feature-log-management-contract.md L105-106 "request | ... duration_ms" / "dependency | ... duration_ms" - - key: duration_ms - type: numeric - source: application_set - required_in: [request, dependency] - http_header_mapping: null - envelope_field: null - propagation: [none] - owner_branch: feature-log-management-contract - cardinality_safe_for_metric: false - case_style: snake - compatibility_impact: none - required_test: contract-verification:log-fields - - # source: feature-log-management-contract.md L106 "dependency | dependency_name, dependency_type, duration_ms, outcome, error_code" - # 참고: metrics L88 "dependency_name | 50" — 값 개수가 정해져 있다 - - key: dependency_name - type: string - source: application_set - required_in: [dependency] - http_header_mapping: null - envelope_field: null - propagation: [none] - owner_branch: feature-log-management-contract - cardinality_safe_for_metric: true - case_style: snake - compatibility_impact: none - required_test: contract-verification:log-fields - - # source: feature-log-management-contract.md L106 "dependency | dependency_name, dependency_type, ..." - - key: dependency_type - type: string - source: application_set - required_in: [dependency] - http_header_mapping: null - envelope_field: null - propagation: [none] - owner_branch: feature-log-management-contract - cardinality_safe_for_metric: true - case_style: snake - compatibility_impact: none - required_test: contract-verification:log-fields - - # source: feature-log-management-contract.md L106 + metrics L91 "outcome (resilience4j) | 5 (SUCCESS/FAILURE/CIRCUIT_OPEN/TIMEOUT/REJECTED)" - - key: outcome - type: string - source: application_set - required_in: [dependency] - http_header_mapping: null - envelope_field: null - propagation: [none] - owner_branch: feature-log-management-contract - cardinality_safe_for_metric: true - case_style: snake - compatibility_impact: none - required_test: contract-verification:log-fields - - # source: feature-log-management-contract.md L106 "dependency | ... error_code (실패 시)" - # 참고: metrics L89 "error_code | 100 — error registry row 상한과 정합" — 값 개수가 정해져 있다 - - key: error_code - type: string - source: application_set - required_in: [dependency] - http_header_mapping: null - envelope_field: null - propagation: [none] - owner_branch: feature-log-management-contract - cardinality_safe_for_metric: true - case_style: snake - compatibility_impact: none - required_test: contract-verification:log-fields - - # source: feature-log-management-contract.md L107 - # "security | event_type, user_principal (pseudonymized), source_ip (anonymized — last octet zeroed)" - - key: event_type - type: string - source: application_set - required_in: [security, audit] - http_header_mapping: null - envelope_field: null - propagation: [none] - owner_branch: feature-log-management-contract - cardinality_safe_for_metric: true - case_style: snake - compatibility_impact: none - required_test: contract-verification:log-fields - - # source: feature-log-management-contract.md L107 "security | ... source_ip (anonymized — last octet zeroed)" - # NOTE: metrics L93 "high-cardinality 금지 tag: ... ip_address" - - key: source_ip_anon - type: string - source: inbound_filter - required_in: [security] - http_header_mapping: null - envelope_field: null - propagation: [none] - owner_branch: feature-log-management-contract - cardinality_safe_for_metric: false - case_style: snake - compatibility_impact: none - required_test: contract-verification:log-fields - - # source: feature-log-management-contract.md L108 "audit | actor, action, target, before_hash, after_hash, occurred_at" - - key: actor - type: string - source: security_context - required_in: [audit] - http_header_mapping: null - envelope_field: null - propagation: [none] - owner_branch: feature-log-management-contract - cardinality_safe_for_metric: false - case_style: snake - compatibility_impact: none - required_test: contract-verification:log-fields - - # source: feature-log-management-contract.md L108 "audit | actor, action, target, ..." - - key: action - type: string - source: application_set - required_in: [audit] - http_header_mapping: null - envelope_field: null - propagation: [none] - owner_branch: feature-log-management-contract - cardinality_safe_for_metric: true - case_style: snake - compatibility_impact: none - required_test: contract-verification:log-fields - - # source: feature-log-management-contract.md L108 "audit | actor, action, target, ..." - - key: target - type: string - source: application_set - required_in: [audit] - http_header_mapping: null - envelope_field: null - propagation: [none] - owner_branch: feature-log-management-contract - cardinality_safe_for_metric: false - case_style: snake - compatibility_impact: none - required_test: contract-verification:log-fields diff --git a/docs/registries/metrics.yaml b/docs/registries/metrics.yaml index 06f17143..4722c3ba 100644 --- a/docs/registries/metrics.yaml +++ b/docs/registries/metrics.yaml @@ -4,7 +4,17 @@ # Last updated: 2026-05-22 # # 읽는 법 -# - 이름: Micrometer dot.case에 단위 접미사(.seconds | .bytes | .total)를 붙인다. +# - 이름: Micrometer dot.case다. 단위는 `unit` 칸(seconds | bytes | total)이 말하고, 새 이름에는 단위를 다시 쓰지 +# 않는다. Prometheus로 내보낼 때 Micrometer가 timer에 _seconds, counter에 _total을 붙인다(gauge에는 base unit이 +# 있을 때만 붙고, unit이 total인 count gauge에는 붙지 않는다). 단위로 끝나는 옛 이름 +# (outbox.publisher.published.total 같은 .total, redis.lane.acquire.wait.seconds, http.client.request.bytes· +# response.bytes, fileserver.transfer.bytes)은 이름을 바꾸면 대시보드가 깨지므로 그대로 둔다. +# - 행은 코드가 실제로 등록하는 meter다. `source`가 그 출처다: 이름 글자를 가진 main 파일 또는 `framework: …` +# (Spring Boot·Micrometer·HikariCP·Spring for GraphQL이 내는 것). 이름이 상수로만 있으면 `constant`와 그 상수를 쓰는 +# `registered_by` 파일도 적는다. MetricsRegistryCodeAlignmentTest가 행 → 코드(상수면 그 상수를 쓰는 곳까지), +# 그리고 글자·상수 이름으로 등록한 meter → 행의 두 방향을 대조한다. capability를 켠 서비스에서만 나오는 meter도 +# 행이 있다(그 행의 절 제목이 말한다). `emitted_when`이 있는 행은 템플릿 코드가 부르지 않는 공개 메서드를 서비스가 +# 부를 때만 생긴다. # - tag 값 개수(cardinality) 상한의 정본은 feature-metrics-alerting-contract의 "Cardinality Bounds" 표다. # - 값 개수가 많은 tag는 어디서도 쓰지 않는다: user_id, request_id, raw_url, raw_query, # raw_header_value, ip_address. 어떤 행에도 나오면 안 된다. @@ -14,10 +24,9 @@ # 기간(5m, 1h)은 원문 그대로 두고, 테스트는 키(p1·p2·p3)와 값이 비어 있지 않은지만 본다. metrics: - # === HTTP 서버 (inbound) === - # source: feature-metrics-alerting-contract — Metric/Alert Defaults - # "HTTP metric | http.server.requests with method/status/uri-template | raw URL or user id tag" + # Spring MVC가 요청마다 기록하는 timer다. tag는 Spring Framework 기본 observation convention 그대로다. + # 지연(p99) 경보도 이 timer로 건다. - name: http.server.requests type: timer unit: seconds @@ -26,155 +35,82 @@ metrics: cardinality_limit: 8 allowed_values: [GET, POST, PUT, PATCH, DELETE, HEAD, OPTIONS, OTHER] - name: status - cardinality_limit: 7 - allowed_values: [1xx, 2xx, 3xx, 4xx, 5xx, ok, other] - - name: uri_template + # 숫자 HTTP status(200, 404, …). 서비스가 실제로 답하는 코드 수가 상한이다. + cardinality_limit: 40 + - name: uri + # 맞는 route의 template(/v1/orders/{id}), 없으면 NOT_FOUND·REDIRECTION·UNKNOWN·root. cardinality_limit: 200 validation: must_be_template_not_raw_uri + - name: outcome + cardinality_limit: 6 + allowed_values: [INFORMATIONAL, SUCCESS, REDIRECTION, CLIENT_ERROR, SERVER_ERROR, UNKNOWN] + - name: exception + # 예외 클래스의 단순 이름, 없으면 none. + cardinality_limit: 50 + - name: error + cardinality_limit: 50 percentiles: [0.5, 0.9, 0.95, 0.99] histogram_buckets: slo_driven alert_severity_thresholds: - p1: "error_rate > 5%가 5m 동안, 또는 > 10%가 1m 동안" - p2: "error_rate > 1%가 10m 동안" - p3: "error_rate > 0.1%가 1h 동안" + p1: "error_rate > 5%가 5m 동안(또는 > 10%가 1m), 또는 p99 > 5s가 5m 동안" + p2: "error_rate > 1%가 10m 동안, 또는 p99 > 1s가 10m 동안" + p3: "error_rate > 0.1%가 1h 동안, 또는 p99 > 500ms가 30m 동안" owner_branch: feature-metrics-alerting-contract log_field_mapping: [method, status, uri_template] compatibility_impact: additive required_test: contract-verification:metrics-cardinality - - # source: feature-metrics-alerting-contract — P1/P2/P3 정량 기준 (HTTP latency p99) - # "P1: p99 > 5s 5분 / P2: p99 > 1s 10분 / P3: p99 > 500ms 30분" - - name: http.server.requests.latency - type: timer - unit: seconds - tags: - - name: method - cardinality_limit: 8 - allowed_values: [GET, POST, PUT, PATCH, DELETE, HEAD, OPTIONS, OTHER] - - name: uri_template - cardinality_limit: 200 - validation: must_be_template_not_raw_uri - percentiles: [0.5, 0.9, 0.95, 0.99] - histogram_buckets: slo_driven - alert_severity_thresholds: - p1: "p99 > 5s가 5m 동안" - p2: "p99 > 1s가 10m 동안" - p3: "p99 > 500ms가 30m 동안" - owner_branch: feature-metrics-alerting-contract - log_field_mapping: [method, uri_template, duration_ms] - compatibility_impact: additive - required_test: contract-verification:metrics-cardinality - - # === HTTP client (outbound 의존 대상) === - # source: feature-metrics-alerting-contract — Metric/Alert Defaults - # "dependency metric | dependency.client.requests with dependency.name/type/outcome | endpoint with secret tag" - # source: feature-outbound-http-client-baseline — "circuit breaker metric은 dependency.name, dependency.type, outcome까지만 tag로 허용" - - name: dependency.client.requests - type: timer - unit: seconds - tags: - - name: dependency_name - cardinality_limit: 50 - - name: dependency_type - cardinality_limit: 10 - allowed_values: [http, grpc, db, cache, queue, broker, other] - - name: outcome - cardinality_limit: 5 - allowed_values: [SUCCESS, FAILURE, CIRCUIT_OPEN, TIMEOUT, REJECTED] - percentiles: [0.5, 0.9, 0.95, 0.99] - histogram_buckets: slo_driven - alert_severity_thresholds: - p1: "필수 의존 대상이 2m 동안 응답하지 않음" - p2: "선택 의존 대상이 5m 동안 저하됨" - p3: "급증 경보(기준선의 10배)" - owner_branch: feature-metrics-alerting-contract - log_field_mapping: [dependency_name, dependency_type, outcome, duration_ms] - compatibility_impact: additive - required_test: contract-verification:metrics-cardinality - - # source: feature-outbound-http-client-baseline — decisions - # "retry/circuit breaker 기본 라이브러리는 Resilience4j" - # source: feature-metrics-alerting-contract — "retry/CB minimum: resilience4j.retry.calls{outcome}" - - name: resilience4j.retry.calls - type: counter - unit: total - tags: - - name: name - cardinality_limit: 50 - - name: outcome - cardinality_limit: 5 - allowed_values: [SUCCESS, FAILURE, CIRCUIT_OPEN, TIMEOUT, REJECTED] - percentiles: null - histogram_buckets: null - alert_severity_thresholds: - p2: "재시도 소진 비율 > 1%가 10m 동안" - owner_branch: feature-outbound-http-client-baseline - log_field_mapping: [dependency_name, outcome, retry_attempt] - compatibility_impact: additive - required_test: contract-verification:metrics-cardinality - - # source: feature-metrics-alerting-contract — "resilience4j.circuitbreaker.state" - - name: resilience4j.circuitbreaker.state - type: gauge - unit: total - tags: - - name: name - cardinality_limit: 50 - - name: state - cardinality_limit: 6 - allowed_values: [CLOSED, OPEN, HALF_OPEN, DISABLED, FORCED_OPEN, METRICS_ONLY] - percentiles: null - histogram_buckets: null - alert_severity_thresholds: - p1: "필수 의존 대상의 state == OPEN이 2m 동안" - p2: "선택 의존 대상의 state == OPEN이 5m 동안" - owner_branch: feature-outbound-http-client-baseline - log_field_mapping: [dependency_name] - compatibility_impact: additive - required_test: contract-verification:metrics-cardinality - - # source: feature-metrics-alerting-contract — "resilience4j.circuitbreaker.calls{outcome}" - - name: resilience4j.circuitbreaker.calls - type: timer - unit: seconds - tags: - - name: name - cardinality_limit: 50 - - name: outcome - cardinality_limit: 5 - allowed_values: [SUCCESS, FAILURE, CIRCUIT_OPEN, TIMEOUT, REJECTED] - percentiles: [0.5, 0.9, 0.95, 0.99] - histogram_buckets: slo_driven - alert_severity_thresholds: - p2: "CIRCUIT_OPEN 비율 > 1%가 10m 동안" - owner_branch: feature-outbound-http-client-baseline - log_field_mapping: [dependency_name, outcome, duration_ms] - compatibility_impact: additive - required_test: contract-verification:metrics-cardinality - - # === DB connection pool === - # source: feature-persistence-failure-baseline — Hikari Alert Threshold - # "pool wait p99 > 100ms 5분 지속 → P2 / pool exhaustion (active = max) > 1분 → P1" - # source: feature-metrics-alerting-contract — "hikaricp.connections.acquire{outcome='timeout'} p99 > 100ms" + source: "framework: Spring MVC 요청 observation(Spring Framework 기본 convention)" + # === DB connection pool (HikariCP, persistence-jpa를 켤 때) === + # HikariCP의 Micrometer tracker가 내는 이름 그대로다. acquire timer에는 pool tag 하나뿐이다. 획득 timeout은 + # 따로 센다(hikaricp.connections.timeout). - name: hikaricp.connections.acquire type: timer unit: seconds tags: - name: pool cardinality_limit: 5 - - name: outcome - cardinality_limit: 3 - allowed_values: [SUCCESS, TIMEOUT, FAILURE] - percentiles: [0.5, 0.9, 0.95, 0.99] - histogram_buckets: slo_driven + # histogram을 켜는 것이 없다(MetricsDistributionMeterFilter 대상이 아니다). 경보는 평균(sum/count)으로 건다. + percentiles: null + histogram_buckets: null alert_severity_thresholds: - p1: "pool 고갈(active == max)이 1m 동안" - p2: "acquire p99 > 100ms가 5m 동안" + p2: "평균 acquire 시간(sum/count) > 100ms가 5m 동안" owner_branch: feature-persistence-failure-baseline - log_field_mapping: [pool, outcome] + log_field_mapping: [pool] compatibility_impact: additive required_test: contract-verification:metrics-cardinality + source: "framework: HikariCP Micrometer tracker(persistence-jpa를 켤 때)" + - name: hikaricp.connections.timeout + type: counter + unit: total + tags: + - name: pool + cardinality_limit: 5 + percentiles: null + histogram_buckets: null + alert_severity_thresholds: + p1: "timeout 증가 > 0이 1m 동안(pool 고갈: active == max)" + owner_branch: feature-persistence-failure-baseline + log_field_mapping: [pool] + compatibility_impact: additive + required_test: contract-verification:metrics-cardinality + source: "framework: HikariCP Micrometer tracker(persistence-jpa를 켤 때)" + + - name: hikaricp.connections.pending + type: gauge + unit: total + tags: + - name: pool + cardinality_limit: 5 + percentiles: null + histogram_buckets: null + alert_severity_thresholds: + p2: "pending > 0이 5m 동안" + owner_branch: feature-persistence-failure-baseline + log_field_mapping: [pool] + compatibility_impact: additive + required_test: contract-verification:metrics-cardinality + source: "framework: HikariCP Micrometer tracker(persistence-jpa를 켤 때)" # source: feature-persistence-failure-baseline — In scope "Hikari metric 노출 기준" + Hikari Alert Threshold - name: hikaricp.connections.usage type: timer @@ -182,14 +118,15 @@ metrics: tags: - name: pool cardinality_limit: 5 - percentiles: [0.5, 0.9, 0.95, 0.99] - histogram_buckets: slo_driven + percentiles: null + histogram_buckets: null alert_severity_thresholds: - p2: "usage p99가 10m 넘게 높음" + p2: "평균 usage 시간(sum/count) > 1s가 10m 동안(connection을 오래 쥔 트랜잭션)" owner_branch: feature-persistence-failure-baseline log_field_mapping: [pool] compatibility_impact: additive required_test: contract-verification:metrics-cardinality + source: "framework: HikariCP Micrometer tracker(persistence-jpa를 켤 때)" # source: feature-persistence-failure-baseline — In scope "Hikari metric 노출 기준" - name: hikaricp.connections.active @@ -206,26 +143,7 @@ metrics: log_field_mapping: [pool] compatibility_impact: additive required_test: contract-verification:metrics-cardinality - - # source: feature-metrics-alerting-contract — Histogram Buckets/Percentile "DB query: same" - - name: db.query.duration - type: timer - unit: seconds - tags: - - name: operation - cardinality_limit: 20 - allowed_values: [select, insert, update, delete, batch, ddl, other] - - name: outcome - cardinality_limit: 3 - allowed_values: [SUCCESS, FAILURE, TIMEOUT] - percentiles: [0.5, 0.9, 0.95, 0.99] - histogram_buckets: slo_driven - alert_severity_thresholds: - p2: "p99 > 1s가 10m 동안" - owner_branch: feature-metrics-alerting-contract - log_field_mapping: [operation, outcome, duration_ms] - compatibility_impact: additive - required_test: contract-verification:metrics-cardinality + source: "framework: HikariCP Micrometer tracker(persistence-jpa를 켤 때)" # === 백그라운드 작업 / 비동기 executor === # source: feature-background-job-async-contract — Decisionized Work Items "saturation policy" @@ -245,6 +163,7 @@ metrics: log_field_mapping: [executor_name] compatibility_impact: additive required_test: contract-verification:metrics-cardinality + source: "src/bootstrap/app/src/main/java/dev/caskeleton/bootstrap/async/BackgroundJobMetrics.java" # source: feature-background-job-async-contract — Decisionized Work Items "saturation | bounded executor + rejection log" - name: executor.rejected.total @@ -264,6 +183,7 @@ metrics: log_field_mapping: [executor_name, policy] compatibility_impact: additive required_test: contract-verification:metrics-cardinality + source: "src/bootstrap/app/src/main/java/dev/caskeleton/bootstrap/async/BackgroundJobMetrics.java" # source: feature-background-job-async-contract — Decisionized Work Items "retry/DLQ | exp backoff jitter, max 3, DLQ exhausted" - name: job.retry.total @@ -283,6 +203,8 @@ metrics: log_field_mapping: [job_name, outcome, retry_attempt] compatibility_impact: additive required_test: contract-verification:metrics-cardinality + source: "src/bootstrap/app/src/main/java/dev/caskeleton/bootstrap/async/BackgroundJobMetrics.java" + emitted_when: "서비스가 BackgroundJobMetrics.recordRetryOutcome을 부를 때. 템플릿 코드는 부르지 않는다" # source: feature-background-job-async-contract — "DLQ after exhausted attempts" - name: job.dlq.total @@ -299,11 +221,14 @@ metrics: log_field_mapping: [job_name] compatibility_impact: additive required_test: contract-verification:metrics-cardinality + source: "src/bootstrap/app/src/main/java/dev/caskeleton/bootstrap/async/BackgroundJobMetrics.java" + emitted_when: "서비스가 BackgroundJobMetrics.recordDeadLetter를 부를 때. 템플릿 코드는 부르지 않는다" # === Distributed lock === # source: feature-distributed-lock-contract D7 — "metric lock.acquisition (tag: outcome = - # acquired/timeout/error) — 신규 제안" / D5 — try-lock + 유한 waitTime + lease(TTL). 분산 - # 상호배제(distributedLockProvider) 획득 시도 결과를 센다. key 는 tag 로 넣지 않는다 + # acquired/timeout/error) — 신규 제안" / D5 — try-lock + 유한 waitTime + lease(TTL). lock 획득 + # 시도 결과를 센다: 분산 상호배제(distributedLockProvider)와 단일 인스턴스 in-process lock 모두 + # MeteredDistributedLock으로 감싼다. key 는 tag 로 넣지 않는다 # (무한 cardinality — 위 전역 금지 규칙). timeout outcome 은 LOCK_ACQUISITION_TIMEOUT 발생과 1:1. - name: lock.acquisition type: counter @@ -320,6 +245,7 @@ metrics: log_field_mapping: [outcome] compatibility_impact: additive required_test: contract-verification:metrics-cardinality + source: "src/adapter/outbound/persistence-jpa/src/main/java/dev/caskeleton/adapter/outbound/persistence/lock/MeteredDistributedLock.java" # source: feature-distributed-lock-contract §Edge / D5 (SI-LOCK-C5) — "lease 만료 후 unlock → # ConcurrentModificationException — 삼킴 금지, 로그+metric 후 정상 흐름 복귀". lease가 이미 만료된 뒤의 @@ -338,6 +264,7 @@ metrics: log_field_mapping: [] compatibility_impact: additive required_test: contract-verification:metrics-cardinality + source: "src/adapter/outbound/persistence-jpa/src/main/java/dev/caskeleton/adapter/outbound/persistence/lock/MeteredDistributedLock.java" # === Outbox 발행기 === # source: feature-domain-event-outbox-contract — Outbox Defaults @@ -362,6 +289,7 @@ metrics: log_field_mapping: [event_type, outcome, event_id] compatibility_impact: additive required_test: contract-verification:metrics-cardinality + source: "src/bootstrap/outbox-relay/src/main/java/dev/caskeleton/composition/outbox/OutboxMetrics.java" # source: feature-domain-event-outbox-contract — 행 상태 enum PENDING/IN_FLIGHT/PUBLISHED/FAILED/DEAD - name: outbox.publisher.lag @@ -379,6 +307,7 @@ metrics: log_field_mapping: [event_type] compatibility_impact: additive required_test: contract-verification:metrics-cardinality + source: "src/bootstrap/outbox-relay/src/main/java/dev/caskeleton/composition/outbox/OutboxMetrics.java" # source: feature-domain-event-outbox-contract — row status enum + Outbox Defaults attemptCount # PARKED: 끝없이 재시도되던 행을 운영자가 붙잡아 둠 (re-review 2026-11-03 RR-05-03). @@ -400,6 +329,7 @@ metrics: log_field_mapping: [status] compatibility_impact: additive required_test: contract-verification:metrics-cardinality + source: "src/bootstrap/outbox-relay/src/main/java/dev/caskeleton/composition/outbox/OutboxMetrics.java" # source: re-review 2026-11-03 RR-05-03 — 지금의 거절과 답이 없는 발행은 끝없이 재시도된다. 이 둘은 # 10번 이상 미뤄진 FAILED 행을 세고(OutboxMetrics LONG_RETRY_DEFERRED_CLAIMS, publish-failed runbook §4.1 @@ -416,6 +346,7 @@ metrics: log_field_mapping: [] compatibility_impact: additive required_test: contract-verification:metrics-cardinality + source: "src/bootstrap/outbox-relay/src/main/java/dev/caskeleton/composition/outbox/OutboxMetrics.java" - name: outbox.retry.long.age type: gauge @@ -429,113 +360,7 @@ metrics: log_field_mapping: [] compatibility_impact: additive required_test: contract-verification:metrics-cardinality - - # === 캐시 === - # source: feature-cache-consistency-contract — Decisionized Work Items "cache pattern | cache-aside default" - - name: cache.gets.total - type: counter - unit: total - tags: - - name: cache_name - cardinality_limit: 50 - - name: result - cardinality_limit: 3 - allowed_values: [hit, miss, error] - percentiles: null - histogram_buckets: null - alert_severity_thresholds: - p3: "hit_ratio < 기준선의 0.5배가 1h 동안" - owner_branch: feature-cache-consistency-contract - log_field_mapping: [cache_name, result] - compatibility_impact: additive - required_test: contract-verification:metrics-cardinality - - # source: feature-cache-consistency-contract — "invalidation = after-commit only", "invalidation 실패가 조용히 무시되면 실패" - - name: cache.invalidations.total - type: counter - unit: total - tags: - - name: cache_name - cardinality_limit: 50 - - name: outcome - cardinality_limit: 3 - allowed_values: [SUCCESS, FAILURE, SKIPPED] - percentiles: null - histogram_buckets: null - alert_severity_thresholds: - p2: "FAILURE 비율 > 0이 5m 동안" - owner_branch: feature-cache-consistency-contract - log_field_mapping: [cache_name, outcome] - compatibility_impact: additive - required_test: contract-verification:metrics-cardinality - - # source: redis-production-capability — 크기가 정해진 캐시 전용 L1 (선택) - - name: cache.local.requests.total - type: counter - unit: total - tags: - - name: cache_name - cardinality_limit: 50 - - name: result - cardinality_limit: 4 - allowed_values: [hit, miss, error, bypass] - percentiles: null - histogram_buckets: null - alert_severity_thresholds: - p3: "bypass 또는 error 비율이 기준선보다 높은 상태가 15m 동안" - owner_branch: redis-production-capability - log_field_mapping: [cache_name, result] - compatibility_impact: additive - required_test: contract-verification:metrics-cardinality - - # source: redis-production-capability — 로컬 stale 나이 상한 - - name: cache.local.entry.age.seconds - type: timer - unit: seconds - tags: - - name: cache_name - cardinality_limit: 50 - percentiles: [0.5, 0.9, 0.95, 0.99] - histogram_buckets: slo_driven - alert_severity_thresholds: - p3: "p99가 설정한 로컬 TTL에 가까운 상태가 30m 동안" - owner_branch: redis-production-capability - log_field_mapping: [cache_name] - compatibility_impact: additive - required_test: contract-verification:metrics-cardinality - - # source: redis-production-capability — 크기가 정해진 무효화와 세대 대조(reconciliation) - - name: cache.local.maintenance.total - type: counter - unit: total - tags: - - name: cache_name - cardinality_limit: 50 - - name: event - cardinality_limit: 14 - allowed_values: - - evict_cardinality - - evict_weight - - evict_ttl - - evict_invalidation - - flush_invalidation - - reconcile_generation_changed - - reconcile_unchanged - - reconcile_error - - subscriber_disconnected - - subscriber_overflow - - subscriber_malformed - - subscriber_publish_success - - subscriber_publish_error - - other - percentiles: null - histogram_buckets: null - alert_severity_thresholds: - p2: "reconcile_error, subscriber_overflow, 또는 끊김이 5m 동안 계속" - owner_branch: redis-production-capability - log_field_mapping: [cache_name, event] - compatibility_impact: additive - required_test: contract-verification:metrics-cardinality + source: "src/bootstrap/outbox-relay/src/main/java/dev/caskeleton/composition/outbox/OutboxMetrics.java" # === Redis 연결 lane === # source: review 2026-11-03 RR-04-02 — 캐시의 조건부 쓰기는 자기 lane(CACHE_SCRIPT)을 받았다. 캐시 채우기가 @@ -565,6 +390,7 @@ metrics: log_field_mapping: [lane, outcome] compatibility_impact: additive required_test: contract-verification:metrics-cardinality + source: "src/adapter/outbound/redis-runtime/src/main/java/dev/caskeleton/adapter/outbound/cache/redis/runtime/config/RedisLaneMetrics.java" - name: redis.lane.acquire.refused.total type: counter @@ -586,6 +412,7 @@ metrics: log_field_mapping: [lane, reason] compatibility_impact: additive required_test: contract-verification:metrics-cardinality + source: "src/adapter/outbound/redis-runtime/src/main/java/dev/caskeleton/adapter/outbound/cache/redis/runtime/config/RedisLaneMetrics.java" - name: redis.lane.connections.opened.total type: counter @@ -603,6 +430,7 @@ metrics: log_field_mapping: [lane] compatibility_impact: additive required_test: contract-verification:metrics-cardinality + source: "src/adapter/outbound/redis-runtime/src/main/java/dev/caskeleton/adapter/outbound/cache/redis/runtime/config/RedisLaneMetrics.java" - name: redis.lane.connections.discarded.total type: counter @@ -623,6 +451,7 @@ metrics: log_field_mapping: [lane, reason] compatibility_impact: additive required_test: contract-verification:metrics-cardinality + source: "src/adapter/outbound/redis-runtime/src/main/java/dev/caskeleton/adapter/outbound/cache/redis/runtime/config/RedisLaneMetrics.java" - name: redis.lane.connections.active type: gauge @@ -640,6 +469,7 @@ metrics: log_field_mapping: [lane] compatibility_impact: additive required_test: contract-verification:metrics-cardinality + source: "src/adapter/outbound/redis-runtime/src/main/java/dev/caskeleton/adapter/outbound/cache/redis/runtime/config/RedisLaneMetrics.java" - name: redis.lane.connections.pending type: gauge @@ -657,6 +487,7 @@ metrics: log_field_mapping: [lane] compatibility_impact: additive required_test: contract-verification:metrics-cardinality + source: "src/adapter/outbound/redis-runtime/src/main/java/dev/caskeleton/adapter/outbound/cache/redis/runtime/config/RedisLaneMetrics.java" # === 로그 appender === # source: feature-log-management-contract — Sampling Policy (final) @@ -678,6 +509,7 @@ metrics: log_field_mapping: [appender, level] compatibility_impact: additive required_test: contract-verification:metrics-cardinality + source: "src/bootstrap/app/src/main/java/dev/caskeleton/bootstrap/logging/MetricsAsyncAppender.java" # === 분산 추적 === # source: feature-distributed-tracing-contract — decisions @@ -697,6 +529,7 @@ metrics: log_field_mapping: [profile] compatibility_impact: additive required_test: contract-verification:metrics-cardinality + source: "src/bootstrap/service-runtime/src/main/java/dev/caskeleton/composition/runtime/tracing/TracingSamplingRateGauge.java" # === JVM 기본 === # source: feature-metrics-alerting-contract — In scope "JVM/process metric" @@ -717,6 +550,7 @@ metrics: log_field_mapping: [area, id] compatibility_impact: additive required_test: contract-verification:metrics-cardinality + source: "framework: Micrometer JVM binder(Spring Boot)" # source: feature-metrics-alerting-contract — In scope "JVM/process metric" - name: jvm.gc.pause @@ -735,6 +569,7 @@ metrics: log_field_mapping: [action, cause] compatibility_impact: additive required_test: contract-verification:metrics-cardinality + source: "framework: Micrometer JVM binder(Spring Boot)" # source: feature-metrics-alerting-contract — In scope "JVM/process metric" - name: jvm.threads.live @@ -749,6 +584,7 @@ metrics: log_field_mapping: [] compatibility_impact: additive required_test: contract-verification:metrics-cardinality + source: "framework: Micrometer JVM binder(Spring Boot)" # source: feature-metrics-alerting-contract — In scope "JVM/process metric" (process uptime) - name: process.uptime @@ -763,3 +599,1052 @@ metrics: log_field_mapping: [] compatibility_impact: additive required_test: contract-verification:metrics-cardinality + source: "framework: Micrometer process binder(Spring Boot)" + + # === GraphQL (adapter:inbound:graphql를 켤 때) === + # observation 이름이다. timer 하나와 long-task timer(<이름>.active) 하나가 된다. + + # operation 이름은 observed-operation-names 허용 목록 밖이면 하나로 묶인다(GraphQlOperationNameCardinality). + - name: graphql.request + type: timer + unit: seconds + tags: + - name: graphql.operation.name + cardinality_limit: 50 + - name: graphql.operation.type + cardinality_limit: 3 + allowed_values: [QUERY, MUTATION, SUBSCRIPTION] + - name: graphql.client.profile + cardinality_limit: 10 + - name: graphql.persisted + cardinality_limit: 2 + allowed_values: [true, false] + - name: graphql.outcome + cardinality_limit: 5 + - name: error.type + # 오류가 있을 때만 붙는다. + cardinality_limit: 10 + - name: graphql.complexity.bucket + # 복잡도를 셌을 때만 붙는다. + cardinality_limit: 6 + - name: graphql.depth.bucket + cardinality_limit: 6 + percentiles: null + histogram_buckets: null + alert_severity_thresholds: {} + owner_branch: graphql-platform + log_field_mapping: [] + compatibility_impact: additive + required_test: contract-verification:metrics-cardinality + source: "src/adapter/inbound/graphql/src/main/java/dev/caskeleton/adapter/inbound/graphql/observation/GraphQlObservationNames.java" + constant: GraphQlObservationNames.REQUEST + registered_by: "src/adapter/inbound/graphql/src/main/java/dev/caskeleton/adapter/inbound/graphql/observation/GraphQlRequestObservationConvention.java" + + # Spring for GraphQL 기본 observation. tag는 Spring 기본 convention(DefaultDataFetcherObservationConvention) + # 그대로다. 이 저장소의 convention bean은 이 두 observation에 붙지 않는다. + - name: graphql.datafetcher + type: timer + unit: seconds + tags: + - name: graphql.outcome + cardinality_limit: 2 + allowed_values: [SUCCESS, ERROR] + - name: graphql.field.name + # schema의 field 이름. schema의 field 수가 상한이다. + cardinality_limit: 500 + - name: graphql.error.type + # 예외 클래스의 단순 이름, 없으면 NONE. + cardinality_limit: 20 + percentiles: null + histogram_buckets: null + alert_severity_thresholds: {} + owner_branch: graphql-platform + log_field_mapping: [] + compatibility_impact: additive + required_test: contract-verification:metrics-cardinality + source: "framework: Spring for GraphQL observation(이름은 GraphQlObservationNames와 같다)" + + - name: graphql.dataloader + type: timer + unit: seconds + tags: + - name: graphql.outcome + cardinality_limit: 2 + allowed_values: [SUCCESS, ERROR] + - name: graphql.loader.name + # 등록한 DataLoader 이름, 모르면 unknown. + cardinality_limit: 50 + - name: graphql.error.type + # 예외 클래스의 단순 이름, 없으면 NONE. + cardinality_limit: 20 + percentiles: null + histogram_buckets: null + alert_severity_thresholds: {} + owner_branch: graphql-platform + log_field_mapping: [] + compatibility_impact: additive + required_test: contract-verification:metrics-cardinality + source: "framework: Spring for GraphQL observation(이름은 GraphQlObservationNames와 같다)" + + # DataLoader 없이 부모마다 풀린 field. 임계를 넘은 요청에 한 번 기록한다(GraphQlPerParentFetchInstrumentation). + - name: graphql.per.parent.fetch + type: timer + unit: seconds + tags: + - name: graphql.schema.coordinate + cardinality_limit: 200 + percentiles: null + histogram_buckets: null + alert_severity_thresholds: {} + owner_branch: graphql-platform + log_field_mapping: [] + compatibility_impact: additive + required_test: contract-verification:metrics-cardinality + source: "src/adapter/inbound/graphql/src/main/java/dev/caskeleton/adapter/inbound/graphql/observation/GraphQlObservationNames.java" + constant: GraphQlObservationNames.PER_PARENT_FETCH + registered_by: "src/adapter/inbound/graphql/src/main/java/dev/caskeleton/adapter/inbound/graphql/runtime/GraphQlPerParentFetchInstrumentation.java" + + # === 알림 platform (composition:notification-platform을 켤 때) === + + - name: notification.requested + type: counter + unit: total + tags: + - name: notificationCategory + cardinality_limit: 20 + - name: strategy + cardinality_limit: 5 + percentiles: null + histogram_buckets: null + alert_severity_thresholds: {} + owner_branch: notification-platform + log_field_mapping: [] + compatibility_impact: additive + required_test: contract-verification:metrics-cardinality + source: "src/application/notification/src/main/java/dev/caskeleton/application/notification/platform/observation/NotificationMetricName.java" + constant: NotificationMetricName.REQUESTED + registered_by: "src/application/notification/src/main/java/dev/caskeleton/application/notification/platform/dispatch/NotificationSubmissionService.java" + + - name: notification.suppressed + type: counter + unit: total + tags: + - name: channel + cardinality_limit: 5 + - name: reason + cardinality_limit: 10 + percentiles: null + histogram_buckets: null + alert_severity_thresholds: {} + owner_branch: notification-platform + log_field_mapping: [] + compatibility_impact: additive + required_test: contract-verification:metrics-cardinality + source: "src/application/notification/src/main/java/dev/caskeleton/application/notification/platform/observation/NotificationMetricName.java" + constant: NotificationMetricName.SUPPRESSED + registered_by: "src/application/notification/src/main/java/dev/caskeleton/application/notification/platform/dispatch/DispatchPipeline.java" + + - name: notification.dispatch + type: counter + unit: total + tags: + - name: channel + cardinality_limit: 5 + - name: provider + cardinality_limit: 20 + - name: result + cardinality_limit: 8 + percentiles: null + histogram_buckets: null + alert_severity_thresholds: {} + owner_branch: notification-platform + log_field_mapping: [] + compatibility_impact: additive + required_test: contract-verification:metrics-cardinality + source: "src/application/notification/src/main/java/dev/caskeleton/application/notification/platform/observation/NotificationMetricName.java" + constant: NotificationMetricName.DISPATCH + registered_by: "src/application/notification/src/main/java/dev/caskeleton/application/notification/platform/dispatch/DispatchOutcomeRecorder.java" + + - name: notification.provider.accepted + type: counter + unit: total + tags: + - name: channel + cardinality_limit: 5 + - name: provider + cardinality_limit: 20 + percentiles: null + histogram_buckets: null + alert_severity_thresholds: {} + owner_branch: notification-platform + log_field_mapping: [] + compatibility_impact: additive + required_test: contract-verification:metrics-cardinality + source: "src/application/notification/src/main/java/dev/caskeleton/application/notification/platform/observation/NotificationMetricName.java" + constant: NotificationMetricName.PROVIDER_ACCEPTED + registered_by: "src/application/notification/src/main/java/dev/caskeleton/application/notification/platform/dispatch/DispatchOutcomeRecorder.java" + + - name: notification.ambiguous + type: counter + unit: total + tags: + - name: channel + cardinality_limit: 5 + - name: provider + cardinality_limit: 20 + percentiles: null + histogram_buckets: null + alert_severity_thresholds: {} + owner_branch: notification-platform + log_field_mapping: [] + compatibility_impact: additive + required_test: contract-verification:metrics-cardinality + source: "src/application/notification/src/main/java/dev/caskeleton/application/notification/platform/observation/NotificationMetricName.java" + constant: NotificationMetricName.AMBIGUOUS + registered_by: "src/application/notification/src/main/java/dev/caskeleton/application/notification/platform/dispatch/DispatchOutcomeRecorder.java" + + - name: notification.delivery + type: counter + unit: total + tags: + - name: channel + cardinality_limit: 5 + - name: provider + cardinality_limit: 20 + - name: outcome + cardinality_limit: 6 + percentiles: null + histogram_buckets: null + alert_severity_thresholds: {} + owner_branch: notification-platform + log_field_mapping: [] + compatibility_impact: additive + required_test: contract-verification:metrics-cardinality + source: "src/application/notification/src/main/java/dev/caskeleton/application/notification/platform/observation/NotificationMetricName.java" + constant: NotificationMetricName.DELIVERY + registered_by: "src/application/notification/src/main/java/dev/caskeleton/application/notification/platform/callback/ProviderEventProjectionService.java" + + - name: notification.callback + type: counter + unit: total + tags: + - name: provider + cardinality_limit: 20 + - name: result + cardinality_limit: 8 + percentiles: null + histogram_buckets: null + alert_severity_thresholds: {} + owner_branch: notification-platform + log_field_mapping: [] + compatibility_impact: additive + required_test: contract-verification:metrics-cardinality + source: "src/application/notification/src/main/java/dev/caskeleton/application/notification/platform/observation/NotificationMetricName.java" + constant: NotificationMetricName.CALLBACK + registered_by: "src/application/notification/src/main/java/dev/caskeleton/application/notification/platform/callback/IngestProviderCallbackApplicationUseCase.java" + + - name: notification.reconciliation + type: counter + unit: total + tags: + - name: provider + cardinality_limit: 20 + - name: correction + cardinality_limit: 5 + percentiles: null + histogram_buckets: null + alert_severity_thresholds: {} + owner_branch: notification-platform + log_field_mapping: [] + compatibility_impact: additive + required_test: contract-verification:metrics-cardinality + source: "src/application/notification/src/main/java/dev/caskeleton/application/notification/platform/observation/NotificationMetricName.java" + constant: NotificationMetricName.RECONCILIATION + registered_by: "src/application/notification/src/main/java/dev/caskeleton/application/notification/platform/dispatch/ReconciliationService.java" + + - name: notification.queue.depth + type: gauge + unit: total + tags: [] + percentiles: null + histogram_buckets: null + alert_severity_thresholds: {} + owner_branch: notification-platform + log_field_mapping: [] + compatibility_impact: additive + required_test: contract-verification:metrics-cardinality + source: "src/application/notification/src/main/java/dev/caskeleton/application/notification/platform/observation/NotificationMetricName.java" + constant: NotificationMetricName.QUEUE_DEPTH + registered_by: "src/adapter/outbound/notification/src/main/java/dev/caskeleton/adapter/outbound/notification/platform/dispatch/NotificationSchedulerWorker.java" + + # === outbound HTTP client (adapter:outbound:httpclient를 켤 때) === + # Spring의 RestClient observation은 끄고 이 이름으로 기록한다(tag 이름도 이 leaf 것이다). attempt의 outcome은 + # success|failure, 논리 호출은 ambiguous도 있다. + + - name: http.client.attempts + type: counter + unit: total + tags: + - name: clientName + cardinality_limit: 50 + - name: operationName + cardinality_limit: 100 + - name: method + cardinality_limit: 8 + - name: uriTemplate + cardinality_limit: 200 + - name: transport + cardinality_limit: 3 + - name: protocol + cardinality_limit: 3 + percentiles: null + histogram_buckets: null + alert_severity_thresholds: {} + owner_branch: httpclient + log_field_mapping: [] + compatibility_impact: additive + required_test: contract-verification:metrics-cardinality + source: "src/adapter/outbound/httpclient/src/main/java/dev/caskeleton/adapter/outbound/httpclient/observation/HttpClientObservationNames.java" + constant: HttpClientObservationNames.ATTEMPT_COUNTER + registered_by: "src/adapter/outbound/httpclient/src/main/java/dev/caskeleton/adapter/outbound/httpclient/observation/AttemptObservation.java" + + - name: http.client.requests + type: timer + unit: seconds + tags: + - name: clientName + cardinality_limit: 50 + - name: operationName + cardinality_limit: 100 + - name: method + cardinality_limit: 8 + - name: uriTemplate + cardinality_limit: 200 + - name: transport + cardinality_limit: 3 + - name: protocol + cardinality_limit: 3 + - name: status + cardinality_limit: 40 + - name: evidence + cardinality_limit: 5 + - name: outcome + cardinality_limit: 2 + allowed_values: [success, failure] + percentiles: null + histogram_buckets: null + alert_severity_thresholds: {} + owner_branch: httpclient + log_field_mapping: [] + compatibility_impact: additive + required_test: contract-verification:metrics-cardinality + source: "src/adapter/outbound/httpclient/src/main/java/dev/caskeleton/adapter/outbound/httpclient/observation/HttpClientObservationNames.java" + constant: HttpClientObservationNames.ATTEMPT_TIMER + registered_by: "src/adapter/outbound/httpclient/src/main/java/dev/caskeleton/adapter/outbound/httpclient/observation/AttemptObservation.java" + + - name: http.client.timeout + type: counter + unit: total + tags: + - name: clientName + cardinality_limit: 50 + - name: operationName + cardinality_limit: 100 + - name: method + cardinality_limit: 8 + - name: uriTemplate + cardinality_limit: 200 + - name: transport + cardinality_limit: 3 + - name: protocol + cardinality_limit: 3 + - name: timeoutType + cardinality_limit: 5 + percentiles: null + histogram_buckets: null + alert_severity_thresholds: {} + owner_branch: httpclient + log_field_mapping: [] + compatibility_impact: additive + required_test: contract-verification:metrics-cardinality + source: "src/adapter/outbound/httpclient/src/main/java/dev/caskeleton/adapter/outbound/httpclient/observation/HttpClientObservationNames.java" + constant: HttpClientObservationNames.TIMEOUT + registered_by: "src/adapter/outbound/httpclient/src/main/java/dev/caskeleton/adapter/outbound/httpclient/observation/AttemptObservation.java" + + - name: http.client.request.bytes + type: distribution_summary + unit: bytes + tags: + - name: clientName + cardinality_limit: 50 + - name: operationName + cardinality_limit: 100 + - name: method + cardinality_limit: 8 + - name: uriTemplate + cardinality_limit: 200 + - name: transport + cardinality_limit: 3 + - name: protocol + cardinality_limit: 3 + percentiles: null + histogram_buckets: null + alert_severity_thresholds: {} + owner_branch: httpclient + log_field_mapping: [] + compatibility_impact: additive + required_test: contract-verification:metrics-cardinality + source: "src/adapter/outbound/httpclient/src/main/java/dev/caskeleton/adapter/outbound/httpclient/observation/HttpClientObservationNames.java" + constant: HttpClientObservationNames.REQUEST_BYTES + registered_by: "src/adapter/outbound/httpclient/src/main/java/dev/caskeleton/adapter/outbound/httpclient/observation/AttemptObservation.java" + + - name: http.client.response.bytes + type: distribution_summary + unit: bytes + tags: + - name: clientName + cardinality_limit: 50 + - name: operationName + cardinality_limit: 100 + - name: method + cardinality_limit: 8 + - name: uriTemplate + cardinality_limit: 200 + - name: transport + cardinality_limit: 3 + - name: protocol + cardinality_limit: 3 + percentiles: null + histogram_buckets: null + alert_severity_thresholds: {} + owner_branch: httpclient + log_field_mapping: [] + compatibility_impact: additive + required_test: contract-verification:metrics-cardinality + source: "src/adapter/outbound/httpclient/src/main/java/dev/caskeleton/adapter/outbound/httpclient/observation/HttpClientObservationNames.java" + constant: HttpClientObservationNames.RESPONSE_BYTES + registered_by: "src/adapter/outbound/httpclient/src/main/java/dev/caskeleton/adapter/outbound/httpclient/observation/AttemptObservation.java" + + - name: http.client.logical.calls + type: timer + unit: seconds + tags: + - name: clientName + cardinality_limit: 50 + - name: operationName + cardinality_limit: 100 + - name: method + cardinality_limit: 8 + - name: uriTemplate + cardinality_limit: 200 + - name: status + cardinality_limit: 40 + - name: evidence + cardinality_limit: 5 + - name: outcome + cardinality_limit: 3 + allowed_values: [success, failure, ambiguous] + percentiles: null + histogram_buckets: null + alert_severity_thresholds: {} + owner_branch: httpclient + log_field_mapping: [] + compatibility_impact: additive + required_test: contract-verification:metrics-cardinality + source: "src/adapter/outbound/httpclient/src/main/java/dev/caskeleton/adapter/outbound/httpclient/observation/HttpClientObservationNames.java" + constant: HttpClientObservationNames.LOGICAL_CALL_TIMER + registered_by: "src/adapter/outbound/httpclient/src/main/java/dev/caskeleton/adapter/outbound/httpclient/observation/LogicalCallObservation.java" + + - name: http.client.retry.count + type: counter + unit: total + tags: + - name: clientName + cardinality_limit: 50 + - name: operationName + cardinality_limit: 100 + - name: method + cardinality_limit: 8 + - name: uriTemplate + cardinality_limit: 200 + - name: retryReason + cardinality_limit: 10 + percentiles: null + histogram_buckets: null + alert_severity_thresholds: {} + owner_branch: httpclient + log_field_mapping: [] + compatibility_impact: additive + required_test: contract-verification:metrics-cardinality + source: "src/adapter/outbound/httpclient/src/main/java/dev/caskeleton/adapter/outbound/httpclient/observation/HttpClientObservationNames.java" + constant: HttpClientObservationNames.RETRY_COUNT + registered_by: "src/adapter/outbound/httpclient/src/main/java/dev/caskeleton/adapter/outbound/httpclient/observation/LogicalCallObservation.java" + + - name: http.client.retry.exhausted + type: counter + unit: total + tags: + - name: clientName + cardinality_limit: 50 + - name: operationName + cardinality_limit: 100 + - name: method + cardinality_limit: 8 + - name: uriTemplate + cardinality_limit: 200 + percentiles: null + histogram_buckets: null + alert_severity_thresholds: {} + owner_branch: httpclient + log_field_mapping: [] + compatibility_impact: additive + required_test: contract-verification:metrics-cardinality + source: "src/adapter/outbound/httpclient/src/main/java/dev/caskeleton/adapter/outbound/httpclient/observation/HttpClientObservationNames.java" + constant: HttpClientObservationNames.RETRY_EXHAUSTED + registered_by: "src/adapter/outbound/httpclient/src/main/java/dev/caskeleton/adapter/outbound/httpclient/observation/LogicalCallObservation.java" + + - name: http.client.ambiguous + type: counter + unit: total + tags: + - name: clientName + cardinality_limit: 50 + - name: operationName + cardinality_limit: 100 + - name: method + cardinality_limit: 8 + - name: uriTemplate + cardinality_limit: 200 + percentiles: null + histogram_buckets: null + alert_severity_thresholds: {} + owner_branch: httpclient + log_field_mapping: [] + compatibility_impact: additive + required_test: contract-verification:metrics-cardinality + source: "src/adapter/outbound/httpclient/src/main/java/dev/caskeleton/adapter/outbound/httpclient/observation/HttpClientObservationNames.java" + constant: HttpClientObservationNames.AMBIGUOUS + registered_by: "src/adapter/outbound/httpclient/src/main/java/dev/caskeleton/adapter/outbound/httpclient/observation/LogicalCallObservation.java" + + # 값 1 = open, 0 = 그 밖. + - name: http.client.circuit.state + type: gauge + unit: total + tags: + - name: clientName + cardinality_limit: 50 + percentiles: null + histogram_buckets: null + alert_severity_thresholds: {} + owner_branch: httpclient + log_field_mapping: [] + compatibility_impact: additive + required_test: contract-verification:metrics-cardinality + source: "src/adapter/outbound/httpclient/src/main/java/dev/caskeleton/adapter/outbound/httpclient/observation/HttpClientObservationNames.java" + constant: HttpClientObservationNames.CIRCUIT_STATE + registered_by: "src/adapter/outbound/httpclient/src/main/java/dev/caskeleton/adapter/outbound/httpclient/resilience/ResilienceRejectionRecorder.java" + + - name: http.client.circuit.state.rejected + type: counter + unit: total + tags: + - name: clientName + cardinality_limit: 50 + percentiles: null + histogram_buckets: null + alert_severity_thresholds: {} + owner_branch: httpclient + log_field_mapping: [] + compatibility_impact: additive + required_test: contract-verification:metrics-cardinality + source: "src/adapter/outbound/httpclient/src/main/java/dev/caskeleton/adapter/outbound/httpclient/observation/HttpClientObservationNames.java" + constant: HttpClientObservationNames.CIRCUIT_REJECTED + registered_by: "src/adapter/outbound/httpclient/src/main/java/dev/caskeleton/adapter/outbound/httpclient/resilience/ResilienceRejectionRecorder.java" + + - name: http.client.ratelimit.rejected + type: counter + unit: total + tags: + - name: clientName + cardinality_limit: 50 + percentiles: null + histogram_buckets: null + alert_severity_thresholds: {} + owner_branch: httpclient + log_field_mapping: [] + compatibility_impact: additive + required_test: contract-verification:metrics-cardinality + source: "src/adapter/outbound/httpclient/src/main/java/dev/caskeleton/adapter/outbound/httpclient/observation/HttpClientObservationNames.java" + constant: HttpClientObservationNames.RATE_LIMIT_REJECTED + registered_by: "src/adapter/outbound/httpclient/src/main/java/dev/caskeleton/adapter/outbound/httpclient/resilience/ResilienceRejectionRecorder.java" + + - name: http.client.bulkhead.rejected + type: counter + unit: total + tags: + - name: clientName + cardinality_limit: 50 + percentiles: null + histogram_buckets: null + alert_severity_thresholds: {} + owner_branch: httpclient + log_field_mapping: [] + compatibility_impact: additive + required_test: contract-verification:metrics-cardinality + source: "src/adapter/outbound/httpclient/src/main/java/dev/caskeleton/adapter/outbound/httpclient/observation/HttpClientObservationNames.java" + constant: HttpClientObservationNames.BULKHEAD_REJECTED + registered_by: "src/adapter/outbound/httpclient/src/main/java/dev/caskeleton/adapter/outbound/httpclient/resilience/ResilienceRejectionRecorder.java" + + - name: http.client.ssrf.rejected + type: counter + unit: total + tags: + - name: clientName + cardinality_limit: 50 + - name: outcome + cardinality_limit: 5 + percentiles: null + histogram_buckets: null + alert_severity_thresholds: {} + owner_branch: httpclient + log_field_mapping: [] + compatibility_impact: additive + required_test: contract-verification:metrics-cardinality + source: "src/adapter/outbound/httpclient/src/main/java/dev/caskeleton/adapter/outbound/httpclient/observation/HttpClientObservationNames.java" + constant: HttpClientObservationNames.SSRF_REJECTED + registered_by: "src/adapter/outbound/httpclient/src/main/java/dev/caskeleton/adapter/outbound/httpclient/dynamic/DefaultDynamicTargetGateway.java" + + - name: http.client.pool.connections + type: gauge + unit: total + tags: + - name: clientName + cardinality_limit: 50 + - name: transport + cardinality_limit: 3 + - name: state + cardinality_limit: 3 + percentiles: null + histogram_buckets: null + alert_severity_thresholds: {} + owner_branch: httpclient + log_field_mapping: [] + compatibility_impact: additive + required_test: contract-verification:metrics-cardinality + source: "src/adapter/outbound/httpclient/src/main/java/dev/caskeleton/adapter/outbound/httpclient/observation/HttpClientObservationNames.java" + constant: HttpClientObservationNames.POOL_CONNECTIONS + registered_by: "src/adapter/outbound/httpclient/src/main/java/dev/caskeleton/adapter/outbound/httpclient/apache/ApachePoolMetricsBinder.java" + + - name: http.client.pool.pending + type: gauge + unit: total + tags: + - name: clientName + cardinality_limit: 50 + - name: transport + cardinality_limit: 3 + percentiles: null + histogram_buckets: null + alert_severity_thresholds: {} + owner_branch: httpclient + log_field_mapping: [] + compatibility_impact: additive + required_test: contract-verification:metrics-cardinality + source: "src/adapter/outbound/httpclient/src/main/java/dev/caskeleton/adapter/outbound/httpclient/observation/HttpClientObservationNames.java" + constant: HttpClientObservationNames.POOL_PENDING + registered_by: "src/adapter/outbound/httpclient/src/main/java/dev/caskeleton/adapter/outbound/httpclient/apache/ApachePoolMetricsBinder.java" + + # === 파일 서버 (composition:fileserver-platform을 켤 때) === + + - name: fileserver.upload.duration + type: timer + unit: seconds + tags: + - name: mode + cardinality_limit: 5 + - name: storage + cardinality_limit: 5 + - name: result + cardinality_limit: 6 + - name: size_bucket + cardinality_limit: 6 + percentiles: null + histogram_buckets: null + alert_severity_thresholds: {} + owner_branch: fileserver-platform + log_field_mapping: [] + compatibility_impact: additive + required_test: contract-verification:metrics-cardinality + source: "src/adapter/outbound/fileserver/src/main/java/dev/caskeleton/adapter/outbound/fileserver/platform/observability/MicrometerFileserverMetrics.java" + + - name: fileserver.download.duration + type: timer + unit: seconds + tags: + - name: transfer_mode + cardinality_limit: 5 + - name: range_type + cardinality_limit: 4 + - name: result + cardinality_limit: 6 + - name: size_bucket + cardinality_limit: 6 + percentiles: null + histogram_buckets: null + alert_severity_thresholds: {} + owner_branch: fileserver-platform + log_field_mapping: [] + compatibility_impact: additive + required_test: contract-verification:metrics-cardinality + source: "src/adapter/outbound/fileserver/src/main/java/dev/caskeleton/adapter/outbound/fileserver/platform/observability/MicrometerFileserverMetrics.java" + + - name: fileserver.transfer.bytes + type: counter + unit: bytes + tags: + - name: direction + cardinality_limit: 2 + allowed_values: [upload, download] + - name: storage + cardinality_limit: 5 + percentiles: null + histogram_buckets: null + alert_severity_thresholds: {} + owner_branch: fileserver-platform + log_field_mapping: [] + compatibility_impact: additive + required_test: contract-verification:metrics-cardinality + source: "src/adapter/outbound/fileserver/src/main/java/dev/caskeleton/adapter/outbound/fileserver/platform/observability/MicrometerFileserverMetrics.java" + + - name: fileserver.transfer.active + type: gauge + unit: total + tags: + - name: direction + cardinality_limit: 2 + allowed_values: [upload, download] + - name: instance + cardinality_limit: 50 + percentiles: null + histogram_buckets: null + alert_severity_thresholds: {} + owner_branch: fileserver-platform + log_field_mapping: [] + compatibility_impact: additive + required_test: contract-verification:metrics-cardinality + source: "src/adapter/outbound/fileserver/src/main/java/dev/caskeleton/adapter/outbound/fileserver/platform/observability/MicrometerFileserverMetrics.java" + + - name: fileserver.transfer.interruption + type: counter + unit: total + tags: + - name: direction + cardinality_limit: 2 + allowed_values: [upload, download] + - name: reason + cardinality_limit: 10 + percentiles: null + histogram_buckets: null + alert_severity_thresholds: {} + owner_branch: fileserver-platform + log_field_mapping: [] + compatibility_impact: additive + required_test: contract-verification:metrics-cardinality + source: "src/adapter/outbound/fileserver/src/main/java/dev/caskeleton/adapter/outbound/fileserver/platform/observability/MicrometerFileserverMetrics.java" + + - name: fileserver.upload.offset.mismatch + type: counter + unit: total + tags: + - name: mode + cardinality_limit: 5 + - name: client_type + cardinality_limit: 5 + percentiles: null + histogram_buckets: null + alert_severity_thresholds: {} + owner_branch: fileserver-platform + log_field_mapping: [] + compatibility_impact: additive + required_test: contract-verification:metrics-cardinality + source: "src/adapter/outbound/fileserver/src/main/java/dev/caskeleton/adapter/outbound/fileserver/platform/observability/MicrometerFileserverMetrics.java" + + - name: fileserver.checksum.failure + type: counter + unit: total + tags: + - name: algorithm + cardinality_limit: 5 + - name: stage + cardinality_limit: 5 + percentiles: null + histogram_buckets: null + alert_severity_thresholds: {} + owner_branch: fileserver-platform + log_field_mapping: [] + compatibility_impact: additive + required_test: contract-verification:metrics-cardinality + source: "src/adapter/outbound/fileserver/src/main/java/dev/caskeleton/adapter/outbound/fileserver/platform/observability/MicrometerFileserverMetrics.java" + + - name: fileserver.verification.queue + type: counter + unit: total + tags: + - name: verifier + cardinality_limit: 5 + - name: verdict + cardinality_limit: 5 + - name: age_bucket + cardinality_limit: 6 + percentiles: null + histogram_buckets: null + alert_severity_thresholds: {} + owner_branch: fileserver-platform + log_field_mapping: [] + compatibility_impact: additive + required_test: contract-verification:metrics-cardinality + source: "src/adapter/outbound/fileserver/src/main/java/dev/caskeleton/adapter/outbound/fileserver/platform/observability/MicrometerFileserverMetrics.java" + + - name: fileserver.quota + type: counter + unit: total + tags: + - name: scope_type + cardinality_limit: 5 + - name: result + cardinality_limit: 5 + percentiles: null + histogram_buckets: null + alert_severity_thresholds: {} + owner_branch: fileserver-platform + log_field_mapping: [] + compatibility_impact: additive + required_test: contract-verification:metrics-cardinality + source: "src/adapter/outbound/fileserver/src/main/java/dev/caskeleton/adapter/outbound/fileserver/platform/observability/MicrometerFileserverMetrics.java" + + - name: fileserver.cleanup + type: counter + unit: total + tags: + - name: type + cardinality_limit: 5 + - name: result + cardinality_limit: 5 + percentiles: null + histogram_buckets: null + alert_severity_thresholds: {} + owner_branch: fileserver-platform + log_field_mapping: [] + compatibility_impact: additive + required_test: contract-verification:metrics-cardinality + source: "src/adapter/outbound/fileserver/src/main/java/dev/caskeleton/adapter/outbound/fileserver/platform/observability/MicrometerFileserverMetrics.java" + + - name: fileserver.download.delegation + type: counter + unit: total + tags: + - name: size_bucket + cardinality_limit: 6 + - name: delegated + cardinality_limit: 2 + allowed_values: [true, false] + percentiles: null + histogram_buckets: null + alert_severity_thresholds: {} + owner_branch: fileserver-platform + log_field_mapping: [] + compatibility_impact: additive + required_test: contract-verification:metrics-cardinality + source: "src/adapter/outbound/fileserver/src/main/java/dev/caskeleton/adapter/outbound/fileserver/platform/observability/MicrometerFileserverMetrics.java" + + - name: fileserver.access.denial + type: counter + unit: total + tags: + - name: operation + cardinality_limit: 10 + - name: policy_code + cardinality_limit: 20 + percentiles: null + histogram_buckets: null + alert_severity_thresholds: {} + owner_branch: fileserver-platform + log_field_mapping: [] + compatibility_impact: additive + required_test: contract-verification:metrics-cardinality + source: "src/adapter/outbound/fileserver/src/main/java/dev/caskeleton/adapter/outbound/fileserver/platform/observability/MicrometerFileserverMetrics.java" + + # === MongoDB (adapter:outbound:persistence-mongo를 켤 때) === + + - name: mongodb.driver.command + type: timer + unit: seconds + tags: + - name: mongoProfile + cardinality_limit: 10 + - name: operationType + cardinality_limit: 20 + - name: result + cardinality_limit: 3 + percentiles: null + histogram_buckets: null + alert_severity_thresholds: {} + owner_branch: persistence-mongo + log_field_mapping: [] + compatibility_impact: additive + required_test: contract-verification:metrics-cardinality + source: "src/adapter/outbound/persistence-mongo/src/main/java/dev/caskeleton/adapter/outbound/mongo/observation/MongoCommandObservationListener.java" + + - name: mongodb.driver.pool.checkedout + type: gauge + unit: total + tags: + - name: mongoProfile + cardinality_limit: 10 + percentiles: null + histogram_buckets: null + alert_severity_thresholds: {} + owner_branch: persistence-mongo + log_field_mapping: [] + compatibility_impact: additive + required_test: contract-verification:metrics-cardinality + source: "src/adapter/outbound/persistence-mongo/src/main/java/dev/caskeleton/adapter/outbound/mongo/observation/MongoPoolObservationListener.java" + + - name: mongodb.driver.pool.checkout.waiting + type: gauge + unit: total + tags: + - name: mongoProfile + cardinality_limit: 10 + percentiles: null + histogram_buckets: null + alert_severity_thresholds: {} + owner_branch: persistence-mongo + log_field_mapping: [] + compatibility_impact: additive + required_test: contract-verification:metrics-cardinality + source: "src/adapter/outbound/persistence-mongo/src/main/java/dev/caskeleton/adapter/outbound/mongo/observation/MongoPoolObservationListener.java" + + - name: mongodb.driver.pool.checkout.failed + type: counter + unit: total + tags: + - name: mongoProfile + cardinality_limit: 10 + - name: failureCategory + cardinality_limit: 10 + percentiles: null + histogram_buckets: null + alert_severity_thresholds: {} + owner_branch: persistence-mongo + log_field_mapping: [] + compatibility_impact: additive + required_test: contract-verification:metrics-cardinality + source: "src/adapter/outbound/persistence-mongo/src/main/java/dev/caskeleton/adapter/outbound/mongo/observation/MongoPoolObservationListener.java" + + - name: mongodb.driver.topology.changed + type: counter + unit: total + tags: + - name: mongoProfile + cardinality_limit: 10 + percentiles: null + histogram_buckets: null + alert_severity_thresholds: {} + owner_branch: persistence-mongo + log_field_mapping: [] + compatibility_impact: additive + required_test: contract-verification:metrics-cardinality + source: "src/adapter/outbound/persistence-mongo/src/main/java/dev/caskeleton/adapter/outbound/mongo/observation/MongoSdamObservationListener.java" + + - name: mongodb.driver.primary.changed + type: counter + unit: total + tags: + - name: mongoProfile + cardinality_limit: 10 + percentiles: null + histogram_buckets: null + alert_severity_thresholds: {} + owner_branch: persistence-mongo + log_field_mapping: [] + compatibility_impact: additive + required_test: contract-verification:metrics-cardinality + source: "src/adapter/outbound/persistence-mongo/src/main/java/dev/caskeleton/adapter/outbound/mongo/observation/MongoSdamObservationListener.java" + + - name: mongodb.operation + type: timer + unit: seconds + tags: + - name: mongoProfile + cardinality_limit: 10 + - name: databaseProfile + cardinality_limit: 10 + - name: collectionProfile + cardinality_limit: 50 + - name: operationName + cardinality_limit: 50 + - name: operationType + cardinality_limit: 20 + - name: consistencyProfile + cardinality_limit: 5 + - name: result + cardinality_limit: 8 + - name: failureCategory + cardinality_limit: 21 + percentiles: null + histogram_buckets: null + alert_severity_thresholds: {} + owner_branch: persistence-mongo + log_field_mapping: [] + compatibility_impact: additive + required_test: contract-verification:metrics-cardinality + source: "src/adapter/outbound/persistence-mongo/src/main/java/dev/caskeleton/adapter/outbound/mongo/observation/MicrometerMongoOperationObserver.java" + + # === JPA query 관측 (adapter:outbound:persistence-jpa, 실행기가 MicrometerQueryObservation을 쓸 때) === + - name: jpa.query.duration + type: timer + unit: seconds + tags: + - name: persistence.unit + cardinality_limit: 5 + - name: persistence.operation + cardinality_limit: 10 + - name: persistence.query + # 등록한 QueryName이다(SQL 원문이 아니다). + cardinality_limit: 200 + - name: outcome + cardinality_limit: 6 + - name: failure.category + cardinality_limit: 20 + percentiles: null + histogram_buckets: null + alert_severity_thresholds: {} + owner_branch: persistence-jpa + log_field_mapping: [] + compatibility_impact: additive + required_test: contract-verification:metrics-cardinality + source: "src/adapter/outbound/persistence-jpa/src/main/java/dev/caskeleton/adapter/outbound/persistence/observation/MicrometerQueryObservation.java" + + - name: jpa.query.rows + type: distribution_summary + unit: total + tags: + - name: persistence.unit + cardinality_limit: 5 + - name: persistence.operation + cardinality_limit: 10 + - name: persistence.query + cardinality_limit: 200 + - name: outcome + cardinality_limit: 6 + - name: failure.category + cardinality_limit: 20 + percentiles: null + histogram_buckets: null + alert_severity_thresholds: {} + owner_branch: persistence-jpa + log_field_mapping: [] + compatibility_impact: additive + required_test: contract-verification:metrics-cardinality + source: "src/adapter/outbound/persistence-jpa/src/main/java/dev/caskeleton/adapter/outbound/persistence/observation/MicrometerQueryObservation.java" diff --git a/docs/runbooks/lock-acquisition-timeout.md b/docs/runbooks/lock-acquisition-timeout.md index 4230ae9c..374faa65 100644 --- a/docs/runbooks/lock-acquisition-timeout.md +++ b/docs/runbooks/lock-acquisition-timeout.md @@ -25,11 +25,10 @@ status: 초안 ## 진단 -1. 경합 비율. `APP_MULTI_INSTANCE_ENABLED=true`이면 JDBC 락에 메트릭이 붙는다. +1. 경합 비율. 프로세스 안 락(`APP_MULTI_INSTANCE_ENABLED=false`, 기본값)과 JDBC 락 모두 메트릭이 붙는다. `lock.acquisition{outcome="timeout"}`을 `lock.acquisition{outcome="acquired"}`와 비교한다. 이 counter에는 - 락 이름도 대기 시간도 없다. 키는 메트릭 태그가 되지 않는다(`docs/registries/metrics.yaml`). 프로세스 안 락 - (`APP_MULTI_INSTANCE_ENABLED=false`, 기본값)은 메트릭이 없고 DB에 아무것도 남기지 않는다. 아래 단계는 JDBC - 락에만 해당한다. + 락 이름도 대기 시간도 없다. 키는 메트릭 태그가 되지 않는다(`docs/registries/metrics.yaml`). 프로세스 안 락은 + DB에 아무것도 남기지 않는다. 아래 단계는 JDBC 락에만 해당한다. 2. 지금 누가 락을 쥐고 있나. `INT_LOCK`에는 쥔 락마다 한 행이 있다. `CLIENT_ID`는 쥔 인스턴스의 락 저장소 id이고, 시각은 UTC다. `EXPIRED_AFTER`가 지난 행은 죽은 lease다. 그 키를 다음에 잡는 쪽이 넘겨받는다. 가장 오래된 살아 있는 행이 오래 쥔 쪽이다. diff --git a/docs/upgrade/2026-11-round9-observability.md b/docs/upgrade/2026-11-round9-observability.md index 35ff8444..8de169ca 100644 --- a/docs/upgrade/2026-11-round9-observability.md +++ b/docs/upgrade/2026-11-round9-observability.md @@ -1,8 +1,9 @@ # 업그레이드 안내 — 2026-11 관측 registry·로그 내용·로컬 비밀 기본값 -- 범위: 2026-11-12 전체 리뷰의 로그 내용 묶음과 비밀·IP 문서 묶음을 고친 변경. 로그 줄이 원본 경로·SQL·요청 값을 덜 - 싣게 했고, 로컬 비밀 기본값을 `__LOCAL_DEV_` 규칙에 맞췄다. -- 근거: 2026-11-12 전체 리뷰(`docs/reviews/2026-11-12-round9-review/`)의 RR9-02-09·10·12·13·15·21·22·26 +- 범위: 2026-11-12 전체 리뷰의 관측 registry 묶음, 로그 내용 묶음, 비밀·IP 문서 묶음을 고친 변경. 지표·MDC·헤더 registry를 코드에 + 맞췄고, 로그 줄이 원본 경로·SQL·요청 값을 덜 싣게 했고, 로컬 비밀 기본값을 `__LOCAL_DEV_` 규칙에 맞췄다. +- 근거: 2026-11-12 전체 리뷰(`docs/reviews/2026-11-12-round9-review/`)의 RR9-02-05·07·08·09·10·12·13·15·21·22·26·27, + RR9-03-05 ## 1. 누가 읽나 @@ -20,7 +21,13 @@ compose로 개발하는 사람이다. | 5 | web `HeaderSanitizer`(로그용 헤더 가림 공개 유틸리티)와 그 시험을 지웠다. 운영 코드가 부르지 않았고, 요청 로그는 헤더를 싣지 않는다 | 그 클래스를 부르던 서비스는 자기 코드로 옮긴다(마지막으로 있던 커밋은 `docs/capability-status.md` 제거 표) | 컴파일이 실패한다 | `HeaderSanitizer`를 부르던 서비스 | | 6 | 로컬 비밀 기본값이 `__LOCAL_DEV_` 접두어다: DB 비밀번호 `__LOCAL_DEV_db_password__`(compose·lane·`application-local.yml`), MinIO root `__LOCAL_DEV_minio_root__`, Keycloak admin `__LOCAL_DEV_keycloak_admin__`. prod에서 이 값이면 기동이 실패한다 | 옛 기본값으로 만든 로컬 volume(Postgres `caskeleton-db-data`, MinIO `caskeleton-minio-data`)이 있으면 저장소 루트 `.env`(compose 보간과 `bootRun`이 읽는 파일)에 옛 값을 `APP_DATASOURCE_PASSWORD`·`MINIO_ROOT_PASSWORD`로 두거나 volume을 지우고 다시 만든다(`docker compose down -v`). Keycloak은 volume이 없어 컨테이너를 다시 만들면 새 비밀번호로 들어간다 | 로컬 앱이 DB 인증에 실패하고 `minio-init`이 bucket을 만들지 못한다 | 로컬 compose를 쓰는 사람 | | 7 | `docs/registries/secrets-classification.yaml`의 `masking_rule`은 `full`·`userinfo`·`none`뿐이다(`full_except_last_4`는 구현이 없었다). Layer 1 가림이 salt·PEM·HMAC/signing/encryption/credential/private key 이름, 여러 줄 PEM 블록 전체, URI 사용자 정보 비밀번호(`scheme://user:****@host`)도 가린다. `required_test`는 app-bootstrap 시험 폴더에 있는 시험 클래스 이름이어야 한다(옛 `secrets-contract:*` 표지는 안 된다). 쓰지 않는 행 넷(`APP_PROFILE`, `SERVER_PORT`, `APP_NOTIFICATION_SLACK_WEBHOOK_URL`, `APP_SECURITY_GOOGLE_OAUTH_CLIENT_ID`)을 지웠다 | 자기 행에 `full_except_last_4`를 쓴 서비스는 `full`이나 `userinfo`로, `secrets-contract:*`는 시험 클래스 이름으로 바꾼다 | `SecretsClassificationRegistryTest`가 실패한다 | registry에 행을 더한 서비스 | +| 8 | 지표 이름 둘을 이름 규칙(dot.case, 밑줄 없음)에 맞췄다: `http.client.rate_limit.rejected` → `http.client.ratelimit.rejected`, `fileserver.upload.offset_mismatch` → `fileserver.upload.offset.mismatch` | 그 이름을 쓰는 대시보드·경보를 새 이름으로 바꾼다 | 그 패널이 비고 경보가 울리지 않는다 | httpclient·파일 서버를 켠 서비스 | +| 9 | `docs/registries/metrics.yaml`에서 아무 코드도 등록하지 않던 행 열하나를 지웠다(`http.server.requests.latency`, `dependency.client.requests`, `db.query.duration`, `resilience4j.*` 셋, `cache.*` 다섯). `http.server.requests`의 tag는 Spring Framework 기본 convention이 다는 값(`method`, `uri`, 숫자 `status`, `outcome`, `exception`, `error`)이다 | 그 이름·tag(`uri_template`, `2xx`)로 만든 경보를 실제 이름으로 바꾼다. 지연 경보는 `http.server.requests`로 건다 | 경보가 없는 시계열을 본다 | registry를 보고 경보를 만든 서비스 | +| 10 | `MetricsDistributionMeterFilter`의 SLO 대상이 `http.server.requests`·`jvm.gc.pause` 둘이다. 아무도 등록하지 않던 `http.server.requests.latency`·`dependency.client.requests`·`db.query.duration`은 대상에서 빠졌다 | 그 셋 이름으로 timer를 직접 만들었으면 자기 `MeterFilter`로 SLO histogram을 켠다 | 그 timer의 `_bucket` 시계열(인스턴스 사이 p99)이 사라진다 | 그 셋 이름으로 timer를 직접 만든 서비스 | +| 11 | 단일 인스턴스(in-process) lock도 `lock.acquisition`을 센다 | 단일 인스턴스 배포의 lock 경보 기준선을 다시 잡는다 | 새로 생긴 값으로 경보가 울릴 수 있다 | persistence-jpa를 켠 서비스 | +| 12 | `docs/registries/mdc-keys.yaml`은 코드가 MDC에 넣는 다섯 key(`request_id`, `trace_id`, `span_id`, `correlation_id`, `user_principal`)만 적는다. `headers.yaml`에서 아무도 쓰지 않던 `Deprecation`·`Sunset`을 지우고 쓰는 헤더 행을 더했다(`tracestate`는 W3C 전파가 쓰므로 `framework:` 행으로 둔다). 지표·헤더 행마다 `source`가 있다 | 자기 MDC key·헤더·지표를 더한 서비스는 `MdcKeys`·행을 같이 더하고 `source`를 적는다 | 새 정합 시험이 실패한다 | registry에 행을 더한 서비스 | +| 13 | testFixtures `CardinalityBounds`: `STATUS_CODE`가 7에서 40(숫자 HTTP status)으로, `RESILIENCE4J_OUTCOME`(5)이 `OUTCOME`(6)으로 바뀌었고 `uri` key가 더해졌다. `MetricNaming.ALLOWED_UNITS`는 이름 접미사가 아니라 `unit` 칸의 값이다 | 그 상수를 쓰던 시험을 새 이름·값으로 고친다 | 시험 컴파일이 실패한다 | 그 fixture를 쓰는 서비스 | ## 3. 바뀌지 않은 것 -- 지표·MDC key·헤더 이름과 응답 모양은 그대로다. +- 지표·MDC key·헤더 이름은 위 8번 둘 말고 그대로다. 응답 모양도 그대로다. diff --git a/src/adapter/inbound/grpc/src/main/java/dev/caskeleton/adapter/inbound/grpc/GrpcCorrelationInterceptor.java b/src/adapter/inbound/grpc/src/main/java/dev/caskeleton/adapter/inbound/grpc/GrpcCorrelationInterceptor.java index 2100cdde..c95b49a5 100644 --- a/src/adapter/inbound/grpc/src/main/java/dev/caskeleton/adapter/inbound/grpc/GrpcCorrelationInterceptor.java +++ b/src/adapter/inbound/grpc/src/main/java/dev/caskeleton/adapter/inbound/grpc/GrpcCorrelationInterceptor.java @@ -14,7 +14,11 @@ import java.util.function.Supplier; import org.slf4j.MDC; /** - * Puts a call's correlation on MDC under the same keys the HTTP path uses (review X2-P2-5). + * Puts a call's correlation on MDC under the HTTP path's key names (review X2-P2-5) — a subset of + * them: {@code request_id}, and {@code trace_id}/{@code span_id} only when the caller sent a {@code + * traceparent}. No {@code correlation_id}: a gRPC call has no session to join, and the + * per-transport table in {@code docs/registries/mdc-keys.yaml} records that (this line used to say + * "the same keys", review 2026-11-12). * *
Correlation used to stop at the servlet filter: a log line written while a gRPC call ran * carried no {@code request_id}, so it could not be joined to anything. Each call gets its own diff --git a/src/adapter/inbound/web/README.md b/src/adapter/inbound/web/README.md index 9c596966..bdc316bc 100644 --- a/src/adapter/inbound/web/README.md +++ b/src/adapter/inbound/web/README.md @@ -484,13 +484,15 @@ commit-aware response wrapper 계약을 구현한다. 또한 이 모듈은 HTML 필터는 KEYED 가 아닌 엔드포인트까지 막아 버렸다. ### ApiHeaders -- 인바운드 web 어댑터 전역의 HTTP 헤더명 상수 — `docs/registries/headers.yaml`의 - 코드 미러. 리터럴을 중앙집중해 controller/filter/advice 가 casing 으로 drift 하지 않게 하고, 레지스트리 - 일관성 테스트가 단일 출처를 참조하게 한다. -- 소유권: `X-Api-Version`(D2)·`Idempotency-Key`(D3 — 이름만; key shape/scope/replay 정책은 - application-core 소유)는 여기서 생산. conditional-request(D15)·cache(D16)·method/negotiation/ - LRO(D12/D17)·pagination(D18)·rate-limit signaling(generated_if_missing=true; Limit/Remaining 은 numeric, Reset 은 - rfc3339 = fixed-window end)·deep-offset deprecation marker(D18)·always-emitted(D24)는 표준 RFC 9110/9111 이름 참조. +- 인바운드 web 어댑터가 쓰는 HTTP 헤더명 상수. 리터럴을 한 곳에 모아 controller/filter/advice 가 casing 으로 + drift 하지 않게 한다. `docs/registries/headers.yaml` 의 web 행은 이 파일을 `source` 로 가리키고, + `HeaderRegistryRuntimeAlignmentTest` 가 그 이름 글자가 이 파일에 있고 그 상수를 쓰는 main 코드가 있는지 본다 + (`service:` 행은 서비스용 상수라 쓰는 곳을 보지 않는다). +- 템플릿 main 이 쓰는 상수: `Retry-After`, rate-limit 신호(`X-RateLimit-Limit`·`-Remaining` 은 숫자, `-Reset` 은 + rfc3339 = fixed window 끝), `Cache-Control`·`Vary`, `ETag`·`Location`(CORS 노출 목록). +- 서비스 controller 가 읽으라고 둔 상수: `Idempotency-Key`(이름만. key shape/scope/replay 정책은 application-core 소유), + `If-Match`·`If-None-Match`(조건부 요청, 비교는 `conditional.ETags`), `Allow`. `Deprecation`·`Date` 상수는 있지만 아무 + 코드도 쓰지 않는다. `X-Api-Version` 은 없다. --- diff --git a/src/adapter/outbound/fileserver/src/main/java/dev/caskeleton/adapter/outbound/fileserver/platform/observability/MicrometerFileserverMetrics.java b/src/adapter/outbound/fileserver/src/main/java/dev/caskeleton/adapter/outbound/fileserver/platform/observability/MicrometerFileserverMetrics.java index 19f38bc9..0c71f03b 100644 --- a/src/adapter/outbound/fileserver/src/main/java/dev/caskeleton/adapter/outbound/fileserver/platform/observability/MicrometerFileserverMetrics.java +++ b/src/adapter/outbound/fileserver/src/main/java/dev/caskeleton/adapter/outbound/fileserver/platform/observability/MicrometerFileserverMetrics.java @@ -103,7 +103,7 @@ public final class MicrometerFileserverMetrics implements FileserverMetricsPort @Override public void recordOffsetMismatch(UploadMode mode, String clientType) { increment( - "fileserver.upload.offset_mismatch", + "fileserver.upload.offset.mismatch", Tags.of("mode", tag(mode.name()), "client_type", tag(clientType))); } diff --git a/src/adapter/outbound/httpclient/src/main/java/dev/caskeleton/adapter/outbound/httpclient/observation/HttpClientObservationNames.java b/src/adapter/outbound/httpclient/src/main/java/dev/caskeleton/adapter/outbound/httpclient/observation/HttpClientObservationNames.java index 07b76d79..96a687c9 100644 --- a/src/adapter/outbound/httpclient/src/main/java/dev/caskeleton/adapter/outbound/httpclient/observation/HttpClientObservationNames.java +++ b/src/adapter/outbound/httpclient/src/main/java/dev/caskeleton/adapter/outbound/httpclient/observation/HttpClientObservationNames.java @@ -26,8 +26,9 @@ public final class HttpClientObservationNames { public static final String CONNECT_DURATION = "http.client.connect.duration"; public static final String TLS_DURATION = "http.client.tls.duration"; public static final String CIRCUIT_STATE = "http.client.circuit.state"; + public static final String CIRCUIT_REJECTED = "http.client.circuit.state.rejected"; public static final String BULKHEAD_REJECTED = "http.client.bulkhead.rejected"; - public static final String RATE_LIMIT_REJECTED = "http.client.rate_limit.rejected"; + public static final String RATE_LIMIT_REJECTED = "http.client.ratelimit.rejected"; public static final String OAUTH_REFRESH = "http.client.oauth.refresh"; public static final String SSRF_REJECTED = "http.client.ssrf.rejected"; diff --git a/src/adapter/outbound/httpclient/src/main/java/dev/caskeleton/adapter/outbound/httpclient/resilience/ResilienceRejectionRecorder.java b/src/adapter/outbound/httpclient/src/main/java/dev/caskeleton/adapter/outbound/httpclient/resilience/ResilienceRejectionRecorder.java index a7738acb..5938ff86 100644 --- a/src/adapter/outbound/httpclient/src/main/java/dev/caskeleton/adapter/outbound/httpclient/resilience/ResilienceRejectionRecorder.java +++ b/src/adapter/outbound/httpclient/src/main/java/dev/caskeleton/adapter/outbound/httpclient/resilience/ResilienceRejectionRecorder.java @@ -11,7 +11,7 @@ import java.util.function.Supplier; /** * Publishes the local back-pressure signals the platform already produced but never recorded. * - *
{@code http.client.rate_limit.rejected}, {@code http.client.bulkhead.rejected} and {@code + *
{@code http.client.ratelimit.rejected}, {@code http.client.bulkhead.rejected} and {@code * http.client.circuit.state} were declared in the metric vocabulary, documented in the support * matrix, and emitted by nothing. That is the worst arrangement of the three possibilities: an * operator building a dashboard finds the names, charts them, and sees a flat zero during the exact @@ -76,7 +76,7 @@ public interface ResilienceRejectionRecorder { return new ResilienceRejectionRecorder() { @Override public void circuitOpen() { - registry.counter(HttpClientObservationNames.CIRCUIT_STATE + ".rejected", tags).increment(); + registry.counter(HttpClientObservationNames.CIRCUIT_REJECTED, tags).increment(); } @Override diff --git a/src/adapter/outbound/persistence-jpa/README.md b/src/adapter/outbound/persistence-jpa/README.md index 403e60e6..a22a79a2 100644 --- a/src/adapter/outbound/persistence-jpa/README.md +++ b/src/adapter/outbound/persistence-jpa/README.md @@ -24,8 +24,8 @@ bean과 함께 Boot의 JDBC·Hibernate·Spring Data JPA·Flyway 자동 설정도 필요한 다른 capability(notification·fileserver 저장소, messaging JDBC outbox/inbox 등)와 함께 끌 수는 없다) - 포트 구현: `TransactionPort`(`SpringTransaction`), `DistributedLockPort`(단일 인스턴스는 - in-process, `APP_MULTI_INSTANCE_ENABLED=true`면 JDBC `INT_LOCK` + `MeteredDistributedLock` - 데코레이터 = `distributedLockProvider`), outbox store(`OutboxStorePort`, `APP_OUTBOX_ENABLED=true`), + in-process, `APP_MULTI_INSTANCE_ENABLED=true`면 JDBC `INT_LOCK` = `distributedLockProvider`. 둘 다 + `MeteredDistributedLock` 데코레이터 안에 있다), outbox store(`OutboxStorePort`, `APP_OUTBOX_ENABLED=true`), PostgreSQL 멱등성 store(`APP_IDEMPOTENCY_PROVIDER=postgresql`), audit·operation·live-event store. - 엔티티/리포지토리 스캔, vendor 조립(PostgreSQL 기본, H2는 개발용), Flyway 위치. - 기동 정책(`startup` 패키지, `JpaStartupAutoConfiguration`): Flyway 마이그레이션 전략 @@ -540,7 +540,8 @@ provider 선택(flag → bean → registry)의 정본은 설정 코드 `lock/Dis ### `DistributedLockPersistenceConfig` 와이어링 결정 - `jdbcDistributedLock` 빈은 일부러 `@Primary` 가 아니다. 같은 설정이 이를 metrics 데코레이터(`MeteredDistributedLock`, package-private)로 감싸 `@Primary` - `distributedLockProvider` 빈으로 노출한다. 예전에는 app-bootstrap의 `DistributedLockConfig`가 + `distributedLockProvider` 빈으로 노출한다. 단일 인스턴스의 `inProcessDistributedLock` 도 같은 데코레이터 안에 있어 + `lock.acquisition` 이 두 경로를 모두 센다. 예전에는 app-bootstrap의 `DistributedLockConfig`가 감쌌지만, JDBC 락이 있을 때만 의미가 있는 코드라 이 모듈로 옮겼다. SI 타입은 여전히 이 모듈 밖으로 보이지 않는다(SI 는 `implementation` 의존). - `DefaultLockRepository` 는 `InitializingBean`/`SmartLifecycle` 을 구현해 Spring 이 lifecycle 을 diff --git a/src/adapter/outbound/persistence-jpa/src/main/java/dev/caskeleton/adapter/outbound/persistence/lock/DistributedLockPersistenceConfig.java b/src/adapter/outbound/persistence-jpa/src/main/java/dev/caskeleton/adapter/outbound/persistence/lock/DistributedLockPersistenceConfig.java index dd242484..c730b058 100644 --- a/src/adapter/outbound/persistence-jpa/src/main/java/dev/caskeleton/adapter/outbound/persistence/lock/DistributedLockPersistenceConfig.java +++ b/src/adapter/outbound/persistence-jpa/src/main/java/dev/caskeleton/adapter/outbound/persistence/lock/DistributedLockPersistenceConfig.java @@ -37,7 +37,14 @@ public class DistributedLockPersistenceConfig { return LockSettings.bind(environment); } - /** In-process adapter; active when {@code multi-instance-enabled} is {@code false} or absent. */ + /** + * In-process adapter; active when {@code multi-instance-enabled} is {@code false} or absent. + * + *
Behind the same metrics decorator as the JDBC one, so {@code lock.acquisition} counts every
+ * call on both paths. It used to be the bare adapter, and on the default single-instance path the
+ * meter stayed at zero while the registry and README said it counted every call (review
+ * 2026-11-12).
+ */
@Bean
@Primary
@ConditionalOnProperty(
@@ -45,8 +52,11 @@ public class DistributedLockPersistenceConfig {
name = "multi-instance-enabled",
havingValue = "false",
matchIfMissing = true)
- DistributedLockPort inProcessDistributedLock(LockSettings settings) {
- return new LockRegistryDistributedLock(new DefaultLockRegistry(), settings.leaseTtl());
+ DistributedLockPort inProcessDistributedLock(
+ LockSettings settings, ObjectProvider