chore: initialize from backend template 0a6dd0e

2026-08-13 20:31:02 +09:00
commit e64e701fe5
3223 changed files with 388401 additions and 0 deletions
@@ -0,0 +1,34 @@
+---
+title: Runbook — ADAPTER_DISABLED (런타임 어댑터 비활성화 호출)
+category: INTERNAL
+error_codes: [ADAPTER_DISABLED]
+severity: P2
+owner: oncall
+last_updated: 2026-06-15
+status: stub
+---
+
+# Runbook: ADAPTER_DISABLED (`runbook://adapter/adapter-disabled`)
+
+## Symptoms
+
+- HTTP 500 with `error.code=ADAPTER_DISABLED`
+- Code invoked an optional adapter (Kafka/Redis/Slack/Email) that is disabled in this deployment
+
+## Diagnosis
+
+- Check adapter name in log (`adapter_name` field)
+- Review deployment config — which optional adapters are enabled?
+
+## Action
+
+- Enable the adapter in deployment configuration (env flag)
+- Or update application logic to skip disabled-adapter paths
+
+## Escalation
+
+- Escalate to deployment team if adapter should be enabled but isn't
+
+---
+
+> **Stub**: Phase D2 — author body after domain adoption. (feature-operational-runbook-contract D9)
@@ -0,0 +1,34 @@
+---
+title: Runbook — AUTH_AUDIENCE_MISMATCH (대상 불일치)
+category: AUTH
+error_codes: [AUTH_AUDIENCE_MISMATCH]
+severity: P2
+owner: oncall
+last_updated: 2026-06-15
+status: stub
+---
+
+# Runbook: AUTH_AUDIENCE_MISMATCH (`runbook://auth/audience-mismatch`)
+
+## Symptoms
+
+- HTTP 401 with `error.code=AUTH_AUDIENCE_MISMATCH`
+- Token `aud` claim does not include this service's expected audience
+
+## Diagnosis
+
+- Check token `aud` claim value
+- Compare against configured `spring.security.oauth2.resourceserver.jwt.audiences`
+
+## Action
+
+- Verify client is requesting tokens scoped to the correct audience
+- Update audience configuration if service identifier changed
+
+## Escalation
+
+- Escalate to auth-platform team if misconfiguration is system-wide
+
+---
+
+> **Stub**: Phase D2 — author body after domain adoption. (feature-operational-runbook-contract D9)
@@ -0,0 +1,34 @@
+---
+title: Runbook — AUTH_CLAIM_MAPPING_FAILED (클레임 매핑 실패)
+category: AUTH
+error_codes: [AUTH_CLAIM_MAPPING_FAILED]
+severity: P2
+owner: oncall
+last_updated: 2026-06-15
+status: stub
+---
+
+# Runbook: AUTH_CLAIM_MAPPING_FAILED (`runbook://auth/claim-mapping-failed`)
+
+## Symptoms
+
+- HTTP 401 with `error.code=AUTH_CLAIM_MAPPING_FAILED`
+- Token validated but required claims (sub, roles, tenant) missing or unexpected type
+
+## Diagnosis
+
+- Inspect token payload claims via logs
+- Check claim extractor configuration
+
+## Action
+
+- Verify IdP token template includes required claims
+- Update claim mapping configuration if IdP schema changed
+
+## Escalation
+
+- Escalate to auth-platform team if IdP changed claim schema
+
+---
+
+> **Stub**: Phase D2 — author body after domain adoption. (feature-operational-runbook-contract D9)
@@ -0,0 +1,34 @@
+---
+title: Runbook — AUTH_ISSUER_MISMATCH (발급자 불일치)
+category: AUTH
+error_codes: [AUTH_ISSUER_MISMATCH]
+severity: P2
+owner: oncall
+last_updated: 2026-06-15
+status: stub
+---
+
+# Runbook: AUTH_ISSUER_MISMATCH (`runbook://auth/issuer-mismatch`)
+
+## Symptoms
+
+- HTTP 401 with `error.code=AUTH_ISSUER_MISMATCH`
+- Token `iss` claim does not match configured expected issuer
+
+## Diagnosis
+
+- Compare token `iss` against `spring.security.oauth2.resourceserver.jwt.issuer-uri`
+- Check if IdP environment changed
+
+## Action
+
+- Update issuer config if IdP migrated
+- Reject tokens from unexpected issuers
+
+## Escalation
+
+- Escalate to platform-security if unexpected issuer detected
+
+---
+
+> **Stub**: Phase D2 — author body after domain adoption. (feature-operational-runbook-contract D9)
@@ -0,0 +1,34 @@
+---
+title: Runbook — AUTH_JWKS_UNAVAILABLE (JWKS 엔드포인트 장애)
+category: TRANSIENT_DEPENDENCY
+error_codes: [AUTH_JWKS_UNAVAILABLE]
+severity: P1
+owner: oncall
+last_updated: 2026-06-15
+status: stub
+---
+
+# Runbook: AUTH_JWKS_UNAVAILABLE (`runbook://auth/jwks-unavailable`)
+
+## Symptoms
+
+- HTTP 503 with `error.code=AUTH_JWKS_UNAVAILABLE`
+- All authentication failing; JWKS refresh attempts failing
+
+## Diagnosis
+
+- Check IdP JWKS endpoint health: `curl -sf https://<idp-host>/.well-known/jwks.json`
+- Check network connectivity from app pods to IdP
+
+## Action
+
+- Enable cached JWKS fallback if available
+- Coordinate with IdP team for restoration
+
+## Escalation
+
+- P1 page: IdP team immediately if JWKS endpoint unreachable > 2 minutes
+
+---
+
+> **Stub**: Phase D2 — author body after domain adoption. (feature-operational-runbook-contract D9)
@@ -0,0 +1,34 @@
+---
+title: Runbook — AUTH_KID_UNKNOWN (키 ID 미인식)
+category: AUTH
+error_codes: [AUTH_KID_UNKNOWN]
+severity: P2
+owner: oncall
+last_updated: 2026-06-15
+status: stub
+---
+
+# Runbook: AUTH_KID_UNKNOWN (`runbook://auth/kid-unknown`)
+
+## Symptoms
+
+- HTTP 401 with `error.code=AUTH_KID_UNKNOWN`, `retryable=true`
+- Token `kid` header not present in cached JWKS
+
+## Diagnosis
+
+- Check if IdP key rotation occurred recently
+- Verify JWKS cache TTL and refresh timing
+
+## Action
+
+- Force JWKS cache refresh
+- Confirm new key is published in IdP JWKS endpoint
+
+## Escalation
+
+- Escalate to IdP team if new kid not appearing in JWKS after 10 minutes
+
+---
+
+> **Stub**: Phase D2 — author body after domain adoption. (feature-operational-runbook-contract D9)
@@ -0,0 +1,34 @@
+---
+title: Runbook — INTERNAL_AUTH_MISCONFIGURATION (공개 경로 설정 오류)
+category: INTERNAL
+error_codes: [INTERNAL_AUTH_MISCONFIGURATION]
+severity: P1
+owner: oncall
+last_updated: 2026-06-15
+status: stub
+---
+
+# Runbook: INTERNAL_AUTH_MISCONFIGURATION (`runbook://auth/public-path-misconfiguration`)
+
+## Symptoms
+
+- HTTP 500 with `error.code=INTERNAL_AUTH_MISCONFIGURATION`
+- Security filter misconfiguration detected at runtime
+
+## Diagnosis
+
+- Check `verifyPublicPathSnapshot` output in CI
+- Review recent changes to `SecurityConfig` or `application.yml` public path list
+
+## Action
+
+- Revert misconfigured public path change
+- Run `./gradlew verifyPublicPathSnapshot` to compare snapshot
+
+## Escalation
+
+- P1 immediate: if auth bypass is possible due to misconfiguration
+
+---
+
+> **Stub**: Phase D2 — author body after domain adoption. (feature-operational-runbook-contract D9)
@@ -0,0 +1,34 @@
+---
+title: Runbook — AUTH_TOKEN_EXPIRED (토큰 만료)
+category: AUTH
+error_codes: [AUTH_TOKEN_EXPIRED]
+severity: P3
+owner: oncall
+last_updated: 2026-06-15
+status: stub
+---
+
+# Runbook: AUTH_TOKEN_EXPIRED (`runbook://auth/token-expired`)
+
+## Symptoms
+
+- HTTP 401 with `error.code=AUTH_TOKEN_EXPIRED`
+- Spike may indicate clock skew or long-lived token usage
+
+## Diagnosis
+
+- Check `exp` claim vs server clock
+- Check NTP sync on token-issuing host
+
+## Action
+
+- Client must refresh tokens before expiry
+- Verify clock skew tolerance is configured (default 60s)
+
+## Escalation
+
+- Escalate if spike is widespread or clock drift is confirmed
+
+---
+
+> **Stub**: Phase D2 — author body after domain adoption. (feature-operational-runbook-contract D9)
@@ -0,0 +1,34 @@
+---
+title: Runbook — AUTH_TOKEN_INVALID_SIGNATURE (서명 검증 실패)
+category: AUTH
+error_codes: [AUTH_TOKEN_INVALID_SIGNATURE]
+severity: P1
+owner: oncall
+last_updated: 2026-06-15
+status: stub
+---
+
+# Runbook: AUTH_TOKEN_INVALID_SIGNATURE (`runbook://auth/token-invalid-signature`)
+
+## Symptoms
+
+- HTTP 401 with `error.code=AUTH_TOKEN_INVALID_SIGNATURE`
+- `log_level=ERROR` — may indicate forged tokens or wrong signing key
+
+## Diagnosis
+
+- Check if JWKS endpoint returned a new key set
+- Check for token forgery attempts in logs
+
+## Action
+
+- Verify JWKS key IDs match token headers
+- Alert security team if forgery suspected
+
+## Escalation
+
+- Immediate P1 escalation if forgery indicators present
+
+---
+
+> **Stub**: Phase D2 — author body after domain adoption. (feature-operational-runbook-contract D9)
@@ -0,0 +1,34 @@
+---
+title: Runbook — AUTH_TOKEN_MALFORMED (토큰 파싱 실패)
+category: AUTH
+error_codes: [AUTH_TOKEN_MALFORMED]
+severity: P3
+owner: oncall
+last_updated: 2026-06-15
+status: stub
+---
+
+# Runbook: AUTH_TOKEN_MALFORMED (`runbook://auth/token-malformed`)
+
+## Symptoms
+
+- HTTP 401 responses with `error.code=AUTH_TOKEN_MALFORMED`
+- Token present but fails JWT parse (not 3-part, non-base64, etc.)
+
+## Diagnosis
+
+- Inspect raw Authorization header value in logs
+- Check if token generation tooling has a bug
+
+## Action
+
+- Identify source of malformed tokens
+- Fix or update client token generation
+
+## Escalation
+
+- Escalate if spike suggests infrastructure issue
+
+---
+
+> **Stub**: Phase D2 — author body after domain adoption. (feature-operational-runbook-contract D9)
@@ -0,0 +1,34 @@
+---
+title: Runbook — AUTH_TOKEN_MISSING (인증 토큰 누락)
+category: AUTH
+error_codes: [AUTH_TOKEN_MISSING]
+severity: P3
+owner: oncall
+last_updated: 2026-06-15
+status: stub
+---
+
+# Runbook: AUTH_TOKEN_MISSING (`runbook://auth/token-missing`)
+
+## Symptoms
+
+- HTTP 401 responses with `error.code=AUTH_TOKEN_MISSING`
+- Client missing Authorization header or Bearer token
+
+## Diagnosis
+
+- Check request logs for missing Authorization header
+- Verify client SDK configuration
+
+## Action
+
+- Confirm API clients are sending Authorization header
+- Check gateway/proxy configuration for header stripping
+
+## Escalation
+
+- Escalate if widespread or affecting critical workflows
+
+---
+
+> **Stub**: Phase D2 — author body after domain adoption. (feature-operational-runbook-contract D9)
@@ -0,0 +1,69 @@
+---
+title: Runbook — JWT key rotation 시 인증 실패 spike
+category: AUTH
+error_codes: [AUTH_TOKEN_EXPIRED, AUTH_KID_UNKNOWN, AUTH_JWKS_UNAVAILABLE, AUTH_TOKEN_INVALID_SIGNATURE]
+severity: P1
+owner: oncall
+last_updated: 2026-05-22
+status: stub
+---
+
+# Runbook: JWT key rotation 시 인증 실패 spike
+
+## 1. Trigger
+
+이 runbook은 다음 alert에서 발동됩니다.
+
+- alert name: `auth_401_error_rate_high` 또는 `jwks_refresh_failure_spike`
+- alert payload 필수 field: `operation`, `error.code`, `error.category`, `runbook_link`, `dependency_name`
+- 임계: 401 error rate > 5% 5분 지속 OR JWKS refresh failure count > 10건/분
+
+## 2. First Response (5분 이내)
+
+### Step 1 — 확인
+1. JWKS endpoint health check: `curl -sf https://<idp-host>/.well-known/jwks.json | jq '.keys | length'`
+2. log query에서 `error.code` 분포 확인 — `AUTH_KID_UNKNOWN` 비중이 높으면 rotation 원인 강력 시사
+3. IdP rotation schedule 확인 (직전 24h 내 rotation 이벤트가 있었는지)
+
+### Step 2 — 임시 격리
+- JWKS cache TTL을 짧게(예: 60s) 강제하여 새 kid 전파 가속
+- 새 kid가 JWKS에 publish되어 있는지 확인. 누락이면 IdP에 republish 요청
+
+## 3. Diagnosis
+
+- log query (Loki/CloudWatch): `{service="auth"} | error.category="AUTH" | dependency_name="jwks-endpoint"`
+- metric panel: `auth_jwks_cache_hit_ratio`, `auth_jwks_refresh_failure_total`, `auth_kid_unknown_total`
+- trace: 실패한 request 1건에서 `traceId` 추출 → IdP outbound span 확인
+- 가능한 원인:
+  - 새 kid가 JWKS에 publish되기 전 token 발급 → 24h overlap window 안에 있는지 확인
+  - JWKS endpoint 장애 (5xx, timeout) → IdP status page 확인
+  - 시계 skew로 인한 만료 오판 → NTP sync 상태 확인
+
+## 4. Mitigation
+
+- 단기: old kid를 임시 재허용 (rollback). overlap window를 48h로 일시 확장
+- IdP에 새 JWKS publish 재시도 요청
+- 장기: rotation 절차에 "publish → 24h 대기 → switch" 단계 강제. observability에 kid 분포 metric 추가
+
+## 5. Escalation
+
+- P2 → P1 격상 조건: 401 error rate > 20% 또는 다중 tenant에 동시 발생
+- 다음 on-call로 page: 10분 내 회복 안 되면 IdP team 또는 platform-security team page
+
+## 6. Recovery / Verification
+
+- 회복 확인 metric: `auth_401_error_rate < 1%` 5분 지속, `AUTH_KID_UNKNOWN` 건수 0
+- post-incident:
+  - rotation 절차 RCA 작성
+  - JWKS overlap window 정책 문서 업데이트
+  - kid 분포 dashboard 영구화
+
+## 7. Related
+
+- error-codes.yaml rows: `AUTH_TOKEN_EXPIRED`, `AUTH_KID_UNKNOWN`, `AUTH_JWKS_UNAVAILABLE`, `AUTH_TOKEN_INVALID_SIGNATURE`
+- metrics.yaml: `auth_jwks_cache_hit_ratio`, `auth_jwks_refresh_failure_total`
+- 관련 branch: [[feature-security-operational-baseline]]
+
+---
+
+> **Stub 상태 안내**: 이 runbook은 skeleton 단계의 stub. 도메인 도입 시 실제 IdP 종류·rotation 정책·JWKS endpoint URL·dashboard 링크로 보강 필요.
@@ -0,0 +1,72 @@
+---
+title: Runbook — cross-tenant 접근 시도 감지
+category: AUTHZ
+error_codes: [AUTHZ_INSUFFICIENT_PERMISSION, AUTHZ_TENANT_MISMATCH]
+severity: P2
+owner: oncall
+last_updated: 2026-05-22
+status: stub
+---
+
+# Runbook: cross-tenant 접근 시도 감지
+
+## 1. Trigger
+
+이 runbook은 다음 alert에서 발동됩니다.
+
+- alert name: `authz_cross_tenant_violation` 또는 `authz_403_spike`
+- alert payload 필수 field: `operation`, `error.code`, `error.category`, `principal_id_pseudonymized`, `tenant_id`, `runbook_link`
+- 임계:
+  - P2: 403 with `error.code=AUTHZ_TENANT_MISMATCH` > 10건/5분
+  - P1 격상: 동일 principal에서 3개 이상 tenant 시도 OR 5분 내 100건 초과
+
+## 2. First Response (5분 이내)
+
+### Step 1 — 확인
+1. log query로 위반 principal 식별 (pseudonymized): `error.code=AUTHZ_TENANT_MISMATCH`
+2. principal의 정상 tenant scope 확인 (IdP claim 또는 entitlement table)
+3. `CROSS_TENANT_ADMIN` capability 보유 여부 확인 — 보유자라면 false positive 가능성
+
+### Step 2 — 임시 격리
+- 명백한 위반 패턴이면 principal session 강제 만료 (token revocation list 추가)
+- security incident channel 통보 (`#sec-incident`)
+- 위반 request의 source IP / user-agent 기록
+
+## 3. Diagnosis
+
+- log query: `{service="api"} | error.category="AUTHZ" | principal_id_pseudonymized="<hash>"`
+- metric panel: `authz_denied_total{reason="tenant_mismatch"}`, `authz_principal_tenant_distribution`
+- trace: 위반 request의 `traceId`로 호출 chain 확인. token claim의 `tenant_id`와 요청 path의 `tenant_id` 비교
+- 가능한 원인:
+  - account takeover (계정 탈취) → 즉시 session revoke + 비밀번호 reset 요구
+  - client bug (잘못된 tenant id 전송) → product team에 통보
+  - 정상 admin operation 누락된 capability → entitlement 보정
+
+## 4. Mitigation
+
+- 단기: principal session revoke, source IP rate-limit 강화
+- 위반이 client bug면 client patch release 협조
+- 장기: tenant boundary 검증 layer를 controller가 아닌 repository 진입점에서 강제 ([[feature-repository-access-permission-contract]])
+
+## 5. Escalation
+
+- 다음 on-call로 page: 보안 incident channel 즉시 page. 5분 내 security on-call 응답 없으면 CISO escalation
+- legal/compliance 통보 필요 여부 판단 (개인정보 noted시)
+
+## 6. Recovery / Verification
+
+- 회복 확인 metric: `AUTHZ_TENANT_MISMATCH` 건수 정상 baseline 복귀
+- post-incident:
+  - account takeover면 forensics 수행 + audit log 보존
+  - cross-tenant 검증 unit test 추가
+  - 위반 패턴 detection rule 영구화
+
+## 7. Related
+
+- error-codes.yaml rows: `AUTHZ_INSUFFICIENT_PERMISSION`, `AUTHZ_TENANT_MISMATCH`
+- metrics.yaml: `authz_denied_total`, `authz_principal_tenant_distribution`
+- 관련 branch: [[feature-tenant-context-policy]], [[feature-repository-access-permission-contract]]
+
+---
+
+> **Stub 상태 안내**: 이 runbook은 skeleton 단계의 stub. 도메인 도입 시 실제 tenant 모델·capability 정의·security team 연락 체계로 보강 필요.
@@ -0,0 +1,34 @@
+---
+title: Runbook — AUTHZ_INSUFFICIENT_PERMISSION (권한 부족)
+category: AUTHZ
+error_codes: [AUTHZ_INSUFFICIENT_PERMISSION]
+severity: P3
+owner: oncall
+last_updated: 2026-06-15
+status: stub
+---
+
+# Runbook: AUTHZ_INSUFFICIENT_PERMISSION (`runbook://authz/insufficient-permission`)
+
+## Symptoms
+
+- HTTP 403 with `error.code=AUTHZ_INSUFFICIENT_PERMISSION`
+- Valid token but missing required role or permission
+
+## Diagnosis
+
+- Check user's assigned roles in IdP
+- Review endpoint's required permission annotation
+
+## Action
+
+- Grant correct role/permission to user
+- Verify endpoint permission requirement is correct
+
+## Escalation
+
+- Escalate to access-management team if bulk users affected
+
+---
+
+> **Stub**: Phase D2 — author body after domain adoption. (feature-operational-runbook-contract D9)
@@ -0,0 +1,34 @@
+---
+title: Runbook — AUTHZ_TENANT_MISMATCH (테넌트 cross-access 시도)
+category: AUTHZ
+error_codes: [AUTHZ_TENANT_MISMATCH]
+severity: P1
+owner: oncall
+last_updated: 2026-06-15
+status: stub
+---
+
+# Runbook: AUTHZ_TENANT_MISMATCH (`runbook://authz/tenant-mismatch`)
+
+## Symptoms
+
+- HTTP 403 with `error.code=AUTHZ_TENANT_MISMATCH`
+- `log_level=ERROR` — cross-tenant access attempt detected
+
+## Diagnosis
+
+- Extract `traceId`, check `X-Tenant-Id` vs token tenant claim
+- Determine if this is misconfigured client or intentional attack
+
+## Action
+
+- Block repeat offenders at gateway level
+- Alert security team for investigation
+
+## Escalation
+
+- P1 if confirmed malicious cross-tenant access attempt
+
+---
+
+> **Stub**: Phase D2 — author body after domain adoption. (feature-operational-runbook-contract D9)
@@ -0,0 +1,34 @@
+---
+title: Runbook — CACHE_STAMPEDE_LOCK_TIMEOUT (캐시 스탬피드 락 타임아웃)
+category: TRANSIENT_DEPENDENCY
+error_codes: [CACHE_STAMPEDE_LOCK_TIMEOUT]
+severity: P2
+owner: oncall
+last_updated: 2026-06-15
+status: stub
+---
+
+# Runbook: CACHE_STAMPEDE_LOCK_TIMEOUT (`runbook://cache/stampede-lock-timeout`)
+
+## Symptoms
+
+- HTTP 503 with `error.code=CACHE_STAMPEDE_LOCK_TIMEOUT`
+- Multiple concurrent cache misses on same key; lock contention
+
+## Diagnosis
+
+- Check cache hit ratio metrics
+- Identify cache keys with high miss rates
+
+## Action
+
+- Verify stampede lock TTL is configured appropriately
+- Pre-warm cache for high-traffic keys on startup
+
+## Escalation
+
+- Escalate if backend load spike accompanies stampede
+
+---
+
+> **Stub**: Phase D2 — author body after domain adoption. (feature-operational-runbook-contract D9)
@@ -0,0 +1,34 @@
+---
+title: Runbook — CACHE_UNAVAILABLE (캐시 연결 불가)
+category: TRANSIENT_DEPENDENCY
+error_codes: [CACHE_UNAVAILABLE]
+severity: P1
+owner: oncall
+last_updated: 2026-06-15
+status: stub
+---
+
+# Runbook: CACHE_UNAVAILABLE (`runbook://cache/unavailable`)
+
+## Symptoms
+
+- HTTP 503 with `error.code=CACHE_UNAVAILABLE`
+- Redis connection errors in logs
+
+## Diagnosis
+
+- Check Redis cluster health
+- Verify network connectivity from app to Redis
+
+## Action
+
+- Check Redis sentinel/cluster status
+- Enable cache degradation path if configured for optional caches
+
+## Escalation
+
+- P1 if required cache is down and no degradation path exists
+
+---
+
+> **Stub**: Phase D2 — author body after domain adoption. (feature-operational-runbook-contract D9)
@@ -0,0 +1,34 @@
+---
+title: Runbook — DB_DEADLOCK (데드락)
+category: CONFLICT
+error_codes: [DB_DEADLOCK]
+severity: P2
+owner: oncall
+last_updated: 2026-06-15
+status: stub
+---
+
+# Runbook: DB_DEADLOCK (`runbook://db/deadlock`)
+
+## Symptoms
+
+- HTTP 409 with `error.code=DB_DEADLOCK`
+- SQLState 40P01 in Postgres logs
+
+## Diagnosis
+
+- Check `pg_locks` and `pg_stat_activity` during deadlock
+- Identify conflicting transaction lock order
+
+## Action
+
+- Client should retry (retryable=true)
+- Fix lock ordering in code if recurring
+
+## Escalation
+
+- Escalate to DBA if deadlock rate is sustained > 1% of transactions
+
+---
+
+> **Stub**: Phase D2 — author body after domain adoption. (feature-operational-runbook-contract D9)
@@ -0,0 +1,34 @@
+---
+title: Runbook — DB_IDLE_IN_TX_TIMEOUT (트랜잭션 idle 타임아웃)
+category: TRANSIENT_DEPENDENCY
+error_codes: [DB_IDLE_IN_TX_TIMEOUT]
+severity: P2
+owner: oncall
+last_updated: 2026-06-15
+status: stub
+---
+
+# Runbook: DB_IDLE_IN_TX_TIMEOUT (`runbook://db/idle-in-tx-timeout`)
+
+## Symptoms
+
+- HTTP 503 with `error.code=DB_IDLE_IN_TX_TIMEOUT`
+- SQLState 25P03; transaction held open too long without activity
+
+## Diagnosis
+
+- Check `idle_in_transaction_session_timeout` Postgres setting
+- Look for application-level long-running transaction holders
+
+## Action
+
+- Reduce transaction scope in application code
+- Verify `spring.jpa.properties.hibernate.connection.timeout` is bounded
+
+## Escalation
+
+- Escalate to DBA if connection pool exhaustion results
+
+---
+
+> **Stub**: Phase D2 — author body after domain adoption. (feature-operational-runbook-contract D9)
@@ -0,0 +1,34 @@
+---
+title: Runbook — DB_QUERY_CANCELED (쿼리 취소)
+category: TRANSIENT_DEPENDENCY
+error_codes: [DB_QUERY_CANCELED]
+severity: P2
+owner: oncall
+last_updated: 2026-06-15
+status: stub
+---
+
+# Runbook: DB_QUERY_CANCELED (`runbook://db/query-canceled`)
+
+## Symptoms
+
+- HTTP 503 with `error.code=DB_QUERY_CANCELED`
+- SQLState 57014; query exceeds statement timeout
+
+## Diagnosis
+
+- Check `statement_timeout` in Postgres
+- Identify slow queries in `pg_stat_statements`
+
+## Action
+
+- Optimize slow query or add index
+- Adjust statement timeout if query is legitimately long
+
+## Escalation
+
+- Escalate to DBA for query optimization if recurring
+
+---
+
+> **Stub**: Phase D2 — author body after domain adoption. (feature-operational-runbook-contract D9)
@@ -0,0 +1,34 @@
+---
+title: Runbook — DB_SERIALIZATION_FAILURE (직렬화 실패)
+category: CONFLICT
+error_codes: [DB_SERIALIZATION_FAILURE]
+severity: P3
+owner: oncall
+last_updated: 2026-06-15
+status: stub
+---
+
+# Runbook: DB_SERIALIZATION_FAILURE (`runbook://db/serialization-failure`)
+
+## Symptoms
+
+- HTTP 409 with `error.code=DB_SERIALIZATION_FAILURE`
+- SQLState 40001; high concurrent transaction contention
+
+## Diagnosis
+
+- Check DB transaction isolation level
+- Identify hot rows / hot tables under high concurrency
+
+## Action
+
+- Client should retry with exponential backoff (retryable=true)
+- Optimize transaction scope if spike is sustained
+
+## Escalation
+
+- Escalate to DBA if sustained serialization failure rate > 5%
+
+---
+
+> **Stub**: Phase D2 — author body after domain adoption. (feature-operational-runbook-contract D9)
@@ -0,0 +1,34 @@
+---
+title: Runbook — DB_UNAVAILABLE (데이터베이스 연결 불가)
+category: TRANSIENT_DEPENDENCY
+error_codes: [DB_UNAVAILABLE]
+severity: P1
+owner: oncall
+last_updated: 2026-06-15
+status: stub
+---
+
+# Runbook: DB_UNAVAILABLE (`runbook://db/unavailable`)
+
+## Symptoms
+
+- HTTP 503 with `error.code=DB_UNAVAILABLE`
+- SQLState 08* connection errors in logs
+
+## Diagnosis
+
+- Check DB server health and connection pool exhaustion
+- Review network connectivity from app pods to DB
+
+## Action
+
+- Check DB primary health; failover to replica if available
+- Drain connection pool and reconnect
+
+## Escalation
+
+- P1: immediate if DB primary is down
+
+---
+
+> **Stub**: Phase D2 — author body after domain adoption. (feature-operational-runbook-contract D9)
@@ -0,0 +1,34 @@
+---
+title: Runbook — DEPENDENCY_4XX_CLIENT (업스트림 클라이언트 오류)
+category: PERMANENT_DEPENDENCY
+error_codes: [DEPENDENCY_4XX_CLIENT]
+severity: P2
+owner: oncall
+last_updated: 2026-06-15
+status: stub
+---
+
+# Runbook: DEPENDENCY_4XX_CLIENT (`runbook://dependency/4xx-client`)
+
+## Symptoms
+
+- HTTP 502 with `error.code=DEPENDENCY_4XX_CLIENT`
+- Upstream returned 401/403/400 — credential, scope, or request format issue
+
+## Diagnosis
+
+- Check upstream response body in logs for error detail
+- Verify API credentials and scopes are valid
+
+## Action
+
+- Rotate credentials if expired
+- Fix request format if API contract changed
+
+## Escalation
+
+- Escalate to upstream API owner if contract change is suspected
+
+---
+
+> **Stub**: Phase D2 — author body after domain adoption. (feature-operational-runbook-contract D9)
@@ -0,0 +1,34 @@
+---
+title: Runbook — DEPENDENCY_5XX_SERVER (업스트림 서버 오류)
+category: TRANSIENT_DEPENDENCY
+error_codes: [DEPENDENCY_5XX_SERVER]
+severity: P1
+owner: oncall
+last_updated: 2026-06-15
+status: stub
+---
+
+# Runbook: DEPENDENCY_5XX_SERVER (`runbook://dependency/5xx-server`)
+
+## Symptoms
+
+- HTTP 502 with `error.code=DEPENDENCY_5XX_SERVER`
+- Upstream returned 5xx; transient server-side failure
+
+## Diagnosis
+
+- Check `dependency_name` tag for which upstream is failing
+- Review upstream service status page
+
+## Action
+
+- Client should retry (retryable=true)
+- Monitor upstream recovery
+
+## Escalation
+
+- P1 if critical upstream is in sustained 5xx state
+
+---
+
+> **Stub**: Phase D2 — author body after domain adoption. (feature-operational-runbook-contract D9)
@@ -0,0 +1,34 @@
+---
+title: Runbook — DEPENDENCY_CIRCUIT_OPEN (서킷 브레이커 개방)
+category: TRANSIENT_DEPENDENCY
+error_codes: [DEPENDENCY_CIRCUIT_OPEN]
+severity: P2
+owner: oncall
+last_updated: 2026-06-15
+status: stub
+---
+
+# Runbook: DEPENDENCY_CIRCUIT_OPEN (`runbook://dependency/circuit-open`)
+
+## Symptoms
+
+- HTTP 503 with `error.code=DEPENDENCY_CIRCUIT_OPEN`
+- Circuit breaker (Resilience4j) in OPEN state for a dependency
+
+## Diagnosis
+
+- Check Resilience4j circuit breaker metrics for the dependency
+- Check upstream health; circuit opens after failure threshold breached
+
+## Action
+
+- Wait for circuit half-open probe (automatic after wait duration)
+- Resolve upstream issue to allow circuit to close
+
+## Escalation
+
+- P1 if circuit remains open > 5 minutes on a critical dependency
+
+---
+
+> **Stub**: Phase D2 — author body after domain adoption. (feature-operational-runbook-contract D9)
@@ -0,0 +1,34 @@
+---
+title: Runbook — DEPENDENCY_CONNECT_FAILED (외부 의존성 연결 실패)
+category: TRANSIENT_DEPENDENCY
+error_codes: [DEPENDENCY_CONNECT_FAILED]
+severity: P1
+owner: oncall
+last_updated: 2026-06-15
+status: stub
+---
+
+# Runbook: DEPENDENCY_CONNECT_FAILED (`runbook://dependency/connect-failed`)
+
+## Symptoms
+
+- HTTP 503 with `error.code=DEPENDENCY_CONNECT_FAILED`
+- TCP connection refused or network unreachable to upstream
+
+## Diagnosis
+
+- Check `dependency_name` tag for which upstream is unreachable
+- Verify network path and firewall rules
+
+## Action
+
+- Check upstream service availability
+- Verify service discovery / DNS resolution
+
+## Escalation
+
+- P1 if upstream is a critical service dependency
+
+---
+
+> **Stub**: Phase D2 — author body after domain adoption. (feature-operational-runbook-contract D9)
@@ -0,0 +1,34 @@
+---
+title: Runbook — DEPENDENCY_DNS_FAILED (DNS 조회 실패)
+category: TRANSIENT_DEPENDENCY
+error_codes: [DEPENDENCY_DNS_FAILED]
+severity: P1
+owner: oncall
+last_updated: 2026-06-15
+status: stub
+---
+
+# Runbook: DEPENDENCY_DNS_FAILED (`runbook://dependency/dns-failed`)
+
+## Symptoms
+
+- HTTP 503 with `error.code=DEPENDENCY_DNS_FAILED`
+- DNS resolution failure for upstream hostname
+
+## Diagnosis
+
+- Test DNS resolution from app pod: `nslookup <upstream-host>`
+- Check cluster DNS (CoreDNS) health
+
+## Action
+
+- Verify upstream hostname configuration
+- Check CoreDNS / cluster DNS health
+
+## Escalation
+
+- P1 if cluster DNS is degraded
+
+---
+
+> **Stub**: Phase D2 — author body after domain adoption. (feature-operational-runbook-contract D9)
@@ -0,0 +1,34 @@
+---
+title: Runbook — DEPENDENCY_TIMEOUT (외부 의존성 타임아웃)
+category: TRANSIENT_DEPENDENCY
+error_codes: [DEPENDENCY_TIMEOUT]
+severity: P1
+owner: oncall
+last_updated: 2026-06-15
+status: stub
+---
+
+# Runbook: DEPENDENCY_TIMEOUT (`runbook://dependency/timeout`)
+
+## Symptoms
+
+- HTTP 504 with `error.code=DEPENDENCY_TIMEOUT`
+- Upstream service did not respond within configured timeout (default: global 10s)
+
+## Diagnosis
+
+- Check `dependency_name` in log for which upstream is timing out
+- Review upstream service latency metrics
+
+## Action
+
+- Check upstream service health
+- Verify timeout settings match SLA expectations
+
+## Escalation
+
+- P1 if critical upstream is timing out at scale
+
+---
+
+> **Stub**: Phase D2 — author body after domain adoption. (feature-operational-runbook-contract D9)
@@ -0,0 +1,81 @@
+---
+title: Runbook — 외부 의존성 unavailable
+category: TRANSIENT_DEPENDENCY
+error_codes: [DEPENDENCY_TIMEOUT, DEPENDENCY_CONNECT_FAILED, DEPENDENCY_DNS_FAILED, DEPENDENCY_CIRCUIT_OPEN, DEPENDENCY_5XX_SERVER, CACHE_UNAVAILABLE, DB_UNAVAILABLE]
+severity: P1
+owner: oncall
+last_updated: 2026-05-22
+status: stub
+---
+
+# Runbook: 외부 의존성 unavailable
+
+## 1. Trigger
+
+이 runbook은 다음 alert에서 발동됩니다.
+
+- alert name: `dependency_error_rate_critical` 또는 `circuit_breaker_open`
+- alert payload 필수 field: `operation`, `error.code`, `error.category`, `dependency_name`, `dependency_kind`(required|optional), `runbook_link`
+- 임계:
+  - P1: required dependency의 error rate > 50% 1분 OR circuit_open state 활성
+  - P2: optional dependency degraded (fail-open으로 동작 중)
+
+## 2. First Response (5분 이내)
+
+### Step 1 — 확인
+1. `dependency_name` 별 status page 확인 (외부 SaaS면 vendor status, internal이면 해당 service dashboard)
+2. log query로 실패 패턴 확인: timeout / connect / DNS / 5xx 중 어떤 모드인지
+3. runtime-health Dependency Matrix에서 required vs optional 분류 확인
+4. circuit breaker state 확인 (Resilience4j metric)
+
+### Step 2 — 임시 격리
+- required dep이면 readiness probe로 traffic 차단 (회복 대기) — cascade failure 방지
+- optional dep이면 fail-open with degraded mode 확인. degraded banner를 client에 노출
+- DNS failure면 resolver/coredns 상태 확인. cache 강제 flush 검토
+
+## 3. Diagnosis
+
+- log query: `{service="app"} | dependency_name="<name>" | stats count by error.code`
+- metric panel:
+  - `resilience4j_circuitbreaker_state{name="<name>"}`
+  - `resilience4j_retry_calls_total{kind="failed_without_retry"}`
+  - `hikaricp_connections_active`, `hikaricp_connections_pending` (DB_UNAVAILABLE)
+  - `http_client_requests_seconds_count{outcome="SERVER_ERROR"}`
+- trace: 실패 request의 outbound span에서 timeout/connect/DNS 분류, target endpoint 확인
+- 가능한 원인:
+  - vendor outage → status page 확인, 회복 대기
+  - 네트워크 문제 (DNS, security group, NAT) → infra team 확인
+  - connection pool 고갈 (Hikari) → pool size/timeout 점검
+  - circuit breaker open 후 half-open 전환 실패 → 수동 reset 검토
+  - retry-storm으로 인한 self-DoS → retry budget 축소
+
+## 4. Mitigation
+
+- 단기: required면 회복 대기 + traffic 차단, optional이면 degraded mode로 유지
+- pool 고갈이면 일시 pool size 상향 + leak detection 활성화
+- circuit이 stuck이면 수동 reset (`actuator/circuitbreakerevents`)
+- 장기: retry budget·timeout·circuit 임계 재조정, fallback path 보강, vendor SLA 재협상
+
+## 5. Escalation
+
+- 다음 on-call로 page: required dep 5분 내 회복 안 되면 외부 dep team 또는 vendor에 page
+- 다중 dep 동시 장애면 incident commander 호출 (네트워크 전반 문제 의심)
+
+## 6. Recovery / Verification
+
+- 회복 확인 metric: dependency error rate < 1% 5분 지속, circuit_breaker_state = CLOSED, pool utilization 정상
+- post-incident:
+  - vendor postmortem 요청 (외부 SaaS면)
+  - timeout/retry/circuit 설정 재검토
+  - degraded mode가 사용자 경험에 미친 영향 측정
+  - chaos test에 해당 시나리오 추가
+
+## 7. Related
+
+- error-codes.yaml rows: `DEPENDENCY_TIMEOUT`, `DEPENDENCY_CONNECT_FAILED`, `DEPENDENCY_DNS_FAILED`, `DEPENDENCY_CIRCUIT_OPEN`, `DEPENDENCY_5XX_SERVER`, `CACHE_UNAVAILABLE`, `DB_UNAVAILABLE`
+- metrics.yaml: `resilience4j_circuitbreaker_state`, `hikaricp_connections_active`, `http_client_requests_seconds_count`
+- 관련 branch: [[feature-outbound-http-client-baseline]], [[feature-persistence-failure-baseline]]
+
+---
+
+> **Stub 상태 안내**: 이 runbook은 skeleton 단계의 stub. 도메인 도입 시 실제 dependency 목록·required/optional 분류·vendor 연락 체계·circuit/timeout 임계로 보강 필요.
@@ -0,0 +1,34 @@
+---
+title: Runbook — DOWNLOAD_STREAMING_FAILURE (스트리밍 다운로드 실패)
+category: TRANSIENT_DEPENDENCY
+error_codes: [DOWNLOAD_STREAMING_FAILURE]
+severity: P2
+owner: oncall
+last_updated: 2026-06-15
+status: stub
+---
+
+# Runbook: DOWNLOAD_STREAMING_FAILURE (`runbook://file/download-streaming-failure`)
+
+## Symptoms
+
+- HTTP 503 with `error.code=DOWNLOAD_STREAMING_FAILURE`
+- Streaming response truncated; backpressure or timeout (60s / 100MB limit)
+
+## Diagnosis
+
+- Check streaming response timeout configuration
+- Review download size vs 100MB limit
+
+## Action
+
+- Verify storage backend is reachable
+- Check for network congestion on download path
+
+## Escalation
+
+- Escalate to infra if storage backend is degraded
+
+---
+
+> **Stub**: Phase D2 — author body after domain adoption. (feature-operational-runbook-contract D9)
@@ -0,0 +1,74 @@
+---
+title: Runbook — 5xx Internal error spike
+category: INTERNAL
+error_codes: [INTERNAL_ERROR, INTERNAL_AUTH_MISCONFIGURATION, JVM_OOM]
+severity: P1
+owner: oncall
+last_updated: 2026-05-22
+status: stub
+---
+
+# Runbook: 5xx Internal error spike
+
+## 1. Trigger
+
+이 runbook은 다음 alert에서 발동됩니다.
+
+- alert name: `http_5xx_error_rate_critical`
+- alert payload 필수 field: `operation`, `error.code`, `error.category`, `request_id`, `traceId`, `runbook_link`
+- 임계: 5xx error rate > 5% 5분 지속 OR > 10% 1분
+
+## 2. First Response (5분 이내)
+
+### Step 1 — 확인
+1. 가장 최근 deploy 시각 확인 (CI/CD dashboard, artifact registry digest)
+2. JVM metric 확인: heap usage, GC pause, CPU, thread count
+3. log에서 실패 request 1건 추출 → `request_id`, `traceId` 확보
+4. error.code 분포 확인: `INTERNAL_ERROR` vs `JVM_OOM` vs `INTERNAL_AUTH_MISCONFIGURATION`
+
+### Step 2 — 임시 격리
+- 직전 deploy가 의심되면 즉시 rollback (artifact registry에서 직전 image digest pin)
+- OOM 패턴이면 affected pod evict → ASG/HPA로 replacement 유도
+- LB에서 unhealthy pod 격리 (readiness probe failure 유도)
+
+## 3. Diagnosis
+
+- log query: `{service="app"} | http.status>=500 | stats count by error.code`
+- metric panel: `jvm_memory_used_bytes{area="heap"}`, `jvm_gc_pause_seconds`, `process_cpu_seconds_total`, `http_server_requests_seconds_count{status=~"5.."}`
+- trace: 실패 request의 `traceId`로 span chain 확인 → stack trace에서 root exception 추출
+- heap dump 위치: `/var/tmp/heap/heapdump-<pid>.hprof` (JVM ergonomics: `-XX:MaxRAMPercentage=75 -XX:+HeapDumpOnOutOfMemoryError`)
+- 가능한 원인:
+  - 직전 deploy의 회귀 버그 → rollback
+  - JVM OOM (메모리 leak 또는 부하 증가) → heap dump 분석
+  - 외부 의존성 설정 오류 (`INTERNAL_AUTH_MISCONFIGURATION`) → config secret 확인
+  - thread starvation (pool 고갈) → thread dump (`jstack <pid>`)
+
+## 4. Mitigation
+
+- 단기: 직전 deploy rollback, OOM pod replacement, traffic 일시 감소(scale-out 또는 rate-limit 강화)
+- config 오류면 secret/configmap rollback
+- 장기: heap dump 기반 leak 수정, capacity planning 재검토
+
+## 5. Escalation
+
+- 다음 on-call로 page: 10분 내 회복 안 되면 incident commander 호출, severity 1 incident 선언
+- 데이터 손상 의심되면 DBA team page
+
+## 6. Recovery / Verification
+
+- 회복 확인 metric: 5xx rate < 0.5% 5분 지속, JVM heap usage < 70%, GC pause p99 < 500ms
+- post-incident:
+  - rollback 원인 RCA 작성 (배포 게이트 강화 필요 여부)
+  - heap dump 분석 결과 공유
+  - JVM ergonomics(`-XX:MaxRAMPercentage`) 재검토
+  - rollback 자동화 절차 점검
+
+## 7. Related
+
+- error-codes.yaml rows: `INTERNAL_ERROR`, `INTERNAL_AUTH_MISCONFIGURATION`, `JVM_OOM`
+- metrics.yaml: `jvm_memory_used_bytes`, `jvm_gc_pause_seconds`, `http_server_requests_seconds_count`
+- 관련 branch: [[feature-operational-error-observability-foundation]], [[feature-container-runtime-contract]]
+
+---
+
+> **Stub 상태 안내**: 이 runbook은 skeleton 단계의 stub. 도메인 도입 시 실제 deploy 파이프라인·heap dump 보관 경로·rollback 자동화 명령으로 보강 필요.
@@ -0,0 +1,67 @@
+---
+title: Runbook — background job dead letter
+category: INTERNAL
+error_codes: [JOB_DEAD_LETTER]
+severity: P1
+owner: oncall
+last_updated: 2026-06-13
+status: stub
+---
+
+# Runbook: background job dead letter (`runbook://job/dead-letter`)
+
+## 1. Trigger
+
+이 runbook은 다음 alert에서 발동됩니다.
+
+- alert name: `job_dead_letter`
+- alert payload 필수 field: `error.code=JOB_DEAD_LETTER`, `job_name`, `correlation_id`, `runbook_link`
+- 임계: `job.dlq.total` > 0 for 5m (p1) — retry 소진 후 DLQ 진입은 자동 회복이 없으므로 점검 대상
+
+## 2. First Response (5분 이내)
+
+### Step 1 — 확인
+1. ERROR log에서 `JOB_DEAD_LETTER` 라인 확인: `job_name`, 최종 실패 원인 예외, `correlation_id` 추출
+2. `job.retry.total{outcome=EXHAUSTED}` 추이로 DLQ 유입 규모 파악
+3. DLQ 적재 위치(향후 retry carrier 확정 시 DB 테이블/큐) 확인 — 현재 skeleton은 vocabulary 단계
+
+### Step 2 — 임시 격리
+- DLQ는 max attempts(3) 소진의 최종 상태 — 자동 재시도 없음, 수동 개입 필수
+- 비즈니스 크리티컬 job이면 §4의 수동 처분(재처리 또는 폐기)을 우선 수행
+
+## 3. Diagnosis
+
+- log query: `{service="app"} | error.code="JOB_DEAD_LETTER" | stats count by job_name`
+- metric panel: `job.dlq.total{job_name}`, `job.retry.total{job_name, outcome}`
+- 최종 실패 원인 분류:
+  - poison input(직렬화/계약 위반) → 입력 결함, 재처리해도 실패 — 수정 후 재처리 또는 폐기
+  - 외부 의존성 장기 outage 중 attempts 소진 → 의존성 회복 후 재처리로 해결 가능
+  - non-transient error(권한/도메인/스키마)인데 retry된 경우 → 분류기 보강 필요(WAF-REL05-C3: 즉시 DLQ가 정답)
+
+## 4. Mitigation (수동 처분 — 둘 중 하나)
+
+- **재처리 (기본)**: 원인 해소 후 해당 job을 다시 enqueue. 소비자는 멱등(idempotencyKey dedupe) 의무가 있으므로 중복 처리 안전
+- **폐기 (영구)**: 작업이 더 이상 유효하지 않으면 DLQ에서 제거. ⚠ 비즈니스 오너 승인 후에만 수행하고 incident 기록에 남김
+- 장기: poison input 재발 방지(입력 계약 테스트 보강), non-transient error는 retry 없이 즉시 DLQ로 분류
+
+## 5. Escalation
+
+- 처분 판단(재처리 vs 폐기)이 불가하면 해당 job의 비즈니스 오너에게 escalate
+- DLQ 누적이 특정 `job_name`에 집중되면 해당 job 코드 오너에게 page
+
+## 6. Recovery / Verification
+
+- 회복 확인: `job.dlq.total` 증가 멈춤, 재처리분의 소비자 dedupe 동작 확인
+- post-incident: DLQ 원인 분류 기록, 같은 원인의 재발 방지 테스트 추가
+
+## 7. Related
+
+- error-codes.yaml rows: `JOB_DEAD_LETTER` (INTERNAL, 500, retryable=false)
+- metrics.yaml: `job.dlq.total{job_name}`, `job.retry.total{job_name, outcome=DLQ}`
+- 코드: `app-bootstrap` `async/BackgroundJobMetrics`(retry/DLQ vocabulary 기록 seam — D2/D4)
+- 관련 runbook: [[job-executor-rejected]], [[job-timeout]], [[outbox-dead-letter]]
+- 관련 branch: [[feature-background-job-async-contract]] (D4 retry/DLQ vocabulary SSOT — outbox/outbound가 consume)
+
+---
+
+> **Stub 상태 안내**: 이 runbook은 skeleton 단계의 stub. retry carrier(Spring Retry / Resilience4j / 자체) 확정 후 DLQ 저장소·재처리 절차 보강 필요.
@@ -0,0 +1,71 @@
+---
+title: Runbook — async executor rejected
+category: TRANSIENT_DEPENDENCY
+error_codes: [JOB_EXECUTOR_REJECTED]
+severity: P1
+owner: oncall
+last_updated: 2026-06-13
+status: stub
+---
+
+# Runbook: async executor rejected (`runbook://job/executor-rejected`)
+
+## 1. Trigger
+
+이 runbook은 다음 alert에서 발동됩니다.
+
+- alert name: `executor_rejected`
+- alert payload 필수 field: `error.code=JOB_EXECUTOR_REJECTED`, `executor_name`, `policy`, `runbook_link`
+- 임계: `executor.rejected.total` > 0 for 1m (p1) — bounded pool이 saturation으로 task를 거부
+- 보조 신호: `executor.saturation` gauge > queue capacity의 80% for 5m (p2)
+
+## 2. First Response (5분 이내)
+
+### Step 1 — 확인
+1. ERROR log에서 `JOB_EXECUTOR_REJECTED` 라인 확인: `executor_name`, `policy=AbortPolicy`, `queue_size` 추출
+2. `executor.saturation` 패널에서 큐 점유율 추이 확인 — 일시적 burst인지 지속 saturation인지 판별
+3. 동시 유입 원인 파악: 신규 배포 / 트래픽 spike / 다운스트림 지연으로 worker가 장기 점유되는지
+
+### Step 2 — 임시 격리
+- AbortPolicy 거부는 호출부에 `RejectedExecutionException`으로 surface됨 — fire-and-forget `@Async` 호출이면 호출부의 async-exception 처리(log/metric)로 흡수됐는지 확인
+- 지속 saturation이면 유입 측(트래픽/스케줄러 빈도)을 우선 감속
+
+## 3. Diagnosis
+
+- log query: `{service="app"} | error.code="JOB_EXECUTOR_REJECTED" | stats count by executor_name`
+- metric panel: `executor.saturation{executor_name}`, `executor.rejected.total{executor_name, policy}`
+- 가능한 원인 우선순위:
+  - 다운스트림 의존성 지연 → worker가 반납되지 않아 큐 포화 (가장 흔함)
+  - 트래픽 spike → 정상 부하 한계 초과
+  - pool 과소 설정 (`APP_ASYNC_EXECUTOR_*`)
+  - non-idempotent 작업이 retry로 누적
+
+## 4. Mitigation
+
+- 단기: 유입 감속(상위 rate-limit / 스케줄러 interval 확대) 또는 다운스트림 의존성 회복
+- pool 재조정(restart-only): `APP_ASYNC_EXECUTOR_CORE_SIZE` / `APP_ASYNC_EXECUTOR_MAX_SIZE` / `APP_ASYNC_EXECUTOR_QUEUE_CAPACITY`
+  — ⚠ queue를 무한정 키우지 말 것(unbounded 금지, D7). 부하테스트로 수치 검증 후 변경
+- CallerRunsPolicy로의 전환은 use-case 차원의 명시적 결정 필요(request thread latency 침식 — TPE-JDK21-C6)
+
+## 5. Escalation
+
+- 다운스트림 의존성 장애가 근본 원인이면 해당 의존성 오너에게 escalate
+- pool 재조정으로도 saturation이 지속되면 용량 계획(capacity planning) 오너에게 page
+
+## 6. Recovery / Verification
+
+- 회복 확인: `executor.rejected.total` 증가 멈춤, `executor.saturation` < 80% 정상화
+- 거부된 작업의 재처리 경로(멱등 retry / 다음 스케줄 cycle) 정상 동작 확인
+
+## 7. Related
+
+- error-codes.yaml rows: `JOB_EXECUTOR_REJECTED` (TRANSIENT_DEPENDENCY, 503, retryable=true, retry_after 5s)
+- metrics.yaml: `executor.rejected.total{executor_name, policy}`, `executor.saturation{executor_name}`
+- 코드: `app-bootstrap` `async/AsyncExecutorConfig`(bounded executor), `async/LoggingAbortPolicy`(reject log+metric), `async/BackgroundJobMetrics`
+- env: `APP_ASYNC_EXECUTOR_CORE_SIZE` / `APP_ASYNC_EXECUTOR_MAX_SIZE` / `APP_ASYNC_EXECUTOR_QUEUE_CAPACITY`
+- 관련 runbook: [[job-timeout]], [[job-dead-letter]]
+- 관련 branch: [[feature-background-job-async-contract]] (D7 saturation policy)
+
+---
+
+> **Stub 상태 안내**: 이 runbook은 skeleton 단계의 stub. 실제 부하 프로파일·alert 채널·pool 수치 확정 시 보강 필요.
@@ -0,0 +1,69 @@
+---
+title: Runbook — background job timeout
+category: TRANSIENT_DEPENDENCY
+error_codes: [JOB_TIMEOUT]
+severity: P2
+owner: oncall
+last_updated: 2026-06-13
+status: stub
+---
+
+# Runbook: background job timeout (`runbook://job/timeout`)
+
+## 1. Trigger
+
+이 runbook은 다음 alert에서 발동됩니다.
+
+- alert name: `job_timeout`
+- alert payload 필수 field: `error.code=JOB_TIMEOUT`, `job_name`, `correlation_id`, `runbook_link`
+- 임계: `job.retry.total{outcome=RETRY}` 급증 또는 graceful-shutdown 중 in-flight job interrupt 발생
+- 연관: shutdown phase에서 19s await 초과로 interrupt된 job (D8)
+
+## 2. First Response (10분 이내)
+
+### Step 1 — 확인
+1. ERROR log에서 `JOB_TIMEOUT` 라인 확인: `job_name`, 마지막 단계, 소요 시간 추출
+2. timeout이 정상 실행 중 발생인지, graceful-shutdown(배포/스케일다운) 중 interrupt인지 구분
+3. 해당 job이 멱등(retry-on-next-cycle 안전)인지 확인 — 비멱등이면 §4에서 신중히 처리
+
+### Step 2 — 임시 격리
+- shutdown 중 interrupt면: 다음 기동 시 재시도 대상인지(멱등 전제) 확인, 중복 부작용 여부 점검
+- 정상 실행 중 timeout이면: 해당 job의 외부 의존성(DB/HTTP) 지연 여부 확인
+
+## 3. Diagnosis
+
+- log query: `{service="app"} | error.code="JOB_TIMEOUT" | stats count by job_name`
+- metric panel: `job.retry.total{job_name, outcome}`
+- 가능한 원인 우선순위:
+  - 외부 의존성(DB lock / 느린 HTTP) 지연으로 job p99 상승
+  - job 작업량 증가로 단일 cycle이 19s 예산 초과 (D8 — interrupt 노출)
+  - interrupt 미반응 blocking call(JDBC 등) → awaitTermination 초과 (K8S-POD-LC-C2 SIGKILL 경로)
+
+## 4. Mitigation
+
+- 단기: 의존성 회복 / job 입력 배치 크기 축소
+- job p99가 구조적으로 19s를 넘으면: 작업을 분할하거나, grace period 연장 검토(parent project 운영 계약 소유자 승인 필요 — OUT_OF_BRANCH_SCOPE)
+- 비멱등 job이 재시도로 중복 부작용을 내면 멱등키/dedupe 도입 우선
+
+## 5. Escalation
+
+- 의존성 지연이 근본 원인이면 해당 의존성 오너에게 escalate
+- shutdown 예산(20s) vs k8s `terminationGracePeriodSeconds`(30s) 정합 이슈면 플랫폼/런타임 오너에게 escalate
+
+## 6. Recovery / Verification
+
+- 회복 확인: `JOB_TIMEOUT` 신규 발생 멈춤, `job.retry.total{outcome=SUCCESS}` 정상 비율 회복
+- 멱등 재시도분의 부작용 중복 없음 확인
+
+## 7. Related
+
+- error-codes.yaml rows: `JOB_TIMEOUT` (TRANSIENT_DEPENDENCY, 500, retryable=true, retry_after 10s)
+- metrics.yaml: `job.retry.total{job_name, outcome}`
+- 코드: `app-bootstrap` `async/AsyncExecutorConfig`(awaitTermination 19s — D8 graceful shutdown)
+- env: `APP_SERVER_SHUTDOWN_TIMEOUT`(owner: feature-env-driven-runtime-configuration D2)
+- 관련 runbook: [[job-executor-rejected]], [[job-dead-letter]]
+- 관련 branch: [[feature-background-job-async-contract]] (D4 retry / D8 shutdown)
+
+---
+
+> **Stub 상태 안내**: 이 runbook은 skeleton 단계의 stub. 실제 retry carrier·job p99·shutdown 예산 확정 시 보강 필요.
@@ -0,0 +1,34 @@
+---
+title: Runbook — LOCK_ACQUISITION_TIMEOUT (분산 락 획득 타임아웃)
+category: CONFLICT
+error_codes: [LOCK_ACQUISITION_TIMEOUT]
+severity: P2
+owner: oncall
+last_updated: 2026-06-15
+status: stub
+---
+
+# Runbook: LOCK_ACQUISITION_TIMEOUT (`runbook://lock/acquisition-timeout`)
+
+## Symptoms
+
+- HTTP 409 with `error.code=LOCK_ACQUISITION_TIMEOUT`
+- Distributed lock wait exceeded configured timeout; high contention on a resource
+
+## Diagnosis
+
+- Check `lock.acquisition` metric for lock name and duration
+- Identify lock holders (check DB `integration_lock` table)
+
+## Action
+
+- Client should retry with backoff (retryable=true)
+- Optimize critical section holding time if lock contention is systemic
+
+## Escalation
+
+- Escalate if lock holder appears stuck (potential deadlock in distributed lock)
+
+---
+
+> **Stub**: Phase D2 — author body after domain adoption. (feature-operational-runbook-contract D9)
@@ -0,0 +1,34 @@
+---
+title: Runbook — ACTUATOR_FORBIDDEN (Actuator 접근 거부)
+category: AUTHZ
+error_codes: [ACTUATOR_FORBIDDEN]
+severity: P2
+owner: oncall
+last_updated: 2026-06-15
+status: stub
+---
+
+# Runbook: ACTUATOR_FORBIDDEN (`runbook://management/actuator-forbidden`)
+
+## Symptoms
+
+- HTTP 403 with `error.code=ACTUATOR_FORBIDDEN`
+- Attempt to access restricted actuator endpoint (env/configprops/heapdump/shutdown)
+
+## Diagnosis
+
+- Identify which actuator endpoint was accessed
+- Check caller identity (internal tooling vs external)
+
+## Action
+
+- Verify management port is not exposed externally
+- For heapdump/threaddump: follow break-glass runbook procedure
+
+## Escalation
+
+- P1 if forbidden actuator access appears to be external attack
+
+---
+
+> **Stub**: Phase D2 — author body after domain adoption. (feature-operational-runbook-contract D9)
@@ -0,0 +1,96 @@
+---
+title: Runbook — MIGRATION_FAILED (DB 마이그레이션 실패)
+category: INTERNAL
+error_codes: [MIGRATION_FAILED]
+severity: P1
+owner: oncall
+last_updated: 2026-07-29
+status: active
+---
+
+# Runbook: MIGRATION_FAILED (`runbook://migration/failed`)
+
+## Symptoms
+
+- Container exits with code 70 (migration failure exit)
+- Structured log with `error.code=MIGRATION_FAILED`, `startup.phase=migration`
+- App refuses to start (fail-fast)
+- JPA capability adapter refuses activation because its
+  `capability_schema_registry.lifecycle_state` is not `ACTIVE`
+
+## Diagnosis
+
+1. Stop rollout and keep the failed revision out of readiness. Do not route traffic to a partially
+   migrated instance.
+2. Identify the exact stream from `src/config/jpa/readiness-cards.yaml`. Each stream has an
+   independent `location` and `history-table`; do not infer ownership from a broad
+   `classpath:db/migration` scan.
+3. From a privileged migration session, capture the stream state before changing anything:
+
+   ```sql
+   select installed_rank, version, description, success
+   from <owned_history_table>
+   order by installed_rank;
+
+   select capability_id, installation_origin, core_epoch, feature_revision, lifecycle_state
+   from capability_schema_registry
+   where capability_id = '<card-id>';
+   ```
+
+4. Check whether any owned relation was created without a successful history entry. Compare only
+   against the owned tables in the reviewed migration; do not drop unrelated relations.
+5. Classify the failure:
+   - lock/statement timeout: remove the blocker or reduce rollout concurrency, then rerun;
+   - SQL/data precondition: create a new forward migration that makes the precondition explicit;
+   - checksum mismatch: compare the deployed artifact with the already applied script before
+     considering repair;
+   - connection/TLS failure: fix transport or credentials without changing Flyway history.
+
+## Action
+
+1. Prefer forward recovery. Fix the environmental blocker or add a new immutable migration, then
+   rerun the same owned stream with its exact history table.
+2. For an optional stream that never installed successfully, keep the capability marker absent and
+   the runtime adapter disabled until migration succeeds.
+3. After a successful migration, validate:
+   - the history contains only successful expected versions;
+   - `core_epoch` and `feature_revision` match the readiness registry;
+   - the marker is `INSTALLED_INACTIVE`;
+   - owned objects and constraints exist.
+4. Change the marker to `ACTIVE` only after the compatible application revision is deployed and its
+   readiness check succeeds. Disabling or rolling back application code changes the marker to
+   `INSTALLED_INACTIVE`; it does not drop history or owned data.
+5. Re-run the candidate evidence task before promoting:
+
+   ```bash
+   cd src
+   ./gradlew :adapter:outbound:persistence-jpa:verifyJpaCandidateEvidence --console=plain
+   ```
+
+6. Record the failed revision, stream/history table, root cause, recovery migration, elapsed time
+   and verification artifact in the incident.
+
+Do not:
+
+- edit an already applied migration;
+- delete or rewrite Flyway history to make validation green;
+- run `flyway repair` before checksum provenance is proven and reviewed;
+- use `clean`, destructive rollback, or schema-wide restore as the first response;
+- mark a capability `ACTIVE` before its migration and adapter readiness succeed.
+
+If commit outcome was indeterminate during the failure, reconcile by the application
+`OperationId`/idempotency reference before retrying business work. Never blind-retry a commit whose
+result is unknown.
+
+## Escalation
+
+- P1 immediate: the required application revision cannot become ready.
+- Escalate to the database owner before Flyway history repair, destructive DDL, point-in-time
+  recovery, or primary failover.
+- R3 restore/PITR and failover rehearsal requires a target-like backup topology; local
+  Testcontainers evidence is not a substitute.
+
+---
+
+This runbook is forward-only. The reviewed migration artifact and the per-card evidence manifest
+are the audit sources.
@@ -0,0 +1,73 @@
+---
+title: Runbook — outbox dead letter
+category: INTERNAL
+error_codes: [OUTBOX_DEAD_LETTER]
+severity: P1
+owner: oncall
+last_updated: 2026-06-11
+status: stub
+---
+
+# Runbook: outbox dead letter (`runbook://outbox/dead-letter`)
+
+## 1. Trigger
+
+이 runbook은 다음 alert에서 발동됩니다.
+
+- alert name: `outbox_dead_letter`
+- alert payload 필수 field: `error.code=OUTBOX_DEAD_LETTER`, `event_type`, `event_id`, `correlation_id`, `runbook_link`
+- 임계: `outbox.publisher.published.total{outcome=DEAD}` > 0 (DEAD 전이는 자동 회복이 없으므로 단건도 점검 대상)
+
+## 2. First Response (5분 이내)
+
+### Step 1 — 확인
+1. ERROR log에서 `OUTBOX_DEAD_LETTER` 라인 확인: `event_id`, `event_type`, `correlation_id`, 마지막 실패 원인 예외 추출
+2. DB에서 DEAD row 확인: `SELECT * FROM outbox_event WHERE status = 'DEAD' ORDER BY occurred_at;`
+3. **차단 영향 파악 (중요)**: strict per-aggregate FIFO 정책상 DEAD row는 같은 `aggregate_id`의 후행 이벤트를 계속 차단함 —
+   `SELECT count(*) FROM outbox_event b WHERE b.status <> 'PUBLISHED' AND EXISTS (SELECT 1 FROM outbox_event d WHERE d.status='DEAD' AND d.aggregate_id=b.aggregate_id AND d.occurred_at < b.occurred_at);`
+
+### Step 2 — 임시 격리
+- DEAD는 max attempts(3) 소진의 최종 상태 — 자동 재시도 없음, 수동 개입 필수
+- 차단된 aggregate가 비즈니스 크리티컬하면 아래 §4의 수동 처분(재발행 또는 skip)을 우선 수행
+
+## 3. Diagnosis
+
+- log query: `{service="app"} | error.code="OUTBOX_DEAD_LETTER" | stats count by event_type`
+- 마지막 실패 원인 분류:
+  - poison event (payload 직렬화/계약 위반) → payload 자체 결함, 재발행해도 실패 — 수정 후 재발행 또는 skip
+  - broker 장기 outage 중 attempts 소진 → broker 회복 후 재발행으로 해결 가능
+  - 구성 오류 (Kafka disabled 상태에서 producer 활성) → 구성 수정 후 재발행
+- 가능한 원인 우선순위: 구성 오류 > broker outage > poison payload
+
+## 4. Mitigation (수동 처분 — 둘 중 하나)
+
+- **재발행 (기본)**: 원인 해소 후 해당 row를 다시 claim 가능 상태로 되돌림 —
+  `UPDATE outbox_event SET status = 'PENDING', attempt_count = 0, next_attempt_at = now() WHERE event_id = '<id>' AND status = 'DEAD';`
+  (consumer는 at-least-once + idempotencyKey dedupe 의무가 있으므로 중복 발행은 안전)
+- **skip (영구 폐기)**: 이벤트가 더 이상 유효하지 않으면 PUBLISHED로 마킹해 FIFO 차단을 해제 —
+  `UPDATE outbox_event SET status = 'PUBLISHED' WHERE event_id = '<id>' AND status = 'DEAD';`
+  ⚠️ skip은 다운스트림에 영구 이벤트 갭을 만든다 — 비즈니스 오너 승인 후에만 수행하고 incident 기록에 남김
+- 장기: poison event 재발 방지(payload 계약 테스트 보강), DEAD 빈발 event_type의 producer 검증 강화
+
+## 5. Escalation
+
+- 처분 판단(재발행 vs skip)이 불가하면 해당 이벤트의 비즈니스 오너에게 escalate
+- DEAD 누적이 특정 event_type에 집중되면 producer 코드 오너에게 page
+
+## 6. Recovery / Verification
+
+- 회복 확인: `SELECT count(*) FROM outbox_event WHERE status='DEAD';` = 0, 차단됐던 aggregate의 후행 이벤트가 PUBLISHED로 전이
+- `outbox.publisher.lag` 정상화(< 60s), 재발행분의 consumer dedupe 동작 확인
+- post-incident: DEAD 원인 분류 기록, 같은 원인의 재발 방지 테스트 추가
+
+## 7. Related
+
+- error-codes.yaml rows: `OUTBOX_DEAD_LETTER` (INTERNAL, retryable=false)
+- metrics.yaml: `outbox.publisher.published.total{outcome=DEAD}`, `outbox.pending.size{status=DEAD}`, `outbox.publisher.lag`
+- 코드: `application-core` `PublishPendingOutboxEventsUseCase`(FAILED→DEAD 전이), `adapter-persistence` `outbox/OutboxEventJpaRepository`(FIFO 게이트 — DEAD가 후행 차단)
+- 관련 runbook: [[outbox-publish-failed]]
+- 관련 branch: [[feature-domain-event-outbox-contract]], [[feature-background-job-async-contract]] (max attempts/DLQ vocabulary SSOT)
+
+---
+
+> **Stub 상태 안내**: 이 runbook은 skeleton 단계의 stub. 실제 broker·DLQ 토픽·승인 체계 확정 시 보강 필요.
@@ -0,0 +1,77 @@
+---
+title: Runbook — outbox publish 일시 실패
+category: TRANSIENT_DEPENDENCY
+error_codes: [OUTBOX_PUBLISH_FAILED]
+severity: P2
+owner: oncall
+last_updated: 2026-06-11
+status: stub
+---
+
+# Runbook: outbox publish 일시 실패 (`runbook://outbox/publish-failed`)
+
+## 1. Trigger
+
+이 runbook은 다음 alert에서 발동됩니다.
+
+- alert name: `outbox_publish_failed_rate` 또는 `outbox_publisher_lag`
+- alert payload 필수 field: `error.code=OUTBOX_PUBLISH_FAILED`, `event_type`, `correlation_id`, `runbook_link`
+- 임계 (metrics.yaml verbatim):
+  - P2: `outbox.publisher.published.total{outcome=FAILED}` rate > 1% for 10m
+  - P2: `outbox.publisher.lag` > 60s for 10m / P1: > 300s for 5m
+  - P2: `outbox.pending.size{status=PENDING}` growing for 10m
+
+## 2. First Response (5분 이내)
+
+### Step 1 — 확인
+1. ERROR log에서 `OUTBOX_PUBLISH_FAILED` 라인 확인: `event_type`, `event_id`, `correlation_id`, `attempt_count` 추출
+2. broker(기본 Kafka adapter) 상태 확인: `APP_MESSAGING_KAFKA_ENABLED` 값과 broker endpoint 가용성
+   - Kafka disabled(default) 상태에서 outbox 이벤트가 append 되고 있으면 publish 경로가 `AdapterDisabledException`으로 전부 실패하는 구성 오류 — 이 경우 producer use case 쪽 활성화/구성을 먼저 의심
+3. `outbox.pending.size` status 분포 확인 (FAILED 누적 vs PENDING 누적)
+
+### Step 2 — 임시 격리
+- 일시 실패는 자동 backoff 재시도(30s × 2^(attempt-1) + jitter, max attempts 3)가 동작 — 즉시 수동 개입 불필요
+- broker 장기 다운이면 DEAD 전이 누적 전에 broker 회복을 우선 (max attempts 소진 시 `runbook://outbox/dead-letter`로 이관)
+- relay 자체를 멈춰야 하면 `ca-skeleton.outbox.relay-enabled=false`로 스케줄러 비활성 (이벤트는 outbox 테이블에 안전하게 보존됨 — 유실 없음)
+
+## 3. Diagnosis
+
+- log query: `{service="app"} | error.code="OUTBOX_PUBLISH_FAILED" | stats count by event_type`
+- metric panel:
+  - `outbox.publisher.published.total{outcome}` — FAILED 비율
+  - `outbox.publisher.lag{event_type}` — 최고령 미발행 이벤트 age
+  - `outbox.pending.size{status}` — 상태별 분포
+- DB 확인: `SELECT status, count(*) FROM outbox_event GROUP BY status;`
+- 가능한 원인:
+  - broker outage/네트워크 → broker 측 회복 대기
+  - Kafka adapter 미구성(enabled인데 brokers 누락은 기동 시 차단됨) / disabled 상태에서 producer 활성화
+  - poison event (직렬화 불가/payload 계약 위반) → 재시도 무의미, attempts 소진 후 DEAD로 흘러감 (의도된 동작)
+  - 동일 aggregate head 실패로 후행 이벤트가 FIFO 게이트에 차단되어 lag 증가 (strict per-aggregate FIFO — 설계 의도)
+
+## 4. Mitigation
+
+- 단기: broker 회복 후 backoff 만료 시 자동 재발행 — `outcome=PUBLISHED` 회복 확인
+- IN_FLIGHT orphan(claim 후 crash)은 in-flight-timeout(기본 PT5M) 경과 후 자동 재claim — at-least-once이므로 중복 발행 가능, consumer dedupe(idempotencyKey)가 흡수
+- 장기: `ca-skeleton.outbox.poll-interval`/`batch-size` 조정, broker 가용성 SLA 점검, 빈발 event_type의 payload 계약 검토
+
+## 5. Escalation
+
+- P1 lag(>300s 5m) 지속 + broker 회복 불가면 broker/infra 팀에 page
+- DEAD 전이가 발생하기 시작하면 `runbook://outbox/dead-letter` 절차로 이관
+
+## 6. Recovery / Verification
+
+- 회복 확인 metric: `outcome=FAILED` rate < 1% 10분 지속, `outbox.publisher.lag` < 60s, `outbox.pending.size{status=FAILED}` 감소 추세
+- post-incident: 실패 구간의 DEAD row 유무 확인, consumer 측 중복 처리량 확인(dedupe 동작 검증), backoff/attempts 상수 재평가
+
+## 7. Related
+
+- error-codes.yaml rows: `OUTBOX_PUBLISH_FAILED` (TRANSIENT_DEPENDENCY, retryable=true, retry_after 30s)
+- metrics.yaml: `outbox.publisher.published.total`, `outbox.publisher.lag`, `outbox.pending.size`
+- 코드: `application-core` `PublishPendingOutboxEventsUseCase`(상태머신), `adapter-persistence` `outbox/OutboxEventJpaRepository`(SKIP LOCKED claim + FIFO 게이트), `adapter-outbound` `messaging/outbox/KafkaOutboxMessagePublishAdapter`(fail-closed)
+- 관련 runbook: [[outbox-dead-letter]]
+- 관련 branch: [[feature-domain-event-outbox-contract]], [[feature-background-job-async-contract]] (retry/DLQ vocabulary SSOT)
+
+---
+
+> **Stub 상태 안내**: 이 runbook은 skeleton 단계의 stub. 실제 broker 채택·alert 라우팅·대시보드 링크 확정 시 보강 필요.
@@ -0,0 +1,73 @@
+---
+title: Runbook — Rate limit 초과 spike
+category: RATE_LIMIT
+error_codes: [RATE_LIMIT_EXCEEDED, IDEMPOTENT_IN_FLIGHT]
+severity: P3
+owner: oncall
+last_updated: 2026-05-22
+status: stub
+---
+
+# Runbook: Rate limit 초과 spike
+
+## 1. Trigger
+
+이 runbook은 다음 alert에서 발동됩니다.
+
+- alert name: `rate_limit_429_high`
+- alert payload 필수 field: `operation`, `error.code`, `error.category`, `rate_limit_key_type`(ip|principal|tenant), `runbook_link`
+- 임계:
+  - P3: 429 rate > 1% 10분 지속 (일상적 abuse 차단 효과 정상)
+  - P2 격상: 정상 client(known principal/tenant)에서 spike 또는 spike와 함께 5xx 동반
+
+## 2. First Response (5분 이내)
+
+### Step 1 — 확인
+1. rate-limit key 분포 확인: IP/principal/tenant 중 어디서 spike가 발생했는지
+   - log query: `error.code=RATE_LIMIT_EXCEEDED | stats count by rate_limit_key_type, rate_limit_key`
+2. top-N offending key 추출 (상위 10건)
+3. 정상 client 식별 — 알려진 partner/internal service면 P2 격상
+
+### Step 2 — 임시 격리
+- abuse traffic 패턴이면 WAF/gateway에서 IP block (geo, ASN 단위)
+- IDEMPOTENT_IN_FLIGHT 다발이면 client의 retry-storm 의심 → client에 retry-after 협조 요청
+
+## 3. Diagnosis
+
+- log query: `{service="gateway"} | error.code="RATE_LIMIT_EXCEEDED" | stats count by rate_limit_key`
+- metric panel: `gateway_rate_limit_dropped_total`, `gateway_rate_limit_bucket_utilization`
+- trace: 429 응답의 `Retry-After` 헤더 값, `rate_limit_remaining` header 확인
+- 가능한 원인:
+  - abuse / bot traffic → IP/ASN block
+  - 정상 client의 traffic 증가 (캠페인, 신규 feature) → limit 일시 상향
+  - retry-storm (client backoff 미적용) → client에 idempotency-key + exponential backoff 권고
+  - limit 설정 오류 (잘못된 정량 threshold) → config rollback
+
+## 4. Mitigation
+
+- 단기: abuse면 IP/ASN block, 정상 client면 해당 key의 limit 일시 상향(예: 2x, 1시간 TTL)
+- IDEMPOTENT_IN_FLIGHT 다발: idempotency-key 정책 점검, client 협조 요청
+- 장기: limit 정책을 tenant tier별 차등으로 재설계, abuse pattern detection 자동화
+
+## 5. Escalation
+
+- 다음 on-call로 page: 30분 내 정상 client 회복 안 되면 product team 통보
+- 정상 client에 SLO 위반 가능성 있으면 CSM/계정담당 통보
+
+## 6. Recovery / Verification
+
+- 회복 확인 metric: 429 rate < 0.5% 10분 지속, 정상 client의 success rate 정상화
+- post-incident:
+  - 일시 상향한 limit 원복 (TTL 만료 확인)
+  - abuse pattern을 detection rule에 영구 등록
+  - retry-storm이면 client SDK 가이드 보완
+
+## 7. Related
+
+- error-codes.yaml rows: `RATE_LIMIT_EXCEEDED`, `IDEMPOTENT_IN_FLIGHT`
+- metrics.yaml: `gateway_rate_limit_dropped_total`, `gateway_rate_limit_bucket_utilization`
+- 관련 branch: [[feature-rate-limit-idempotency-contract]]
+
+---
+
+> **Stub 상태 안내**: 이 runbook은 skeleton 단계의 stub. 도메인 도입 시 실제 gateway 제품(NGINX/Envoy/Kong 등)·tenant tier 정책·WAF 연동 절차로 보강 필요.
@@ -0,0 +1,229 @@
+---
+title: Runbook — Redis capability incident
+category: TRANSIENT_DEPENDENCY
+error_codes: []
+severity: P1
+owner: oncall
+last_updated: 2026-07-29
+status: active
+---
+
+# Runbook: Redis capability incident (`runbook://redis/capability-incident`)
+
+이 runbook은 Redis 전체를 하나의 상태로 취급하지 않는다. 먼저 영향받은 capability와 role을
+식별한다.
+
+| Role | Capability | 기본 안전 결정 |
+| --- | --- | --- |
+| `CACHE` | cache, cache refresh soft lease | source fallback 예산 안에서 degraded serving 허용 |
+| `COORDINATION` | edge rate limit, request-replay idempotency, efficiency lease | 새 mutation/claim을 fail closed하고 결과 불확실성을 보존 |
+| `SESSION` | Redis session | 인증을 fail open하지 않고 재인증 또는 503으로 전환 |
+
+Redis liveness 실패만으로 pod를 재시작하지 않는다. 재시작 폭주는 reconnect와 source fallback
+부하를 키울 수 있다.
+
+## Detection
+
+- readiness detail에서 affected role과 `required` 여부를 확인한다. endpoint, key, token, secret
+  reference는 detail에 포함되면 안 된다.
+- semantic reason을 구분한다: read/write failure, program ACL denial, unsupported server
+  version, program failure, command saturation, recent command failure, probe-in-progress,
+  stale observation, closed route, command unavailable. `semanticObservedAt`,
+  `semanticAgeMillis`, `semanticStale`를 함께 확인한다. PING 성공만으로 role이 ready라는 뜻은
+  아니다.
+- `evictionValidation=CONFIGURED_EXPECTATION_ONLY`와
+  `externalEvictionAttestation=INCOMPLETE`는 effective server policy가 증명되지 않았다는
+  뜻이다. 이를 정상 attestation으로 해석하지 않는다.
+- `redis.capability.operations.total`과 `redis.capability.duration.seconds`에서 affected
+  capability/role/operation의 실제 반환 outcome을 확인한다. mutation의
+  `certainty=indeterminate`는 timeout이나 연결 끊김을 미실행 증거로 바꾸지 않는다.
+- `redis.capability.admission.rejected.total`에서 `rejected_saturated`와
+  `rejected_closed`를 구분하고, `redis.capability.inflight.total`의 같은 role에 대해 현재 0이
+  아닌 state를 확인한다. in-flight gauge는 bounded command count이며 byte 수나 queue depth가
+  아니다.
+- `redis.capability.readiness.total`은 현재 상태 gauge가 아니라 exact sanitized
+  `RoleHealth` 관측 횟수다. 최신 health detail의 state/reason/requirement와 함께 해석한다.
+  optional cache의 degraded serving과 required coordination/session의 fail-closed 결정을
+  같은 availability 의미로 합치지 않는다.
+- 종료 시 `redis.capability.lifecycle.drain.total`에서 `drained`,
+  `forced_after_timeout`, `interrupted`를 구분한다. repeated close는 새 drain을 시작하거나
+  중복 관측을 만들지 않는다.
+- reconnect, cache source-load, session repository error 지표의 변화를 함께 본다.
+- Redis server 측에서는 memory/eviction, rejected clients, replication link/lag,
+  persistence error, Cluster coverage를 operator dashboard에서 확인한다.
+- `NOSCRIPT`, result-schema mismatch, ACL denial, TLS/auth failure, OOM, timeout을 서로 다른
+  incident category로 분류한다. timeout은 command 미실행 증거가 아니다.
+
+### Observability and lifecycle boundaries
+
+- 여섯 `redis.capability.*` meter의 tag는 닫힌 enum에서만 생성된다. key, subject, session id,
+  token, endpoint, exception text, script/SHA, value 같은 identity/wire material을 metric이나
+  ticket에 복사하지 않는다.
+- semantic operation 계측은 logical provider가 실제로 반환한 hit/miss/denied/conflict/
+  unavailable/indeterminate 결과를 기록한다. cache의 `stale`/`skipped`, session의
+  `tombstoned`/`absolute_expired`도 정상 hit/miss와 분리한다. meter registry, classifier,
+  monotonic ticker 장애는 command 결과나 원래 exception instance를 바꾸지 않는다.
+- route 응답이 설정된 byte/collection bound를 넘으면 동일 logical operation을
+  `unavailable`로 종료한다. GET/read-only 응답은 `not_applied`, mutation VALUE/MULTI 응답은
+  서버 실행 여부를 되돌릴 수 없으므로 `indeterminate`다. 앞선 `success` 표본과 이 실패를 두
+  operation으로 합산하지 않는다.
+- Spring 종료의 dependency order는 invalidation subscription 같은 capability dependent를 먼저
+  닫고, capability bean을 닫은 다음 canonical registry가 router admission을 닫아 in-flight를
+  bounded drain하고 마지막에 runtime을 닫는 순서다. 종료 중 새 command를 허용하거나 drain
+  timeout 뒤 무기한 기다리지 않는다.
+- 현재 composition에는 active Redis scheduler나 dormant credential-rotation coordinator가 없다.
+  존재하지 않는 lifecycle coordinator를 복구 절차에서 찾거나 수동 호출하지 않는다.
+- 이 meter와 단일-process lifecycle test는 Sentinel/Cluster failover, TLS/ACL 배포 적합성,
+  k3s multi-node, L1/L2 분산 일관성, distributed session 동작의 qualification 증거가 아니다.
+  해당 label은 별도 topology/conformance lane의 실제 증거가 있어야 한다.
+
+## Immediate mitigation
+
+1. 새 배포나 credential/program 전환 직후라면 해당 rollout을 중지한다. 이미 실행된 mutation을
+   무조건 재시도하지 않는다.
+2. optional cache만 영향을 받으면 source bulkhead와 stale/source fallback 예산을 확인한 뒤
+   degraded serving을 유지한다. source가 포화되면 cache miss를 더 많은 source 요청으로
+   증폭시키지 않는다.
+3. rate limit이 불확실하면 정책에 정의된 fail-closed 또는 bounded local-emergency만 사용한다.
+   local provider를 조용한 primary fallback으로 바꾸지 않는다.
+4. idempotency claim/complete 응답이 유실됐으면 같은 operation token으로 inspect/reconcile한다.
+   record를 삭제하거나 새 owner를 추측하지 않는다.
+5. lease 결과가 불확실하면 소유권이 있다고 가정하지 않는다. fencing 없는 efficiency lease를
+   correctness lock으로 승격하지 않는다.
+6. session repository 장애에서는 기존 요청을 인증된 것으로 간주하지 않는다. fail closed 또는
+   재인증으로 전환하고 JWT와 Redis Session filter를 동시에 활성화하지 않는다.
+
+## Diagnosis
+
+### Connectivity, TLS, ACL
+
+- 배포 설정이 올바른 role을 참조하고 TLS, hostname verification, explicit trust bundle, named ACL
+  user를 사용하는지 확인한다.
+- runtime identity로 `CONFIG`, `KEYS`, `FLUSH*`, arbitrary program deployment를 시도하지
+  않는다. Catalog digest로 닫힌 recovery 외 ACL 점검은 별도 operator/deployer identity의
+  `ACL DRYRUN` 또는 동등한 관리 절차로 수행한다.
+- runtime readiness identity에는 bounded probe namespace `~ca-health:*`, SET/GET/DEL,
+  PING/EVALSHA와 catalog recovery에 필요한 SCRIPT LOAD, 그리고 선택 capability manifest의 exact
+  command set이 필요하다. broad `~*`/`+@all`로 장애를 우회하지 않는다.
+- readiness probe는 5초 TTL의 opaque key만 사용한다. `ca-health:*` key가 5초를 넘겨 남는다면
+  cleanup/expiry 이상으로 분류하되 key나 value를 ticket/log에 복사하지 않는다.
+- 기본 semantic cadence는 minimum interval 5초, maximum staleness 15초다. refresh follower는
+  blocking하지 않는다. maximum staleness를 넘은 관측을 backend 정상으로 해석하지 말고,
+  probe 부하를 줄이기 위해 interval을 1초 미만으로 낮추지 않는다.
+- optional CACHE의 typed temporary connect/PING outage만 dormant degraded startup과
+  health-triggered reconnect를 허용한다. reconnect 후보는 full semantic qualification 뒤에만
+  설치된다. auth/TLS/material/version/ACL/schema mismatch를 transient로 재분류하거나 required
+  role에 같은 fallback을 적용하지 않는다.
+- credential rotation 중이라면 new credential 검증, traffic switch, old connection drain,
+  old credential revoke 순서를 확인한다. secret 값은 ticket, log, shell history에 복사하지 않는다.
+
+### Program or schema
+
+- checked-in program manifest digest와 배포 artifact digest를 대조한다.
+- `semantic-capability-acl-v1` contract와 Redis minimum 7.2를 확인한다. 이 프로그램은
+  Redis Lua API의 `redis.acl_check_cmd`로 선택 capability의 exact command/key 권한을
+  비변경 방식으로 검사하고 `redis.REDIS_VERSION_NUM`의 explicit >=7.2 gate를 먼저 적용한다.
+  두 API는 7.0부터 존재하지만 repository support policy minimum은 7.2다.
+- `NOSCRIPT`는 bounded `SCRIPT LOAD -> digest verify -> EVALSHA` recovery가 수행됐는지 확인한다.
+  arbitrary `EVAL`로 우회하지 않는다.
+- result-schema/key/codec future version은 장애가 아니라 호환성 위반으로 분류하고 writer rollout을
+  중지한다.
+- `BUSY` 또는 slow program이면 affected capability admission을 줄이고 isolated environment에서만
+  재현한다. shared Redis에 장시간 script를 추가 실행하지 않는다.
+
+### Memory and eviction
+
+- `CACHE` 배포와 `COORDINATION`/`SESSION` 배포가 물리적으로 분리됐는지 확인한다.
+- correctness role에서 eviction이 관측되면 P1이다. 새 write를 중지하고 record loss를 전제로
+  idempotency/session reconciliation 또는 재인증 범위를 산정한다.
+- noeviction OOM은 성공으로 변환하지 않는다. cache write는 degraded/indeterminate, coordination
+  mutation은 unavailable/indeterminate로 유지한다.
+- big key를 찾을 때 production request path에서 `KEYS`나 unbounded collection read를 사용하지
+  않는다. 승인된 operator job의 bounded `SCAN`/sampling을 사용한다.
+
+### Topology and persistence
+
+- 현재 구현 후보 card의 promotion topology는 readiness registry의 `selected-topology`가 정본이다.
+  이는 selection 또는 R2 qualification을 뜻하지 않는다. Sentinel/Cluster evidence가 없는
+  상태에서 standalone 증거를 HA 증거로 재사용하지 않는다.
+- Cluster same-slot semantic probe는 해당 hash slot owner 한 노드만 검증한다. 이를 cluster-wide
+  또는 failover target version/ACL/program 증거로 해석하지 말고, promotion 전에 모든 target을
+  별도 conformance lane으로 검증한다.
+- failover 뒤에는 in-flight mutation의 certainty, primary role, program availability, replication
+  offset/lag, persistence status를 각각 확인한다.
+- restore 후 session/idempotency/lease record를 자동으로 신뢰하지 않는다. security epoch,
+  tombstone, durable receipt/fencing high-watermark가 필요한 capability는 별도 reconciliation을
+  수행한다.
+
+### Sentinel failover
+
+1. affected role의 semantic readiness가 unavailable인지 확인하고 단순 PING success로 정상 판정하지
+   않는다. required coordination/session은 새 mutation admission을 닫는다.
+2. 세 Sentinel 중 응답 수와 같은 master에 동의한 수를 확인한다. 2-of-3 동의 전에는 임의 endpoint,
+   최초 응답 또는 DNS 추측으로 data runtime을 바꾸지 않는다.
+3. Sentinel discovery credential/CA와 Redis data credential/CA가 분리되어 있는지 확인한다.
+   장애 우회를 위해 trust-all, hostname verification off, plaintext 또는 broad ACL을 열지 않는다.
+4. election, discovered primary qualification, new runtime install, old runtime admission close/drain의
+   순서를 확인한다. old runtime을 강제로 닫아야 했다면 그 시점의 mutation을 성공/미실행으로
+   추정하지 않는다.
+5. response-only cut, timeout, disconnect가 있었던 rate/idempotency/session mutation은
+   `INDETERMINATE`를 보존한다. rate evaluation replay, 같은 idempotency/session operation token의
+   inspect/reconcile 또는 재인증을 사용하고 blind retry하지 않는다.
+6. semantic readiness 복구 전에는 traffic을 정상화하지 않는다. 복구 뒤 old primary의 replica
+   재합류, replication lag/acknowledgement, program digest, actor runtime generation을 확인한다.
+
+Sentinel은 asynchronous replication의 zero-data-loss나 strong consistency를 보장하지 않는다.
+`min-replicas-to-write`, lag bound, replica acknowledgement가 설정돼도 acknowledgement 결과가
+불명확한 mutation은 여전히 `INDETERMINATE`다.
+
+`min-replicas-to-write 1` + `min-replicas-max-lag 1`은 선택이 아니라 **필수**다. 미설정 시
+promotion 중 교체된 구 primary가 계속 `+OK`를 반환하고 그 write는 resync에서 폐기된다. 7.4
+레인 실측: 승격 후 강등까지 11초, 그 사이 **2,086건이 acknowledge된 뒤 소실**, 실패한 명령은
+1건. 클라이언트는 이를 감지할 수단이 없다 — 서버가 응답했으므로 driver·SDK·호출자 모두
+정상 성공으로 기록한다. 설정 후 동일 promotion에서 소실 1건, 나머지 2,020건은 `NOREPLICAS`로
+명시 거부됐다. 근거: `docs/redis/operations.md`, `LiveRedisSentinelPromotionTest`.
+
+### Disposable Multipass k3s qualification safety
+
+qualification lab은 host k3s incident 조치 도구가 아니다. VM exact allowlist는
+`ca-redis-lab-server`, `ca-redis-lab-agent-1`, `ca-redis-lab-agent-2`이며 전용 kubeconfig와
+`ca-redis-lab` context만 사용한다.
+
+- 시작 전 host context/API/node/CIDR/NodePort와 Multipass inventory fingerprint를 기록한다.
+- lab pod/service CIDR `10.52.0.0/16`, `10.53.0.0/16`이 host와 겹치면 생성하지 않는다.
+- default kubeconfig를 merge/overwrite하거나 host context에 write command를 실행하지 않는다.
+- cleanup은 exact 세 VM만 대상으로 한다. global `multipass purge`, wildcard delete를 사용하지
+  않는다.
+- 성공/실패 뒤 postflight fingerprint와 VM resource 0을 확인한다. local retain-on-failure가
+  명시적으로 활성화됐으면 보존 이유와 exact inventory를 기록하며 CI에서는 보존하지 않는다.
+- 이 한 물리 host의 3 VM 결과를 k3s control-plane HA, physical host/AZ failure 또는
+  multi-region 증거로 승격하지 않는다.
+
+## Recovery and verification
+
+1. affected role의 connection/auth/TLS와 `ca-health:` SET/GET/cleanup probe가 정상인지
+   확인한다. probe 잔여 key가 있으면 최대 TTL 5초 뒤 소멸하는지도 확인한다.
+2. 선택 capability의 대표 program digest/result schema, semantic ACL contract와 Redis minimum
+   version 7.2를 재확인한다.
+3. capability별 smoke를 수행한다: cache generation guarded write, rate evaluation replay,
+   idempotency same-operation inspect, lease stale-owner reject, session create/read/logout.
+4. queue saturation, indeterminate outcome, source fallback, re-auth 지표가 incident 전 범위로
+   돌아온 뒤에만 rollout을 재개한다.
+5. `CONFIGURED_EXPECTATION_ONLY`인 eviction은 operator/deployer identity의 외부 conformance
+   job 또는 서명 attestation으로 effective policy를 별도 검증한다. runtime user에 CONFIG/ACL
+   권한을 추가하지 않는다.
+6. production label을 변경하기 전 repository readiness task를 실행한다. Sentinel/Cluster task가
+   zero-evidence로 실패한다면 topology를 낮춰 표기하거나 실제 evidence를 먼저 추가한다.
+7. Sentinel qualification에서는 actual image ID/digest와 fault/election/runtime-swap/readiness
+   timeline, capability certainty, teardown 결과가 sanitizer/reconciler를 통과했는지 확인한다.
+   clean committed source와 실제 remote CI가 없으면 `implemented-candidate`,
+   `releaseQualification=NOT_CLAIMED`를 유지한다.
+
+## Escalation
+
+- `COORDINATION` 또는 `SESSION` required role이 5분 이상 unavailable이면 P1로 Redis/platform,
+  application on-call을 동시에 호출한다.
+- data loss, stale session resurrection, conflicting idempotency completion, duplicate correctness
+  side effect가 의심되면 security/business owner까지 즉시 확대한다.
+- 한 물리 host의 VM 세 개 또는 standalone container 결과를 AZ/host failure 증거로 해석하지
+  않는다. 그 증거가 필요한 release는 별도 disposable multi-node qualification을 요구한다.
@@ -0,0 +1,36 @@
+---
+title: Runbook — JVM_OOM (JVM OutOfMemoryError)
+category: INTERNAL
+error_codes: [JVM_OOM]
+severity: P1
+owner: oncall
+last_updated: 2026-06-15
+status: stub
+---
+
+# Runbook: JVM_OOM (`runbook://runtime/jvm-oom`)
+
+## Symptoms
+
+- Container exits with code 137 (ExitOnOutOfMemoryError triggered)
+- Structured log entry with `error.code=JVM_OOM` before exit
+
+## Diagnosis
+
+- Check heap dump if `-XX:HeapDumpOnOutOfMemoryError` is configured
+- Review memory usage trends before crash
+- Check for memory leaks: large cache growth, unbounded lists, session accumulation
+
+## Action
+
+- Restart container immediately (k8s will auto-restart with liveness probe)
+- If recurring: increase heap `-Xmx` or fix memory leak
+
+## Escalation
+
+- P1: immediate if multiple pods crashing simultaneously
+- Page SRE / infra team for heap analysis
+
+---
+
+> **Stub**: Phase D2 — author body after domain adoption. (feature-operational-runbook-contract D9)
@@ -0,0 +1,35 @@
+---
+title: Runbook — PROFILE_MISMATCH (프로파일 불일치)
+category: INTERNAL
+error_codes: [PROFILE_MISMATCH]
+severity: P1
+owner: oncall
+last_updated: 2026-06-15
+status: stub
+---
+
+# Runbook: PROFILE_MISMATCH (`runbook://startup/profile-mismatch`)
+
+## Symptoms
+
+- Container exits with code 71 (profile mismatch exit)
+- Structured log with `error.code=PROFILE_MISMATCH`, `startup.phase=profile-check`
+- Production profile active with local-only settings enabled
+
+## Diagnosis
+
+- Check active Spring profiles (`spring.profiles.active`)
+- Identify which local-only setting is incorrectly enabled in prod profile
+
+## Action
+
+- Remove local-only setting from production deployment config
+- Ensure prod profile does not inherit local/dev profile settings
+
+## Escalation
+
+- P1: security risk if local settings expose debug endpoints in production
+
+---
+
+> **Stub**: Phase D2 — author body after domain adoption. (feature-operational-runbook-contract D9)
@@ -0,0 +1,34 @@
+---
+title: Runbook — REQUIRED_ADAPTER_DISABLED (필수 어댑터 비활성화)
+category: INTERNAL
+error_codes: [REQUIRED_ADAPTER_DISABLED]
+severity: P1
+owner: oncall
+last_updated: 2026-06-15
+status: stub
+---
+
+# Runbook: REQUIRED_ADAPTER_DISABLED (`runbook://startup/required-adapter-disabled`)
+
+## Symptoms
+
+- Container exits with code 72 (required adapter disabled exit)
+- Structured log with `error.code=REQUIRED_ADAPTER_DISABLED`, `startup.phase=adapter-check`
+
+## Diagnosis
+
+- Identify which adapter is disabled but required
+- Check adapter enable flags in environment config
+
+## Action
+
+- Enable required adapter in deployment configuration
+- If adapter is intentionally disabled, update the required/optional designation
+
+## Escalation
+
+- P1: app cannot start; coordinate with deployment team
+
+---
+
+> **Stub**: Phase D2 — author body after domain adoption. (feature-operational-runbook-contract D9)
@@ -0,0 +1,34 @@
+---
+title: Runbook — STARTUP_VALIDATION_FAILED (환경 변수 검증 실패)
+category: INTERNAL
+error_codes: [STARTUP_VALIDATION_FAILED]
+severity: P1
+owner: oncall
+last_updated: 2026-06-15
+status: stub
+---
+
+# Runbook: STARTUP_VALIDATION_FAILED (`runbook://startup/validation-failed`)
+
+## Symptoms
+
+- Container exits with code 78 (env validation failure exit)
+- Structured log with `error.code=STARTUP_VALIDATION_FAILED`, `startup.phase=env-validation`
+
+## Diagnosis
+
+- Check which required env variable is missing or malformed
+- Review container environment and secrets injection
+
+## Action
+
+- Supply missing environment variables to deployment
+- Verify secrets are correctly mounted / injected
+
+## Escalation
+
+- P1: app cannot start; coordinate with deployment/secrets team
+
+---
+
+> **Stub**: Phase D2 — author body after domain adoption. (feature-operational-runbook-contract D9)
@@ -0,0 +1,44 @@
+---
+title: Runbook — <TITLE>
+category: <CATEGORY>
+error_codes: [<ERROR_CODE_1>, <ERROR_CODE_2>]
+severity: <P1|P2|P3>
+owner: oncall
+last_updated: <YYYY-MM-DD>
+status: <stub|active>
+---
+
+# Runbook: <TITLE> (`runbook://<area>/<scenario>`)
+
+## Symptoms
+
+- What observable signals trigger this runbook?
+- Alert name, metric thresholds, log patterns
+
+## Diagnosis
+
+- Step-by-step diagnostic commands and queries
+- Log queries (Loki/CloudWatch)
+- Metric panels to check
+- Trace investigation approach
+
+## Action
+
+- Immediate mitigation steps
+- Configuration changes
+- Manual intervention procedures
+
+## Escalation
+
+- Conditions for severity upgrade (e.g., P2 → P1)
+- Who to page and when
+- Fallback procedures if on-call cannot resolve
+
+---
+
+> **Note**: This is the canonical runbook template.
+> Copy this file, rename it to match the `runbook://area/scenario` pattern (→ `area-scenario.md`),
+> fill in the frontmatter fields, replace section bodies with operational content,
+> then set `status: active`. `LEGACY_STUB_DEBT` in `RunbookCoverageContractTest` is temporary
+> containment for existing debt only; do not add a new stub there. Complete the runbook or adopt
+> the future owned, expiring debt ledger.