init: 클린 아키텍처 백엔드

2026-07-24 14:29:36 +09:00
parent 9eed16d097
commit 821fe00c32
971 changed files with 74769 additions and 1 deletions
@@ -0,0 +1,34 @@
+---
+title: Runbook — ADAPTER_DISABLED (런타임 어댑터 비활성화 호출)
+category: INTERNAL
+error_codes: [ADAPTER_DISABLED]
+severity: P2
+owner: oncall
+last_updated: 2026-06-15
+status: stub
+---
+
+# Runbook: ADAPTER_DISABLED (`runbook://adapter/adapter-disabled`)
+
+## Symptoms
+
+- HTTP 500 with `error.code=ADAPTER_DISABLED`
+- Code invoked an optional adapter (Kafka/Redis/Slack/Email) that is disabled in this deployment
+
+## Diagnosis
+
+- Check adapter name in log (`adapter_name` field)
+- Review deployment config — which optional adapters are enabled?
+
+## Action
+
+- Enable the adapter in deployment configuration (env flag)
+- Or update application logic to skip disabled-adapter paths
+
+## Escalation
+
+- Escalate to deployment team if adapter should be enabled but isn't
+
+---
+
+> **Stub**: Phase D2 — author body after domain adoption. (feature-operational-runbook-contract D9)
@@ -0,0 +1,34 @@
+---
+title: Runbook — AUTH_AUDIENCE_MISMATCH (대상 불일치)
+category: AUTH
+error_codes: [AUTH_AUDIENCE_MISMATCH]
+severity: P2
+owner: oncall
+last_updated: 2026-06-15
+status: stub
+---
+
+# Runbook: AUTH_AUDIENCE_MISMATCH (`runbook://auth/audience-mismatch`)
+
+## Symptoms
+
+- HTTP 401 with `error.code=AUTH_AUDIENCE_MISMATCH`
+- Token `aud` claim does not include this service's expected audience
+
+## Diagnosis
+
+- Check token `aud` claim value
+- Compare against configured `spring.security.oauth2.resourceserver.jwt.audiences`
+
+## Action
+
+- Verify client is requesting tokens scoped to the correct audience
+- Update audience configuration if service identifier changed
+
+## Escalation
+
+- Escalate to auth-platform team if misconfiguration is system-wide
+
+---
+
+> **Stub**: Phase D2 — author body after domain adoption. (feature-operational-runbook-contract D9)
@@ -0,0 +1,34 @@
+---
+title: Runbook — AUTH_CLAIM_MAPPING_FAILED (클레임 매핑 실패)
+category: AUTH
+error_codes: [AUTH_CLAIM_MAPPING_FAILED]
+severity: P2
+owner: oncall
+last_updated: 2026-06-15
+status: stub
+---
+
+# Runbook: AUTH_CLAIM_MAPPING_FAILED (`runbook://auth/claim-mapping-failed`)
+
+## Symptoms
+
+- HTTP 401 with `error.code=AUTH_CLAIM_MAPPING_FAILED`
+- Token validated but required claims (sub, roles, tenant) missing or unexpected type
+
+## Diagnosis
+
+- Inspect token payload claims via logs
+- Check claim extractor configuration
+
+## Action
+
+- Verify IdP token template includes required claims
+- Update claim mapping configuration if IdP schema changed
+
+## Escalation
+
+- Escalate to auth-platform team if IdP changed claim schema
+
+---
+
+> **Stub**: Phase D2 — author body after domain adoption. (feature-operational-runbook-contract D9)
@@ -0,0 +1,34 @@
+---
+title: Runbook — AUTH_ISSUER_MISMATCH (발급자 불일치)
+category: AUTH
+error_codes: [AUTH_ISSUER_MISMATCH]
+severity: P2
+owner: oncall
+last_updated: 2026-06-15
+status: stub
+---
+
+# Runbook: AUTH_ISSUER_MISMATCH (`runbook://auth/issuer-mismatch`)
+
+## Symptoms
+
+- HTTP 401 with `error.code=AUTH_ISSUER_MISMATCH`
+- Token `iss` claim does not match configured expected issuer
+
+## Diagnosis
+
+- Compare token `iss` against `spring.security.oauth2.resourceserver.jwt.issuer-uri`
+- Check if IdP environment changed
+
+## Action
+
+- Update issuer config if IdP migrated
+- Reject tokens from unexpected issuers
+
+## Escalation
+
+- Escalate to platform-security if unexpected issuer detected
+
+---
+
+> **Stub**: Phase D2 — author body after domain adoption. (feature-operational-runbook-contract D9)
@@ -0,0 +1,34 @@
+---
+title: Runbook — AUTH_JWKS_UNAVAILABLE (JWKS 엔드포인트 장애)
+category: TRANSIENT_DEPENDENCY
+error_codes: [AUTH_JWKS_UNAVAILABLE]
+severity: P1
+owner: oncall
+last_updated: 2026-06-15
+status: stub
+---
+
+# Runbook: AUTH_JWKS_UNAVAILABLE (`runbook://auth/jwks-unavailable`)
+
+## Symptoms
+
+- HTTP 503 with `error.code=AUTH_JWKS_UNAVAILABLE`
+- All authentication failing; JWKS refresh attempts failing
+
+## Diagnosis
+
+- Check IdP JWKS endpoint health: `curl -sf https://<idp-host>/.well-known/jwks.json`
+- Check network connectivity from app pods to IdP
+
+## Action
+
+- Enable cached JWKS fallback if available
+- Coordinate with IdP team for restoration
+
+## Escalation
+
+- P1 page: IdP team immediately if JWKS endpoint unreachable > 2 minutes
+
+---
+
+> **Stub**: Phase D2 — author body after domain adoption. (feature-operational-runbook-contract D9)
@@ -0,0 +1,34 @@
+---
+title: Runbook — AUTH_KID_UNKNOWN (키 ID 미인식)
+category: AUTH
+error_codes: [AUTH_KID_UNKNOWN]
+severity: P2
+owner: oncall
+last_updated: 2026-06-15
+status: stub
+---
+
+# Runbook: AUTH_KID_UNKNOWN (`runbook://auth/kid-unknown`)
+
+## Symptoms
+
+- HTTP 401 with `error.code=AUTH_KID_UNKNOWN`, `retryable=true`
+- Token `kid` header not present in cached JWKS
+
+## Diagnosis
+
+- Check if IdP key rotation occurred recently
+- Verify JWKS cache TTL and refresh timing
+
+## Action
+
+- Force JWKS cache refresh
+- Confirm new key is published in IdP JWKS endpoint
+
+## Escalation
+
+- Escalate to IdP team if new kid not appearing in JWKS after 10 minutes
+
+---
+
+> **Stub**: Phase D2 — author body after domain adoption. (feature-operational-runbook-contract D9)
@@ -0,0 +1,34 @@
+---
+title: Runbook — INTERNAL_AUTH_MISCONFIGURATION (공개 경로 설정 오류)
+category: INTERNAL
+error_codes: [INTERNAL_AUTH_MISCONFIGURATION]
+severity: P1
+owner: oncall
+last_updated: 2026-06-15
+status: stub
+---
+
+# Runbook: INTERNAL_AUTH_MISCONFIGURATION (`runbook://auth/public-path-misconfiguration`)
+
+## Symptoms
+
+- HTTP 500 with `error.code=INTERNAL_AUTH_MISCONFIGURATION`
+- Security filter misconfiguration detected at runtime
+
+## Diagnosis
+
+- Check `verifyPublicPathSnapshot` output in CI
+- Review recent changes to `SecurityConfig` or `application.yml` public path list
+
+## Action
+
+- Revert misconfigured public path change
+- Run `./gradlew verifyPublicPathSnapshot` to compare snapshot
+
+## Escalation
+
+- P1 immediate: if auth bypass is possible due to misconfiguration
+
+---
+
+> **Stub**: Phase D2 — author body after domain adoption. (feature-operational-runbook-contract D9)
@@ -0,0 +1,34 @@
+---
+title: Runbook — AUTH_TOKEN_EXPIRED (토큰 만료)
+category: AUTH
+error_codes: [AUTH_TOKEN_EXPIRED]
+severity: P3
+owner: oncall
+last_updated: 2026-06-15
+status: stub
+---
+
+# Runbook: AUTH_TOKEN_EXPIRED (`runbook://auth/token-expired`)
+
+## Symptoms
+
+- HTTP 401 with `error.code=AUTH_TOKEN_EXPIRED`
+- Spike may indicate clock skew or long-lived token usage
+
+## Diagnosis
+
+- Check `exp` claim vs server clock
+- Check NTP sync on token-issuing host
+
+## Action
+
+- Client must refresh tokens before expiry
+- Verify clock skew tolerance is configured (default 60s)
+
+## Escalation
+
+- Escalate if spike is widespread or clock drift is confirmed
+
+---
+
+> **Stub**: Phase D2 — author body after domain adoption. (feature-operational-runbook-contract D9)
@@ -0,0 +1,34 @@
+---
+title: Runbook — AUTH_TOKEN_INVALID_SIGNATURE (서명 검증 실패)
+category: AUTH
+error_codes: [AUTH_TOKEN_INVALID_SIGNATURE]
+severity: P1
+owner: oncall
+last_updated: 2026-06-15
+status: stub
+---
+
+# Runbook: AUTH_TOKEN_INVALID_SIGNATURE (`runbook://auth/token-invalid-signature`)
+
+## Symptoms
+
+- HTTP 401 with `error.code=AUTH_TOKEN_INVALID_SIGNATURE`
+- `log_level=ERROR` — may indicate forged tokens or wrong signing key
+
+## Diagnosis
+
+- Check if JWKS endpoint returned a new key set
+- Check for token forgery attempts in logs
+
+## Action
+
+- Verify JWKS key IDs match token headers
+- Alert security team if forgery suspected
+
+## Escalation
+
+- Immediate P1 escalation if forgery indicators present
+
+---
+
+> **Stub**: Phase D2 — author body after domain adoption. (feature-operational-runbook-contract D9)
@@ -0,0 +1,34 @@
+---
+title: Runbook — AUTH_TOKEN_MALFORMED (토큰 파싱 실패)
+category: AUTH
+error_codes: [AUTH_TOKEN_MALFORMED]
+severity: P3
+owner: oncall
+last_updated: 2026-06-15
+status: stub
+---
+
+# Runbook: AUTH_TOKEN_MALFORMED (`runbook://auth/token-malformed`)
+
+## Symptoms
+
+- HTTP 401 responses with `error.code=AUTH_TOKEN_MALFORMED`
+- Token present but fails JWT parse (not 3-part, non-base64, etc.)
+
+## Diagnosis
+
+- Inspect raw Authorization header value in logs
+- Check if token generation tooling has a bug
+
+## Action
+
+- Identify source of malformed tokens
+- Fix or update client token generation
+
+## Escalation
+
+- Escalate if spike suggests infrastructure issue
+
+---
+
+> **Stub**: Phase D2 — author body after domain adoption. (feature-operational-runbook-contract D9)
@@ -0,0 +1,34 @@
+---
+title: Runbook — AUTH_TOKEN_MISSING (인증 토큰 누락)
+category: AUTH
+error_codes: [AUTH_TOKEN_MISSING]
+severity: P3
+owner: oncall
+last_updated: 2026-06-15
+status: stub
+---
+
+# Runbook: AUTH_TOKEN_MISSING (`runbook://auth/token-missing`)
+
+## Symptoms
+
+- HTTP 401 responses with `error.code=AUTH_TOKEN_MISSING`
+- Client missing Authorization header or Bearer token
+
+## Diagnosis
+
+- Check request logs for missing Authorization header
+- Verify client SDK configuration
+
+## Action
+
+- Confirm API clients are sending Authorization header
+- Check gateway/proxy configuration for header stripping
+
+## Escalation
+
+- Escalate if widespread or affecting critical workflows
+
+---
+
+> **Stub**: Phase D2 — author body after domain adoption. (feature-operational-runbook-contract D9)
@@ -0,0 +1,69 @@
+---
+title: Runbook — JWT key rotation 시 인증 실패 spike
+category: AUTH
+error_codes: [AUTH_TOKEN_EXPIRED, AUTH_KID_UNKNOWN, AUTH_JWKS_UNAVAILABLE, AUTH_TOKEN_INVALID_SIGNATURE]
+severity: P1
+owner: oncall
+last_updated: 2026-05-22
+status: stub
+---
+
+# Runbook: JWT key rotation 시 인증 실패 spike
+
+## 1. Trigger
+
+이 runbook은 다음 alert에서 발동됩니다.
+
+- alert name: `auth_401_error_rate_high` 또는 `jwks_refresh_failure_spike`
+- alert payload 필수 field: `operation`, `error.code`, `error.category`, `runbook_link`, `dependency_name`
+- 임계: 401 error rate > 5% 5분 지속 OR JWKS refresh failure count > 10건/분
+
+## 2. First Response (5분 이내)
+
+### Step 1 — 확인
+1. JWKS endpoint health check: `curl -sf https://<idp-host>/.well-known/jwks.json | jq '.keys | length'`
+2. log query에서 `error.code` 분포 확인 — `AUTH_KID_UNKNOWN` 비중이 높으면 rotation 원인 강력 시사
+3. IdP rotation schedule 확인 (직전 24h 내 rotation 이벤트가 있었는지)
+
+### Step 2 — 임시 격리
+- JWKS cache TTL을 짧게(예: 60s) 강제하여 새 kid 전파 가속
+- 새 kid가 JWKS에 publish되어 있는지 확인. 누락이면 IdP에 republish 요청
+
+## 3. Diagnosis
+
+- log query (Loki/CloudWatch): `{service="auth"} | error.category="AUTH" | dependency_name="jwks-endpoint"`
+- metric panel: `auth_jwks_cache_hit_ratio`, `auth_jwks_refresh_failure_total`, `auth_kid_unknown_total`
+- trace: 실패한 request 1건에서 `traceId` 추출 → IdP outbound span 확인
+- 가능한 원인:
+  - 새 kid가 JWKS에 publish되기 전 token 발급 → 24h overlap window 안에 있는지 확인
+  - JWKS endpoint 장애 (5xx, timeout) → IdP status page 확인
+  - 시계 skew로 인한 만료 오판 → NTP sync 상태 확인
+
+## 4. Mitigation
+
+- 단기: old kid를 임시 재허용 (rollback). overlap window를 48h로 일시 확장
+- IdP에 새 JWKS publish 재시도 요청
+- 장기: rotation 절차에 "publish → 24h 대기 → switch" 단계 강제. observability에 kid 분포 metric 추가
+
+## 5. Escalation
+
+- P2 → P1 격상 조건: 401 error rate > 20% 또는 다중 tenant에 동시 발생
+- 다음 on-call로 page: 10분 내 회복 안 되면 IdP team 또는 platform-security team page
+
+## 6. Recovery / Verification
+
+- 회복 확인 metric: `auth_401_error_rate < 1%` 5분 지속, `AUTH_KID_UNKNOWN` 건수 0
+- post-incident:
+  - rotation 절차 RCA 작성
+  - JWKS overlap window 정책 문서 업데이트
+  - kid 분포 dashboard 영구화
+
+## 7. Related
+
+- error-codes.yaml rows: `AUTH_TOKEN_EXPIRED`, `AUTH_KID_UNKNOWN`, `AUTH_JWKS_UNAVAILABLE`, `AUTH_TOKEN_INVALID_SIGNATURE`
+- metrics.yaml: `auth_jwks_cache_hit_ratio`, `auth_jwks_refresh_failure_total`
+- 관련 branch: [[feature-security-operational-baseline]]
+
+---
+
+> **Stub 상태 안내**: 이 runbook은 skeleton 단계의 stub. 도메인 도입 시 실제 IdP 종류·rotation 정책·JWKS endpoint URL·dashboard 링크로 보강 필요.
@@ -0,0 +1,72 @@
+---
+title: Runbook — cross-tenant 접근 시도 감지
+category: AUTHZ
+error_codes: [AUTHZ_INSUFFICIENT_PERMISSION, AUTHZ_TENANT_MISMATCH]
+severity: P2
+owner: oncall
+last_updated: 2026-05-22
+status: stub
+---
+
+# Runbook: cross-tenant 접근 시도 감지
+
+## 1. Trigger
+
+이 runbook은 다음 alert에서 발동됩니다.
+
+- alert name: `authz_cross_tenant_violation` 또는 `authz_403_spike`
+- alert payload 필수 field: `operation`, `error.code`, `error.category`, `principal_id_pseudonymized`, `tenant_id`, `runbook_link`
+- 임계:
+  - P2: 403 with `error.code=AUTHZ_TENANT_MISMATCH` > 10건/5분
+  - P1 격상: 동일 principal에서 3개 이상 tenant 시도 OR 5분 내 100건 초과
+
+## 2. First Response (5분 이내)
+
+### Step 1 — 확인
+1. log query로 위반 principal 식별 (pseudonymized): `error.code=AUTHZ_TENANT_MISMATCH`
+2. principal의 정상 tenant scope 확인 (IdP claim 또는 entitlement table)
+3. `CROSS_TENANT_ADMIN` capability 보유 여부 확인 — 보유자라면 false positive 가능성
+
+### Step 2 — 임시 격리
+- 명백한 위반 패턴이면 principal session 강제 만료 (token revocation list 추가)
+- security incident channel 통보 (`#sec-incident`)
+- 위반 request의 source IP / user-agent 기록
+
+## 3. Diagnosis
+
+- log query: `{service="api"} | error.category="AUTHZ" | principal_id_pseudonymized="<hash>"`
+- metric panel: `authz_denied_total{reason="tenant_mismatch"}`, `authz_principal_tenant_distribution`
+- trace: 위반 request의 `traceId`로 호출 chain 확인. token claim의 `tenant_id`와 요청 path의 `tenant_id` 비교
+- 가능한 원인:
+  - account takeover (계정 탈취) → 즉시 session revoke + 비밀번호 reset 요구
+  - client bug (잘못된 tenant id 전송) → product team에 통보
+  - 정상 admin operation 누락된 capability → entitlement 보정
+
+## 4. Mitigation
+
+- 단기: principal session revoke, source IP rate-limit 강화
+- 위반이 client bug면 client patch release 협조
+- 장기: tenant boundary 검증 layer를 controller가 아닌 repository 진입점에서 강제 ([[feature-repository-access-permission-contract]])
+
+## 5. Escalation
+
+- 다음 on-call로 page: 보안 incident channel 즉시 page. 5분 내 security on-call 응답 없으면 CISO escalation
+- legal/compliance 통보 필요 여부 판단 (개인정보 noted시)
+
+## 6. Recovery / Verification
+
+- 회복 확인 metric: `AUTHZ_TENANT_MISMATCH` 건수 정상 baseline 복귀
+- post-incident:
+  - account takeover면 forensics 수행 + audit log 보존
+  - cross-tenant 검증 unit test 추가
+  - 위반 패턴 detection rule 영구화
+
+## 7. Related
+
+- error-codes.yaml rows: `AUTHZ_INSUFFICIENT_PERMISSION`, `AUTHZ_TENANT_MISMATCH`
+- metrics.yaml: `authz_denied_total`, `authz_principal_tenant_distribution`
+- 관련 branch: [[feature-tenant-context-policy]], [[feature-repository-access-permission-contract]]
+
+---
+
+> **Stub 상태 안내**: 이 runbook은 skeleton 단계의 stub. 도메인 도입 시 실제 tenant 모델·capability 정의·security team 연락 체계로 보강 필요.
@@ -0,0 +1,34 @@
+---
+title: Runbook — AUTHZ_INSUFFICIENT_PERMISSION (권한 부족)
+category: AUTHZ
+error_codes: [AUTHZ_INSUFFICIENT_PERMISSION]
+severity: P3
+owner: oncall
+last_updated: 2026-06-15
+status: stub
+---
+
+# Runbook: AUTHZ_INSUFFICIENT_PERMISSION (`runbook://authz/insufficient-permission`)
+
+## Symptoms
+
+- HTTP 403 with `error.code=AUTHZ_INSUFFICIENT_PERMISSION`
+- Valid token but missing required role or permission
+
+## Diagnosis
+
+- Check user's assigned roles in IdP
+- Review endpoint's required permission annotation
+
+## Action
+
+- Grant correct role/permission to user
+- Verify endpoint permission requirement is correct
+
+## Escalation
+
+- Escalate to access-management team if bulk users affected
+
+---
+
+> **Stub**: Phase D2 — author body after domain adoption. (feature-operational-runbook-contract D9)
@@ -0,0 +1,34 @@
+---
+title: Runbook — AUTHZ_TENANT_MISMATCH (테넌트 cross-access 시도)
+category: AUTHZ
+error_codes: [AUTHZ_TENANT_MISMATCH]
+severity: P1
+owner: oncall
+last_updated: 2026-06-15
+status: stub
+---
+
+# Runbook: AUTHZ_TENANT_MISMATCH (`runbook://authz/tenant-mismatch`)
+
+## Symptoms
+
+- HTTP 403 with `error.code=AUTHZ_TENANT_MISMATCH`
+- `log_level=ERROR` — cross-tenant access attempt detected
+
+## Diagnosis
+
+- Extract `traceId`, check `X-Tenant-Id` vs token tenant claim
+- Determine if this is misconfigured client or intentional attack
+
+## Action
+
+- Block repeat offenders at gateway level
+- Alert security team for investigation
+
+## Escalation
+
+- P1 if confirmed malicious cross-tenant access attempt
+
+---
+
+> **Stub**: Phase D2 — author body after domain adoption. (feature-operational-runbook-contract D9)
@@ -0,0 +1,34 @@
+---
+title: Runbook — CACHE_STAMPEDE_LOCK_TIMEOUT (캐시 스탬피드 락 타임아웃)
+category: TRANSIENT_DEPENDENCY
+error_codes: [CACHE_STAMPEDE_LOCK_TIMEOUT]
+severity: P2
+owner: oncall
+last_updated: 2026-06-15
+status: stub
+---
+
+# Runbook: CACHE_STAMPEDE_LOCK_TIMEOUT (`runbook://cache/stampede-lock-timeout`)
+
+## Symptoms
+
+- HTTP 503 with `error.code=CACHE_STAMPEDE_LOCK_TIMEOUT`
+- Multiple concurrent cache misses on same key; lock contention
+
+## Diagnosis
+
+- Check cache hit ratio metrics
+- Identify cache keys with high miss rates
+
+## Action
+
+- Verify stampede lock TTL is configured appropriately
+- Pre-warm cache for high-traffic keys on startup
+
+## Escalation
+
+- Escalate if backend load spike accompanies stampede
+
+---
+
+> **Stub**: Phase D2 — author body after domain adoption. (feature-operational-runbook-contract D9)
@@ -0,0 +1,34 @@
+---
+title: Runbook — CACHE_UNAVAILABLE (캐시 연결 불가)
+category: TRANSIENT_DEPENDENCY
+error_codes: [CACHE_UNAVAILABLE]
+severity: P1
+owner: oncall
+last_updated: 2026-06-15
+status: stub
+---
+
+# Runbook: CACHE_UNAVAILABLE (`runbook://cache/unavailable`)
+
+## Symptoms
+
+- HTTP 503 with `error.code=CACHE_UNAVAILABLE`
+- Redis connection errors in logs
+
+## Diagnosis
+
+- Check Redis cluster health
+- Verify network connectivity from app to Redis
+
+## Action
+
+- Check Redis sentinel/cluster status
+- Enable cache degradation path if configured for optional caches
+
+## Escalation
+
+- P1 if required cache is down and no degradation path exists
+
+---
+
+> **Stub**: Phase D2 — author body after domain adoption. (feature-operational-runbook-contract D9)
@@ -0,0 +1,34 @@
+---
+title: Runbook — DB_DEADLOCK (데드락)
+category: CONFLICT
+error_codes: [DB_DEADLOCK]
+severity: P2
+owner: oncall
+last_updated: 2026-06-15
+status: stub
+---
+
+# Runbook: DB_DEADLOCK (`runbook://db/deadlock`)
+
+## Symptoms
+
+- HTTP 409 with `error.code=DB_DEADLOCK`
+- SQLState 40P01 in Postgres logs
+
+## Diagnosis
+
+- Check `pg_locks` and `pg_stat_activity` during deadlock
+- Identify conflicting transaction lock order
+
+## Action
+
+- Client should retry (retryable=true)
+- Fix lock ordering in code if recurring
+
+## Escalation
+
+- Escalate to DBA if deadlock rate is sustained > 1% of transactions
+
+---
+
+> **Stub**: Phase D2 — author body after domain adoption. (feature-operational-runbook-contract D9)
@@ -0,0 +1,34 @@
+---
+title: Runbook — DB_IDLE_IN_TX_TIMEOUT (트랜잭션 idle 타임아웃)
+category: TRANSIENT_DEPENDENCY
+error_codes: [DB_IDLE_IN_TX_TIMEOUT]
+severity: P2
+owner: oncall
+last_updated: 2026-06-15
+status: stub
+---
+
+# Runbook: DB_IDLE_IN_TX_TIMEOUT (`runbook://db/idle-in-tx-timeout`)
+
+## Symptoms
+
+- HTTP 503 with `error.code=DB_IDLE_IN_TX_TIMEOUT`
+- SQLState 25P03; transaction held open too long without activity
+
+## Diagnosis
+
+- Check `idle_in_transaction_session_timeout` Postgres setting
+- Look for application-level long-running transaction holders
+
+## Action
+
+- Reduce transaction scope in application code
+- Verify `spring.jpa.properties.hibernate.connection.timeout` is bounded
+
+## Escalation
+
+- Escalate to DBA if connection pool exhaustion results
+
+---
+
+> **Stub**: Phase D2 — author body after domain adoption. (feature-operational-runbook-contract D9)
@@ -0,0 +1,34 @@
+---
+title: Runbook — DB_QUERY_CANCELED (쿼리 취소)
+category: TRANSIENT_DEPENDENCY
+error_codes: [DB_QUERY_CANCELED]
+severity: P2
+owner: oncall
+last_updated: 2026-06-15
+status: stub
+---
+
+# Runbook: DB_QUERY_CANCELED (`runbook://db/query-canceled`)
+
+## Symptoms
+
+- HTTP 503 with `error.code=DB_QUERY_CANCELED`
+- SQLState 57014; query exceeds statement timeout
+
+## Diagnosis
+
+- Check `statement_timeout` in Postgres
+- Identify slow queries in `pg_stat_statements`
+
+## Action
+
+- Optimize slow query or add index
+- Adjust statement timeout if query is legitimately long
+
+## Escalation
+
+- Escalate to DBA for query optimization if recurring
+
+---
+
+> **Stub**: Phase D2 — author body after domain adoption. (feature-operational-runbook-contract D9)
@@ -0,0 +1,34 @@
+---
+title: Runbook — DB_SERIALIZATION_FAILURE (직렬화 실패)
+category: CONFLICT
+error_codes: [DB_SERIALIZATION_FAILURE]
+severity: P3
+owner: oncall
+last_updated: 2026-06-15
+status: stub
+---
+
+# Runbook: DB_SERIALIZATION_FAILURE (`runbook://db/serialization-failure`)
+
+## Symptoms
+
+- HTTP 409 with `error.code=DB_SERIALIZATION_FAILURE`
+- SQLState 40001; high concurrent transaction contention
+
+## Diagnosis
+
+- Check DB transaction isolation level
+- Identify hot rows / hot tables under high concurrency
+
+## Action
+
+- Client should retry with exponential backoff (retryable=true)
+- Optimize transaction scope if spike is sustained
+
+## Escalation
+
+- Escalate to DBA if sustained serialization failure rate > 5%
+
+---
+
+> **Stub**: Phase D2 — author body after domain adoption. (feature-operational-runbook-contract D9)
@@ -0,0 +1,34 @@
+---
+title: Runbook — DB_UNAVAILABLE (데이터베이스 연결 불가)
+category: TRANSIENT_DEPENDENCY
+error_codes: [DB_UNAVAILABLE]
+severity: P1
+owner: oncall
+last_updated: 2026-06-15
+status: stub
+---
+
+# Runbook: DB_UNAVAILABLE (`runbook://db/unavailable`)
+
+## Symptoms
+
+- HTTP 503 with `error.code=DB_UNAVAILABLE`
+- SQLState 08* connection errors in logs
+
+## Diagnosis
+
+- Check DB server health and connection pool exhaustion
+- Review network connectivity from app pods to DB
+
+## Action
+
+- Check DB primary health; failover to replica if available
+- Drain connection pool and reconnect
+
+## Escalation
+
+- P1: immediate if DB primary is down
+
+---
+
+> **Stub**: Phase D2 — author body after domain adoption. (feature-operational-runbook-contract D9)
@@ -0,0 +1,34 @@
+---
+title: Runbook — DEPENDENCY_4XX_CLIENT (업스트림 클라이언트 오류)
+category: PERMANENT_DEPENDENCY
+error_codes: [DEPENDENCY_4XX_CLIENT]
+severity: P2
+owner: oncall
+last_updated: 2026-06-15
+status: stub
+---
+
+# Runbook: DEPENDENCY_4XX_CLIENT (`runbook://dependency/4xx-client`)
+
+## Symptoms
+
+- HTTP 502 with `error.code=DEPENDENCY_4XX_CLIENT`
+- Upstream returned 401/403/400 — credential, scope, or request format issue
+
+## Diagnosis
+
+- Check upstream response body in logs for error detail
+- Verify API credentials and scopes are valid
+
+## Action
+
+- Rotate credentials if expired
+- Fix request format if API contract changed
+
+## Escalation
+
+- Escalate to upstream API owner if contract change is suspected
+
+---
+
+> **Stub**: Phase D2 — author body after domain adoption. (feature-operational-runbook-contract D9)
@@ -0,0 +1,34 @@
+---
+title: Runbook — DEPENDENCY_5XX_SERVER (업스트림 서버 오류)
+category: TRANSIENT_DEPENDENCY
+error_codes: [DEPENDENCY_5XX_SERVER]
+severity: P1
+owner: oncall
+last_updated: 2026-06-15
+status: stub
+---
+
+# Runbook: DEPENDENCY_5XX_SERVER (`runbook://dependency/5xx-server`)
+
+## Symptoms
+
+- HTTP 502 with `error.code=DEPENDENCY_5XX_SERVER`
+- Upstream returned 5xx; transient server-side failure
+
+## Diagnosis
+
+- Check `dependency_name` tag for which upstream is failing
+- Review upstream service status page
+
+## Action
+
+- Client should retry (retryable=true)
+- Monitor upstream recovery
+
+## Escalation
+
+- P1 if critical upstream is in sustained 5xx state
+
+---
+
+> **Stub**: Phase D2 — author body after domain adoption. (feature-operational-runbook-contract D9)
@@ -0,0 +1,34 @@
+---
+title: Runbook — DEPENDENCY_CIRCUIT_OPEN (서킷 브레이커 개방)
+category: TRANSIENT_DEPENDENCY
+error_codes: [DEPENDENCY_CIRCUIT_OPEN]
+severity: P2
+owner: oncall
+last_updated: 2026-06-15
+status: stub
+---
+
+# Runbook: DEPENDENCY_CIRCUIT_OPEN (`runbook://dependency/circuit-open`)
+
+## Symptoms
+
+- HTTP 503 with `error.code=DEPENDENCY_CIRCUIT_OPEN`
+- Circuit breaker (Resilience4j) in OPEN state for a dependency
+
+## Diagnosis
+
+- Check Resilience4j circuit breaker metrics for the dependency
+- Check upstream health; circuit opens after failure threshold breached
+
+## Action
+
+- Wait for circuit half-open probe (automatic after wait duration)
+- Resolve upstream issue to allow circuit to close
+
+## Escalation
+
+- P1 if circuit remains open > 5 minutes on a critical dependency
+
+---
+
+> **Stub**: Phase D2 — author body after domain adoption. (feature-operational-runbook-contract D9)
@@ -0,0 +1,34 @@
+---
+title: Runbook — DEPENDENCY_CONNECT_FAILED (외부 의존성 연결 실패)
+category: TRANSIENT_DEPENDENCY
+error_codes: [DEPENDENCY_CONNECT_FAILED]
+severity: P1
+owner: oncall
+last_updated: 2026-06-15
+status: stub
+---
+
+# Runbook: DEPENDENCY_CONNECT_FAILED (`runbook://dependency/connect-failed`)
+
+## Symptoms
+
+- HTTP 503 with `error.code=DEPENDENCY_CONNECT_FAILED`
+- TCP connection refused or network unreachable to upstream
+
+## Diagnosis
+
+- Check `dependency_name` tag for which upstream is unreachable
+- Verify network path and firewall rules
+
+## Action
+
+- Check upstream service availability
+- Verify service discovery / DNS resolution
+
+## Escalation
+
+- P1 if upstream is a critical service dependency
+
+---
+
+> **Stub**: Phase D2 — author body after domain adoption. (feature-operational-runbook-contract D9)
@@ -0,0 +1,34 @@
+---
+title: Runbook — DEPENDENCY_DNS_FAILED (DNS 조회 실패)
+category: TRANSIENT_DEPENDENCY
+error_codes: [DEPENDENCY_DNS_FAILED]
+severity: P1
+owner: oncall
+last_updated: 2026-06-15
+status: stub
+---
+
+# Runbook: DEPENDENCY_DNS_FAILED (`runbook://dependency/dns-failed`)
+
+## Symptoms
+
+- HTTP 503 with `error.code=DEPENDENCY_DNS_FAILED`
+- DNS resolution failure for upstream hostname
+
+## Diagnosis
+
+- Test DNS resolution from app pod: `nslookup <upstream-host>`
+- Check cluster DNS (CoreDNS) health
+
+## Action
+
+- Verify upstream hostname configuration
+- Check CoreDNS / cluster DNS health
+
+## Escalation
+
+- P1 if cluster DNS is degraded
+
+---
+
+> **Stub**: Phase D2 — author body after domain adoption. (feature-operational-runbook-contract D9)
@@ -0,0 +1,34 @@
+---
+title: Runbook — DEPENDENCY_TIMEOUT (외부 의존성 타임아웃)
+category: TRANSIENT_DEPENDENCY
+error_codes: [DEPENDENCY_TIMEOUT]
+severity: P1
+owner: oncall
+last_updated: 2026-06-15
+status: stub
+---
+
+# Runbook: DEPENDENCY_TIMEOUT (`runbook://dependency/timeout`)
+
+## Symptoms
+
+- HTTP 504 with `error.code=DEPENDENCY_TIMEOUT`
+- Upstream service did not respond within configured timeout (default: global 10s)
+
+## Diagnosis
+
+- Check `dependency_name` in log for which upstream is timing out
+- Review upstream service latency metrics
+
+## Action
+
+- Check upstream service health
+- Verify timeout settings match SLA expectations
+
+## Escalation
+
+- P1 if critical upstream is timing out at scale
+
+---
+
+> **Stub**: Phase D2 — author body after domain adoption. (feature-operational-runbook-contract D9)
@@ -0,0 +1,81 @@
+---
+title: Runbook — 외부 의존성 unavailable
+category: TRANSIENT_DEPENDENCY
+error_codes: [DEPENDENCY_TIMEOUT, DEPENDENCY_CONNECT_FAILED, DEPENDENCY_DNS_FAILED, DEPENDENCY_CIRCUIT_OPEN, DEPENDENCY_5XX_SERVER, CACHE_UNAVAILABLE, DB_UNAVAILABLE]
+severity: P1
+owner: oncall
+last_updated: 2026-05-22
+status: stub
+---
+
+# Runbook: 외부 의존성 unavailable
+
+## 1. Trigger
+
+이 runbook은 다음 alert에서 발동됩니다.
+
+- alert name: `dependency_error_rate_critical` 또는 `circuit_breaker_open`
+- alert payload 필수 field: `operation`, `error.code`, `error.category`, `dependency_name`, `dependency_kind`(required|optional), `runbook_link`
+- 임계:
+  - P1: required dependency의 error rate > 50% 1분 OR circuit_open state 활성
+  - P2: optional dependency degraded (fail-open으로 동작 중)
+
+## 2. First Response (5분 이내)
+
+### Step 1 — 확인
+1. `dependency_name` 별 status page 확인 (외부 SaaS면 vendor status, internal이면 해당 service dashboard)
+2. log query로 실패 패턴 확인: timeout / connect / DNS / 5xx 중 어떤 모드인지
+3. runtime-health Dependency Matrix에서 required vs optional 분류 확인
+4. circuit breaker state 확인 (Resilience4j metric)
+
+### Step 2 — 임시 격리
+- required dep이면 readiness probe로 traffic 차단 (회복 대기) — cascade failure 방지
+- optional dep이면 fail-open with degraded mode 확인. degraded banner를 client에 노출
+- DNS failure면 resolver/coredns 상태 확인. cache 강제 flush 검토
+
+## 3. Diagnosis
+
+- log query: `{service="app"} | dependency_name="<name>" | stats count by error.code`
+- metric panel:
+  - `resilience4j_circuitbreaker_state{name="<name>"}`
+  - `resilience4j_retry_calls_total{kind="failed_without_retry"}`
+  - `hikaricp_connections_active`, `hikaricp_connections_pending` (DB_UNAVAILABLE)
+  - `http_client_requests_seconds_count{outcome="SERVER_ERROR"}`
+- trace: 실패 request의 outbound span에서 timeout/connect/DNS 분류, target endpoint 확인
+- 가능한 원인:
+  - vendor outage → status page 확인, 회복 대기
+  - 네트워크 문제 (DNS, security group, NAT) → infra team 확인
+  - connection pool 고갈 (Hikari) → pool size/timeout 점검
+  - circuit breaker open 후 half-open 전환 실패 → 수동 reset 검토
+  - retry-storm으로 인한 self-DoS → retry budget 축소
+
+## 4. Mitigation
+
+- 단기: required면 회복 대기 + traffic 차단, optional이면 degraded mode로 유지
+- pool 고갈이면 일시 pool size 상향 + leak detection 활성화
+- circuit이 stuck이면 수동 reset (`actuator/circuitbreakerevents`)
+- 장기: retry budget·timeout·circuit 임계 재조정, fallback path 보강, vendor SLA 재협상
+
+## 5. Escalation
+
+- 다음 on-call로 page: required dep 5분 내 회복 안 되면 외부 dep team 또는 vendor에 page
+- 다중 dep 동시 장애면 incident commander 호출 (네트워크 전반 문제 의심)
+
+## 6. Recovery / Verification
+
+- 회복 확인 metric: dependency error rate < 1% 5분 지속, circuit_breaker_state = CLOSED, pool utilization 정상
+- post-incident:
+  - vendor postmortem 요청 (외부 SaaS면)
+  - timeout/retry/circuit 설정 재검토
+  - degraded mode가 사용자 경험에 미친 영향 측정
+  - chaos test에 해당 시나리오 추가
+
+## 7. Related
+
+- error-codes.yaml rows: `DEPENDENCY_TIMEOUT`, `DEPENDENCY_CONNECT_FAILED`, `DEPENDENCY_DNS_FAILED`, `DEPENDENCY_CIRCUIT_OPEN`, `DEPENDENCY_5XX_SERVER`, `CACHE_UNAVAILABLE`, `DB_UNAVAILABLE`
+- metrics.yaml: `resilience4j_circuitbreaker_state`, `hikaricp_connections_active`, `http_client_requests_seconds_count`
+- 관련 branch: [[feature-outbound-http-client-baseline]], [[feature-persistence-failure-baseline]]
+
+---
+
+> **Stub 상태 안내**: 이 runbook은 skeleton 단계의 stub. 도메인 도입 시 실제 dependency 목록·required/optional 분류·vendor 연락 체계·circuit/timeout 임계로 보강 필요.
@@ -0,0 +1,34 @@
+---
+title: Runbook — DOWNLOAD_STREAMING_FAILURE (스트리밍 다운로드 실패)
+category: TRANSIENT_DEPENDENCY
+error_codes: [DOWNLOAD_STREAMING_FAILURE]
+severity: P2
+owner: oncall
+last_updated: 2026-06-15
+status: stub
+---
+
+# Runbook: DOWNLOAD_STREAMING_FAILURE (`runbook://file/download-streaming-failure`)
+
+## Symptoms
+
+- HTTP 503 with `error.code=DOWNLOAD_STREAMING_FAILURE`
+- Streaming response truncated; backpressure or timeout (60s / 100MB limit)
+
+## Diagnosis
+
+- Check streaming response timeout configuration
+- Review download size vs 100MB limit
+
+## Action
+
+- Verify storage backend is reachable
+- Check for network congestion on download path
+
+## Escalation
+
+- Escalate to infra if storage backend is degraded
+
+---
+
+> **Stub**: Phase D2 — author body after domain adoption. (feature-operational-runbook-contract D9)
@@ -0,0 +1,74 @@
+---
+title: Runbook — 5xx Internal error spike
+category: INTERNAL
+error_codes: [INTERNAL_ERROR, INTERNAL_AUTH_MISCONFIGURATION, JVM_OOM]
+severity: P1
+owner: oncall
+last_updated: 2026-05-22
+status: stub
+---
+
+# Runbook: 5xx Internal error spike
+
+## 1. Trigger
+
+이 runbook은 다음 alert에서 발동됩니다.
+
+- alert name: `http_5xx_error_rate_critical`
+- alert payload 필수 field: `operation`, `error.code`, `error.category`, `request_id`, `traceId`, `runbook_link`
+- 임계: 5xx error rate > 5% 5분 지속 OR > 10% 1분
+
+## 2. First Response (5분 이내)
+
+### Step 1 — 확인
+1. 가장 최근 deploy 시각 확인 (CI/CD dashboard, artifact registry digest)
+2. JVM metric 확인: heap usage, GC pause, CPU, thread count
+3. log에서 실패 request 1건 추출 → `request_id`, `traceId` 확보
+4. error.code 분포 확인: `INTERNAL_ERROR` vs `JVM_OOM` vs `INTERNAL_AUTH_MISCONFIGURATION`
+
+### Step 2 — 임시 격리
+- 직전 deploy가 의심되면 즉시 rollback (artifact registry에서 직전 image digest pin)
+- OOM 패턴이면 affected pod evict → ASG/HPA로 replacement 유도
+- LB에서 unhealthy pod 격리 (readiness probe failure 유도)
+
+## 3. Diagnosis
+
+- log query: `{service="app"} | http.status>=500 | stats count by error.code`
+- metric panel: `jvm_memory_used_bytes{area="heap"}`, `jvm_gc_pause_seconds`, `process_cpu_seconds_total`, `http_server_requests_seconds_count{status=~"5.."}`
+- trace: 실패 request의 `traceId`로 span chain 확인 → stack trace에서 root exception 추출
+- heap dump 위치: `/var/tmp/heap/heapdump-<pid>.hprof` (JVM ergonomics: `-XX:MaxRAMPercentage=75 -XX:+HeapDumpOnOutOfMemoryError`)
+- 가능한 원인:
+  - 직전 deploy의 회귀 버그 → rollback
+  - JVM OOM (메모리 leak 또는 부하 증가) → heap dump 분석
+  - 외부 의존성 설정 오류 (`INTERNAL_AUTH_MISCONFIGURATION`) → config secret 확인
+  - thread starvation (pool 고갈) → thread dump (`jstack <pid>`)
+
+## 4. Mitigation
+
+- 단기: 직전 deploy rollback, OOM pod replacement, traffic 일시 감소(scale-out 또는 rate-limit 강화)
+- config 오류면 secret/configmap rollback
+- 장기: heap dump 기반 leak 수정, capacity planning 재검토
+
+## 5. Escalation
+
+- 다음 on-call로 page: 10분 내 회복 안 되면 incident commander 호출, severity 1 incident 선언
+- 데이터 손상 의심되면 DBA team page
+
+## 6. Recovery / Verification
+
+- 회복 확인 metric: 5xx rate < 0.5% 5분 지속, JVM heap usage < 70%, GC pause p99 < 500ms
+- post-incident:
+  - rollback 원인 RCA 작성 (배포 게이트 강화 필요 여부)
+  - heap dump 분석 결과 공유
+  - JVM ergonomics(`-XX:MaxRAMPercentage`) 재검토
+  - rollback 자동화 절차 점검
+
+## 7. Related
+
+- error-codes.yaml rows: `INTERNAL_ERROR`, `INTERNAL_AUTH_MISCONFIGURATION`, `JVM_OOM`
+- metrics.yaml: `jvm_memory_used_bytes`, `jvm_gc_pause_seconds`, `http_server_requests_seconds_count`
+- 관련 branch: [[feature-operational-error-observability-foundation]], [[feature-container-runtime-contract]]
+
+---
+
+> **Stub 상태 안내**: 이 runbook은 skeleton 단계의 stub. 도메인 도입 시 실제 deploy 파이프라인·heap dump 보관 경로·rollback 자동화 명령으로 보강 필요.
@@ -0,0 +1,67 @@
+---
+title: Runbook — background job dead letter
+category: INTERNAL
+error_codes: [JOB_DEAD_LETTER]
+severity: P1
+owner: oncall
+last_updated: 2026-06-13
+status: stub
+---
+
+# Runbook: background job dead letter (`runbook://job/dead-letter`)
+
+## 1. Trigger
+
+이 runbook은 다음 alert에서 발동됩니다.
+
+- alert name: `job_dead_letter`
+- alert payload 필수 field: `error.code=JOB_DEAD_LETTER`, `job_name`, `correlation_id`, `runbook_link`
+- 임계: `job.dlq.total` > 0 for 5m (p1) — retry 소진 후 DLQ 진입은 자동 회복이 없으므로 점검 대상
+
+## 2. First Response (5분 이내)
+
+### Step 1 — 확인
+1. ERROR log에서 `JOB_DEAD_LETTER` 라인 확인: `job_name`, 최종 실패 원인 예외, `correlation_id` 추출
+2. `job.retry.total{outcome=EXHAUSTED}` 추이로 DLQ 유입 규모 파악
+3. DLQ 적재 위치(향후 retry carrier 확정 시 DB 테이블/큐) 확인 — 현재 skeleton은 vocabulary 단계
+
+### Step 2 — 임시 격리
+- DLQ는 max attempts(3) 소진의 최종 상태 — 자동 재시도 없음, 수동 개입 필수
+- 비즈니스 크리티컬 job이면 §4의 수동 처분(재처리 또는 폐기)을 우선 수행
+
+## 3. Diagnosis
+
+- log query: `{service="app"} | error.code="JOB_DEAD_LETTER" | stats count by job_name`
+- metric panel: `job.dlq.total{job_name}`, `job.retry.total{job_name, outcome}`
+- 최종 실패 원인 분류:
+  - poison input(직렬화/계약 위반) → 입력 결함, 재처리해도 실패 — 수정 후 재처리 또는 폐기
+  - 외부 의존성 장기 outage 중 attempts 소진 → 의존성 회복 후 재처리로 해결 가능
+  - non-transient error(권한/도메인/스키마)인데 retry된 경우 → 분류기 보강 필요(WAF-REL05-C3: 즉시 DLQ가 정답)
+
+## 4. Mitigation (수동 처분 — 둘 중 하나)
+
+- **재처리 (기본)**: 원인 해소 후 해당 job을 다시 enqueue. 소비자는 멱등(idempotencyKey dedupe) 의무가 있으므로 중복 처리 안전
+- **폐기 (영구)**: 작업이 더 이상 유효하지 않으면 DLQ에서 제거. ⚠ 비즈니스 오너 승인 후에만 수행하고 incident 기록에 남김
+- 장기: poison input 재발 방지(입력 계약 테스트 보강), non-transient error는 retry 없이 즉시 DLQ로 분류
+
+## 5. Escalation
+
+- 처분 판단(재처리 vs 폐기)이 불가하면 해당 job의 비즈니스 오너에게 escalate
+- DLQ 누적이 특정 `job_name`에 집중되면 해당 job 코드 오너에게 page
+
+## 6. Recovery / Verification
+
+- 회복 확인: `job.dlq.total` 증가 멈춤, 재처리분의 소비자 dedupe 동작 확인
+- post-incident: DLQ 원인 분류 기록, 같은 원인의 재발 방지 테스트 추가
+
+## 7. Related
+
+- error-codes.yaml rows: `JOB_DEAD_LETTER` (INTERNAL, 500, retryable=false)
+- metrics.yaml: `job.dlq.total{job_name}`, `job.retry.total{job_name, outcome=DLQ}`
+- 코드: `app-bootstrap` `async/BackgroundJobMetrics`(retry/DLQ vocabulary 기록 seam — D2/D4)
+- 관련 runbook: [[job-executor-rejected]], [[job-timeout]], [[outbox-dead-letter]]
+- 관련 branch: [[feature-background-job-async-contract]] (D4 retry/DLQ vocabulary SSOT — outbox/outbound가 consume)
+
+---
+
+> **Stub 상태 안내**: 이 runbook은 skeleton 단계의 stub. retry carrier(Spring Retry / Resilience4j / 자체) 확정 후 DLQ 저장소·재처리 절차 보강 필요.
@@ -0,0 +1,71 @@
+---
+title: Runbook — async executor rejected
+category: TRANSIENT_DEPENDENCY
+error_codes: [JOB_EXECUTOR_REJECTED]
+severity: P1
+owner: oncall
+last_updated: 2026-06-13
+status: stub
+---
+
+# Runbook: async executor rejected (`runbook://job/executor-rejected`)
+
+## 1. Trigger
+
+이 runbook은 다음 alert에서 발동됩니다.
+
+- alert name: `executor_rejected`
+- alert payload 필수 field: `error.code=JOB_EXECUTOR_REJECTED`, `executor_name`, `policy`, `runbook_link`
+- 임계: `executor.rejected.total` > 0 for 1m (p1) — bounded pool이 saturation으로 task를 거부
+- 보조 신호: `executor.saturation` gauge > queue capacity의 80% for 5m (p2)
+
+## 2. First Response (5분 이내)
+
+### Step 1 — 확인
+1. ERROR log에서 `JOB_EXECUTOR_REJECTED` 라인 확인: `executor_name`, `policy=AbortPolicy`, `queue_size` 추출
+2. `executor.saturation` 패널에서 큐 점유율 추이 확인 — 일시적 burst인지 지속 saturation인지 판별
+3. 동시 유입 원인 파악: 신규 배포 / 트래픽 spike / 다운스트림 지연으로 worker가 장기 점유되는지
+
+### Step 2 — 임시 격리
+- AbortPolicy 거부는 호출부에 `RejectedExecutionException`으로 surface됨 — fire-and-forget `@Async` 호출이면 호출부의 async-exception 처리(log/metric)로 흡수됐는지 확인
+- 지속 saturation이면 유입 측(트래픽/스케줄러 빈도)을 우선 감속
+
+## 3. Diagnosis
+
+- log query: `{service="app"} | error.code="JOB_EXECUTOR_REJECTED" | stats count by executor_name`
+- metric panel: `executor.saturation{executor_name}`, `executor.rejected.total{executor_name, policy}`
+- 가능한 원인 우선순위:
+  - 다운스트림 의존성 지연 → worker가 반납되지 않아 큐 포화 (가장 흔함)
+  - 트래픽 spike → 정상 부하 한계 초과
+  - pool 과소 설정 (`APP_ASYNC_EXECUTOR_*`)
+  - non-idempotent 작업이 retry로 누적
+
+## 4. Mitigation
+
+- 단기: 유입 감속(상위 rate-limit / 스케줄러 interval 확대) 또는 다운스트림 의존성 회복
+- pool 재조정(restart-only): `APP_ASYNC_EXECUTOR_CORE_SIZE` / `APP_ASYNC_EXECUTOR_MAX_SIZE` / `APP_ASYNC_EXECUTOR_QUEUE_CAPACITY`
+  — ⚠ queue를 무한정 키우지 말 것(unbounded 금지, D7). 부하테스트로 수치 검증 후 변경
+- CallerRunsPolicy로의 전환은 use-case 차원의 명시적 결정 필요(request thread latency 침식 — TPE-JDK21-C6)
+
+## 5. Escalation
+
+- 다운스트림 의존성 장애가 근본 원인이면 해당 의존성 오너에게 escalate
+- pool 재조정으로도 saturation이 지속되면 용량 계획(capacity planning) 오너에게 page
+
+## 6. Recovery / Verification
+
+- 회복 확인: `executor.rejected.total` 증가 멈춤, `executor.saturation` < 80% 정상화
+- 거부된 작업의 재처리 경로(멱등 retry / 다음 스케줄 cycle) 정상 동작 확인
+
+## 7. Related
+
+- error-codes.yaml rows: `JOB_EXECUTOR_REJECTED` (TRANSIENT_DEPENDENCY, 503, retryable=true, retry_after 5s)
+- metrics.yaml: `executor.rejected.total{executor_name, policy}`, `executor.saturation{executor_name}`
+- 코드: `app-bootstrap` `async/AsyncExecutorConfig`(bounded executor), `async/LoggingAbortPolicy`(reject log+metric), `async/BackgroundJobMetrics`
+- env: `APP_ASYNC_EXECUTOR_CORE_SIZE` / `APP_ASYNC_EXECUTOR_MAX_SIZE` / `APP_ASYNC_EXECUTOR_QUEUE_CAPACITY`
+- 관련 runbook: [[job-timeout]], [[job-dead-letter]]
+- 관련 branch: [[feature-background-job-async-contract]] (D7 saturation policy)
+
+---
+
+> **Stub 상태 안내**: 이 runbook은 skeleton 단계의 stub. 실제 부하 프로파일·alert 채널·pool 수치 확정 시 보강 필요.
@@ -0,0 +1,69 @@
+---
+title: Runbook — background job timeout
+category: TRANSIENT_DEPENDENCY
+error_codes: [JOB_TIMEOUT]
+severity: P2
+owner: oncall
+last_updated: 2026-06-13
+status: stub
+---
+
+# Runbook: background job timeout (`runbook://job/timeout`)
+
+## 1. Trigger
+
+이 runbook은 다음 alert에서 발동됩니다.
+
+- alert name: `job_timeout`
+- alert payload 필수 field: `error.code=JOB_TIMEOUT`, `job_name`, `correlation_id`, `runbook_link`
+- 임계: `job.retry.total{outcome=RETRY}` 급증 또는 graceful-shutdown 중 in-flight job interrupt 발생
+- 연관: shutdown phase에서 19s await 초과로 interrupt된 job (D8)
+
+## 2. First Response (10분 이내)
+
+### Step 1 — 확인
+1. ERROR log에서 `JOB_TIMEOUT` 라인 확인: `job_name`, 마지막 단계, 소요 시간 추출
+2. timeout이 정상 실행 중 발생인지, graceful-shutdown(배포/스케일다운) 중 interrupt인지 구분
+3. 해당 job이 멱등(retry-on-next-cycle 안전)인지 확인 — 비멱등이면 §4에서 신중히 처리
+
+### Step 2 — 임시 격리
+- shutdown 중 interrupt면: 다음 기동 시 재시도 대상인지(멱등 전제) 확인, 중복 부작용 여부 점검
+- 정상 실행 중 timeout이면: 해당 job의 외부 의존성(DB/HTTP) 지연 여부 확인
+
+## 3. Diagnosis
+
+- log query: `{service="app"} | error.code="JOB_TIMEOUT" | stats count by job_name`
+- metric panel: `job.retry.total{job_name, outcome}`
+- 가능한 원인 우선순위:
+  - 외부 의존성(DB lock / 느린 HTTP) 지연으로 job p99 상승
+  - job 작업량 증가로 단일 cycle이 19s 예산 초과 (D8 — interrupt 노출)
+  - interrupt 미반응 blocking call(JDBC 등) → awaitTermination 초과 (K8S-POD-LC-C2 SIGKILL 경로)
+
+## 4. Mitigation
+
+- 단기: 의존성 회복 / job 입력 배치 크기 축소
+- job p99가 구조적으로 19s를 넘으면: 작업을 분할하거나, grace period 연장 검토(parent project 운영 계약 소유자 승인 필요 — OUT_OF_BRANCH_SCOPE)
+- 비멱등 job이 재시도로 중복 부작용을 내면 멱등키/dedupe 도입 우선
+
+## 5. Escalation
+
+- 의존성 지연이 근본 원인이면 해당 의존성 오너에게 escalate
+- shutdown 예산(20s) vs k8s `terminationGracePeriodSeconds`(30s) 정합 이슈면 플랫폼/런타임 오너에게 escalate
+
+## 6. Recovery / Verification
+
+- 회복 확인: `JOB_TIMEOUT` 신규 발생 멈춤, `job.retry.total{outcome=SUCCESS}` 정상 비율 회복
+- 멱등 재시도분의 부작용 중복 없음 확인
+
+## 7. Related
+
+- error-codes.yaml rows: `JOB_TIMEOUT` (TRANSIENT_DEPENDENCY, 500, retryable=true, retry_after 10s)
+- metrics.yaml: `job.retry.total{job_name, outcome}`
+- 코드: `app-bootstrap` `async/AsyncExecutorConfig`(awaitTermination 19s — D8 graceful shutdown)
+- env: `APP_SERVER_SHUTDOWN_TIMEOUT`(owner: feature-env-driven-runtime-configuration D2)
+- 관련 runbook: [[job-executor-rejected]], [[job-dead-letter]]
+- 관련 branch: [[feature-background-job-async-contract]] (D4 retry / D8 shutdown)
+
+---
+
+> **Stub 상태 안내**: 이 runbook은 skeleton 단계의 stub. 실제 retry carrier·job p99·shutdown 예산 확정 시 보강 필요.
@@ -0,0 +1,34 @@
+---
+title: Runbook — LOCK_ACQUISITION_TIMEOUT (분산 락 획득 타임아웃)
+category: CONFLICT
+error_codes: [LOCK_ACQUISITION_TIMEOUT]
+severity: P2
+owner: oncall
+last_updated: 2026-06-15
+status: stub
+---
+
+# Runbook: LOCK_ACQUISITION_TIMEOUT (`runbook://lock/acquisition-timeout`)
+
+## Symptoms
+
+- HTTP 409 with `error.code=LOCK_ACQUISITION_TIMEOUT`
+- Distributed lock wait exceeded configured timeout; high contention on a resource
+
+## Diagnosis
+
+- Check `lock.acquisition` metric for lock name and duration
+- Identify lock holders (check DB `integration_lock` table)
+
+## Action
+
+- Client should retry with backoff (retryable=true)
+- Optimize critical section holding time if lock contention is systemic
+
+## Escalation
+
+- Escalate if lock holder appears stuck (potential deadlock in distributed lock)
+
+---
+
+> **Stub**: Phase D2 — author body after domain adoption. (feature-operational-runbook-contract D9)
@@ -0,0 +1,34 @@
+---
+title: Runbook — ACTUATOR_FORBIDDEN (Actuator 접근 거부)
+category: AUTHZ
+error_codes: [ACTUATOR_FORBIDDEN]
+severity: P2
+owner: oncall
+last_updated: 2026-06-15
+status: stub
+---
+
+# Runbook: ACTUATOR_FORBIDDEN (`runbook://management/actuator-forbidden`)
+
+## Symptoms
+
+- HTTP 403 with `error.code=ACTUATOR_FORBIDDEN`
+- Attempt to access restricted actuator endpoint (env/configprops/heapdump/shutdown)
+
+## Diagnosis
+
+- Identify which actuator endpoint was accessed
+- Check caller identity (internal tooling vs external)
+
+## Action
+
+- Verify management port is not exposed externally
+- For heapdump/threaddump: follow break-glass runbook procedure
+
+## Escalation
+
+- P1 if forbidden actuator access appears to be external attack
+
+---
+
+> **Stub**: Phase D2 — author body after domain adoption. (feature-operational-runbook-contract D9)
@@ -0,0 +1,35 @@
+---
+title: Runbook — MIGRATION_FAILED (DB 마이그레이션 실패)
+category: INTERNAL
+error_codes: [MIGRATION_FAILED]
+severity: P1
+owner: oncall
+last_updated: 2026-06-15
+status: stub
+---
+
+# Runbook: MIGRATION_FAILED (`runbook://migration/failed`)
+
+## Symptoms
+
+- Container exits with code 70 (migration failure exit)
+- Structured log with `error.code=MIGRATION_FAILED`, `startup.phase=migration`
+- App refuses to start (fail-fast)
+
+## Diagnosis
+
+- Check Flyway migration log for which script failed and why
+- Review latest migration script for SQL errors
+
+## Action
+
+- Fix migration script or roll back to previous migration version
+- Run migration manually in repair mode if checksum mismatch
+
+## Escalation
+
+- P1 immediate: app cannot start until migration is resolved
+
+---
+
+> **Stub**: Phase D2 — author body after domain adoption. (feature-operational-runbook-contract D9)
@@ -0,0 +1,73 @@
+---
+title: Runbook — outbox dead letter
+category: INTERNAL
+error_codes: [OUTBOX_DEAD_LETTER]
+severity: P1
+owner: oncall
+last_updated: 2026-06-11
+status: stub
+---
+
+# Runbook: outbox dead letter (`runbook://outbox/dead-letter`)
+
+## 1. Trigger
+
+이 runbook은 다음 alert에서 발동됩니다.
+
+- alert name: `outbox_dead_letter`
+- alert payload 필수 field: `error.code=OUTBOX_DEAD_LETTER`, `event_type`, `event_id`, `correlation_id`, `runbook_link`
+- 임계: `outbox.publisher.published.total{outcome=DEAD}` > 0 (DEAD 전이는 자동 회복이 없으므로 단건도 점검 대상)
+
+## 2. First Response (5분 이내)
+
+### Step 1 — 확인
+1. ERROR log에서 `OUTBOX_DEAD_LETTER` 라인 확인: `event_id`, `event_type`, `correlation_id`, 마지막 실패 원인 예외 추출
+2. DB에서 DEAD row 확인: `SELECT * FROM outbox_event WHERE status = 'DEAD' ORDER BY occurred_at;`
+3. **차단 영향 파악 (중요)**: strict per-aggregate FIFO 정책상 DEAD row는 같은 `aggregate_id`의 후행 이벤트를 계속 차단함 —
+   `SELECT count(*) FROM outbox_event b WHERE b.status <> 'PUBLISHED' AND EXISTS (SELECT 1 FROM outbox_event d WHERE d.status='DEAD' AND d.aggregate_id=b.aggregate_id AND d.occurred_at < b.occurred_at);`
+
+### Step 2 — 임시 격리
+- DEAD는 max attempts(3) 소진의 최종 상태 — 자동 재시도 없음, 수동 개입 필수
+- 차단된 aggregate가 비즈니스 크리티컬하면 아래 §4의 수동 처분(재발행 또는 skip)을 우선 수행
+
+## 3. Diagnosis
+
+- log query: `{service="app"} | error.code="OUTBOX_DEAD_LETTER" | stats count by event_type`
+- 마지막 실패 원인 분류:
+  - poison event (payload 직렬화/계약 위반) → payload 자체 결함, 재발행해도 실패 — 수정 후 재발행 또는 skip
+  - broker 장기 outage 중 attempts 소진 → broker 회복 후 재발행으로 해결 가능
+  - 구성 오류 (Kafka disabled 상태에서 producer 활성) → 구성 수정 후 재발행
+- 가능한 원인 우선순위: 구성 오류 > broker outage > poison payload
+
+## 4. Mitigation (수동 처분 — 둘 중 하나)
+
+- **재발행 (기본)**: 원인 해소 후 해당 row를 다시 claim 가능 상태로 되돌림 —
+  `UPDATE outbox_event SET status = 'PENDING', attempt_count = 0, next_attempt_at = now() WHERE event_id = '<id>' AND status = 'DEAD';`
+  (consumer는 at-least-once + idempotencyKey dedupe 의무가 있으므로 중복 발행은 안전)
+- **skip (영구 폐기)**: 이벤트가 더 이상 유효하지 않으면 PUBLISHED로 마킹해 FIFO 차단을 해제 —
+  `UPDATE outbox_event SET status = 'PUBLISHED' WHERE event_id = '<id>' AND status = 'DEAD';`
+  ⚠️ skip은 다운스트림에 영구 이벤트 갭을 만든다 — 비즈니스 오너 승인 후에만 수행하고 incident 기록에 남김
+- 장기: poison event 재발 방지(payload 계약 테스트 보강), DEAD 빈발 event_type의 producer 검증 강화
+
+## 5. Escalation
+
+- 처분 판단(재발행 vs skip)이 불가하면 해당 이벤트의 비즈니스 오너에게 escalate
+- DEAD 누적이 특정 event_type에 집중되면 producer 코드 오너에게 page
+
+## 6. Recovery / Verification
+
+- 회복 확인: `SELECT count(*) FROM outbox_event WHERE status='DEAD';` = 0, 차단됐던 aggregate의 후행 이벤트가 PUBLISHED로 전이
+- `outbox.publisher.lag` 정상화(< 60s), 재발행분의 consumer dedupe 동작 확인
+- post-incident: DEAD 원인 분류 기록, 같은 원인의 재발 방지 테스트 추가
+
+## 7. Related
+
+- error-codes.yaml rows: `OUTBOX_DEAD_LETTER` (INTERNAL, retryable=false)
+- metrics.yaml: `outbox.publisher.published.total{outcome=DEAD}`, `outbox.pending.size{status=DEAD}`, `outbox.publisher.lag`
+- 코드: `application-core` `PublishPendingOutboxEventsUseCase`(FAILED→DEAD 전이), `adapter-persistence` `outbox/OutboxEventJpaRepository`(FIFO 게이트 — DEAD가 후행 차단)
+- 관련 runbook: [[outbox-publish-failed]]
+- 관련 branch: [[feature-domain-event-outbox-contract]], [[feature-background-job-async-contract]] (max attempts/DLQ vocabulary SSOT)
+
+---
+
+> **Stub 상태 안내**: 이 runbook은 skeleton 단계의 stub. 실제 broker·DLQ 토픽·승인 체계 확정 시 보강 필요.
@@ -0,0 +1,77 @@
+---
+title: Runbook — outbox publish 일시 실패
+category: TRANSIENT_DEPENDENCY
+error_codes: [OUTBOX_PUBLISH_FAILED]
+severity: P2
+owner: oncall
+last_updated: 2026-06-11
+status: stub
+---
+
+# Runbook: outbox publish 일시 실패 (`runbook://outbox/publish-failed`)
+
+## 1. Trigger
+
+이 runbook은 다음 alert에서 발동됩니다.
+
+- alert name: `outbox_publish_failed_rate` 또는 `outbox_publisher_lag`
+- alert payload 필수 field: `error.code=OUTBOX_PUBLISH_FAILED`, `event_type`, `correlation_id`, `runbook_link`
+- 임계 (metrics.yaml verbatim):
+  - P2: `outbox.publisher.published.total{outcome=FAILED}` rate > 1% for 10m
+  - P2: `outbox.publisher.lag` > 60s for 10m / P1: > 300s for 5m
+  - P2: `outbox.pending.size{status=PENDING}` growing for 10m
+
+## 2. First Response (5분 이내)
+
+### Step 1 — 확인
+1. ERROR log에서 `OUTBOX_PUBLISH_FAILED` 라인 확인: `event_type`, `event_id`, `correlation_id`, `attempt_count` 추출
+2. broker(기본 Kafka adapter) 상태 확인: `APP_MESSAGING_KAFKA_ENABLED` 값과 broker endpoint 가용성
+   - Kafka disabled(default) 상태에서 outbox 이벤트가 append 되고 있으면 publish 경로가 `AdapterDisabledException`으로 전부 실패하는 구성 오류 — 이 경우 producer use case 쪽 활성화/구성을 먼저 의심
+3. `outbox.pending.size` status 분포 확인 (FAILED 누적 vs PENDING 누적)
+
+### Step 2 — 임시 격리
+- 일시 실패는 자동 backoff 재시도(30s × 2^(attempt-1) + jitter, max attempts 3)가 동작 — 즉시 수동 개입 불필요
+- broker 장기 다운이면 DEAD 전이 누적 전에 broker 회복을 우선 (max attempts 소진 시 `runbook://outbox/dead-letter`로 이관)
+- relay 자체를 멈춰야 하면 `ca-skeleton.outbox.relay-enabled=false`로 스케줄러 비활성 (이벤트는 outbox 테이블에 안전하게 보존됨 — 유실 없음)
+
+## 3. Diagnosis
+
+- log query: `{service="app"} | error.code="OUTBOX_PUBLISH_FAILED" | stats count by event_type`
+- metric panel:
+  - `outbox.publisher.published.total{outcome}` — FAILED 비율
+  - `outbox.publisher.lag{event_type}` — 최고령 미발행 이벤트 age
+  - `outbox.pending.size{status}` — 상태별 분포
+- DB 확인: `SELECT status, count(*) FROM outbox_event GROUP BY status;`
+- 가능한 원인:
+  - broker outage/네트워크 → broker 측 회복 대기
+  - Kafka adapter 미구성(enabled인데 brokers 누락은 기동 시 차단됨) / disabled 상태에서 producer 활성화
+  - poison event (직렬화 불가/payload 계약 위반) → 재시도 무의미, attempts 소진 후 DEAD로 흘러감 (의도된 동작)
+  - 동일 aggregate head 실패로 후행 이벤트가 FIFO 게이트에 차단되어 lag 증가 (strict per-aggregate FIFO — 설계 의도)
+
+## 4. Mitigation
+
+- 단기: broker 회복 후 backoff 만료 시 자동 재발행 — `outcome=PUBLISHED` 회복 확인
+- IN_FLIGHT orphan(claim 후 crash)은 in-flight-timeout(기본 PT5M) 경과 후 자동 재claim — at-least-once이므로 중복 발행 가능, consumer dedupe(idempotencyKey)가 흡수
+- 장기: `ca-skeleton.outbox.poll-interval`/`batch-size` 조정, broker 가용성 SLA 점검, 빈발 event_type의 payload 계약 검토
+
+## 5. Escalation
+
+- P1 lag(>300s 5m) 지속 + broker 회복 불가면 broker/infra 팀에 page
+- DEAD 전이가 발생하기 시작하면 `runbook://outbox/dead-letter` 절차로 이관
+
+## 6. Recovery / Verification
+
+- 회복 확인 metric: `outcome=FAILED` rate < 1% 10분 지속, `outbox.publisher.lag` < 60s, `outbox.pending.size{status=FAILED}` 감소 추세
+- post-incident: 실패 구간의 DEAD row 유무 확인, consumer 측 중복 처리량 확인(dedupe 동작 검증), backoff/attempts 상수 재평가
+
+## 7. Related
+
+- error-codes.yaml rows: `OUTBOX_PUBLISH_FAILED` (TRANSIENT_DEPENDENCY, retryable=true, retry_after 30s)
+- metrics.yaml: `outbox.publisher.published.total`, `outbox.publisher.lag`, `outbox.pending.size`
+- 코드: `application-core` `PublishPendingOutboxEventsUseCase`(상태머신), `adapter-persistence` `outbox/OutboxEventJpaRepository`(SKIP LOCKED claim + FIFO 게이트), `adapter-outbound` `messaging/outbox/KafkaOutboxMessagePublishAdapter`(fail-closed)
+- 관련 runbook: [[outbox-dead-letter]]
+- 관련 branch: [[feature-domain-event-outbox-contract]], [[feature-background-job-async-contract]] (retry/DLQ vocabulary SSOT)
+
+---
+
+> **Stub 상태 안내**: 이 runbook은 skeleton 단계의 stub. 실제 broker 채택·alert 라우팅·대시보드 링크 확정 시 보강 필요.
@@ -0,0 +1,73 @@
+---
+title: Runbook — Rate limit 초과 spike
+category: RATE_LIMIT
+error_codes: [RATE_LIMIT_EXCEEDED, IDEMPOTENT_IN_FLIGHT]
+severity: P3
+owner: oncall
+last_updated: 2026-05-22
+status: stub
+---
+
+# Runbook: Rate limit 초과 spike
+
+## 1. Trigger
+
+이 runbook은 다음 alert에서 발동됩니다.
+
+- alert name: `rate_limit_429_high`
+- alert payload 필수 field: `operation`, `error.code`, `error.category`, `rate_limit_key_type`(ip|principal|tenant), `runbook_link`
+- 임계:
+  - P3: 429 rate > 1% 10분 지속 (일상적 abuse 차단 효과 정상)
+  - P2 격상: 정상 client(known principal/tenant)에서 spike 또는 spike와 함께 5xx 동반
+
+## 2. First Response (5분 이내)
+
+### Step 1 — 확인
+1. rate-limit key 분포 확인: IP/principal/tenant 중 어디서 spike가 발생했는지
+   - log query: `error.code=RATE_LIMIT_EXCEEDED | stats count by rate_limit_key_type, rate_limit_key`
+2. top-N offending key 추출 (상위 10건)
+3. 정상 client 식별 — 알려진 partner/internal service면 P2 격상
+
+### Step 2 — 임시 격리
+- abuse traffic 패턴이면 WAF/gateway에서 IP block (geo, ASN 단위)
+- IDEMPOTENT_IN_FLIGHT 다발이면 client의 retry-storm 의심 → client에 retry-after 협조 요청
+
+## 3. Diagnosis
+
+- log query: `{service="gateway"} | error.code="RATE_LIMIT_EXCEEDED" | stats count by rate_limit_key`
+- metric panel: `gateway_rate_limit_dropped_total`, `gateway_rate_limit_bucket_utilization`
+- trace: 429 응답의 `Retry-After` 헤더 값, `rate_limit_remaining` header 확인
+- 가능한 원인:
+  - abuse / bot traffic → IP/ASN block
+  - 정상 client의 traffic 증가 (캠페인, 신규 feature) → limit 일시 상향
+  - retry-storm (client backoff 미적용) → client에 idempotency-key + exponential backoff 권고
+  - limit 설정 오류 (잘못된 정량 threshold) → config rollback
+
+## 4. Mitigation
+
+- 단기: abuse면 IP/ASN block, 정상 client면 해당 key의 limit 일시 상향(예: 2x, 1시간 TTL)
+- IDEMPOTENT_IN_FLIGHT 다발: idempotency-key 정책 점검, client 협조 요청
+- 장기: limit 정책을 tenant tier별 차등으로 재설계, abuse pattern detection 자동화
+
+## 5. Escalation
+
+- 다음 on-call로 page: 30분 내 정상 client 회복 안 되면 product team 통보
+- 정상 client에 SLO 위반 가능성 있으면 CSM/계정담당 통보
+
+## 6. Recovery / Verification
+
+- 회복 확인 metric: 429 rate < 0.5% 10분 지속, 정상 client의 success rate 정상화
+- post-incident:
+  - 일시 상향한 limit 원복 (TTL 만료 확인)
+  - abuse pattern을 detection rule에 영구 등록
+  - retry-storm이면 client SDK 가이드 보완
+
+## 7. Related
+
+- error-codes.yaml rows: `RATE_LIMIT_EXCEEDED`, `IDEMPOTENT_IN_FLIGHT`
+- metrics.yaml: `gateway_rate_limit_dropped_total`, `gateway_rate_limit_bucket_utilization`
+- 관련 branch: [[feature-rate-limit-idempotency-contract]]
+
+---
+
+> **Stub 상태 안내**: 이 runbook은 skeleton 단계의 stub. 도메인 도입 시 실제 gateway 제품(NGINX/Envoy/Kong 등)·tenant tier 정책·WAF 연동 절차로 보강 필요.
@@ -0,0 +1,36 @@
+---
+title: Runbook — JVM_OOM (JVM OutOfMemoryError)
+category: INTERNAL
+error_codes: [JVM_OOM]
+severity: P1
+owner: oncall
+last_updated: 2026-06-15
+status: stub
+---
+
+# Runbook: JVM_OOM (`runbook://runtime/jvm-oom`)
+
+## Symptoms
+
+- Container exits with code 137 (ExitOnOutOfMemoryError triggered)
+- Structured log entry with `error.code=JVM_OOM` before exit
+
+## Diagnosis
+
+- Check heap dump if `-XX:HeapDumpOnOutOfMemoryError` is configured
+- Review memory usage trends before crash
+- Check for memory leaks: large cache growth, unbounded lists, session accumulation
+
+## Action
+
+- Restart container immediately (k8s will auto-restart with liveness probe)
+- If recurring: increase heap `-Xmx` or fix memory leak
+
+## Escalation
+
+- P1: immediate if multiple pods crashing simultaneously
+- Page SRE / infra team for heap analysis
+
+---
+
+> **Stub**: Phase D2 — author body after domain adoption. (feature-operational-runbook-contract D9)
@@ -0,0 +1,35 @@
+---
+title: Runbook — PROFILE_MISMATCH (프로파일 불일치)
+category: INTERNAL
+error_codes: [PROFILE_MISMATCH]
+severity: P1
+owner: oncall
+last_updated: 2026-06-15
+status: stub
+---
+
+# Runbook: PROFILE_MISMATCH (`runbook://startup/profile-mismatch`)
+
+## Symptoms
+
+- Container exits with code 71 (profile mismatch exit)
+- Structured log with `error.code=PROFILE_MISMATCH`, `startup.phase=profile-check`
+- Production profile active with local-only settings enabled
+
+## Diagnosis
+
+- Check active Spring profiles (`spring.profiles.active`)
+- Identify which local-only setting is incorrectly enabled in prod profile
+
+## Action
+
+- Remove local-only setting from production deployment config
+- Ensure prod profile does not inherit local/dev profile settings
+
+## Escalation
+
+- P1: security risk if local settings expose debug endpoints in production
+
+---
+
+> **Stub**: Phase D2 — author body after domain adoption. (feature-operational-runbook-contract D9)
@@ -0,0 +1,34 @@
+---
+title: Runbook — REQUIRED_ADAPTER_DISABLED (필수 어댑터 비활성화)
+category: INTERNAL
+error_codes: [REQUIRED_ADAPTER_DISABLED]
+severity: P1
+owner: oncall
+last_updated: 2026-06-15
+status: stub
+---
+
+# Runbook: REQUIRED_ADAPTER_DISABLED (`runbook://startup/required-adapter-disabled`)
+
+## Symptoms
+
+- Container exits with code 72 (required adapter disabled exit)
+- Structured log with `error.code=REQUIRED_ADAPTER_DISABLED`, `startup.phase=adapter-check`
+
+## Diagnosis
+
+- Identify which adapter is disabled but required
+- Check adapter enable flags in environment config
+
+## Action
+
+- Enable required adapter in deployment configuration
+- If adapter is intentionally disabled, update the required/optional designation
+
+## Escalation
+
+- P1: app cannot start; coordinate with deployment team
+
+---
+
+> **Stub**: Phase D2 — author body after domain adoption. (feature-operational-runbook-contract D9)
@@ -0,0 +1,34 @@
+---
+title: Runbook — STARTUP_VALIDATION_FAILED (환경 변수 검증 실패)
+category: INTERNAL
+error_codes: [STARTUP_VALIDATION_FAILED]
+severity: P1
+owner: oncall
+last_updated: 2026-06-15
+status: stub
+---
+
+# Runbook: STARTUP_VALIDATION_FAILED (`runbook://startup/validation-failed`)
+
+## Symptoms
+
+- Container exits with code 78 (env validation failure exit)
+- Structured log with `error.code=STARTUP_VALIDATION_FAILED`, `startup.phase=env-validation`
+
+## Diagnosis
+
+- Check which required env variable is missing or malformed
+- Review container environment and secrets injection
+
+## Action
+
+- Supply missing environment variables to deployment
+- Verify secrets are correctly mounted / injected
+
+## Escalation
+
+- P1: app cannot start; coordinate with deployment/secrets team
+
+---
+
+> **Stub**: Phase D2 — author body after domain adoption. (feature-operational-runbook-contract D9)
@@ -0,0 +1,42 @@
+---
+title: Runbook — <TITLE>
+category: <CATEGORY>
+error_codes: [<ERROR_CODE_1>, <ERROR_CODE_2>]
+severity: <P1|P2|P3>
+owner: oncall
+last_updated: <YYYY-MM-DD>
+status: <stub|active>
+---
+
+# Runbook: <TITLE> (`runbook://<area>/<scenario>`)
+
+## Symptoms
+
+- What observable signals trigger this runbook?
+- Alert name, metric thresholds, log patterns
+
+## Diagnosis
+
+- Step-by-step diagnostic commands and queries
+- Log queries (Loki/CloudWatch)
+- Metric panels to check
+- Trace investigation approach
+
+## Action
+
+- Immediate mitigation steps
+- Configuration changes
+- Manual intervention procedures
+
+## Escalation
+
+- Conditions for severity upgrade (e.g., P2 → P1)
+- Who to page and when
+- Fallback procedures if on-call cannot resolve
+
+---
+
+> **Note**: This is the canonical runbook template.
+> Copy this file, rename it to match the `runbook://area/scenario` pattern (→ `area-scenario.md`),
+> fill in the frontmatter fields, replace section bodies with operational content,
+> then set `status: active` and remove from `STUB_ALLOWLIST` in `RunbookCoverageContractTest`.