feat: grpc 기능 deep 구현

docs/runbooks/grpc-advanced-capabilities.md

@@ -0,0 +1,98 @@
+# Runbook: gRPC advanced capabilities
+
+Scope: the `:grpc-advanced:*` family. Everything here is off by default and stays off until a
+deployment names it. Nothing in this family ships in a runtime composition today.
+
+Flags are `ca-skeleton.grpc.advanced.<capability>.enabled`. `GrpcAdvancedCapability` owns the list of
+capability names; `GrpcAdvancedSupportMatrix` owns their current grades.
+
+---
+
+## A capability refuses to start
+
+`GrpcAdvancedModuleGuard` gives three different refusals, and the remedy differs:
+
+| Message contains | Meaning | Remedy |
+| --- | --- | --- |
+| "its feature flag is not set" | Nobody enabled it | Set the property named in the message |
+| "tracked rather than implemented" | Grade is `WATCH` | Nothing to do here; the capability is not implemented |
+| "uncharacterised failure modes" | Grade is `EXPERIMENTAL` and this is production | Record a production approval, or run it outside production |
+
+The refusal message always names the property key, so the first case is a configuration line rather
+than a support question.
+
+---
+
+## xDS: the control plane went away
+
+**What you are seeing.** The control plane is unreachable and clients are still routing.
+
+**What it means.** `GrpcXdsFailurePolicy` serves the last-known-good snapshot, up to its staleness
+bound.
+
+**What to do.**
+
+1. Check the snapshot's age. `SERVE_LAST_KNOWN_GOOD` is the healthy degraded state.
+2. `STALE_BEYOND_BOUND` means the snapshot is older than the policy allows and is no longer trusted.
+   Beyond that bound, a decommissioned backend would otherwise keep receiving traffic indefinitely.
+3. `NO_SNAPSHOT_YET` on a starting instance means it never reached the control plane. It fails after
+   the initial fetch timeout rather than starting with no routing.
+
+**Do not** add application-level retry policy while xDS is enabled. `GrpcXdsStartupGuard` refuses it,
+because retry defined in two places has a winner that depends on resolution order rather than on a
+decision.
+
+---
+
+## gRPC-Web: a browser call hangs and then fails with no status
+
+**Almost always the proxy.** A gRPC status arrives as a trailer, and a browser cannot read a trailer
+the proxy did not expose. Check that the proxy's CORS `expose_headers` includes `grpc-status` and
+`grpc-message`; `GrpcWebProxyContract.violations` reports exactly this, and the reference
+configuration in `envoy/envoy.yaml` shows it in place.
+
+**If the method is client- or bidirectional-streaming**, it cannot work over gRPC-Web at all — a
+browser has no way to send a stream of messages. `GrpcWebCompatibilityGate` reports such a method
+before it is exposed.
+
+---
+
+## Servlet: a transport setting appears to be ignored
+
+It is ignored. The container owns the socket, so keepalive tuning, maximum connection age and
+flow-control window tuning belong to it. `GrpcServletStartupValidator` refuses those settings at
+startup rather than accepting and dropping them, because a setting that is silently ignored sends the
+investigation somewhere else.
+
+A Servlet run never substitutes for Netty certification.
+
+---
+
+## Hedging: backend load doubled
+
+**Expected, within a bound.** Hedging trades duplicate load for tail latency.
+`GrpcHedgingResult` records both `duplicateBackendCalls` and `cancelledLoserAttempts`; a dashboard
+showing only the latency improvement makes the trade look free.
+
+**What to check.** `GrpcHedgingBudget` caps hedges as a fraction of completed calls. If duplicate
+load is above that fraction, the budget is not being consumed — which means something is issuing
+hedges outside the coordinator.
+
+**Hedging is refused** for anything but a read-only unary method. A hedged mutation runs twice by
+design, and an idempotency key does not help: the second attempt duplicates a success in progress
+rather than retrying a failure.
+
+---
+
+## Promoting a capability
+
+`GrpcAdvancedPromotionGate.evaluate` names every missing item. Promotion to `ADVANCED_STABLE` needs
+compatibility evidence, a security review, fault evidence, performance evidence, an ADR, a runbook, a
+real-environment test and seven days of soak. A Stable default needs thirty.
+
+Promotions are independent: promoting one capability changes no other's grade, and
+`GrpcAdvancedSupportMatrix.apply` refuses a decision made against a different matrix state.
+
+Before citing a suite as evidence, check `GrpcAdvancedInfrastructureTestkit.missingInfrastructure`.
+A suite that ran without the proxy, the container, the control plane or the toolchain it needs passed
+and established nothing.

docs/runbooks/grpc-platform-operations.md

@@ -0,0 +1,142 @@
+# Runbook: gRPC platform operations
+
+Scope: the `:grpc:*` family. All of it is build-only today — every leaf's `runtime_memberships` is
+empty — so nothing here fires in production yet. It is written now because the states it covers are
+the ones an on-call cannot work out from first principles at three in the morning, and shipping the
+behaviour before the runbook means the first person to meet one is doing that.
+
+Configuration lives under `ca-skeleton.grpc.platform.*` and is bound by `GrpcPlatformProperties`.
+The platform does not start unless `ca-skeleton.grpc.platform.enabled=true`.
+
+---
+
+## COMPLETION_UNKNOWN on a mutation
+
+**What you are seeing.** A client received `DEADLINE_EXCEEDED`, `UNAVAILABLE` or `INTERNAL` on a
+state-changing call, and the response trailer `completion-outcome` reads `COMPLETION_UNKNOWN`.
+
+**What it means.** The server may have committed. This is not a failure and not a success; the status
+code cannot distinguish them, which is why the outcome is carried separately.
+
+**What not to do.** Do not re-issue the call. Do not tell the caller it failed. Both are wrong half
+the time, and which half is not knowable from the status.
+
+**What to do.**
+
+1. Take the `error-execution-id` from the trailers. It is the only link between what the client saw
+   and what the server did.
+2. If the method is `IDEMPOTENCY_KEY_REQUIRED`, query the operation ledger with the caller
+   fingerprint, the full method name and the caller's key. `GrpcOperationStatusQuery` returns one of
+   `IN_PROGRESS`, `COMMITTED`, `FAILED_TERMINAL`, `NOT_FOUND` or `UNKNOWN`.
+3. `COMMITTED` means return the stored outcome reference, not a freshly computed answer — the resource
+   may have changed since, and a new answer would describe the state at reconciliation time rather
+   than the state the caller's own call produced.
+4. `NOT_FOUND` means the operation never started and is safe to re-issue. `FAILED_TERMINAL` means the
+   same.
+5. `UNKNOWN` means the ledger could not be consulted. Nothing may be concluded. The case is queued by
+   `GrpcCompletionReconciler` and retried later.
+6. If the method is not keyed, there is no ledger row. Resolve it against the business resource, or
+   escalate to the service owner. This is the case the keyed profile exists to avoid.
+
+**Escalate when** the reconciler's pending list grows across passes. That means the ledger is
+unreachable rather than slow.
+
+---
+
+## A stream ended with FULL_RESYNC_REQUIRED
+
+**What you are seeing.** A client's resume was refused and it was told to resynchronise.
+
+**What it means.** The server can no longer replay from the client's cursor. Either the snapshot
+version moved, or the cursor predates retained history.
+
+**What to do.** Nothing on the server. The client is expected to discard its position and start a new
+stream from a fresh snapshot. A client that instead retries the same token will keep receiving the
+same answer.
+
+**Escalate when** it is happening to many clients at once. That usually means history retention was
+reduced, or snapshots are rotating faster than clients reconnect.
+
+---
+
+## A stream ended with SLOW_CONSUMER
+
+**What you are seeing.** Streams terminating with `SLOW_CONSUMER`, and
+`grpc.stream.flow_control_stalls` rising.
+
+**What it means.** The consumer could not keep up with the bounded queue. The stream was terminated
+rather than silently dropping messages, because a client cannot detect drops — the sequence numbers
+it sees are the ones it was sent.
+
+**What to do.**
+
+1. Check whether the consumer is slow or the producer is fast. `grpc.stream.messages` against
+   `grpc.stream.lifetime` tells you the rate.
+2. If the consumer is slow, the fix is on the consumer. Raising the queue bound moves the failure
+   later and makes it larger.
+3. A resume is not available after this ending: the messages that overflowed the queue are gone, so
+   continuing from the last delivered sequence would silently skip them. The client resynchronises.
+
+---
+
+## RESOURCE_EXHAUSTED under load
+
+**What you are seeing.** Calls refused with `RESOURCE_EXHAUSTED` and `GrpcAdmissionController`
+reporting rejections.
+
+**What it means.** The server is at its concurrency and queue bounds and is shedding rather than
+queueing. This is the designed behaviour: accepting work whose callers have already given up spends
+capacity on nothing.
+
+**What to do.**
+
+1. Read `grpc.rpc.duration` and `grpc.rpc.queue_wait` separately. Queue time rising with duration flat
+   means the bottleneck is admission, not the work.
+2. Check which saturation counter is moving — executor, channel or flow control. They look identical
+   in a latency graph and have different fixes.
+3. Raising `ca-skeleton.grpc.platform.executor-queue-capacity` defers the problem; it does not remove
+   it. `GrpcExecutorProfile` refuses a queue above ten thousand for that reason.
+
+---
+
+## A rollout is producing errors at every deploy
+
+**What you are seeing.** A burst of `UNAVAILABLE` or `CANCELLED` each time an instance goes away.
+
+**What it means.** The drain sequence is not completing, or is running out of order.
+
+**What to do.**
+
+1. `GrpcDrainResult` records what each drain achieved: completed and cancelled unary calls, signalled
+   and cancelled streams, and which phases ran. A drain that routinely force-cancels is the cause.
+2. The order matters. Readiness flips first and nothing is refused during that window, because there
+   is a gap between an instance reporting unready and routing acting on it. Refusing during that gap
+   turns a clean rollout into a burst of errors at every deploy.
+3. For long streams, check the Kubernetes profile's `streamReconnectBudget` and
+   `readinessDrainGrace`. A stream is pinned to one pod for its whole life, so every rollout ends it;
+   a profile with long streams and no reconnect budget has not decided what clients do next.
+
+---
+
+## Verifying a deployment's configuration
+
+`GrpcPlatformSnapshotService` produces a secret-free snapshot for a caller on the admin network
+holding an admin role. Both gates are required.
+
+`GrpcPlatformSnapshotService.driftAgainstRelease` compares a running snapshot with the release
+manifest and reports schema version, method policy hash and per-channel profile differences. An
+instance running a configuration the release did not ship is behind a whole class of incidents that
+are otherwise diagnosed by reading logs.
+
+The snapshot carries hashes and names only. A field whose name looks like a credential is refused at
+construction rather than redacted.
+
+---
+
+## Things that are deliberately off
+
+- **Reflection in production.** `GrpcReflectionMode.defaultFor` returns `DISABLED` for `STAGE` and
+  `PROD`. Reflection publishes the whole schema to anyone who can open a connection.
+- **Every advanced capability.** See `docs/runbooks/grpc-advanced-capabilities.md`.
+- **The platform itself.** `ca-skeleton.grpc.platform.enabled` defaults to false, and every `:grpc:*`
+  leaf is build-only in the registry.