# Structured Slack Observability Notifications Implementation Plan > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. **Goal:** Replace Alertmanager's terse default Slack attachment with a structured, severity-aware message and route `InfoInhibitor` to a null receiver while preserving the existing webhook, transactional apply, and rollback boundaries. **Architecture:** Keep formatting inside `AlertmanagerConfig/platform-alertmanager`; do not add a formatter service. Strengthen the existing source, core-render, access-render, and apply assertions so the same exact configuration is enforced at every boundary. Runtime acceptance changes from one qualified receiver to the exact qualified pair for Slack and null. **Tech Stack:** Kubernetes `AlertmanagerConfig` v1alpha1, Prometheus Alertmanager 0.33.1 Go templates, Bash, embedded Python/PyYAML, existing renderer and rules-alerts transaction. ## Global Constraints - Approved design: `/home/donghyeon/workspace/docs/platform/plans/2026-08-15-observability-structured-slack-notifications-design.md`. - The Slack Secret selector remains exactly `alertmanager-slack-webhook` / `url`; never read, print, copy, decode, or persist its payload. - Do not set a Slack `channel`, automatic mention, webhook literal, or new credential source. - Route grouping is the exact set `cluster`, `namespace`, `alertname`, `severity`; timing remains `30s`, `5m`, `4h`. - Source receivers are exactly `platform-null` and `platform-slack`; `platform-null` has no integrations. - Runtime receivers are the order-independent exact set `observability/platform-alertmanager/platform-null` and `observability/platform-alertmanager/platform-slack`. - `InfoInhibitor` routes to `platform-null`; its inhibition rule targets `severity=info` with equality on `namespace`. Preserve the upstream conditional semantics rather than claiming all info alerts are always dropped. - Colors are resolved=`good`, emergency/critical=`danger`, warning=`warning`, and info/unknown=`#439FE0`. - Target text uses all labels after removing group-label names, sorted by label name; missing values use explicit fallbacks. - No alert expression, threshold, duration, NetworkPolicy, dashboard JSON, retention setting, or Slack destination changes. - No new formatter workload or generic automation script. - This is a non-Git workspace. Record starting/final SHA-256 and metadata in task reports instead of creating commits or worktrees. - Production changes require a real failing test first. Run each expensive focused suite at most once for RED and once for final GREEN unless a genuine failure requires a fix. - Do not execute live apply until source review is clean. Direct `kubectl apply` is forbidden; use the existing rules-alerts transaction exactly once with a fresh rollback ID and fresh verified handoff. ## Exact AlertmanagerConfig Contract The canonical source and every exact oracle use this `spec` value. YAML block scalars preserve the template text exactly. ```yaml spec: route: receiver: platform-slack groupBy: - cluster - namespace - alertname - severity groupWait: 30s groupInterval: 5m repeatInterval: 4h routes: - receiver: platform-null matchers: - name: alertname matchType: "=" value: InfoInhibitor inhibitRules: - sourceMatch: - name: alertname matchType: "=" value: InfoInhibitor targetMatch: - name: severity matchType: "=" value: info equal: - namespace receivers: - name: platform-null - name: platform-slack slackConfigs: - apiURL: name: alertmanager-slack-webhook key: url sendResolved: true linkNames: false mrkdwnIn: - text - fields fallback: >- {{ if eq .Status "firing" }}FIRING{{ else }}RESOLVED{{ end }}: {{ if .CommonLabels.severity }}{{ .CommonLabels.severity | toUpper }}{{ else }}UNKNOWN{{ end }} · {{ .CommonLabels.alertname }} title: >- {{ if eq .Status "firing" }}[FIRING:{{ .Alerts.Firing | len }}]{{ else }}[RESOLVED]{{ end }} {{ if .CommonLabels.severity }}{{ .CommonLabels.severity | toUpper }}{{ else }}UNKNOWN{{ end }} · {{ .CommonLabels.alertname }} titleLink: https://grafana.learn.hyeonworks.com/ color: >- {{ if eq .Status "resolved" }}good{{ else if or (eq .CommonLabels.severity "emergency") (eq .CommonLabels.severity "critical") }}danger{{ else if eq .CommonLabels.severity "warning" }}warning{{ else }}#439FE0{{ end }} fields: - title: Status value: '{{ .Status | toUpper }}' short: true - title: Severity value: '{{ if .CommonLabels.severity }}{{ .CommonLabels.severity | toUpper }}{{ else }}UNKNOWN{{ end }}' short: true - title: Location value: '{{ if .CommonLabels.cluster }}{{ .CommonLabels.cluster }}{{ else }}unknown-cluster{{ end }} / {{ if .CommonLabels.namespace }}{{ .CommonLabels.namespace }}{{ else }}cluster-scoped{{ end }}' short: true - title: Alert count value: '{{ len .Alerts }}' short: true text: |- {{ range .Alerts }} *Alert status:* {{ .Status | toUpper }} *Target:* {{ $target := .Labels.Remove $.GroupLabels.Names }}{{ if $target }}{{ range $target.SortedPairs }}{{ .Name }}={{ .Value }} {{ end }}{{ else }}unknown{{ end }} *Summary:* {{ with .Annotations.summary }}{{ . }}{{ else }}No summary provided{{ end }} *Details:* {{ with .Annotations.description }}{{ . }}{{ else }}No description provided{{ end }} *Started:* {{ .StartsAt.Format "2006-01-02T15:04:05Z07:00" }} {{ if eq .Status "resolved" }}*Ended:* {{ .EndsAt.Format "2006-01-02T15:04:05Z07:00" }}{{ end }} {{ with .Annotations.runbook_url }}*Runbook:* <{{ . }}|대응 절차 열기>{{ end }} {{ end }} · footer: hyeonworks observability · Alertmanager ``` --- ### Task 1: Source and Core Renderer Exact Contract **Files:** - Modify: `scripts/validate/test-observability-alerting.sh` - Modify: `scripts/validate/test-render-observability-core.sh` - Modify: `services/observability/alerting/platform-alertmanager.yaml` - Modify: `scripts/validate/render-observability-core.sh` - Create: `.superpowers/sdd/2026-08-15-observability-structured-slack-notifications/task-1-report.md` **Interfaces:** - Consumes: the exact AlertmanagerConfig contract above. - Produces: source YAML and core renderer assertion that accept only that contract; later tasks rely on `assert_alertmanager_routing_contract` as the canonical oracle. - [ ] **Step 1: Record the starting identities** Record SHA-256, mode, owner, nlink, and size for the four modified files in the task report. Starting SHA-256 values must be: ```text platform-alertmanager.yaml 76db5e6dd3653bc8afb9ecffe3472fc57eb07c5a2b33d218f3f4a41fb202caa2 test-observability-alerting.sh c5c578435cfeaac0d4221dd335797e09aaa010a9f63c397d21936a252e461db2 render-observability-core.sh 14460b985489ca59ba2377a61300a187d84fe795c8307fe2ba32bee9674d4bb2 test-render-observability-core.sh 9dd935e403968bf063ec0ddd44cd32ba853fe9f121a2764fbb01206d3c561d35 ``` - [ ] **Step 2: Write the failing source and core tests** In `test-observability-alerting.sh`, replace the old exact `alert` oracle with the exact contract above, retaining exact metadata and API identity. In `test-render-observability-core.sh`, update the valid Alertmanager fixture to the exact contract and add independently hand-authored negative fixtures for: ```text missing severity from groupBy missing title color that maps critical to warning InfoInhibitor child route pointing to platform-slack missing inhibit rule platform-null containing a Slack integration channel override text containing @channel extra receiver ``` Each negative must exercise `assert_alertmanager_routing_contract` and expect rejection. - [ ] **Step 3: Run RED against unchanged production** Run: ```bash bash -n scripts/validate/test-observability-alerting.sh bash -n scripts/validate/test-render-observability-core.sh bash scripts/validate/test-observability-alerting.sh bash scripts/validate/test-render-observability-core.sh ``` Expected: both syntax checks return 0; at least one behavioral command returns nonzero because production still has the old three-key group and terse singleton receiver. Capture the exact first failing leaf. A syntax or fixture-construction error is not an acceptable RED. - [ ] **Step 4: Implement the minimal source and canonical oracle** Use `apply_patch`. Replace only the `spec` in `platform-alertmanager.yaml` with the exact contract above. Replace `render-observability-core.sh` mode `alertmanager_routing`'s `expected_config` with the exact full object, including receiver order, every template string, child route, and inhibit rule. Do not change the NetworkPolicy oracle or Alertmanager global-reference checks. - [ ] **Step 5: Run GREEN and mutation checks** Run both commands from Step 3 again. Expected exact terminal markers: ```text OBSERVABILITY ALERTING STATIC CONTRACT PASS OBSERVABILITY ALERTMANAGER GLOBAL RENDER CONTRACT PASS OBSERVABILITY CORE STATIC RENDER PASS ``` Confirm every negative fixture returns nonzero while the canonical fixture returns zero. Record assertion counts and final identities in `task-1-report.md`. ### Task 2: Access Renderer and Transaction Runtime Contract **Files:** - Modify: `scripts/validate/test-render-observability-access.sh` - Modify: `scripts/validate/render-observability-access.sh` - Modify: `scripts/validate/test-apply-observability-access.sh` - Modify: `scripts/bootstrap/apply-observability-access.sh` - Create: `.superpowers/sdd/2026-08-15-observability-structured-slack-notifications/task-2-report.md` **Interfaces:** - Consumes: Task 1's `assert_alertmanager_routing_contract` and exact source configuration. - Produces: an access handoff that rejects altered templates/routes and a transaction postcheck that accepts only the qualified receiver pair. - [ ] **Step 1: Record starting identities** Starting SHA-256 values must be: ```text render-observability-access.sh ae5373b1b20be42e573d58e7384dcc654b1d3cac75a6c269cf684f4a14f3ca1a test-render-observability-access.sh 5dfdc52b9f72b6d2e754893e799f5baf27c2b87e701958452dc170627ff21435 apply-observability-access.sh f797c3c678e45adb08c407f9a55b6da10fa49d93038cd0b301f589555b11263c test-apply-observability-access.sh dd78ef4601c89652b9270a7fcb257232a9f54431f65b96669062cc0efa920e4f ``` - [ ] **Step 2: Write access RED cases** Update the valid fixtures to the exact Task 1 contract. In `test-render-observability-access.sh`, add mutations that remove `title`, change `groupBy`, route `InfoInhibitor` to Slack, add `channel`, and add an extra receiver. Each must fail at the published handoff assertion with no partial output mutation. Run: ```bash bash -n scripts/validate/test-render-observability-access.sh bash scripts/validate/test-render-observability-access.sh ``` Expected: syntax 0 and behavioral nonzero against the current loose `rules-alerts` access assertion; capture the leaf showing an invalid structured contract was accepted. - [ ] **Step 3: Write apply RED cases** Update the fake `/api/v2/receivers` default to the qualified pair. Define literal cases: ```python qualified_pair = [ {"name": "observability/platform-alertmanager/platform-null"}, {"name": "observability/platform-alertmanager/platform-slack"}, ] ``` Add one accepted reversed-order case and rejected cases for old Slack singleton, null singleton, empty list, raw pair, duplicate member, extra receiver, malformed JSON, and wrong namespace/config/local name for either member. Add a structured-handoff mutation that must fail before prompt, ledger creation, dry-run, or apply. Run the smallest named positive case or the bounded focused suite until the new qualified pair reaches the old singleton predicate. Expected RED is nonzero at the receiver postcheck, not a fixture or syntax error. - [ ] **Step 4: Strengthen production access and apply checks** In `render-observability-access.sh`, replace the one-line shell wrapper with: ```bash assert_access_rules_alerts_contract() { (( $# == 3 )) || return 1 _access_assert rules-alerts "$@" || return 1 assert_alertmanager_routing_contract "$3" } ``` Keep the existing dashboard, rule, runbook, scope, and NetworkPolicy checks. In the apply postcheck Python, replace singleton logic with: ```python expected_receivers = { "observability/platform-alertmanager/platform-null", "observability/platform-alertmanager/platform-slack", } if ( not isinstance(receiver_payload, list) or len(receiver_payload) != 2 or any(not isinstance(item, dict) or set(item) != {"name"} for item in receiver_payload) or {item["name"] for item in receiver_payload} != expected_receivers ): raise SystemExit(1) ``` Do not change rollback, ledger, resource rows, Slack gate, Secret name/key validation, or acceptance schema. - [ ] **Step 5: Run final focused GREEN once** Run syntax for all four files, then: ```bash bash scripts/validate/test-render-observability-access.sh timeout 330s bash scripts/validate/test-apply-observability-access.sh ``` Expected: access terminal PASS; apply terminal `APPLY OBSERVABILITY ACCESS TEST PASS`, the documented assertion count, suite wall at most 300 seconds, outer RC0, and relevant orphan count 0. Verify every receiver mutation fails at its intended leaf. Record final identities and evidence in `task-2-report.md`. ### Task 3: Integrated Schema, Template, and Independent Source Review **Files:** - Modify only if the RED demands it: `scripts/validate/test-observability-alerting.sh` - Create: `.superpowers/sdd/2026-08-15-observability-structured-slack-notifications/task-3-report.md` **Interfaces:** - Consumes: Tasks 1-2 frozen candidate bytes. - Produces: pre-live Ready/Not Ready verdict and exact candidate identities. - [ ] **Step 1: Run complete syntax and source-focused matrix** Run Bash syntax on all eight changed scripts. Run `test-observability-alerting.sh` without live mode once, then the already-green core/access/apply commands only if their final Task reports do not contain unambiguous RC0 evidence for the same hashes. Do not rerun an expensive suite merely to duplicate evidence. - [ ] **Step 2: Validate CRD admission without mutation** Run: ```bash PLATFORM_KUBECTL_BIN=/usr/local/bin/kubectl \ bash scripts/validate/test-observability-alerting.sh --server-dry-run ``` Expected: both server-side dry runs and all static markers pass. This is schema evidence only. - [ ] **Step 3: Render real templates with pinned Alertmanager 0.33.1** Extract `fallback`, `title`, `color`, all field values, and `text` from the candidate YAML without exposing any Secret value. Use `/bin/amtool` in the existing Alertmanager 0.33.1 container with `amtool template render`; pass a hand-authored JSON fixture through `/dev/stdin`, not a persistent pod file. The fixture must cover: ```text warning firing with cluster and namespace resolved alert with EndsAt missing cluster, namespace, summary, description, and runbook two alerts with different non-group labels ``` Assert literal results include the correct severity color, `unknown-cluster`, `cluster-scoped`, sorted target labels, both fallbacks, FIRING count, RESOLVED title, runbook when present, Grafana links, and footer. No Secret or webhook URL enters the fixture or output. - [ ] **Step 4: Perform independent read-only review** The reviewer reads the design, plan, task reports, final changed files, and current CRD. Required verdict fields are Spec PASS/FAIL, Critical/Important/Minor counts, and Ready for one transactional live apply YES/NO. Any Critical or Important finding must be fixed with a new failing test and re-reviewed before Task 4. - [ ] **Step 5: Freeze pre-live identities** Record SHA-256 and metadata for all eight source/test files plus the design, plan, and Task reports. Confirm no webhook literal, decoded Secret, `channel`, `@channel`, or `@here` occurs in governed source. Record process/temp residue and mark Task 3 Ready only when all checks pass. ### Task 4: Transactional Rollout, Synthetic Delivery, and Documentation **Files:** - Modify after observed results: `/home/donghyeon/workspace/docs/platform/observability/03-alerting-and-slack.md` - Modify after observed results: `/home/donghyeon/workspace/docs/platform/observability/07-build-history-and-decisions.md` - Create: `.superpowers/sdd/2026-08-15-observability-structured-slack-notifications/task-4-report.md` **Interfaces:** - Consumes: Task 3 Ready verdict and frozen candidate identities. - Produces: live structured Slack configuration, observed FIRING/RESOLVED acceptance, cleanup evidence, and final operator documentation. - [ ] **Step 1: Prepare one fresh transaction** Use a fresh UTC rollback ID unequal to every previous ID and create its root once as root-owned mode 0700. Preserve all prior rollback roots. Generate one fresh private handoff from the authoritative immutable inventory pair: ```text target-initial 79688d017d38eec9a6f100f8d0f784a5474e79802046ef1c2c11b30d170b0b0c 21 post-substrate b1c3049206a1a88165ee672ae9aceac7945673a3bb9c3cf3670b7f0d56c3f291 30 ``` Render rules-alerts into the fresh root and verify exact source hashes, metadata, topology, and new structured AlertmanagerConfig. Run the no-argument dry mode once. Do not reuse a prior handoff or rollback ID. - [ ] **Step 2: Execute the existing transaction once** Build a six-token Bash array to avoid paste splitting: ```bash TASK4_CMD=(bash) TASK4_CMD+=(scripts/bootstrap/apply-observability-access.sh) TASK4_CMD+=(--execute) TASK4_CMD+=(--rules-alerts) TASK4_CMD+=(--verified-output-dir) TASK4_CMD+=("$METRIC_ROOT") ``` Export the fresh `PLATFORM_OBSERVABILITY_ROLLBACK_ID` and exact `PLATFORM_HELM_BIN=/home/donghyeon/.local/bin/helm`. Invoke `"${TASK4_CMD[@]}"` once, enter exact `APPLY`, and make the immediate next statement `TASK4_APPLY_RC=$?`. Require RC0, exact transaction PASS, both inventory SHA markers, no rollback ambiguity, and accepted v2 ledger. A nonzero or ambiguous result means preserve evidence and stop; never retry the same ID. - [ ] **Step 3: Verify runtime reconciliation** Require Alertmanager CR Available/Reconciled, StatefulSet ready, `alertmanager_config_last_reload_successful == 1`, no relevant operator/config-reloader error, production rules healthy, and the exact qualified receiver pair. AlertmanagerConfig itself has no usable status condition and must not be reported as Available/Reconciled. - [ ] **Step 4: Verify one warning FIRING and RESOLVED message** Create one uniquely named, temporary warning `PrometheusRule` with the required `observability.hyeonworks.com/instance=home` selector label, literal summary/description/runbook, and a unique target label. Wait until Prometheus and Alertmanager each show exactly one matching firing alert. The user verifies that Slack contains the agreed title, severity, fields, target labels, summary, details, start time, runbook, Grafana links, and footer. Delete the temporary rule once. Wait for the matching RESOLVED message and have the user verify its resolved title, green color, end time, and same context. The synthetic rule name and labels must never collide with production rules. - [ ] **Step 5: Clean up and document only observed facts** Require synthetic resource absent, matching pending/firing/active alerts zero, production rules healthy, receiver pair exact, and no mutator process or task-created temp residue. Confirm `InfoInhibitor` is not delivered to Slack. Document the conditional upstream info-inhibition semantics; do not claim every info alert is always suppressed. Update chapter 03 with the exact source contract and observed FIRING/RESOLVED results. Update chapter 07 with the original terse-message problem, the decision to keep formatting in Alertmanager, and the final transaction result. Record final file identities and any limitation, including long grouped Slack text, in `task-4-report.md`.