platform-core/docs/superpowers/plans/2026-08-15-observability-structured-slack-notifications.md

Structured Slack Observability Notifications Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Replace Alertmanager's terse default Slack attachment with a structured, severity-aware message and route InfoInhibitor to a null receiver while preserving the existing webhook, transactional apply, and rollback boundaries.

Architecture: Keep formatting inside AlertmanagerConfig/platform-alertmanager; do not add a formatter service. Strengthen the existing source, core-render, access-render, and apply assertions so the same exact configuration is enforced at every boundary. Runtime acceptance changes from one qualified receiver to the exact qualified pair for Slack and null.

Tech Stack: Kubernetes AlertmanagerConfig v1alpha1, Prometheus Alertmanager 0.33.1 Go templates, Bash, embedded Python/PyYAML, existing renderer and rules-alerts transaction.

Global Constraints

• Approved design: /home/donghyeon/workspace/docs/platform/plans/2026-08-15-observability-structured-slack-notifications-design.md.
• The Slack Secret selector remains exactly alertmanager-slack-webhook / url; never read, print, copy, decode, or persist its payload.
• Do not set a Slack channel, automatic mention, webhook literal, or new credential source.
• Route grouping is the exact set cluster, namespace, alertname, severity; timing remains 30s, 5m, 4h.
• Source receivers are exactly platform-null and platform-slack; platform-null has no integrations.
• Runtime receivers are the order-independent exact set observability/platform-alertmanager/platform-null and observability/platform-alertmanager/platform-slack.
• InfoInhibitor routes to platform-null; its inhibition rule targets severity=info with equality on namespace. Preserve the upstream conditional semantics rather than claiming all info alerts are always dropped.
• Colors are resolved=good, emergency/critical=danger, warning=warning, and info/unknown=#439FE0.
• Target text uses all labels after removing group-label names, sorted by label name; missing values use explicit fallbacks.
• No alert expression, threshold, duration, NetworkPolicy, dashboard JSON, retention setting, or Slack destination changes.
• No new formatter workload or generic automation script.
• This is a non-Git workspace. Record starting/final SHA-256 and metadata in task reports instead of creating commits or worktrees.
• Production changes require a real failing test first. Run each expensive focused suite at most once for RED and once for final GREEN unless a genuine failure requires a fix.
• Do not execute live apply until source review is clean. Direct kubectl apply is forbidden; use the existing rules-alerts transaction exactly once with a fresh rollback ID and fresh verified handoff.

Exact AlertmanagerConfig Contract

The canonical source and every exact oracle use this spec value. YAML block scalars preserve the template text exactly.

spec:
  route:
    receiver: platform-slack
    groupBy:
      - cluster
      - namespace
      - alertname
      - severity
    groupWait: 30s
    groupInterval: 5m
    repeatInterval: 4h
    routes:
      - receiver: platform-null
        matchers:
          - name: alertname
            matchType: "="
            value: InfoInhibitor
  inhibitRules:
    - sourceMatch:
        - name: alertname
          matchType: "="
          value: InfoInhibitor
      targetMatch:
        - name: severity
          matchType: "="
          value: info
      equal:
        - namespace
  receivers:
    - name: platform-null
    - name: platform-slack
      slackConfigs:
        - apiURL:
            name: alertmanager-slack-webhook
            key: url
          sendResolved: true
          linkNames: false
          mrkdwnIn:
            - text
            - fields
          fallback: >-
            {{ if eq .Status "firing" }}FIRING{{ else }}RESOLVED{{ end }}: {{ if .CommonLabels.severity }}{{ .CommonLabels.severity | toUpper }}{{ else }}UNKNOWN{{ end }} · {{ .CommonLabels.alertname }}
          title: >-
            {{ if eq .Status "firing" }}[FIRING:{{ .Alerts.Firing | len }}]{{ else }}[RESOLVED]{{ end }} {{ if .CommonLabels.severity }}{{ .CommonLabels.severity | toUpper }}{{ else }}UNKNOWN{{ end }} · {{ .CommonLabels.alertname }}
          titleLink: https://grafana.learn.hyeonworks.com/
          color: >-
            {{ if eq .Status "resolved" }}good{{ else if or (eq .CommonLabels.severity "emergency") (eq .CommonLabels.severity "critical") }}danger{{ else if eq .CommonLabels.severity "warning" }}warning{{ else }}#439FE0{{ end }}
          fields:
            - title: Status
              value: '{{ .Status | toUpper }}'
              short: true
            - title: Severity
              value: '{{ if .CommonLabels.severity }}{{ .CommonLabels.severity | toUpper }}{{ else }}UNKNOWN{{ end }}'
              short: true
            - title: Location
              value: '{{ if .CommonLabels.cluster }}{{ .CommonLabels.cluster }}{{ else }}unknown-cluster{{ end }} / {{ if .CommonLabels.namespace }}{{ .CommonLabels.namespace }}{{ else }}cluster-scoped{{ end }}'
              short: true
            - title: Alert count
              value: '{{ len .Alerts }}'
              short: true
          text: |-
            {{ range .Alerts }}
            *Alert status:* {{ .Status | toUpper }}
            *Target:* {{ $target := .Labels.Remove $.GroupLabels.Names }}{{ if $target }}{{ range $target.SortedPairs }}{{ .Name }}={{ .Value }} {{ end }}{{ else }}unknown{{ end }}
            *Summary:* {{ with .Annotations.summary }}{{ . }}{{ else }}No summary provided{{ end }}
            *Details:* {{ with .Annotations.description }}{{ . }}{{ else }}No description provided{{ end }}
            *Started:* {{ .StartsAt.Format "2006-01-02T15:04:05Z07:00" }}
            {{ if eq .Status "resolved" }}*Ended:* {{ .EndsAt.Format "2006-01-02T15:04:05Z07:00" }}{{ end }}
            {{ with .Annotations.runbook_url }}*Runbook:* <{{ . }}|대응 절차 열기>{{ end }}

            {{ end }}
            <https://grafana.learn.hyeonworks.com/|Grafana> · <https://grafana.learn.hyeonworks.com/explore|Explore>
          footer: hyeonworks observability · Alertmanager

---

Task 1: Source and Core Renderer Exact Contract

Files:

• Modify: scripts/validate/test-observability-alerting.sh
• Modify: scripts/validate/test-render-observability-core.sh
• Modify: services/observability/alerting/platform-alertmanager.yaml
• Modify: scripts/validate/render-observability-core.sh
• Create: .superpowers/sdd/2026-08-15-observability-structured-slack-notifications/task-1-report.md

Interfaces:

• Consumes: the exact AlertmanagerConfig contract above.

• Produces: source YAML and core renderer assertion that accept only that contract; later tasks rely on assert_alertmanager_routing_contract as the canonical oracle.

• Step 1: Record the starting identities

Record SHA-256, mode, owner, nlink, and size for the four modified files in the task report. Starting SHA-256 values must be:

platform-alertmanager.yaml              76db5e6dd3653bc8afb9ecffe3472fc57eb07c5a2b33d218f3f4a41fb202caa2
test-observability-alerting.sh          c5c578435cfeaac0d4221dd335797e09aaa010a9f63c397d21936a252e461db2
render-observability-core.sh            14460b985489ca59ba2377a61300a187d84fe795c8307fe2ba32bee9674d4bb2
test-render-observability-core.sh       9dd935e403968bf063ec0ddd44cd32ba853fe9f121a2764fbb01206d3c561d35

• Step 2: Write the failing source and core tests

In test-observability-alerting.sh, replace the old exact alert oracle with the exact contract above, retaining exact metadata and API identity. In test-render-observability-core.sh, update the valid Alertmanager fixture to the exact contract and add independently hand-authored negative fixtures for:

missing severity from groupBy
missing title
color that maps critical to warning
InfoInhibitor child route pointing to platform-slack
missing inhibit rule
platform-null containing a Slack integration
channel override
text containing @channel
extra receiver

Each negative must exercise assert_alertmanager_routing_contract and expect rejection.

• Step 3: Run RED against unchanged production

Run:

bash -n scripts/validate/test-observability-alerting.sh
bash -n scripts/validate/test-render-observability-core.sh
bash scripts/validate/test-observability-alerting.sh
bash scripts/validate/test-render-observability-core.sh

Expected: both syntax checks return 0; at least one behavioral command returns nonzero because production still has the old three-key group and terse singleton receiver. Capture the exact first failing leaf. A syntax or fixture-construction error is not an acceptable RED.

• Step 4: Implement the minimal source and canonical oracle

Use apply_patch. Replace only the spec in platform-alertmanager.yaml with the exact contract above. Replace render-observability-core.sh mode alertmanager_routing's expected_config with the exact full object, including receiver order, every template string, child route, and inhibit rule. Do not change the NetworkPolicy oracle or Alertmanager global-reference checks.

• Step 5: Run GREEN and mutation checks

Run both commands from Step 3 again. Expected exact terminal markers:

OBSERVABILITY ALERTING STATIC CONTRACT PASS
OBSERVABILITY ALERTMANAGER GLOBAL RENDER CONTRACT PASS
OBSERVABILITY CORE STATIC RENDER PASS

Confirm every negative fixture returns nonzero while the canonical fixture returns zero. Record assertion counts and final identities in task-1-report.md.

Task 2: Access Renderer and Transaction Runtime Contract

Files:

• Modify: scripts/validate/test-render-observability-access.sh
• Modify: scripts/validate/render-observability-access.sh
• Modify: scripts/validate/test-apply-observability-access.sh
• Modify: scripts/bootstrap/apply-observability-access.sh
• Create: .superpowers/sdd/2026-08-15-observability-structured-slack-notifications/task-2-report.md

Interfaces:

• Consumes: Task 1's assert_alertmanager_routing_contract and exact source configuration.

• Produces: an access handoff that rejects altered templates/routes and a transaction postcheck that accepts only the qualified receiver pair.

• Step 1: Record starting identities

Starting SHA-256 values must be:

render-observability-access.sh          ae5373b1b20be42e573d58e7384dcc654b1d3cac75a6c269cf684f4a14f3ca1a
test-render-observability-access.sh     5dfdc52b9f72b6d2e754893e799f5baf27c2b87e701958452dc170627ff21435
apply-observability-access.sh           f797c3c678e45adb08c407f9a55b6da10fa49d93038cd0b301f589555b11263c
test-apply-observability-access.sh      dd78ef4601c89652b9270a7fcb257232a9f54431f65b96669062cc0efa920e4f

• Step 2: Write access RED cases

Update the valid fixtures to the exact Task 1 contract. In test-render-observability-access.sh, add mutations that remove title, change groupBy, route InfoInhibitor to Slack, add channel, and add an extra receiver. Each must fail at the published handoff assertion with no partial output mutation.

Run:

bash -n scripts/validate/test-render-observability-access.sh
bash scripts/validate/test-render-observability-access.sh

Expected: syntax 0 and behavioral nonzero against the current loose rules-alerts access assertion; capture the leaf showing an invalid structured contract was accepted.

• Step 3: Write apply RED cases

Update the fake /api/v2/receivers default to the qualified pair. Define literal cases:

qualified_pair = [
    {"name": "observability/platform-alertmanager/platform-null"},
    {"name": "observability/platform-alertmanager/platform-slack"},
]

Add one accepted reversed-order case and rejected cases for old Slack singleton, null singleton, empty list, raw pair, duplicate member, extra receiver, malformed JSON, and wrong namespace/config/local name for either member. Add a structured-handoff mutation that must fail before prompt, ledger creation, dry-run, or apply.

Run the smallest named positive case or the bounded focused suite until the new qualified pair reaches the old singleton predicate. Expected RED is nonzero at the receiver postcheck, not a fixture or syntax error.

• Step 4: Strengthen production access and apply checks

In render-observability-access.sh, replace the one-line shell wrapper with:

assert_access_rules_alerts_contract() {
  (( $# == 3 )) || return 1
  _access_assert rules-alerts "$@" || return 1
  assert_alertmanager_routing_contract "$3"
}

Keep the existing dashboard, rule, runbook, scope, and NetworkPolicy checks.

In the apply postcheck Python, replace singleton logic with:

expected_receivers = {
    "observability/platform-alertmanager/platform-null",
    "observability/platform-alertmanager/platform-slack",
}
if (
    not isinstance(receiver_payload, list)
    or len(receiver_payload) != 2
    or any(not isinstance(item, dict) or set(item) != {"name"} for item in receiver_payload)
    or {item["name"] for item in receiver_payload} != expected_receivers
):
    raise SystemExit(1)

Do not change rollback, ledger, resource rows, Slack gate, Secret name/key validation, or acceptance schema.

• Step 5: Run final focused GREEN once

Run syntax for all four files, then:

bash scripts/validate/test-render-observability-access.sh
timeout 330s bash scripts/validate/test-apply-observability-access.sh

Expected: access terminal PASS; apply terminal APPLY OBSERVABILITY ACCESS TEST PASS, the documented assertion count, suite wall at most 300 seconds, outer RC0, and relevant orphan count 0. Verify every receiver mutation fails at its intended leaf. Record final identities and evidence in task-2-report.md.

Task 3: Integrated Schema, Template, and Independent Source Review

Files:

• Modify only if the RED demands it: scripts/validate/test-observability-alerting.sh
• Create: .superpowers/sdd/2026-08-15-observability-structured-slack-notifications/task-3-report.md

Interfaces:

• Consumes: Tasks 1-2 frozen candidate bytes.

• Produces: pre-live Ready/Not Ready verdict and exact candidate identities.

• Step 1: Run complete syntax and source-focused matrix

Run Bash syntax on all eight changed scripts. Run test-observability-alerting.sh without live mode once, then the already-green core/access/apply commands only if their final Task reports do not contain unambiguous RC0 evidence for the same hashes. Do not rerun an expensive suite merely to duplicate evidence.

• Step 2: Validate CRD admission without mutation

Run:

PLATFORM_KUBECTL_BIN=/usr/local/bin/kubectl \
  bash scripts/validate/test-observability-alerting.sh --server-dry-run

Expected: both server-side dry runs and all static markers pass. This is schema evidence only.

• Step 3: Render real templates with pinned Alertmanager 0.33.1

Extract fallback, title, color, all field values, and text from the candidate YAML without exposing any Secret value. Use /bin/amtool in the existing Alertmanager 0.33.1 container with amtool template render; pass a hand-authored JSON fixture through /dev/stdin, not a persistent pod file. The fixture must cover:

warning firing with cluster and namespace
resolved alert with EndsAt
missing cluster, namespace, summary, description, and runbook
two alerts with different non-group labels

Assert literal results include the correct severity color, unknown-cluster, cluster-scoped, sorted target labels, both fallbacks, FIRING count, RESOLVED title, runbook when present, Grafana links, and footer. No Secret or webhook URL enters the fixture or output.

• Step 4: Perform independent read-only review

The reviewer reads the design, plan, task reports, final changed files, and current CRD. Required verdict fields are Spec PASS/FAIL, Critical/Important/Minor counts, and Ready for one transactional live apply YES/NO. Any Critical or Important finding must be fixed with a new failing test and re-reviewed before Task 4.

• Step 5: Freeze pre-live identities

Record SHA-256 and metadata for all eight source/test files plus the design, plan, and Task reports. Confirm no webhook literal, decoded Secret, channel, @channel, or @here occurs in governed source. Record process/temp residue and mark Task 3 Ready only when all checks pass.

Task 4: Transactional Rollout, Synthetic Delivery, and Documentation

Files:

• Modify after observed results: /home/donghyeon/workspace/docs/platform/observability/03-alerting-and-slack.md
• Modify after observed results: /home/donghyeon/workspace/docs/platform/observability/07-build-history-and-decisions.md
• Create: .superpowers/sdd/2026-08-15-observability-structured-slack-notifications/task-4-report.md

Interfaces:

• Consumes: Task 3 Ready verdict and frozen candidate identities.

• Produces: live structured Slack configuration, observed FIRING/RESOLVED acceptance, cleanup evidence, and final operator documentation.

• Step 1: Prepare one fresh transaction

Use a fresh UTC rollback ID unequal to every previous ID and create its root once as root-owned mode 0700. Preserve all prior rollback roots. Generate one fresh private handoff from the authoritative immutable inventory pair:

target-initial  79688d017d38eec9a6f100f8d0f784a5474e79802046ef1c2c11b30d170b0b0c  21
post-substrate  b1c3049206a1a88165ee672ae9aceac7945673a3bb9c3cf3670b7f0d56c3f291  30

Render rules-alerts into the fresh root and verify exact source hashes, metadata, topology, and new structured AlertmanagerConfig. Run the no-argument dry mode once. Do not reuse a prior handoff or rollback ID.

• Step 2: Execute the existing transaction once

Build a six-token Bash array to avoid paste splitting:

TASK4_CMD=(bash)
TASK4_CMD+=(scripts/bootstrap/apply-observability-access.sh)
TASK4_CMD+=(--execute)
TASK4_CMD+=(--rules-alerts)
TASK4_CMD+=(--verified-output-dir)
TASK4_CMD+=("$METRIC_ROOT")

Export the fresh PLATFORM_OBSERVABILITY_ROLLBACK_ID and exact PLATFORM_HELM_BIN=/home/donghyeon/.local/bin/helm. Invoke "${TASK4_CMD[@]}" once, enter exact APPLY, and make the immediate next statement TASK4_APPLY_RC=$?. Require RC0, exact transaction PASS, both inventory SHA markers, no rollback ambiguity, and accepted v2 ledger. A nonzero or ambiguous result means preserve evidence and stop; never retry the same ID.

• Step 3: Verify runtime reconciliation

Require Alertmanager CR Available/Reconciled, StatefulSet ready, alertmanager_config_last_reload_successful == 1, no relevant operator/config-reloader error, production rules healthy, and the exact qualified receiver pair. AlertmanagerConfig itself has no usable status condition and must not be reported as Available/Reconciled.

• Step 4: Verify one warning FIRING and RESOLVED message

Create one uniquely named, temporary warning PrometheusRule with the required observability.hyeonworks.com/instance=home selector label, literal summary/description/runbook, and a unique target label. Wait until Prometheus and Alertmanager each show exactly one matching firing alert. The user verifies that Slack contains the agreed title, severity, fields, target labels, summary, details, start time, runbook, Grafana links, and footer.

Delete the temporary rule once. Wait for the matching RESOLVED message and have the user verify its resolved title, green color, end time, and same context. The synthetic rule name and labels must never collide with production rules.

• Step 5: Clean up and document only observed facts

Require synthetic resource absent, matching pending/firing/active alerts zero, production rules healthy, receiver pair exact, and no mutator process or task-created temp residue. Confirm InfoInhibitor is not delivered to Slack. Document the conditional upstream info-inhibition semantics; do not claim every info alert is always suppressed.

Update chapter 03 with the exact source contract and observed FIRING/RESOLVED results. Update chapter 07 with the original terse-message problem, the decision to keep formatting in Alertmanager, and the final transaction result. Record final file identities and any limitation, including long grouped Slack text, in task-4-report.md.