# Observability Alertmanager Receiver Postcheck Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Correct Task 6's Alertmanager runtime receiver oracle, prove the correction against the pinned Operator behavior, and authorize one fresh rules/alerts transaction without weakening any Secret, rollback, inventory, or reconciliation boundary.

**Architecture:** Keep the rendered `AlertmanagerConfig` unchanged and correct only the runtime `/api/v2/receivers` acceptance predicate. The postcheck accepts an exact singleton receiver named `observability/platform-alertmanager/platform-slack`; focused fakes model that downstream Operator transformation. A fresh source freeze and rollback ID are required before another live execution.

**Tech Stack:** Bash 5.2, Python 3 JSON validation embedded in Bash, Kubernetes/kubectl, Prometheus Operator v0.93.0, Alertmanager v0.33.1, private file-backed source manifests.

## Global Constraints

- Never read, print, hash, copy, or disclose Secret data, the Slack webhook value, or a rendered credential.
- Do not change `services/observability/alerting/platform-alertmanager.yaml`, its raw route/receiver name, or its Secret selector.
- Runtime receiver identity is exactly `observability/platform-alertmanager/platform-slack`.
- `/api/v2/receivers` must contain exactly one receiver object; raw, absent, duplicate, differently qualified, extra, or malformed topologies fail closed.
- Failed rollback ID `20260814T080303Z`, its root ledger, and `/tmp/platform-observability-metrics.LNzksC` remain preserved and are never reused for mutation.
- The missing `kubectl.kubernetes.io/last-applied-configuration` warnings on the two pre-existing resources are not the root cause and require no source change.
- No live retry occurs before RED, GREEN, bounded scans, exact source freeze, and independent review all pass.
- This workspace is non-Git; do not commit, create a branch, or claim a commit. Freeze exact hashes/modes in reports instead.

---

### Task 1: Production-faithful receiver regression

**Files:**
- Modify: `scripts/validate/test-apply-observability-access.sh:230-240`
- Report: `.superpowers/sdd/2026-08-14-observability-alertmanager-receiver-postcheck/task-1-report.md`

**Interfaces:**
- Consumes: fake kubectl `/api/v2/receivers` response and `run_apply` environment forwarding.
- Produces: a focused fake whose default response is the exact Operator-generated singleton and whose closed modes drive negative reconciliation cases.

- [ ] **Step 1: Freeze the unchanged production/test identities**

  Record SHA-256, mode, UID:GID, nlink, type, and size for:

  ```text
  scripts/bootstrap/apply-observability-access.sh
  scripts/validate/test-apply-observability-access.sh
  ```

  Expected starting SHA-256 values:

  ```text
  apply = 1e1c44e349e46229dfe79d461e711940e1f2b07ea8c0b91e907eeb7dc8740a11
  test  = 6f8ec85010b6c0ae1fea0579a343c28a03d3278bb58b99513f58bef4be3f5644
  ```

- [ ] **Step 2: Change only the fake default and observe RED**

  Replace the fake's receiver response with the production-faithful literal:

  ```python
  if raw_path.endswith("/proxy/api/v2/receivers"):
      print(json.dumps([
          {"name": "observability/platform-alertmanager/platform-slack"},
      ]))
      raise SystemExit(0)
  ```

  Do not edit production. Run:

  ```bash
  bash -n scripts/bootstrap/apply-observability-access.sh
  bash -n scripts/validate/test-apply-observability-access.sh
  timeout --signal=TERM --kill-after=5s 240s \
    bash scripts/validate/test-apply-observability-access.sh
  ```

  Expected: syntax RC `0`; suite RC nonzero because a success transaction reaches reconcile, the unchanged raw-name predicate rejects the qualified singleton, and no acceptance marker is created.

- [ ] **Step 3: Add closed fake receiver modes**

  Use only a test-file fake mode, defaulting to `qualified`:

  ```python
  mode = os.environ.get("PLATFORM_TEST_ALERTMANAGER_RECEIVER_MODE", "qualified")
  qualified = "observability/platform-alertmanager/platform-slack"
  receiver_cases = {
      "qualified": [{"name": qualified}],
      "raw": [{"name": "platform-slack"}],
      "empty": [],
      "null": [{"name": "null"}],
      "duplicate": [{"name": qualified}, {"name": qualified}],
      "extra": [{"name": qualified}, {"name": "null"}],
      "wrong-namespace": [{"name": "other/platform-alertmanager/platform-slack"}],
      "wrong-config": [{"name": "observability/other/platform-slack"}],
      "wrong-local": [{"name": "observability/platform-alertmanager/other"}],
  }
  if mode == "malformed":
      print('{"name":')
  elif mode in receiver_cases:
      print(json.dumps(receiver_cases[mode]))
  else:
      raise SystemExit(76)
  raise SystemExit(0)
  ```

  Forward `PLATFORM_TEST_ALERTMANAGER_RECEIVER_MODE` through `run_apply`. Retain the existing receiver-drop case by mapping it to `empty` or replacing that test with the explicit matrix; do not leave two contradictory controls.

- [ ] **Step 4: Add exact behavioral assertions**

  Add one positive exact-singleton transaction and table-driven negative transactions for:

  ```text
  raw empty null duplicate extra wrong-namespace wrong-config wrong-local malformed
  ```

  Each negative must assert transaction RC nonzero, `OBSERVABILITY_ACCESS_RULES_ALERTS_ROLLBACK=PASS`, absent `acceptance.env`, and no leftover newly-created fake state. The positive must still be RED until production changes.

- [ ] **Step 5: Record Task 1 RED evidence**

  Write the exact baseline hashes, command, RC, expected failing leaf, acceptance absence, fake-state cleanup, and process/temp residue counts to the Task 1 report. Do not include API bodies beyond the safe receiver-name literals listed in this plan.

---

### Task 2: Minimal exact-singleton production correction

**Files:**
- Modify: `scripts/bootstrap/apply-observability-access.sh:1209-1215`
- Test: `scripts/validate/test-apply-observability-access.sh`
- Append: `.superpowers/sdd/2026-08-14-observability-alertmanager-receiver-postcheck/task-1-report.md`

**Interfaces:**
- Consumes: JSON bytes returned from Alertmanager `/api/v2/receivers`.
- Produces: reconcile success only for the exact qualified singleton; all other topologies return failure to the existing rollback coordinator.

- [ ] **Step 1: Implement the minimal predicate**

  Replace the raw-name counter with:

  ```python
  receiver_payload = json.loads(receivers_api.read_text(encoding="utf-8"))
  expected_receiver = "observability/platform-alertmanager/platform-slack"
  if (
      not isinstance(receiver_payload, list)
      or len(receiver_payload) != 1
      or not isinstance(receiver_payload[0], dict)
      or receiver_payload[0].get("name") != expected_receiver
  ):
      raise SystemExit(1)
  ```

  Do not add a production environment seam, fallback raw name, prefix match, wildcard, or dynamic discovery.

- [ ] **Step 2: Run GREEN syntax and full focused suite**

  Run exactly:

  ```bash
  bash -n scripts/bootstrap/apply-observability-access.sh
  bash -n scripts/validate/test-apply-observability-access.sh
  timeout --signal=TERM --kill-after=5s 330s \
    bash scripts/validate/test-apply-observability-access.sh
  ```

  The receiver matrix adds nine complete rollback transactions to the prior 176-second suite. Set the test-only internal `SUITE_WALL_BOUND_SECONDS` to `300`; the measured unchanged-bound RED is `222 > 220`. Expected: both syntax RC `0`; focused suite RC `0` within the new internal 300-second bound and outer 330-second supervisor; exact terminal PASS; every receiver negative rolls back; no suite-owned orphan process or fixture residue.

- [ ] **Step 3: Mutation-check the tests**

  In a private temporary copy only, substitute each of the following and prove at least one focused assertion fails for each mutation:

  ```text
  expected_receiver = "platform-slack"
  len(receiver_payload) >= 1
  receiver_payload[0].get("name", "").endswith("/platform-slack")
  ```

  Delete only the private temporary copy afterward. Do not edit production for this check.

- [ ] **Step 4: Append GREEN evidence**

  Append final source/test hashes and modes, the full assertion count, terminal PASS, wall time, orphan count, and residue audit to the Task 1 report.

---

### Task 3: Documentation, scans, review, and retry freeze

**Files:**
- Modify: `bootstrap/manual/phase4-observability-access.md:242-247`
- Modify: `/home/donghyeon/workspace/docs/platform/plans/2026-08-14-observability-authoritative-metric-inventory-implementation.md`
- Regenerate: `.superpowers/sdd/2026-08-14-observability-authoritative-metric-inventory-implementation/task-3-brief.md`
- Append: `.superpowers/sdd/2026-08-14-observability-authoritative-metric-inventory-implementation/task-2-report.md`
- Append: `/home/donghyeon/workspace/docs/platform/runbooks/2026-08-13-observability-phase4-resume-worklog.md`
- Preserve old authority under: `.superpowers/sdd/2026-08-14-observability-alertmanager-receiver-postcheck/baseline/failed-task6-20260814T080303Z/`
- Regenerate: `.superpowers/sdd/2026-08-14-observability-slack-risk-acceptance-implementation/baseline/task-6-prelive-source-manifest.txt`
- Regenerate: `.superpowers/sdd/2026-08-14-observability-slack-risk-acceptance-implementation/baseline/task-6-prelive-source-manifest.sha256`

**Interfaces:**
- Consumes: Task 2 exact hashes and passing evidence.
- Produces: reviewed execution authority for one fresh Task 6 transaction.

- [ ] **Step 1: Correct human and agent acceptance wording**

  Replace only runtime acceptance wording from raw `platform-slack` to:

  ```text
  generated receiver observability/platform-alertmanager/platform-slack = exact singleton
  ```

  Keep the `AlertmanagerConfig` source receiver documented as raw `platform-slack`. Record the failed live RC `1`, stage `reconcile`, rollback `PASS`, no acceptance claim, exact root cause, and preservation of rollback ID `20260814T080303Z`.

- [ ] **Step 2: Regenerate the derived Task 3 brief deterministically**

  Run the canonical extractor once to the derived brief and once to a private comparison file:

  ```bash
  EXTRACTOR=/home/donghyeon/.codex/plugins/cache/openai-curated-remote/superpowers/6.2.0/skills/subagent-driven-development/scripts/task-brief
  CENTRAL_PLAN=/home/donghyeon/workspace/docs/platform/plans/2026-08-14-observability-authoritative-metric-inventory-implementation.md
  DERIVED_BRIEF=/home/donghyeon/workspace/platform/.superpowers/sdd/2026-08-14-observability-authoritative-metric-inventory-implementation/task-3-brief.md
  PRIVATE_BRIEF="$(mktemp /tmp/platform-task6-derived-brief.XXXXXX)"
  "$EXTRACTOR" "$CENTRAL_PLAN" 3 "$DERIVED_BRIEF"
  "$EXTRACTOR" "$CENTRAL_PLAN" 3 "$PRIVATE_BRIEF"
  cmp -s -- "$DERIVED_BRIEF" "$PRIVATE_BRIEF"
  ```

  Require `cmp` RC `0`, record the derived SHA-256, then remove only `PRIVATE_BRIEF`. Do not manually edit the derived brief.

- [ ] **Step 3: Run focused scanner gates**

  Run:

  ```bash
  bash -n scripts/validate/scan-platform-sensitive-source.sh
  timeout --signal=TERM --kill-after=5s 120s \
    bash scripts/validate/test-scan-platform-sensitive-source.sh
  ```

  Expected: RC `0` and `PLATFORM SENSITIVE SOURCE ASSERTION TEST PASS` with no credential output.

- [ ] **Step 4: Run one new authoritative full scan**

  A source change authorizes exactly one new full scan. With tracing disabled, run:

  ```bash
  TASK6_FIX_SCAN_ROOT="$(mktemp -d /tmp/platform-task6-receiver-fix-scan.XXXXXX)"
  chmod 0700 "$TASK6_FIX_SCAN_ROOT"
  : >"$TASK6_FIX_SCAN_ROOT/scan.log"
  chmod 0600 "$TASK6_FIX_SCAN_ROOT/scan.log"
  start_ms="$(date +%s%3N)"
  set +e
  timeout --signal=TERM --kill-after=5s 240s \
    bash scripts/validate/scan-platform-sensitive-source.sh \
    >"$TASK6_FIX_SCAN_ROOT/scan.log" 2>&1
  scan_rc=$?
  set -e
  end_ms="$(date +%s%3N)"
  printf 'RC=%d\nWALL_MS=%d\n' "$scan_rc" "$((end_ms - start_ms))" \
    >"$TASK6_FIX_SCAN_ROOT/result"
  chmod 0600 "$TASK6_FIX_SCAN_ROOT/result"
  ```

  Require RC `0`, exactly one `PLATFORM RENDERED SECRET SCAN PASS`, exactly one `PLATFORM SENSITIVE SOURCE SCAN PASS`, and zero new scanner/renderer processes or temporary artifacts.

- [ ] **Step 5: Preserve the failed execution authority before regeneration**

  Attest the current prelive manifest/sidecar as regular, non-symlink, mode `0664`, UID:GID `1000:1000`, nlink `1`; copy them byte-for-byte into the failed-transaction baseline directory with mode `0600`; verify `cmp -s` and record both old hashes. Amend the central plan to state that this preserved copy is the historical authority for rollback ID `20260814T080303Z` and the canonical prelive path is superseded only for the fresh retry.

- [ ] **Step 6: Regenerate and strictly validate the canonical prelive manifest**

  Rebuild the existing exact 15-row manifest grammar:

  ```text
  sha256|mode4|uid:gid|nlink|regular file|size|/canonical/absolute/path
  ```

  Keep exactly the verifier's `EXPECTED_ROWS`, each once and in order. Regenerate the canonical sidecar, then invoke the unchanged verifier with literal absolute paths and externally reviewed lowercase SHA values. Require 15 rows, 15 canonical unique paths, no row mismatch, and sidecar binding PASS.

- [x] **Step 7: Independent read-only review**

  Review exact source/test/docs/manifest hashes against this plan and the approved design. Required verdicts: spec compliance, Critical/Important/Minor counts, TDD RED provenance, exact-singleton contract, rollback preservation, scan evidence, and live retry Ready YES/NO. Any Critical or Important finding returns to the same implementer for a bounded fix/re-review loop.

---

### Task 4: One fresh live Task 6 transaction

**Files:**
- Preserve: `/var/lib/hyeonworks/platform-rollbacks/observability-20260814T080303Z`
- Preserve read-only: `/tmp/platform-observability-metrics.LNzksC`
- Create: one fresh `/var/lib/hyeonworks/platform-rollbacks/observability-<new UTC ID>` root
- Create and revalidate: one fresh `/tmp/platform-observability-metrics.XXXXXX` root
- Append after terminal result: Task 2 report, central worklog, central implementation plan
- Create after terminal success: `task-6-final-source-manifest.txt` and `.sha256`

**Interfaces:**
- Consumes: reviewed Task 3 freeze and a fresh byte-preserved/revalidated private handoff.
- Produces: Task 6 acceptance schema `platform-observability-rules-alerts-v2` or a preserved fail-closed transaction with no retry.

- [x] **Step 1: Create, attest, and bind a fresh rollback root**

  In the existing attached tmux operator pane, refresh sudo and generate one strict UTC ID. Reject
  `20260814T080303Z`; require its exact rollback path pre-ABSENT and non-symlink; invoke
  `sudo -n /usr/bin/mkdir --mode=0700 -- "$TASK6_ROLLBACK_ROOT"` exactly once; then require
  `directory|0:0|700` and export it as `PLATFORM_OBSERVABILITY_ROLLBACK_ID`. Require exact equality
  between the fresh ID and active exported ID. Any collision, create error, metadata drift, or
  equality failure stops and forbids retry with that ID.

- [x] **Step 2: Re-establish per-ID and global gates**

  For the exact active new ID, run the Blackbox private-edge proof exactly once. Only the operator
  enters exact confirmation `PROVE BLACKBOX PRIVATE EDGE default`. Require exact
  `BLACKBOX PRIVATE EDGE SOURCE PASS`, immediate RC `0`, and normalized proof metadata
  `regular|0:0|600|1` (`regular file|0:0|600|1` from the exact `stat` fields). Any missing,
  ambiguous, or nonzero result stops the transaction, preserves the new ID, and forbids validator
  or Task 6 retry under that ID. Then require encryption RC `0`, restore RC `0`, Slack deployment
  gate `RISK_ACCEPTED`, Secret name-only presence, API `readyz`, and rollback-root metadata.

  Residue is a delta contract. Preserve without reading/deleting the exact preexisting roots
  `/tmp/platform-k3s-encryption.Mskzy3`,
  `/tmp/platform-observability-access-apply.oeNcfI`,
  `/tmp/platform-observability-access-apply.Im02dz`, and
  `/tmp/platform-observability-slack-gate.LYhYbv`. The attested classifications are respectively
  8/1 empty evidence, two 8/12 recorded evidence roots, and today's failed-live evidence containing
  only two private filenames. Require the name-only baseline unchanged, matching executable
  processes `0`, and current preflight/live newly-created matching-root delta `0`. Unknown/new
  residue stops for identity review and is never broadly deleted. Do not read Secret data, proof
  contents, or these evidence-root contents.

- [x] **Step 3: Create a fresh handoff and revalidate source authority**

  Run the strict absolute three-argument source-manifest verifier with the newly reviewed verifier
  and manifest hashes. Preserve `/tmp/platform-observability-metrics.LNzksC` read-only and never use
  it as source, destination, renderer input, or apply input. From canonical preserved source
  `/tmp/platform-observability-metrics.VUpsZn`, create a new mode `0700`
  `/tmp/platform-observability-metrics.XXXXXX` root, copy the two phase inventory pairs with
  `cp --no-dereference --reflink=never`, and set files mode `0600`. Before rendering, require the
  new destination to contain exactly the two phase directories and no other entry; the renderer
  then adds only the reviewed manifest set. Recheck source/destination
  canonical path, owner/mode/nlink, exact two inventory pins/counts `21/30`, exact entry set and
  rendered file set, and source identity unchanged. Bind the resulting exact path to `$METRIC_ROOT`.
  For `LNzksC`, preserve and compare only the already-attested path/fingerprint identity; do not
  reopen inventory bodies or private file content. Any copy or identity gate failure stops the
  transaction without using either handoff for apply.

- [x] **Step 4: Execute exactly once**

  Send this as one unsplit line to the verified idle tmux pane:

  ```bash
  PLATFORM_HELM_BIN=/home/donghyeon/.local/bin/helm bash scripts/bootstrap/apply-observability-access.sh --execute --rules-alerts --verified-output-dir "$METRIC_ROOT"
  ```

  The operator types exactly `APPLY` at the prompt. Immediately after return, run `echo "TASK6_APPLY_RC=$?"`. Any nonzero RC, missing PASS, response loss, or rollback ambiguity stops and preserves the new ID without retry.

- [x] **Step 5: Validate terminal acceptance payload-free**

  Require:

  ```text
  AlertmanagerConfig platform-alertmanager = 1
  platform PrometheusRule exact set = 4
  dashboard ConfigMap exact set = 5
  receiver list = [observability/platform-alertmanager/platform-slack]
  desired rules evaluation health = ok
  Grafana / Blackbox / target readiness unchanged
  acceptance schema = platform-observability-rules-alerts-v2
  acceptance slack_deployment_gate = RISK_ACCEPTED
  ```

  Verify acceptance and ledger metadata only; never read the Slack URL or Secret data.

  Terminal evidence: rollback ID `20260814T145009Z`, fresh handoff
  `/tmp/platform-observability-metrics.dw5gLZ`, exact six-element argv attestation, inventory
  pins/counts `21/30`, apply RC `0`, exact terminal PASS, dashboard `5`, platform rules `4`, desired
  rule health `23/23`, runbook URL `22/22`, AlertmanagerConfig `1`, receiver exact qualified
  singleton, target `30/30`, Ready workloads, v2 acceptance with `RISK_ACCEPTED`, ledger
  object/mutation lines `13/13`, and residue delta `0`. Success did not invoke rollback.

- [x] **Step 6: Terminal documentation and final manifest**

  Append the new rollback ID, RC, exact resource counts, inventory pins/counts, qualified receiver identity, acceptance schema, and residue result. Check only genuinely completed Task 6 steps. Regenerate the supported final manifest/sidecar from terminal bytes and run the strict verifier. Task 7 may begin only after final independent review returns Ready YES.

  Post-review closeout: final manifest SHA-256
  `52c2230f23d0cd7733c2ae685737e7c93d837182b01a4c25ff755e780df308f3`, sidecar file SHA-256
  `655d5b1efc2a6ce04715f11fdc646db392ef3951b24b5fe5f2b21c9703c15ee3`, exact rows/unique paths
  `15/15`, strict literal absolute three-argument verifier RC `0` twice. Independent terminal review
  returned Critical/Important/Minor `0/0/1`, Spec `PASS`, Quality `Approved`, Task 6 complete `YES`,
  and Task 7 start `YES`. The historical-numbering Minor is deferred to Task 8. Task 7 has not been
  executed by this closeout.