donghyeon.kang/platform-core
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
Files
16c337bcc91b27ac0f074154b8337e4c8dc17ebb
platform-core/docs/superpowers/plans/2026-08-14-observability-alertmanager-receiver-postcheck.md
T

20 KiB
Raw Blame History

Observability Alertmanager Receiver Postcheck Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Correct Task 6's Alertmanager runtime receiver oracle, prove the correction against the pinned Operator behavior, and authorize one fresh rules/alerts transaction without weakening any Secret, rollback, inventory, or reconciliation boundary.

Architecture: Keep the rendered AlertmanagerConfig unchanged and correct only the runtime /api/v2/receivers acceptance predicate. The postcheck accepts an exact singleton receiver named observability/platform-alertmanager/platform-slack; focused fakes model that downstream Operator transformation. A fresh source freeze and rollback ID are required before another live execution.

Tech Stack: Bash 5.2, Python 3 JSON validation embedded in Bash, Kubernetes/kubectl, Prometheus Operator v0.93.0, Alertmanager v0.33.1, private file-backed source manifests.

Global Constraints

  • Never read, print, hash, copy, or disclose Secret data, the Slack webhook value, or a rendered credential.
  • Do not change services/observability/alerting/platform-alertmanager.yaml, its raw route/receiver name, or its Secret selector.
  • Runtime receiver identity is exactly observability/platform-alertmanager/platform-slack.
  • /api/v2/receivers must contain exactly one receiver object; raw, absent, duplicate, differently qualified, extra, or malformed topologies fail closed.
  • Failed rollback ID 20260814T080303Z, its root ledger, and /tmp/platform-observability-metrics.LNzksC remain preserved and are never reused for mutation.
  • The missing kubectl.kubernetes.io/last-applied-configuration warnings on the two pre-existing resources are not the root cause and require no source change.
  • No live retry occurs before RED, GREEN, bounded scans, exact source freeze, and independent review all pass.
  • This workspace is non-Git; do not commit, create a branch, or claim a commit. Freeze exact hashes/modes in reports instead.

Task 1: Production-faithful receiver regression

Files:

  • Modify: scripts/validate/test-apply-observability-access.sh:230-240
  • Report: .superpowers/sdd/2026-08-14-observability-alertmanager-receiver-postcheck/task-1-report.md

Interfaces:

  • Consumes: fake kubectl /api/v2/receivers response and run_apply environment forwarding.

  • Produces: a focused fake whose default response is the exact Operator-generated singleton and whose closed modes drive negative reconciliation cases.

  • Step 1: Freeze the unchanged production/test identities

    Record SHA-256, mode, UID:GID, nlink, type, and size for:

    scripts/bootstrap/apply-observability-access.sh
scripts/validate/test-apply-observability-access.sh

    Expected starting SHA-256 values:

    apply = 1e1c44e349e46229dfe79d461e711940e1f2b07ea8c0b91e907eeb7dc8740a11
test  = 6f8ec85010b6c0ae1fea0579a343c28a03d3278bb58b99513f58bef4be3f5644

  • Step 2: Change only the fake default and observe RED

    Replace the fake's receiver response with the production-faithful literal:

    if raw_path.endswith("/proxy/api/v2/receivers"):
    print(json.dumps([
        {"name": "observability/platform-alertmanager/platform-slack"},
    ]))
    raise SystemExit(0)

    Do not edit production. Run:

    bash -n scripts/bootstrap/apply-observability-access.sh
bash -n scripts/validate/test-apply-observability-access.sh
timeout --signal=TERM --kill-after=5s 240s \
  bash scripts/validate/test-apply-observability-access.sh

    Expected: syntax RC 0; suite RC nonzero because a success transaction reaches reconcile, the unchanged raw-name predicate rejects the qualified singleton, and no acceptance marker is created.

  • Step 3: Add closed fake receiver modes

    Use only a test-file fake mode, defaulting to qualified:

    mode = os.environ.get("PLATFORM_TEST_ALERTMANAGER_RECEIVER_MODE", "qualified")
qualified = "observability/platform-alertmanager/platform-slack"
receiver_cases = {
    "qualified": [{"name": qualified}],
    "raw": [{"name": "platform-slack"}],
    "empty": [],
    "null": [{"name": "null"}],
    "duplicate": [{"name": qualified}, {"name": qualified}],
    "extra": [{"name": qualified}, {"name": "null"}],
    "wrong-namespace": [{"name": "other/platform-alertmanager/platform-slack"}],
    "wrong-config": [{"name": "observability/other/platform-slack"}],
    "wrong-local": [{"name": "observability/platform-alertmanager/other"}],
}
if mode == "malformed":
    print('{"name":')
elif mode in receiver_cases:
    print(json.dumps(receiver_cases[mode]))
else:
    raise SystemExit(76)
raise SystemExit(0)

    Forward PLATFORM_TEST_ALERTMANAGER_RECEIVER_MODE through run_apply. Retain the existing receiver-drop case by mapping it to empty or replacing that test with the explicit matrix; do not leave two contradictory controls.

  • Step 4: Add exact behavioral assertions

    Add one positive exact-singleton transaction and table-driven negative transactions for:

    raw empty null duplicate extra wrong-namespace wrong-config wrong-local malformed

    Each negative must assert transaction RC nonzero, OBSERVABILITY_ACCESS_RULES_ALERTS_ROLLBACK=PASS, absent acceptance.env, and no leftover newly-created fake state. The positive must still be RED until production changes.

  • Step 5: Record Task 1 RED evidence

    Write the exact baseline hashes, command, RC, expected failing leaf, acceptance absence, fake-state cleanup, and process/temp residue counts to the Task 1 report. Do not include API bodies beyond the safe receiver-name literals listed in this plan.

Task 2: Minimal exact-singleton production correction

Files:

  • Modify: scripts/bootstrap/apply-observability-access.sh:1209-1215
  • Test: scripts/validate/test-apply-observability-access.sh
  • Append: .superpowers/sdd/2026-08-14-observability-alertmanager-receiver-postcheck/task-1-report.md

Interfaces:

  • Consumes: JSON bytes returned from Alertmanager /api/v2/receivers.

  • Produces: reconcile success only for the exact qualified singleton; all other topologies return failure to the existing rollback coordinator.

  • Step 1: Implement the minimal predicate

    Replace the raw-name counter with:

    receiver_payload = json.loads(receivers_api.read_text(encoding="utf-8"))
expected_receiver = "observability/platform-alertmanager/platform-slack"
if (
    not isinstance(receiver_payload, list)
    or len(receiver_payload) != 1
    or not isinstance(receiver_payload[0], dict)
    or receiver_payload[0].get("name") != expected_receiver
):
    raise SystemExit(1)

    Do not add a production environment seam, fallback raw name, prefix match, wildcard, or dynamic discovery.

  • Step 2: Run GREEN syntax and full focused suite

    Run exactly:

    bash -n scripts/bootstrap/apply-observability-access.sh
bash -n scripts/validate/test-apply-observability-access.sh
timeout --signal=TERM --kill-after=5s 330s \
  bash scripts/validate/test-apply-observability-access.sh

    The receiver matrix adds nine complete rollback transactions to the prior 176-second suite. Set the test-only internal SUITE_WALL_BOUND_SECONDS to 300; the measured unchanged-bound RED is 222 > 220. Expected: both syntax RC 0; focused suite RC 0 within the new internal 300-second bound and outer 330-second supervisor; exact terminal PASS; every receiver negative rolls back; no suite-owned orphan process or fixture residue.

  • Step 3: Mutation-check the tests

    In a private temporary copy only, substitute each of the following and prove at least one focused assertion fails for each mutation:

    expected_receiver = "platform-slack"
len(receiver_payload) >= 1
receiver_payload[0].get("name", "").endswith("/platform-slack")

    Delete only the private temporary copy afterward. Do not edit production for this check.

  • Step 4: Append GREEN evidence

    Append final source/test hashes and modes, the full assertion count, terminal PASS, wall time, orphan count, and residue audit to the Task 1 report.

Task 3: Documentation, scans, review, and retry freeze

Files:

  • Modify: bootstrap/manual/phase4-observability-access.md:242-247
  • Modify: /home/donghyeon/workspace/docs/platform/plans/2026-08-14-observability-authoritative-metric-inventory-implementation.md
  • Regenerate: .superpowers/sdd/2026-08-14-observability-authoritative-metric-inventory-implementation/task-3-brief.md
  • Append: .superpowers/sdd/2026-08-14-observability-authoritative-metric-inventory-implementation/task-2-report.md
  • Append: /home/donghyeon/workspace/docs/platform/runbooks/2026-08-13-observability-phase4-resume-worklog.md
  • Preserve old authority under: .superpowers/sdd/2026-08-14-observability-alertmanager-receiver-postcheck/baseline/failed-task6-20260814T080303Z/
  • Regenerate: .superpowers/sdd/2026-08-14-observability-slack-risk-acceptance-implementation/baseline/task-6-prelive-source-manifest.txt
  • Regenerate: .superpowers/sdd/2026-08-14-observability-slack-risk-acceptance-implementation/baseline/task-6-prelive-source-manifest.sha256

Interfaces:

  • Consumes: Task 2 exact hashes and passing evidence.

  • Produces: reviewed execution authority for one fresh Task 6 transaction.

  • Step 1: Correct human and agent acceptance wording

    Replace only runtime acceptance wording from raw platform-slack to:

    generated receiver observability/platform-alertmanager/platform-slack = exact singleton

    Keep the AlertmanagerConfig source receiver documented as raw platform-slack. Record the failed live RC 1, stage reconcile, rollback PASS, no acceptance claim, exact root cause, and preservation of rollback ID 20260814T080303Z.

  • Step 2: Regenerate the derived Task 3 brief deterministically

    Run the canonical extractor once to the derived brief and once to a private comparison file:

    EXTRACTOR=/home/donghyeon/.codex/plugins/cache/openai-curated-remote/superpowers/6.2.0/skills/subagent-driven-development/scripts/task-brief
CENTRAL_PLAN=/home/donghyeon/workspace/docs/platform/plans/2026-08-14-observability-authoritative-metric-inventory-implementation.md
DERIVED_BRIEF=/home/donghyeon/workspace/platform/.superpowers/sdd/2026-08-14-observability-authoritative-metric-inventory-implementation/task-3-brief.md
PRIVATE_BRIEF="$(mktemp /tmp/platform-task6-derived-brief.XXXXXX)"
"$EXTRACTOR" "$CENTRAL_PLAN" 3 "$DERIVED_BRIEF"
"$EXTRACTOR" "$CENTRAL_PLAN" 3 "$PRIVATE_BRIEF"
cmp -s -- "$DERIVED_BRIEF" "$PRIVATE_BRIEF"

    Require cmp RC 0, record the derived SHA-256, then remove only PRIVATE_BRIEF. Do not manually edit the derived brief.

  • Step 3: Run focused scanner gates

    Run:

    bash -n scripts/validate/scan-platform-sensitive-source.sh
timeout --signal=TERM --kill-after=5s 120s \
  bash scripts/validate/test-scan-platform-sensitive-source.sh

    Expected: RC 0 and PLATFORM SENSITIVE SOURCE ASSERTION TEST PASS with no credential output.

  • Step 4: Run one new authoritative full scan

    A source change authorizes exactly one new full scan. With tracing disabled, run:

    TASK6_FIX_SCAN_ROOT="$(mktemp -d /tmp/platform-task6-receiver-fix-scan.XXXXXX)"
chmod 0700 "$TASK6_FIX_SCAN_ROOT"
: >"$TASK6_FIX_SCAN_ROOT/scan.log"
chmod 0600 "$TASK6_FIX_SCAN_ROOT/scan.log"
start_ms="$(date +%s%3N)"
set +e
timeout --signal=TERM --kill-after=5s 240s \
  bash scripts/validate/scan-platform-sensitive-source.sh \
  >"$TASK6_FIX_SCAN_ROOT/scan.log" 2>&1
scan_rc=$?
set -e
end_ms="$(date +%s%3N)"
printf 'RC=%d\nWALL_MS=%d\n' "$scan_rc" "$((end_ms - start_ms))" \
  >"$TASK6_FIX_SCAN_ROOT/result"
chmod 0600 "$TASK6_FIX_SCAN_ROOT/result"

    Require RC 0, exactly one PLATFORM RENDERED SECRET SCAN PASS, exactly one PLATFORM SENSITIVE SOURCE SCAN PASS, and zero new scanner/renderer processes or temporary artifacts.

  • Step 5: Preserve the failed execution authority before regeneration

    Attest the current prelive manifest/sidecar as regular, non-symlink, mode 0664, UID:GID 1000:1000, nlink 1; copy them byte-for-byte into the failed-transaction baseline directory with mode 0600; verify cmp -s and record both old hashes. Amend the central plan to state that this preserved copy is the historical authority for rollback ID 20260814T080303Z and the canonical prelive path is superseded only for the fresh retry.

  • Step 6: Regenerate and strictly validate the canonical prelive manifest

    Rebuild the existing exact 15-row manifest grammar:

    sha256|mode4|uid:gid|nlink|regular file|size|/canonical/absolute/path

    Keep exactly the verifier's EXPECTED_ROWS, each once and in order. Regenerate the canonical sidecar, then invoke the unchanged verifier with literal absolute paths and externally reviewed lowercase SHA values. Require 15 rows, 15 canonical unique paths, no row mismatch, and sidecar binding PASS.

  • Step 7: Independent read-only review

    Review exact source/test/docs/manifest hashes against this plan and the approved design. Required verdicts: spec compliance, Critical/Important/Minor counts, TDD RED provenance, exact-singleton contract, rollback preservation, scan evidence, and live retry Ready YES/NO. Any Critical or Important finding returns to the same implementer for a bounded fix/re-review loop.

Task 4: One fresh live Task 6 transaction

Files:

  • Preserve: /var/lib/hyeonworks/platform-rollbacks/observability-20260814T080303Z
  • Preserve read-only: /tmp/platform-observability-metrics.LNzksC
  • Create: one fresh /var/lib/hyeonworks/platform-rollbacks/observability-<new UTC ID> root
  • Create and revalidate: one fresh /tmp/platform-observability-metrics.XXXXXX root
  • Append after terminal result: Task 2 report, central worklog, central implementation plan
  • Create after terminal success: task-6-final-source-manifest.txt and .sha256

Interfaces:

  • Consumes: reviewed Task 3 freeze and a fresh byte-preserved/revalidated private handoff.

  • Produces: Task 6 acceptance schema platform-observability-rules-alerts-v2 or a preserved fail-closed transaction with no retry.

  • Step 1: Create, attest, and bind a fresh rollback root

    In the existing attached tmux operator pane, refresh sudo and generate one strict UTC ID. Reject 20260814T080303Z; require its exact rollback path pre-ABSENT and non-symlink; invoke sudo -n /usr/bin/mkdir --mode=0700 -- "$TASK6_ROLLBACK_ROOT" exactly once; then require directory|0:0|700 and export it as PLATFORM_OBSERVABILITY_ROLLBACK_ID. Require exact equality between the fresh ID and active exported ID. Any collision, create error, metadata drift, or equality failure stops and forbids retry with that ID.

  • Step 2: Re-establish per-ID and global gates

    For the exact active new ID, run the Blackbox private-edge proof exactly once. Only the operator enters exact confirmation PROVE BLACKBOX PRIVATE EDGE default. Require exact BLACKBOX PRIVATE EDGE SOURCE PASS, immediate RC 0, and normalized proof metadata regular|0:0|600|1 (regular file|0:0|600|1 from the exact stat fields). Any missing, ambiguous, or nonzero result stops the transaction, preserves the new ID, and forbids validator or Task 6 retry under that ID. Then require encryption RC 0, restore RC 0, Slack deployment gate RISK_ACCEPTED, Secret name-only presence, API readyz, and rollback-root metadata.

    Residue is a delta contract. Preserve without reading/deleting the exact preexisting roots /tmp/platform-k3s-encryption.Mskzy3, /tmp/platform-observability-access-apply.oeNcfI, /tmp/platform-observability-access-apply.Im02dz, and /tmp/platform-observability-slack-gate.LYhYbv. The attested classifications are respectively 8/1 empty evidence, two 8/12 recorded evidence roots, and today's failed-live evidence containing only two private filenames. Require the name-only baseline unchanged, matching executable processes 0, and current preflight/live newly-created matching-root delta 0. Unknown/new residue stops for identity review and is never broadly deleted. Do not read Secret data, proof contents, or these evidence-root contents.

  • Step 3: Create a fresh handoff and revalidate source authority

    Run the strict absolute three-argument source-manifest verifier with the newly reviewed verifier and manifest hashes. Preserve /tmp/platform-observability-metrics.LNzksC read-only and never use it as source, destination, renderer input, or apply input. From canonical preserved source /tmp/platform-observability-metrics.VUpsZn, create a new mode 0700 /tmp/platform-observability-metrics.XXXXXX root, copy the two phase inventory pairs with cp --no-dereference --reflink=never, and set files mode 0600. Before rendering, require the new destination to contain exactly the two phase directories and no other entry; the renderer then adds only the reviewed manifest set. Recheck source/destination canonical path, owner/mode/nlink, exact two inventory pins/counts 21/30, exact entry set and rendered file set, and source identity unchanged. Bind the resulting exact path to $METRIC_ROOT. For LNzksC, preserve and compare only the already-attested path/fingerprint identity; do not reopen inventory bodies or private file content. Any copy or identity gate failure stops the transaction without using either handoff for apply.

  • Step 4: Execute exactly once

    Send this as one unsplit line to the verified idle tmux pane:

    PLATFORM_HELM_BIN=/home/donghyeon/.local/bin/helm bash scripts/bootstrap/apply-observability-access.sh --execute --rules-alerts --verified-output-dir "$METRIC_ROOT"

    The operator types exactly APPLY at the prompt. Immediately after return, run echo "TASK6_APPLY_RC=$?". Any nonzero RC, missing PASS, response loss, or rollback ambiguity stops and preserves the new ID without retry.

  • Step 5: Validate terminal acceptance payload-free

    Require:

    AlertmanagerConfig platform-alertmanager = 1
platform PrometheusRule exact set = 4
dashboard ConfigMap exact set = 5
receiver list = [observability/platform-alertmanager/platform-slack]
desired rules evaluation health = ok
Grafana / Blackbox / target readiness unchanged
acceptance schema = platform-observability-rules-alerts-v2
acceptance slack_deployment_gate = RISK_ACCEPTED

    Verify acceptance and ledger metadata only; never read the Slack URL or Secret data.

    Terminal evidence: rollback ID 20260814T145009Z, fresh handoff /tmp/platform-observability-metrics.dw5gLZ, exact six-element argv attestation, inventory pins/counts 21/30, apply RC 0, exact terminal PASS, dashboard 5, platform rules 4, desired rule health 23/23, runbook URL 22/22, AlertmanagerConfig 1, receiver exact qualified singleton, target 30/30, Ready workloads, v2 acceptance with RISK_ACCEPTED, ledger object/mutation lines 13/13, and residue delta 0. Success did not invoke rollback.

  • Step 6: Terminal documentation and final manifest

    Append the new rollback ID, RC, exact resource counts, inventory pins/counts, qualified receiver identity, acceptance schema, and residue result. Check only genuinely completed Task 6 steps. Regenerate the supported final manifest/sidecar from terminal bytes and run the strict verifier. Task 7 may begin only after final independent review returns Ready YES.

    Post-review closeout: final manifest SHA-256 52c2230f23d0cd7733c2ae685737e7c93d837182b01a4c25ff755e780df308f3, sidecar file SHA-256 655d5b1efc2a6ce04715f11fdc646db392ef3951b24b5fe5f2b21c9703c15ee3, exact rows/unique paths 15/15, strict literal absolute three-argument verifier RC 0 twice. Independent terminal review returned Critical/Important/Minor 0/0/1, Spec PASS, Quality Approved, Task 6 complete YES, and Task 7 start YES. The historical-numbering Minor is deferred to Task 8. Task 7 has not been executed by this closeout.

Reference in New Issue View Git Blame Copy Permalink