Observability Alertmanager Receiver Postcheck Correction

Status

Approved approach: require the exact Prometheus Operator-generated receiver identity and reject every other receiver topology.

Problem

Task 6 applied the rules/alerts resources but its reconcile postcheck rejected the live Alertmanager receiver and rolled the transaction back. The rendered AlertmanagerConfig correctly uses the local receiver name platform-slack. Prometheus Operator v0.93.0 transforms that local name for the generated Alertmanager configuration using:

<namespace>/<AlertmanagerConfig name>/<local receiver name>

The runtime receiver returned by Alertmanager is therefore exactly:

observability/platform-alertmanager/platform-slack

The existing postcheck and its fake API response incorrectly expect the unqualified local name platform-slack.

Scope

Change only the Task 6 runtime receiver oracle and its focused fake/test matrix. Do not change the AlertmanagerConfig, route, Slack Secret selector, webhook data, rendered resources, inventory files, or Kubernetes substrate.

The failed rollback ID 20260814T080303Z, its ledger, and the preserved handoff remain immutable evidence and must not be reused.

Runtime contract

The Alertmanager /api/v2/receivers response must be a JSON array containing exactly one receiver object, and that object's name must be exactly:

observability/platform-alertmanager/platform-slack

Fail closed when the response is malformed or when the receiver set is empty, contains the raw local name, contains the qualified name more than once, contains a differently qualified name, or contains any additional receiver.

This check remains part of the existing combined reconcile gate. Failure writes no acceptance marker and invokes the existing UID-bound rollback path.

Test-first implementation

First change the fake receiver API to model the pinned Operator's qualified runtime name. On unchanged production, the existing success transaction must fail at reconcile; this is the required RED.

Then add explicit behavioral cases:

exact qualified singleton: pass;
raw local name only: fail and rollback;
empty response or null only: fail and rollback;
qualified duplicate: fail and rollback;
qualified receiver plus any extra receiver: fail and rollback;
wrong namespace, AlertmanagerConfig name, or local receiver segment: fail and rollback;
malformed JSON/topology: fail and rollback.

The minimal production change replaces the raw-name count predicate with exact singleton equality. No new production test seam is permitted.

Verification and live recovery

Run Bash syntax checks, the complete focused apply suite, the focused sensitive-source scanner, and the bounded full source scan required by the source-freeze workflow. Perform an independent read-only review of the exact frozen hashes and confirm no Secret payload, webhook value, or rendered credential was read.

After review, create a fresh source manifest and a new root-owned rollback ID. Revalidate Slack deployment evidence, encryption, restore evidence, inventory identity, handoff identity, API readiness, and rollback-root metadata before exactly one new Task 6 execution. Success requires the reconcile checks, acceptance marker, and payload-free live resource counts to pass before Task 7 begins.

Non-goals

accepting both raw and qualified names;
dynamically discovering arbitrary receiver identities;
changing Operator or Alertmanager versions;
changing the Slack routing resource;
retrying the failed rollback ID.

3.5 KiB Raw Blame History