Add platform infrastructure configuration
This commit is contained in:
donghyeon-ka
+69
@@ -0,0 +1,69 @@
|
||||
# Observability Alertmanager Receiver Postcheck Correction
|
||||
|
||||
## Status
|
||||
|
||||
Approved approach: require the exact Prometheus Operator-generated receiver identity and reject every other receiver topology.
|
||||
|
||||
## Problem
|
||||
|
||||
Task 6 applied the rules/alerts resources but its reconcile postcheck rejected the live Alertmanager receiver and rolled the transaction back. The rendered `AlertmanagerConfig` correctly uses the local receiver name `platform-slack`. Prometheus Operator v0.93.0 transforms that local name for the generated Alertmanager configuration using:
|
||||
|
||||
```text
|
||||
<namespace>/<AlertmanagerConfig name>/<local receiver name>
|
||||
```
|
||||
|
||||
The runtime receiver returned by Alertmanager is therefore exactly:
|
||||
|
||||
```text
|
||||
observability/platform-alertmanager/platform-slack
|
||||
```
|
||||
|
||||
The existing postcheck and its fake API response incorrectly expect the unqualified local name `platform-slack`.
|
||||
|
||||
## Scope
|
||||
|
||||
Change only the Task 6 runtime receiver oracle and its focused fake/test matrix. Do not change the `AlertmanagerConfig`, route, Slack Secret selector, webhook data, rendered resources, inventory files, or Kubernetes substrate.
|
||||
|
||||
The failed rollback ID `20260814T080303Z`, its ledger, and the preserved handoff remain immutable evidence and must not be reused.
|
||||
|
||||
## Runtime contract
|
||||
|
||||
The Alertmanager `/api/v2/receivers` response must be a JSON array containing exactly one receiver object, and that object's name must be exactly:
|
||||
|
||||
```text
|
||||
observability/platform-alertmanager/platform-slack
|
||||
```
|
||||
|
||||
Fail closed when the response is malformed or when the receiver set is empty, contains the raw local name, contains the qualified name more than once, contains a differently qualified name, or contains any additional receiver.
|
||||
|
||||
This check remains part of the existing combined reconcile gate. Failure writes no acceptance marker and invokes the existing UID-bound rollback path.
|
||||
|
||||
## Test-first implementation
|
||||
|
||||
First change the fake receiver API to model the pinned Operator's qualified runtime name. On unchanged production, the existing success transaction must fail at reconcile; this is the required RED.
|
||||
|
||||
Then add explicit behavioral cases:
|
||||
|
||||
- exact qualified singleton: pass;
|
||||
- raw local name only: fail and rollback;
|
||||
- empty response or `null` only: fail and rollback;
|
||||
- qualified duplicate: fail and rollback;
|
||||
- qualified receiver plus any extra receiver: fail and rollback;
|
||||
- wrong namespace, AlertmanagerConfig name, or local receiver segment: fail and rollback;
|
||||
- malformed JSON/topology: fail and rollback.
|
||||
|
||||
The minimal production change replaces the raw-name count predicate with exact singleton equality. No new production test seam is permitted.
|
||||
|
||||
## Verification and live recovery
|
||||
|
||||
Run Bash syntax checks, the complete focused apply suite, the focused sensitive-source scanner, and the bounded full source scan required by the source-freeze workflow. Perform an independent read-only review of the exact frozen hashes and confirm no Secret payload, webhook value, or rendered credential was read.
|
||||
|
||||
After review, create a fresh source manifest and a new root-owned rollback ID. Revalidate Slack deployment evidence, encryption, restore evidence, inventory identity, handoff identity, API readiness, and rollback-root metadata before exactly one new Task 6 execution. Success requires the reconcile checks, acceptance marker, and payload-free live resource counts to pass before Task 7 begins.
|
||||
|
||||
## Non-goals
|
||||
|
||||
- accepting both raw and qualified names;
|
||||
- dynamically discovering arbitrary receiver identities;
|
||||
- changing Operator or Alertmanager versions;
|
||||
- changing the Slack routing resource;
|
||||
- retrying the failed rollback ID.
|
||||
@@ -0,0 +1,380 @@
|
||||
# Observability Alertmanager Receiver Postcheck Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Correct Task 6's Alertmanager runtime receiver oracle, prove the correction against the pinned Operator behavior, and authorize one fresh rules/alerts transaction without weakening any Secret, rollback, inventory, or reconciliation boundary.
|
||||
|
||||
**Architecture:** Keep the rendered `AlertmanagerConfig` unchanged and correct only the runtime `/api/v2/receivers` acceptance predicate. The postcheck accepts an exact singleton receiver named `observability/platform-alertmanager/platform-slack`; focused fakes model that downstream Operator transformation. A fresh source freeze and rollback ID are required before another live execution.
|
||||
|
||||
**Tech Stack:** Bash 5.2, Python 3 JSON validation embedded in Bash, Kubernetes/kubectl, Prometheus Operator v0.93.0, Alertmanager v0.33.1, private file-backed source manifests.
|
||||
|
||||
## Global Constraints
|
||||
|
||||
- Never read, print, hash, copy, or disclose Secret data, the Slack webhook value, or a rendered credential.
|
||||
- Do not change `services/observability/alerting/platform-alertmanager.yaml`, its raw route/receiver name, or its Secret selector.
|
||||
- Runtime receiver identity is exactly `observability/platform-alertmanager/platform-slack`.
|
||||
- `/api/v2/receivers` must contain exactly one receiver object; raw, absent, duplicate, differently qualified, extra, or malformed topologies fail closed.
|
||||
- Failed rollback ID `20260814T080303Z`, its root ledger, and `/tmp/platform-observability-metrics.LNzksC` remain preserved and are never reused for mutation.
|
||||
- The missing `kubectl.kubernetes.io/last-applied-configuration` warnings on the two pre-existing resources are not the root cause and require no source change.
|
||||
- No live retry occurs before RED, GREEN, bounded scans, exact source freeze, and independent review all pass.
|
||||
- This workspace is non-Git; do not commit, create a branch, or claim a commit. Freeze exact hashes/modes in reports instead.
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Production-faithful receiver regression
|
||||
|
||||
**Files:**
|
||||
- Modify: `scripts/validate/test-apply-observability-access.sh:230-240`
|
||||
- Report: `.superpowers/sdd/2026-08-14-observability-alertmanager-receiver-postcheck/task-1-report.md`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: fake kubectl `/api/v2/receivers` response and `run_apply` environment forwarding.
|
||||
- Produces: a focused fake whose default response is the exact Operator-generated singleton and whose closed modes drive negative reconciliation cases.
|
||||
|
||||
- [ ] **Step 1: Freeze the unchanged production/test identities**
|
||||
|
||||
Record SHA-256, mode, UID:GID, nlink, type, and size for:
|
||||
|
||||
```text
|
||||
scripts/bootstrap/apply-observability-access.sh
|
||||
scripts/validate/test-apply-observability-access.sh
|
||||
```
|
||||
|
||||
Expected starting SHA-256 values:
|
||||
|
||||
```text
|
||||
apply = 1e1c44e349e46229dfe79d461e711940e1f2b07ea8c0b91e907eeb7dc8740a11
|
||||
test = 6f8ec85010b6c0ae1fea0579a343c28a03d3278bb58b99513f58bef4be3f5644
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Change only the fake default and observe RED**
|
||||
|
||||
Replace the fake's receiver response with the production-faithful literal:
|
||||
|
||||
```python
|
||||
if raw_path.endswith("/proxy/api/v2/receivers"):
|
||||
print(json.dumps([
|
||||
{"name": "observability/platform-alertmanager/platform-slack"},
|
||||
]))
|
||||
raise SystemExit(0)
|
||||
```
|
||||
|
||||
Do not edit production. Run:
|
||||
|
||||
```bash
|
||||
bash -n scripts/bootstrap/apply-observability-access.sh
|
||||
bash -n scripts/validate/test-apply-observability-access.sh
|
||||
timeout --signal=TERM --kill-after=5s 240s \
|
||||
bash scripts/validate/test-apply-observability-access.sh
|
||||
```
|
||||
|
||||
Expected: syntax RC `0`; suite RC nonzero because a success transaction reaches reconcile, the unchanged raw-name predicate rejects the qualified singleton, and no acceptance marker is created.
|
||||
|
||||
- [ ] **Step 3: Add closed fake receiver modes**
|
||||
|
||||
Use only a test-file fake mode, defaulting to `qualified`:
|
||||
|
||||
```python
|
||||
mode = os.environ.get("PLATFORM_TEST_ALERTMANAGER_RECEIVER_MODE", "qualified")
|
||||
qualified = "observability/platform-alertmanager/platform-slack"
|
||||
receiver_cases = {
|
||||
"qualified": [{"name": qualified}],
|
||||
"raw": [{"name": "platform-slack"}],
|
||||
"empty": [],
|
||||
"null": [{"name": "null"}],
|
||||
"duplicate": [{"name": qualified}, {"name": qualified}],
|
||||
"extra": [{"name": qualified}, {"name": "null"}],
|
||||
"wrong-namespace": [{"name": "other/platform-alertmanager/platform-slack"}],
|
||||
"wrong-config": [{"name": "observability/other/platform-slack"}],
|
||||
"wrong-local": [{"name": "observability/platform-alertmanager/other"}],
|
||||
}
|
||||
if mode == "malformed":
|
||||
print('{"name":')
|
||||
elif mode in receiver_cases:
|
||||
print(json.dumps(receiver_cases[mode]))
|
||||
else:
|
||||
raise SystemExit(76)
|
||||
raise SystemExit(0)
|
||||
```
|
||||
|
||||
Forward `PLATFORM_TEST_ALERTMANAGER_RECEIVER_MODE` through `run_apply`. Retain the existing receiver-drop case by mapping it to `empty` or replacing that test with the explicit matrix; do not leave two contradictory controls.
|
||||
|
||||
- [ ] **Step 4: Add exact behavioral assertions**
|
||||
|
||||
Add one positive exact-singleton transaction and table-driven negative transactions for:
|
||||
|
||||
```text
|
||||
raw empty null duplicate extra wrong-namespace wrong-config wrong-local malformed
|
||||
```
|
||||
|
||||
Each negative must assert transaction RC nonzero, `OBSERVABILITY_ACCESS_RULES_ALERTS_ROLLBACK=PASS`, absent `acceptance.env`, and no leftover newly-created fake state. The positive must still be RED until production changes.
|
||||
|
||||
- [ ] **Step 5: Record Task 1 RED evidence**
|
||||
|
||||
Write the exact baseline hashes, command, RC, expected failing leaf, acceptance absence, fake-state cleanup, and process/temp residue counts to the Task 1 report. Do not include API bodies beyond the safe receiver-name literals listed in this plan.
|
||||
|
||||
---
|
||||
|
||||
### Task 2: Minimal exact-singleton production correction
|
||||
|
||||
**Files:**
|
||||
- Modify: `scripts/bootstrap/apply-observability-access.sh:1209-1215`
|
||||
- Test: `scripts/validate/test-apply-observability-access.sh`
|
||||
- Append: `.superpowers/sdd/2026-08-14-observability-alertmanager-receiver-postcheck/task-1-report.md`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: JSON bytes returned from Alertmanager `/api/v2/receivers`.
|
||||
- Produces: reconcile success only for the exact qualified singleton; all other topologies return failure to the existing rollback coordinator.
|
||||
|
||||
- [ ] **Step 1: Implement the minimal predicate**
|
||||
|
||||
Replace the raw-name counter with:
|
||||
|
||||
```python
|
||||
receiver_payload = json.loads(receivers_api.read_text(encoding="utf-8"))
|
||||
expected_receiver = "observability/platform-alertmanager/platform-slack"
|
||||
if (
|
||||
not isinstance(receiver_payload, list)
|
||||
or len(receiver_payload) != 1
|
||||
or not isinstance(receiver_payload[0], dict)
|
||||
or receiver_payload[0].get("name") != expected_receiver
|
||||
):
|
||||
raise SystemExit(1)
|
||||
```
|
||||
|
||||
Do not add a production environment seam, fallback raw name, prefix match, wildcard, or dynamic discovery.
|
||||
|
||||
- [ ] **Step 2: Run GREEN syntax and full focused suite**
|
||||
|
||||
Run exactly:
|
||||
|
||||
```bash
|
||||
bash -n scripts/bootstrap/apply-observability-access.sh
|
||||
bash -n scripts/validate/test-apply-observability-access.sh
|
||||
timeout --signal=TERM --kill-after=5s 330s \
|
||||
bash scripts/validate/test-apply-observability-access.sh
|
||||
```
|
||||
|
||||
The receiver matrix adds nine complete rollback transactions to the prior 176-second suite. Set the test-only internal `SUITE_WALL_BOUND_SECONDS` to `300`; the measured unchanged-bound RED is `222 > 220`. Expected: both syntax RC `0`; focused suite RC `0` within the new internal 300-second bound and outer 330-second supervisor; exact terminal PASS; every receiver negative rolls back; no suite-owned orphan process or fixture residue.
|
||||
|
||||
- [ ] **Step 3: Mutation-check the tests**
|
||||
|
||||
In a private temporary copy only, substitute each of the following and prove at least one focused assertion fails for each mutation:
|
||||
|
||||
```text
|
||||
expected_receiver = "platform-slack"
|
||||
len(receiver_payload) >= 1
|
||||
receiver_payload[0].get("name", "").endswith("/platform-slack")
|
||||
```
|
||||
|
||||
Delete only the private temporary copy afterward. Do not edit production for this check.
|
||||
|
||||
- [ ] **Step 4: Append GREEN evidence**
|
||||
|
||||
Append final source/test hashes and modes, the full assertion count, terminal PASS, wall time, orphan count, and residue audit to the Task 1 report.
|
||||
|
||||
---
|
||||
|
||||
### Task 3: Documentation, scans, review, and retry freeze
|
||||
|
||||
**Files:**
|
||||
- Modify: `bootstrap/manual/phase4-observability-access.md:242-247`
|
||||
- Modify: `/home/donghyeon/workspace/docs/platform/plans/2026-08-14-observability-authoritative-metric-inventory-implementation.md`
|
||||
- Regenerate: `.superpowers/sdd/2026-08-14-observability-authoritative-metric-inventory-implementation/task-3-brief.md`
|
||||
- Append: `.superpowers/sdd/2026-08-14-observability-authoritative-metric-inventory-implementation/task-2-report.md`
|
||||
- Append: `/home/donghyeon/workspace/docs/platform/runbooks/2026-08-13-observability-phase4-resume-worklog.md`
|
||||
- Preserve old authority under: `.superpowers/sdd/2026-08-14-observability-alertmanager-receiver-postcheck/baseline/failed-task6-20260814T080303Z/`
|
||||
- Regenerate: `.superpowers/sdd/2026-08-14-observability-slack-risk-acceptance-implementation/baseline/task-6-prelive-source-manifest.txt`
|
||||
- Regenerate: `.superpowers/sdd/2026-08-14-observability-slack-risk-acceptance-implementation/baseline/task-6-prelive-source-manifest.sha256`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: Task 2 exact hashes and passing evidence.
|
||||
- Produces: reviewed execution authority for one fresh Task 6 transaction.
|
||||
|
||||
- [ ] **Step 1: Correct human and agent acceptance wording**
|
||||
|
||||
Replace only runtime acceptance wording from raw `platform-slack` to:
|
||||
|
||||
```text
|
||||
generated receiver observability/platform-alertmanager/platform-slack = exact singleton
|
||||
```
|
||||
|
||||
Keep the `AlertmanagerConfig` source receiver documented as raw `platform-slack`. Record the failed live RC `1`, stage `reconcile`, rollback `PASS`, no acceptance claim, exact root cause, and preservation of rollback ID `20260814T080303Z`.
|
||||
|
||||
- [ ] **Step 2: Regenerate the derived Task 3 brief deterministically**
|
||||
|
||||
Run the canonical extractor once to the derived brief and once to a private comparison file:
|
||||
|
||||
```bash
|
||||
EXTRACTOR=/home/donghyeon/.codex/plugins/cache/openai-curated-remote/superpowers/6.2.0/skills/subagent-driven-development/scripts/task-brief
|
||||
CENTRAL_PLAN=/home/donghyeon/workspace/docs/platform/plans/2026-08-14-observability-authoritative-metric-inventory-implementation.md
|
||||
DERIVED_BRIEF=/home/donghyeon/workspace/platform/.superpowers/sdd/2026-08-14-observability-authoritative-metric-inventory-implementation/task-3-brief.md
|
||||
PRIVATE_BRIEF="$(mktemp /tmp/platform-task6-derived-brief.XXXXXX)"
|
||||
"$EXTRACTOR" "$CENTRAL_PLAN" 3 "$DERIVED_BRIEF"
|
||||
"$EXTRACTOR" "$CENTRAL_PLAN" 3 "$PRIVATE_BRIEF"
|
||||
cmp -s -- "$DERIVED_BRIEF" "$PRIVATE_BRIEF"
|
||||
```
|
||||
|
||||
Require `cmp` RC `0`, record the derived SHA-256, then remove only `PRIVATE_BRIEF`. Do not manually edit the derived brief.
|
||||
|
||||
- [ ] **Step 3: Run focused scanner gates**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
bash -n scripts/validate/scan-platform-sensitive-source.sh
|
||||
timeout --signal=TERM --kill-after=5s 120s \
|
||||
bash scripts/validate/test-scan-platform-sensitive-source.sh
|
||||
```
|
||||
|
||||
Expected: RC `0` and `PLATFORM SENSITIVE SOURCE ASSERTION TEST PASS` with no credential output.
|
||||
|
||||
- [ ] **Step 4: Run one new authoritative full scan**
|
||||
|
||||
A source change authorizes exactly one new full scan. With tracing disabled, run:
|
||||
|
||||
```bash
|
||||
TASK6_FIX_SCAN_ROOT="$(mktemp -d /tmp/platform-task6-receiver-fix-scan.XXXXXX)"
|
||||
chmod 0700 "$TASK6_FIX_SCAN_ROOT"
|
||||
: >"$TASK6_FIX_SCAN_ROOT/scan.log"
|
||||
chmod 0600 "$TASK6_FIX_SCAN_ROOT/scan.log"
|
||||
start_ms="$(date +%s%3N)"
|
||||
set +e
|
||||
timeout --signal=TERM --kill-after=5s 240s \
|
||||
bash scripts/validate/scan-platform-sensitive-source.sh \
|
||||
>"$TASK6_FIX_SCAN_ROOT/scan.log" 2>&1
|
||||
scan_rc=$?
|
||||
set -e
|
||||
end_ms="$(date +%s%3N)"
|
||||
printf 'RC=%d\nWALL_MS=%d\n' "$scan_rc" "$((end_ms - start_ms))" \
|
||||
>"$TASK6_FIX_SCAN_ROOT/result"
|
||||
chmod 0600 "$TASK6_FIX_SCAN_ROOT/result"
|
||||
```
|
||||
|
||||
Require RC `0`, exactly one `PLATFORM RENDERED SECRET SCAN PASS`, exactly one `PLATFORM SENSITIVE SOURCE SCAN PASS`, and zero new scanner/renderer processes or temporary artifacts.
|
||||
|
||||
- [ ] **Step 5: Preserve the failed execution authority before regeneration**
|
||||
|
||||
Attest the current prelive manifest/sidecar as regular, non-symlink, mode `0664`, UID:GID `1000:1000`, nlink `1`; copy them byte-for-byte into the failed-transaction baseline directory with mode `0600`; verify `cmp -s` and record both old hashes. Amend the central plan to state that this preserved copy is the historical authority for rollback ID `20260814T080303Z` and the canonical prelive path is superseded only for the fresh retry.
|
||||
|
||||
- [ ] **Step 6: Regenerate and strictly validate the canonical prelive manifest**
|
||||
|
||||
Rebuild the existing exact 15-row manifest grammar:
|
||||
|
||||
```text
|
||||
sha256|mode4|uid:gid|nlink|regular file|size|/canonical/absolute/path
|
||||
```
|
||||
|
||||
Keep exactly the verifier's `EXPECTED_ROWS`, each once and in order. Regenerate the canonical sidecar, then invoke the unchanged verifier with literal absolute paths and externally reviewed lowercase SHA values. Require 15 rows, 15 canonical unique paths, no row mismatch, and sidecar binding PASS.
|
||||
|
||||
- [x] **Step 7: Independent read-only review**
|
||||
|
||||
Review exact source/test/docs/manifest hashes against this plan and the approved design. Required verdicts: spec compliance, Critical/Important/Minor counts, TDD RED provenance, exact-singleton contract, rollback preservation, scan evidence, and live retry Ready YES/NO. Any Critical or Important finding returns to the same implementer for a bounded fix/re-review loop.
|
||||
|
||||
---
|
||||
|
||||
### Task 4: One fresh live Task 6 transaction
|
||||
|
||||
**Files:**
|
||||
- Preserve: `/var/lib/hyeonworks/platform-rollbacks/observability-20260814T080303Z`
|
||||
- Preserve read-only: `/tmp/platform-observability-metrics.LNzksC`
|
||||
- Create: one fresh `/var/lib/hyeonworks/platform-rollbacks/observability-<new UTC ID>` root
|
||||
- Create and revalidate: one fresh `/tmp/platform-observability-metrics.XXXXXX` root
|
||||
- Append after terminal result: Task 2 report, central worklog, central implementation plan
|
||||
- Create after terminal success: `task-6-final-source-manifest.txt` and `.sha256`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: reviewed Task 3 freeze and a fresh byte-preserved/revalidated private handoff.
|
||||
- Produces: Task 6 acceptance schema `platform-observability-rules-alerts-v2` or a preserved fail-closed transaction with no retry.
|
||||
|
||||
- [x] **Step 1: Create, attest, and bind a fresh rollback root**
|
||||
|
||||
In the existing attached tmux operator pane, refresh sudo and generate one strict UTC ID. Reject
|
||||
`20260814T080303Z`; require its exact rollback path pre-ABSENT and non-symlink; invoke
|
||||
`sudo -n /usr/bin/mkdir --mode=0700 -- "$TASK6_ROLLBACK_ROOT"` exactly once; then require
|
||||
`directory|0:0|700` and export it as `PLATFORM_OBSERVABILITY_ROLLBACK_ID`. Require exact equality
|
||||
between the fresh ID and active exported ID. Any collision, create error, metadata drift, or
|
||||
equality failure stops and forbids retry with that ID.
|
||||
|
||||
- [x] **Step 2: Re-establish per-ID and global gates**
|
||||
|
||||
For the exact active new ID, run the Blackbox private-edge proof exactly once. Only the operator
|
||||
enters exact confirmation `PROVE BLACKBOX PRIVATE EDGE default`. Require exact
|
||||
`BLACKBOX PRIVATE EDGE SOURCE PASS`, immediate RC `0`, and normalized proof metadata
|
||||
`regular|0:0|600|1` (`regular file|0:0|600|1` from the exact `stat` fields). Any missing,
|
||||
ambiguous, or nonzero result stops the transaction, preserves the new ID, and forbids validator
|
||||
or Task 6 retry under that ID. Then require encryption RC `0`, restore RC `0`, Slack deployment
|
||||
gate `RISK_ACCEPTED`, Secret name-only presence, API `readyz`, and rollback-root metadata.
|
||||
|
||||
Residue is a delta contract. Preserve without reading/deleting the exact preexisting roots
|
||||
`/tmp/platform-k3s-encryption.Mskzy3`,
|
||||
`/tmp/platform-observability-access-apply.oeNcfI`,
|
||||
`/tmp/platform-observability-access-apply.Im02dz`, and
|
||||
`/tmp/platform-observability-slack-gate.LYhYbv`. The attested classifications are respectively
|
||||
8/1 empty evidence, two 8/12 recorded evidence roots, and today's failed-live evidence containing
|
||||
only two private filenames. Require the name-only baseline unchanged, matching executable
|
||||
processes `0`, and current preflight/live newly-created matching-root delta `0`. Unknown/new
|
||||
residue stops for identity review and is never broadly deleted. Do not read Secret data, proof
|
||||
contents, or these evidence-root contents.
|
||||
|
||||
- [x] **Step 3: Create a fresh handoff and revalidate source authority**
|
||||
|
||||
Run the strict absolute three-argument source-manifest verifier with the newly reviewed verifier
|
||||
and manifest hashes. Preserve `/tmp/platform-observability-metrics.LNzksC` read-only and never use
|
||||
it as source, destination, renderer input, or apply input. From canonical preserved source
|
||||
`/tmp/platform-observability-metrics.VUpsZn`, create a new mode `0700`
|
||||
`/tmp/platform-observability-metrics.XXXXXX` root, copy the two phase inventory pairs with
|
||||
`cp --no-dereference --reflink=never`, and set files mode `0600`. Before rendering, require the
|
||||
new destination to contain exactly the two phase directories and no other entry; the renderer
|
||||
then adds only the reviewed manifest set. Recheck source/destination
|
||||
canonical path, owner/mode/nlink, exact two inventory pins/counts `21/30`, exact entry set and
|
||||
rendered file set, and source identity unchanged. Bind the resulting exact path to `$METRIC_ROOT`.
|
||||
For `LNzksC`, preserve and compare only the already-attested path/fingerprint identity; do not
|
||||
reopen inventory bodies or private file content. Any copy or identity gate failure stops the
|
||||
transaction without using either handoff for apply.
|
||||
|
||||
- [x] **Step 4: Execute exactly once**
|
||||
|
||||
Send this as one unsplit line to the verified idle tmux pane:
|
||||
|
||||
```bash
|
||||
PLATFORM_HELM_BIN=/home/donghyeon/.local/bin/helm bash scripts/bootstrap/apply-observability-access.sh --execute --rules-alerts --verified-output-dir "$METRIC_ROOT"
|
||||
```
|
||||
|
||||
The operator types exactly `APPLY` at the prompt. Immediately after return, run `echo "TASK6_APPLY_RC=$?"`. Any nonzero RC, missing PASS, response loss, or rollback ambiguity stops and preserves the new ID without retry.
|
||||
|
||||
- [x] **Step 5: Validate terminal acceptance payload-free**
|
||||
|
||||
Require:
|
||||
|
||||
```text
|
||||
AlertmanagerConfig platform-alertmanager = 1
|
||||
platform PrometheusRule exact set = 4
|
||||
dashboard ConfigMap exact set = 5
|
||||
receiver list = [observability/platform-alertmanager/platform-slack]
|
||||
desired rules evaluation health = ok
|
||||
Grafana / Blackbox / target readiness unchanged
|
||||
acceptance schema = platform-observability-rules-alerts-v2
|
||||
acceptance slack_deployment_gate = RISK_ACCEPTED
|
||||
```
|
||||
|
||||
Verify acceptance and ledger metadata only; never read the Slack URL or Secret data.
|
||||
|
||||
Terminal evidence: rollback ID `20260814T145009Z`, fresh handoff
|
||||
`/tmp/platform-observability-metrics.dw5gLZ`, exact six-element argv attestation, inventory
|
||||
pins/counts `21/30`, apply RC `0`, exact terminal PASS, dashboard `5`, platform rules `4`, desired
|
||||
rule health `23/23`, runbook URL `22/22`, AlertmanagerConfig `1`, receiver exact qualified
|
||||
singleton, target `30/30`, Ready workloads, v2 acceptance with `RISK_ACCEPTED`, ledger
|
||||
object/mutation lines `13/13`, and residue delta `0`. Success did not invoke rollback.
|
||||
|
||||
- [x] **Step 6: Terminal documentation and final manifest**
|
||||
|
||||
Append the new rollback ID, RC, exact resource counts, inventory pins/counts, qualified receiver identity, acceptance schema, and residue result. Check only genuinely completed Task 6 steps. Regenerate the supported final manifest/sidecar from terminal bytes and run the strict verifier. Task 7 may begin only after final independent review returns Ready YES.
|
||||
|
||||
Post-review closeout: final manifest SHA-256
|
||||
`52c2230f23d0cd7733c2ae685737e7c93d837182b01a4c25ff755e780df308f3`, sidecar file SHA-256
|
||||
`655d5b1efc2a6ce04715f11fdc646db392ef3951b24b5fe5f2b21c9703c15ee3`, exact rows/unique paths
|
||||
`15/15`, strict literal absolute three-argument verifier RC `0` twice. Independent terminal review
|
||||
returned Critical/Important/Minor `0/0/1`, Spec `PASS`, Quality `Approved`, Task 6 complete `YES`,
|
||||
and Task 7 start `YES`. The historical-numbering Minor is deferred to Task 8. Task 7 has not been
|
||||
executed by this closeout.
|
||||
@@ -0,0 +1,373 @@
|
||||
# Structured Slack Observability Notifications Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Replace Alertmanager's terse default Slack attachment with a structured, severity-aware message and route `InfoInhibitor` to a null receiver while preserving the existing webhook, transactional apply, and rollback boundaries.
|
||||
|
||||
**Architecture:** Keep formatting inside `AlertmanagerConfig/platform-alertmanager`; do not add a formatter service. Strengthen the existing source, core-render, access-render, and apply assertions so the same exact configuration is enforced at every boundary. Runtime acceptance changes from one qualified receiver to the exact qualified pair for Slack and null.
|
||||
|
||||
**Tech Stack:** Kubernetes `AlertmanagerConfig` v1alpha1, Prometheus Alertmanager 0.33.1 Go templates, Bash, embedded Python/PyYAML, existing renderer and rules-alerts transaction.
|
||||
|
||||
## Global Constraints
|
||||
|
||||
- Approved design: `/home/donghyeon/workspace/docs/platform/plans/2026-08-15-observability-structured-slack-notifications-design.md`.
|
||||
- The Slack Secret selector remains exactly `alertmanager-slack-webhook` / `url`; never read, print, copy, decode, or persist its payload.
|
||||
- Do not set a Slack `channel`, automatic mention, webhook literal, or new credential source.
|
||||
- Route grouping is the exact set `cluster`, `namespace`, `alertname`, `severity`; timing remains `30s`, `5m`, `4h`.
|
||||
- Source receivers are exactly `platform-null` and `platform-slack`; `platform-null` has no integrations.
|
||||
- Runtime receivers are the order-independent exact set `observability/platform-alertmanager/platform-null` and `observability/platform-alertmanager/platform-slack`.
|
||||
- `InfoInhibitor` routes to `platform-null`; its inhibition rule targets `severity=info` with equality on `namespace`. Preserve the upstream conditional semantics rather than claiming all info alerts are always dropped.
|
||||
- Colors are resolved=`good`, emergency/critical=`danger`, warning=`warning`, and info/unknown=`#439FE0`.
|
||||
- Target text uses all labels after removing group-label names, sorted by label name; missing values use explicit fallbacks.
|
||||
- No alert expression, threshold, duration, NetworkPolicy, dashboard JSON, retention setting, or Slack destination changes.
|
||||
- No new formatter workload or generic automation script.
|
||||
- This is a non-Git workspace. Record starting/final SHA-256 and metadata in task reports instead of creating commits or worktrees.
|
||||
- Production changes require a real failing test first. Run each expensive focused suite at most once for RED and once for final GREEN unless a genuine failure requires a fix.
|
||||
- Do not execute live apply until source review is clean. Direct `kubectl apply` is forbidden; use the existing rules-alerts transaction exactly once with a fresh rollback ID and fresh verified handoff.
|
||||
|
||||
## Exact AlertmanagerConfig Contract
|
||||
|
||||
The canonical source and every exact oracle use this `spec` value. YAML block scalars preserve the template text exactly.
|
||||
|
||||
```yaml
|
||||
spec:
|
||||
route:
|
||||
receiver: platform-slack
|
||||
groupBy:
|
||||
- cluster
|
||||
- namespace
|
||||
- alertname
|
||||
- severity
|
||||
groupWait: 30s
|
||||
groupInterval: 5m
|
||||
repeatInterval: 4h
|
||||
routes:
|
||||
- receiver: platform-null
|
||||
matchers:
|
||||
- name: alertname
|
||||
matchType: "="
|
||||
value: InfoInhibitor
|
||||
inhibitRules:
|
||||
- sourceMatch:
|
||||
- name: alertname
|
||||
matchType: "="
|
||||
value: InfoInhibitor
|
||||
targetMatch:
|
||||
- name: severity
|
||||
matchType: "="
|
||||
value: info
|
||||
equal:
|
||||
- namespace
|
||||
receivers:
|
||||
- name: platform-null
|
||||
- name: platform-slack
|
||||
slackConfigs:
|
||||
- apiURL:
|
||||
name: alertmanager-slack-webhook
|
||||
key: url
|
||||
sendResolved: true
|
||||
linkNames: false
|
||||
mrkdwnIn:
|
||||
- text
|
||||
- fields
|
||||
fallback: >-
|
||||
{{ if eq .Status "firing" }}FIRING{{ else }}RESOLVED{{ end }}: {{ if .CommonLabels.severity }}{{ .CommonLabels.severity | toUpper }}{{ else }}UNKNOWN{{ end }} · {{ .CommonLabels.alertname }}
|
||||
title: >-
|
||||
{{ if eq .Status "firing" }}[FIRING:{{ .Alerts.Firing | len }}]{{ else }}[RESOLVED]{{ end }} {{ if .CommonLabels.severity }}{{ .CommonLabels.severity | toUpper }}{{ else }}UNKNOWN{{ end }} · {{ .CommonLabels.alertname }}
|
||||
titleLink: https://grafana.learn.hyeonworks.com/
|
||||
color: >-
|
||||
{{ if eq .Status "resolved" }}good{{ else if or (eq .CommonLabels.severity "emergency") (eq .CommonLabels.severity "critical") }}danger{{ else if eq .CommonLabels.severity "warning" }}warning{{ else }}#439FE0{{ end }}
|
||||
fields:
|
||||
- title: Status
|
||||
value: '{{ .Status | toUpper }}'
|
||||
short: true
|
||||
- title: Severity
|
||||
value: '{{ if .CommonLabels.severity }}{{ .CommonLabels.severity | toUpper }}{{ else }}UNKNOWN{{ end }}'
|
||||
short: true
|
||||
- title: Location
|
||||
value: '{{ if .CommonLabels.cluster }}{{ .CommonLabels.cluster }}{{ else }}unknown-cluster{{ end }} / {{ if .CommonLabels.namespace }}{{ .CommonLabels.namespace }}{{ else }}cluster-scoped{{ end }}'
|
||||
short: true
|
||||
- title: Alert count
|
||||
value: '{{ len .Alerts }}'
|
||||
short: true
|
||||
text: |-
|
||||
{{ range .Alerts }}
|
||||
*Alert status:* {{ .Status | toUpper }}
|
||||
*Target:* {{ $target := .Labels.Remove $.GroupLabels.Names }}{{ if $target }}{{ range $target.SortedPairs }}{{ .Name }}={{ .Value }} {{ end }}{{ else }}unknown{{ end }}
|
||||
*Summary:* {{ with .Annotations.summary }}{{ . }}{{ else }}No summary provided{{ end }}
|
||||
*Details:* {{ with .Annotations.description }}{{ . }}{{ else }}No description provided{{ end }}
|
||||
*Started:* {{ .StartsAt.Format "2006-01-02T15:04:05Z07:00" }}
|
||||
{{ if eq .Status "resolved" }}*Ended:* {{ .EndsAt.Format "2006-01-02T15:04:05Z07:00" }}{{ end }}
|
||||
{{ with .Annotations.runbook_url }}*Runbook:* <{{ . }}|대응 절차 열기>{{ end }}
|
||||
|
||||
{{ end }}
|
||||
<https://grafana.learn.hyeonworks.com/|Grafana> · <https://grafana.learn.hyeonworks.com/explore|Explore>
|
||||
footer: hyeonworks observability · Alertmanager
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Source and Core Renderer Exact Contract
|
||||
|
||||
**Files:**
|
||||
- Modify: `scripts/validate/test-observability-alerting.sh`
|
||||
- Modify: `scripts/validate/test-render-observability-core.sh`
|
||||
- Modify: `services/observability/alerting/platform-alertmanager.yaml`
|
||||
- Modify: `scripts/validate/render-observability-core.sh`
|
||||
- Create: `.superpowers/sdd/2026-08-15-observability-structured-slack-notifications/task-1-report.md`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: the exact AlertmanagerConfig contract above.
|
||||
- Produces: source YAML and core renderer assertion that accept only that contract; later tasks rely on `assert_alertmanager_routing_contract` as the canonical oracle.
|
||||
|
||||
- [ ] **Step 1: Record the starting identities**
|
||||
|
||||
Record SHA-256, mode, owner, nlink, and size for the four modified files in the task report. Starting SHA-256 values must be:
|
||||
|
||||
```text
|
||||
platform-alertmanager.yaml 76db5e6dd3653bc8afb9ecffe3472fc57eb07c5a2b33d218f3f4a41fb202caa2
|
||||
test-observability-alerting.sh c5c578435cfeaac0d4221dd335797e09aaa010a9f63c397d21936a252e461db2
|
||||
render-observability-core.sh 14460b985489ca59ba2377a61300a187d84fe795c8307fe2ba32bee9674d4bb2
|
||||
test-render-observability-core.sh 9dd935e403968bf063ec0ddd44cd32ba853fe9f121a2764fbb01206d3c561d35
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Write the failing source and core tests**
|
||||
|
||||
In `test-observability-alerting.sh`, replace the old exact `alert` oracle with the exact contract above, retaining exact metadata and API identity. In `test-render-observability-core.sh`, update the valid Alertmanager fixture to the exact contract and add independently hand-authored negative fixtures for:
|
||||
|
||||
```text
|
||||
missing severity from groupBy
|
||||
missing title
|
||||
color that maps critical to warning
|
||||
InfoInhibitor child route pointing to platform-slack
|
||||
missing inhibit rule
|
||||
platform-null containing a Slack integration
|
||||
channel override
|
||||
text containing @channel
|
||||
extra receiver
|
||||
```
|
||||
|
||||
Each negative must exercise `assert_alertmanager_routing_contract` and expect rejection.
|
||||
|
||||
- [ ] **Step 3: Run RED against unchanged production**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
bash -n scripts/validate/test-observability-alerting.sh
|
||||
bash -n scripts/validate/test-render-observability-core.sh
|
||||
bash scripts/validate/test-observability-alerting.sh
|
||||
bash scripts/validate/test-render-observability-core.sh
|
||||
```
|
||||
|
||||
Expected: both syntax checks return 0; at least one behavioral command returns nonzero because production still has the old three-key group and terse singleton receiver. Capture the exact first failing leaf. A syntax or fixture-construction error is not an acceptable RED.
|
||||
|
||||
- [ ] **Step 4: Implement the minimal source and canonical oracle**
|
||||
|
||||
Use `apply_patch`. Replace only the `spec` in `platform-alertmanager.yaml` with the exact contract above. Replace `render-observability-core.sh` mode `alertmanager_routing`'s `expected_config` with the exact full object, including receiver order, every template string, child route, and inhibit rule. Do not change the NetworkPolicy oracle or Alertmanager global-reference checks.
|
||||
|
||||
- [ ] **Step 5: Run GREEN and mutation checks**
|
||||
|
||||
Run both commands from Step 3 again. Expected exact terminal markers:
|
||||
|
||||
```text
|
||||
OBSERVABILITY ALERTING STATIC CONTRACT PASS
|
||||
OBSERVABILITY ALERTMANAGER GLOBAL RENDER CONTRACT PASS
|
||||
OBSERVABILITY CORE STATIC RENDER PASS
|
||||
```
|
||||
|
||||
Confirm every negative fixture returns nonzero while the canonical fixture returns zero. Record assertion counts and final identities in `task-1-report.md`.
|
||||
|
||||
### Task 2: Access Renderer and Transaction Runtime Contract
|
||||
|
||||
**Files:**
|
||||
- Modify: `scripts/validate/test-render-observability-access.sh`
|
||||
- Modify: `scripts/validate/render-observability-access.sh`
|
||||
- Modify: `scripts/validate/test-apply-observability-access.sh`
|
||||
- Modify: `scripts/bootstrap/apply-observability-access.sh`
|
||||
- Create: `.superpowers/sdd/2026-08-15-observability-structured-slack-notifications/task-2-report.md`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: Task 1's `assert_alertmanager_routing_contract` and exact source configuration.
|
||||
- Produces: an access handoff that rejects altered templates/routes and a transaction postcheck that accepts only the qualified receiver pair.
|
||||
|
||||
- [ ] **Step 1: Record starting identities**
|
||||
|
||||
Starting SHA-256 values must be:
|
||||
|
||||
```text
|
||||
render-observability-access.sh ae5373b1b20be42e573d58e7384dcc654b1d3cac75a6c269cf684f4a14f3ca1a
|
||||
test-render-observability-access.sh 5dfdc52b9f72b6d2e754893e799f5baf27c2b87e701958452dc170627ff21435
|
||||
apply-observability-access.sh f797c3c678e45adb08c407f9a55b6da10fa49d93038cd0b301f589555b11263c
|
||||
test-apply-observability-access.sh dd78ef4601c89652b9270a7fcb257232a9f54431f65b96669062cc0efa920e4f
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Write access RED cases**
|
||||
|
||||
Update the valid fixtures to the exact Task 1 contract. In `test-render-observability-access.sh`, add mutations that remove `title`, change `groupBy`, route `InfoInhibitor` to Slack, add `channel`, and add an extra receiver. Each must fail at the published handoff assertion with no partial output mutation.
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
bash -n scripts/validate/test-render-observability-access.sh
|
||||
bash scripts/validate/test-render-observability-access.sh
|
||||
```
|
||||
|
||||
Expected: syntax 0 and behavioral nonzero against the current loose `rules-alerts` access assertion; capture the leaf showing an invalid structured contract was accepted.
|
||||
|
||||
- [ ] **Step 3: Write apply RED cases**
|
||||
|
||||
Update the fake `/api/v2/receivers` default to the qualified pair. Define literal cases:
|
||||
|
||||
```python
|
||||
qualified_pair = [
|
||||
{"name": "observability/platform-alertmanager/platform-null"},
|
||||
{"name": "observability/platform-alertmanager/platform-slack"},
|
||||
]
|
||||
```
|
||||
|
||||
Add one accepted reversed-order case and rejected cases for old Slack singleton, null singleton, empty list, raw pair, duplicate member, extra receiver, malformed JSON, and wrong namespace/config/local name for either member. Add a structured-handoff mutation that must fail before prompt, ledger creation, dry-run, or apply.
|
||||
|
||||
Run the smallest named positive case or the bounded focused suite until the new qualified pair reaches the old singleton predicate. Expected RED is nonzero at the receiver postcheck, not a fixture or syntax error.
|
||||
|
||||
- [ ] **Step 4: Strengthen production access and apply checks**
|
||||
|
||||
In `render-observability-access.sh`, replace the one-line shell wrapper with:
|
||||
|
||||
```bash
|
||||
assert_access_rules_alerts_contract() {
|
||||
(( $# == 3 )) || return 1
|
||||
_access_assert rules-alerts "$@" || return 1
|
||||
assert_alertmanager_routing_contract "$3"
|
||||
}
|
||||
```
|
||||
|
||||
Keep the existing dashboard, rule, runbook, scope, and NetworkPolicy checks.
|
||||
|
||||
In the apply postcheck Python, replace singleton logic with:
|
||||
|
||||
```python
|
||||
expected_receivers = {
|
||||
"observability/platform-alertmanager/platform-null",
|
||||
"observability/platform-alertmanager/platform-slack",
|
||||
}
|
||||
if (
|
||||
not isinstance(receiver_payload, list)
|
||||
or len(receiver_payload) != 2
|
||||
or any(not isinstance(item, dict) or set(item) != {"name"} for item in receiver_payload)
|
||||
or {item["name"] for item in receiver_payload} != expected_receivers
|
||||
):
|
||||
raise SystemExit(1)
|
||||
```
|
||||
|
||||
Do not change rollback, ledger, resource rows, Slack gate, Secret name/key validation, or acceptance schema.
|
||||
|
||||
- [ ] **Step 5: Run final focused GREEN once**
|
||||
|
||||
Run syntax for all four files, then:
|
||||
|
||||
```bash
|
||||
bash scripts/validate/test-render-observability-access.sh
|
||||
timeout 330s bash scripts/validate/test-apply-observability-access.sh
|
||||
```
|
||||
|
||||
Expected: access terminal PASS; apply terminal `APPLY OBSERVABILITY ACCESS TEST PASS`, the documented assertion count, suite wall at most 300 seconds, outer RC0, and relevant orphan count 0. Verify every receiver mutation fails at its intended leaf. Record final identities and evidence in `task-2-report.md`.
|
||||
|
||||
### Task 3: Integrated Schema, Template, and Independent Source Review
|
||||
|
||||
**Files:**
|
||||
- Modify only if the RED demands it: `scripts/validate/test-observability-alerting.sh`
|
||||
- Create: `.superpowers/sdd/2026-08-15-observability-structured-slack-notifications/task-3-report.md`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: Tasks 1-2 frozen candidate bytes.
|
||||
- Produces: pre-live Ready/Not Ready verdict and exact candidate identities.
|
||||
|
||||
- [ ] **Step 1: Run complete syntax and source-focused matrix**
|
||||
|
||||
Run Bash syntax on all eight changed scripts. Run `test-observability-alerting.sh` without live mode once, then the already-green core/access/apply commands only if their final Task reports do not contain unambiguous RC0 evidence for the same hashes. Do not rerun an expensive suite merely to duplicate evidence.
|
||||
|
||||
- [ ] **Step 2: Validate CRD admission without mutation**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
PLATFORM_KUBECTL_BIN=/usr/local/bin/kubectl \
|
||||
bash scripts/validate/test-observability-alerting.sh --server-dry-run
|
||||
```
|
||||
|
||||
Expected: both server-side dry runs and all static markers pass. This is schema evidence only.
|
||||
|
||||
- [ ] **Step 3: Render real templates with pinned Alertmanager 0.33.1**
|
||||
|
||||
Extract `fallback`, `title`, `color`, all field values, and `text` from the candidate YAML without exposing any Secret value. Use `/bin/amtool` in the existing Alertmanager 0.33.1 container with `amtool template render`; pass a hand-authored JSON fixture through `/dev/stdin`, not a persistent pod file. The fixture must cover:
|
||||
|
||||
```text
|
||||
warning firing with cluster and namespace
|
||||
resolved alert with EndsAt
|
||||
missing cluster, namespace, summary, description, and runbook
|
||||
two alerts with different non-group labels
|
||||
```
|
||||
|
||||
Assert literal results include the correct severity color, `unknown-cluster`, `cluster-scoped`, sorted target labels, both fallbacks, FIRING count, RESOLVED title, runbook when present, Grafana links, and footer. No Secret or webhook URL enters the fixture or output.
|
||||
|
||||
- [ ] **Step 4: Perform independent read-only review**
|
||||
|
||||
The reviewer reads the design, plan, task reports, final changed files, and current CRD. Required verdict fields are Spec PASS/FAIL, Critical/Important/Minor counts, and Ready for one transactional live apply YES/NO. Any Critical or Important finding must be fixed with a new failing test and re-reviewed before Task 4.
|
||||
|
||||
- [ ] **Step 5: Freeze pre-live identities**
|
||||
|
||||
Record SHA-256 and metadata for all eight source/test files plus the design, plan, and Task reports. Confirm no webhook literal, decoded Secret, `channel`, `@channel`, or `@here` occurs in governed source. Record process/temp residue and mark Task 3 Ready only when all checks pass.
|
||||
|
||||
### Task 4: Transactional Rollout, Synthetic Delivery, and Documentation
|
||||
|
||||
**Files:**
|
||||
- Modify after observed results: `/home/donghyeon/workspace/docs/platform/observability/03-alerting-and-slack.md`
|
||||
- Modify after observed results: `/home/donghyeon/workspace/docs/platform/observability/07-build-history-and-decisions.md`
|
||||
- Create: `.superpowers/sdd/2026-08-15-observability-structured-slack-notifications/task-4-report.md`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: Task 3 Ready verdict and frozen candidate identities.
|
||||
- Produces: live structured Slack configuration, observed FIRING/RESOLVED acceptance, cleanup evidence, and final operator documentation.
|
||||
|
||||
- [ ] **Step 1: Prepare one fresh transaction**
|
||||
|
||||
Use a fresh UTC rollback ID unequal to every previous ID and create its root once as root-owned mode 0700. Preserve all prior rollback roots. Generate one fresh private handoff from the authoritative immutable inventory pair:
|
||||
|
||||
```text
|
||||
target-initial 79688d017d38eec9a6f100f8d0f784a5474e79802046ef1c2c11b30d170b0b0c 21
|
||||
post-substrate b1c3049206a1a88165ee672ae9aceac7945673a3bb9c3cf3670b7f0d56c3f291 30
|
||||
```
|
||||
|
||||
Render rules-alerts into the fresh root and verify exact source hashes, metadata, topology, and new structured AlertmanagerConfig. Run the no-argument dry mode once. Do not reuse a prior handoff or rollback ID.
|
||||
|
||||
- [ ] **Step 2: Execute the existing transaction once**
|
||||
|
||||
Build a six-token Bash array to avoid paste splitting:
|
||||
|
||||
```bash
|
||||
TASK4_CMD=(bash)
|
||||
TASK4_CMD+=(scripts/bootstrap/apply-observability-access.sh)
|
||||
TASK4_CMD+=(--execute)
|
||||
TASK4_CMD+=(--rules-alerts)
|
||||
TASK4_CMD+=(--verified-output-dir)
|
||||
TASK4_CMD+=("$METRIC_ROOT")
|
||||
```
|
||||
|
||||
Export the fresh `PLATFORM_OBSERVABILITY_ROLLBACK_ID` and exact `PLATFORM_HELM_BIN=/home/donghyeon/.local/bin/helm`. Invoke `"${TASK4_CMD[@]}"` once, enter exact `APPLY`, and make the immediate next statement `TASK4_APPLY_RC=$?`. Require RC0, exact transaction PASS, both inventory SHA markers, no rollback ambiguity, and accepted v2 ledger. A nonzero or ambiguous result means preserve evidence and stop; never retry the same ID.
|
||||
|
||||
- [ ] **Step 3: Verify runtime reconciliation**
|
||||
|
||||
Require Alertmanager CR Available/Reconciled, StatefulSet ready, `alertmanager_config_last_reload_successful == 1`, no relevant operator/config-reloader error, production rules healthy, and the exact qualified receiver pair. AlertmanagerConfig itself has no usable status condition and must not be reported as Available/Reconciled.
|
||||
|
||||
- [ ] **Step 4: Verify one warning FIRING and RESOLVED message**
|
||||
|
||||
Create one uniquely named, temporary warning `PrometheusRule` with the required `observability.hyeonworks.com/instance=home` selector label, literal summary/description/runbook, and a unique target label. Wait until Prometheus and Alertmanager each show exactly one matching firing alert. The user verifies that Slack contains the agreed title, severity, fields, target labels, summary, details, start time, runbook, Grafana links, and footer.
|
||||
|
||||
Delete the temporary rule once. Wait for the matching RESOLVED message and have the user verify its resolved title, green color, end time, and same context. The synthetic rule name and labels must never collide with production rules.
|
||||
|
||||
- [ ] **Step 5: Clean up and document only observed facts**
|
||||
|
||||
Require synthetic resource absent, matching pending/firing/active alerts zero, production rules healthy, receiver pair exact, and no mutator process or task-created temp residue. Confirm `InfoInhibitor` is not delivered to Slack. Document the conditional upstream info-inhibition semantics; do not claim every info alert is always suppressed.
|
||||
|
||||
Update chapter 03 with the exact source contract and observed FIRING/RESOLVED results. Update chapter 07 with the original terse-message problem, the decision to keep formatting in Alertmanager, and the final transaction result. Record final file identities and any limitation, including long grouped Slack text, in `task-4-report.md`.
|
||||
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user