# Observability Task 7 Cutover Correction Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Correct the fail-closed Host Nginx/Task 6 evidence interface, preserve the successful Task 6 authority, and complete Task 7 through one Host cutover, one OIDC membership transaction, one monolithic smoke, and one bounded regression pass.

**Architecture:** Change only the Host Nginx production script and its focused test for the v2 evidence, first-cutover, command-boundary, and no-contact defects. Keep OIDC, smoke, and renderer production behavior unchanged; correct their orchestration in the central/manual documents, use a fresh inventory-only renderer root, then freeze exact non-Git identities before live execution.

**Tech Stack:** Bash 5.2, Python 3.12, Nginx/systemd, Kubernetes/k3s/kubectl, Keycloak Admin API, Prometheus/Alertmanager/Grafana, hardened private `/tmp` handoffs, SHA-256 non-Git source manifests.

## Global Constraints

- Do not read, print, hash, copy, or disclose Kubernetes Secret values, Slack webhook data, OAuth codes/state, passwords, cookies, bearer tokens, or private recovery payloads.
- Preserve rollback ID `20260814T145009Z` and `/tmp/platform-observability-metrics.dw5gLZ` through Task 7; never use that handoff as complete-render publication output.
- Preserve failed IDs `20260814T080303Z`, `20260814T140953Z`, `/tmp/platform-observability-metrics.LNzksC`, and all historical evidence without mutation or deletion.
- The accepted inventory pins remain `79688d017d38eec9a6f100f8d0f784a5474e79802046ef1c2c11b30d170b0b0c` and `b1c3049206a1a88165ee672ae9aceac7945673a3bb9c3cf3670b7f0d56c3f291`.
- The exact Nginx deny/full pins remain `dbef6d443bcba58b26a5351ea76f6d09f6da8c2ef07a806e22745cf26c88f518` and `7d2de2a92c3597a0859775da1d2ccbf5a3d72c0b2af5cac2439c82311361f801`.
- Full cutover accepts only `platform-observability-rules-alerts-v2`; legacy v1 has no fallback.
- First full cutover requires exact deny-only active state. `ALREADY_ACTIVE` is a STOP condition, not success.
- Before the first full-cutover live preflight or APPLY prompt, the `host-nginx` subtree, ledger, payload
  directory, and first payload path must each be absent under both existence and symlink tests. Normal
  existing files/directories are residue and fail closed just like symlinks.
- A proof older than 24 hours requires a fresh rollback ID, fresh proof, and fresh Task 6 acceptance. Never recreate only the proof or retry Host under the frozen ID.
- Run `observability-smoke.sh --execute` exactly once, after membership reconciliation and external-client readiness. RC `2` remains partial implementation.
- OIDC rollback is in-process automatic rollback; do not claim a persistent Keycloak reversal ledger under the Task 6 root.
- Every live command gets immediate RC capture. Interpret rollback by stage: pre-transaction/pre-active-config failures have rollback N/A; active transaction failures require exact rollback PASS; ambiguity/manual recovery stops all later tasks.
- Any Host ledger/payload/timestamp staging failure consumes the rollback ID for Host purposes: preserve it,
  create a fresh ID, and repeat source proof plus complete Task 6. Never repair staging or retry Host in place.
- Before any operator-shell preparation command, either install the exact production PATH, export it, run
  `hash -r`, and verify the reviewed command inventory, or use literal absolute paths for every preparation
  executable. Do not mix an ambient-PATH setup command into a fixed-boundary procedure.
- Do not mark existing Task 7 checkboxes until the corresponding live result is observed and independently reviewed. Task 8 remains unstarted.
- This workspace is non-Git. Do not commit, create a branch, or claim worktree cleanliness; record exact file metadata and SHA-256 instead. After freeze this implementation plan is immutable authority; track progress only in reports.

---

### Task 1: Make the Host acceptance fixture production-faithful

**Files:**
- Modify: `scripts/validate/test-apply-host-nginx-observability.sh:102-143`
- Modify: `scripts/bootstrap/apply-host-nginx-observability.sh:34,372-423,517-612`
- Create: `.superpowers/sdd/2026-08-15-observability-task7-cutover-correction/task-1-report.md`

**Interfaces:**
- Consumes: Task 6 `access-rules-alerts/acceptance.env`, the two metric inventory hashes, and the same rollback ID.
- Produces: `validate_full_evidence()` acceptance only for an exact nine-field v2 record and one of two exact Slack gate tuples.

- [ ] **Step 1: Freeze the starting identities**

  Record SHA-256, mode, UID:GID, nlink, type, and size for the two files. Required starting hashes are:

  ```text
  apply = 518119a5e11fe8462c5d41921534351f78a2dd721c1bf69ac638e5bffe83536c
  test  = 1c429f7eb2cb73ec25dacd9fe4fdc5e063f62866c6e36732a3e9116fa68b2f53
  ```

  If either hash differs, stop for source-drift review before editing.

- [ ] **Step 2: Change the focused fixture to v2 and add a legacy-v1 rejection test**

  Replace the acceptance writer with an exact helper whose default is the successful Task 6 risk tuple:

  ```bash
  write_acceptance_v2() {
    local path=$1 initial_sha=$2 post_sha=$3
    local gate=${4:-RISK_ACCEPTED} ref uid
    case "$gate" in
      RECOVERY)
        ref=strict-recovery-evidence-v1
        uid=not-applicable
        ;;
      RISK_ACCEPTED)
        ref=2026-08-14-observability-slack-recovery-risk-acceptance-design
        uid=1000
        ;;
      *) return 2 ;;
    esac
    cat >"$path" <<EOF
  schema=platform-observability-rules-alerts-v2
  rollback_id=$ROLLBACK_ID
  target_initial_sha256=$initial_sha
  post_substrate_sha256=$post_sha
  slack_deployment_gate=$gate
  slack_gate_approval_ref=$ref
  slack_gate_accepted_by_uid=$uid
  accepted_at_utc=$(date -u +%Y-%m-%dT%H:%M:%SZ)
  state=accepted
  EOF
    chmod 0600 "$path"
  }
  ```

  Add a fixture that removes the three Slack fields and changes schema to
  `platform-observability-rules-alerts-v1`. It must expect nonzero RC, exact unchanged deny active bytes,
  no preflight/commit command, no `host-nginx` ledger, and no APPLY prompt.

- [ ] **Step 3: Run RED before changing production**

  Run:

  ```bash
  bash -n scripts/validate/test-apply-host-nginx-observability.sh
  timeout --signal=TERM --kill-after=5s 240s \
    bash scripts/validate/test-apply-host-nginx-observability.sh
  ```

  Expected: syntax RC `0`; suite RC nonzero because the default v2 success fixture is rejected by the current v1 parser and/or legacy v1 is still accepted. Record the exact first failing assertion and prove the active fixture remained deny-only.

- [ ] **Step 4: Implement the exact v2 parser and gate tuple validator**

  Set:

  ```bash
  readonly RULES_ACCEPTANCE_SCHEMA=platform-observability-rules-alerts-v2
  ```

  Read exactly these fields, in any file order but with exact cardinality and no duplicates/extras:

  ```bash
  read_exact_env "$acceptance" \
    schema rollback_id target_initial_sha256 post_substrate_sha256 \
    slack_deployment_gate slack_gate_approval_ref slack_gate_accepted_by_uid \
    accepted_at_utc state
  ```

  Add and call this predicate after schema/hash checks:

  ```bash
  validate_slack_acceptance_tuple() {
    local tuple
    tuple="${PARSED_ENV[slack_deployment_gate]}|${PARSED_ENV[slack_gate_approval_ref]}|${PARSED_ENV[slack_gate_accepted_by_uid]}"
    case "$tuple" in
      'RECOVERY|strict-recovery-evidence-v1|not-applicable'|\
      'RISK_ACCEPTED|2026-08-14-observability-slack-recovery-risk-acceptance-design|1000')
        return 0
        ;;
      *)
        fail 'rules-alerts Slack deployment gate tuple mismatch'
        return 1
        ;;
    esac
  }
  ```

  Do not accept v1, ignore the new fields, infer tuple values, or add an environment-controlled fallback.

- [ ] **Step 5: Add the complete evidence adversarial matrix**

  Add table-driven negatives for:

  ```text
  v1
  wrong-schema
  missing-slack-gate
  extra-field
  duplicate-field
  recovery-with-risk-ref
  recovery-with-uid-1000
  risk-with-strict-ref
  risk-with-not-applicable
  arbitrary-gate
  arbitrary-ref
  arbitrary-uid
  rollback-id
  target-initial-hash
  post-substrate-hash
  state
  malformed-accepted-at
  ```

  Add positive transactions for both exact `RECOVERY` and `RISK_ACCEPTED` tuples. Every negative must prove preflight/mutation `0`, active bytes unchanged, no Host ledger, no prompt, and nonzero RC.

- [ ] **Step 6: Run GREEN and record evidence**

  Run:

  ```bash
  bash -n scripts/bootstrap/apply-host-nginx-observability.sh
  bash -n scripts/validate/test-apply-host-nginx-observability.sh
  timeout --signal=TERM --kill-after=5s 300s \
    bash scripts/validate/test-apply-host-nginx-observability.sh
  ```

  Expected: both syntax RC `0`, focused suite RC `0`, exact terminal `APPLY HOST NGINX OBSERVABILITY TEST PASS`, and no fixture/process residue. Append RED/GREEN commands, RCs, assertion count, final hashes, and metadata to the Task 1 report.

---

### Task 2: Enforce first-cutover state, fixed command boundaries, and no-contact dry-run

**Files:**
- Modify: `scripts/bootstrap/apply-host-nginx-observability.sh:1-35,204-265,315-351,883-921`
- Modify: `scripts/validate/test-apply-host-nginx-observability.sh`
- Append: `.superpowers/sdd/2026-08-15-observability-task7-cutover-correction/task-1-report.md`

**Interfaces:**
- Consumes: exact deny and full candidate hashes plus the v2 evidence validator from Task 1.
- Produces: production execution isolated from ambient PATH, a true no-network dry-run, and a full transition that starts only from deny-only.

- [ ] **Step 1: Add focused RED cases before production changes**

  Add these cases:

  1. Copy the full candidate over the active fixture, invoke full execute, and require failure before preflight/ledger/prompt. Current production emits `HOST_NGINX_FULL_STAGE=ALREADY_ACTIVE`, so this is RED.
  2. Extract `print_plan` with `declare -f` and require that it contains none of `dig`, `curl`, `kubectl`, `sudo`, `nginx`, or `systemctl`. Current production contains `dig`, so this is RED.
  3. Put a logging `dirname` wrapper first in a hostile PATH and run `/usr/bin/bash` with `PLATFORM_HOST_NGINX_TEST_MODE=invalid`. Require nonzero RC and an empty hostile log. Current source resolves `dirname` before rejecting the invalid override, so this is RED.
  4. Retain a third-state active fixture and require unchanged bytes, no ledger, no prompt, and nonzero RC.
  5. Precreate a normal, non-symlink `host-nginx/stages.tsv` with the valid header and safe metadata; require
     rejection before preflight/ledger write/prompt with unchanged deny bytes. Current production appends an
     attempt to it, so this is RED.
  6. In a separate fixture, precreate a normal, non-symlink
     `host-nginx/payloads/full-prior-0001.conf` with safe metadata; require the same prompt-before rejection.
     Retain the existing symlinked parent/ledger/payload and hardlinked-ledger negatives independently.

  Run the focused suite once and record all expected RED leaves.

- [ ] **Step 2: Install the production PATH before any external command**

  Immediately after `umask 077`, before computing `ROOT`, install:

  ```bash
  readonly HOST_NGINX_PRODUCTION_PATH='/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'
  PATH=$HOST_NGINX_PRODUCTION_PATH
  export PATH
  hash -r
  ```

  Compute `ROOT` without `dirname`:

  ```bash
  readonly ROOT="$(cd -- "${BASH_SOURCE[0]%/*}/../.." && pwd -P)"
  ```

  After a test fixture has passed the existing canonical owner/mode/boundary checks, prepend only its validated `bin` directory for test mode. Production never restores the caller PATH.

- [ ] **Step 3: Validate the exact production resolution set**

  Add an exact associative allowlist:

  ```bash
  declare -Ar HOST_NGINX_PRODUCTION_COMMANDS=(
    [awk]=/usr/bin/awk [bash]=/usr/bin/bash [cat]=/usr/bin/cat
    [curl]=/usr/bin/curl [date]=/usr/bin/date [dig]=/usr/bin/dig
    [dirname]=/usr/bin/dirname [grep]=/usr/bin/grep [head]=/usr/bin/head
    [id]=/usr/bin/id [install]=/usr/bin/install [jq]=/usr/bin/jq
    [kubectl]=/usr/local/bin/kubectl [mktemp]=/usr/bin/mktemp
    [openssl]=/usr/bin/openssl [python3]=/usr/bin/python3
    [readlink]=/usr/bin/readlink [rm]=/usr/bin/rm [sed]=/usr/bin/sed
    [sha256sum]=/usr/bin/sha256sum [sleep]=/usr/bin/sleep
    [sort]=/usr/bin/sort [ss]=/usr/bin/ss [stat]=/usr/bin/stat
    [tail]=/usr/bin/tail [test]=/usr/bin/test [tr]=/usr/bin/tr
  )
  ```

  The fixed allowlist has exactly `27` entries. For production, require `command -v` lexical equality for
  every non-builtin entry, and bind the `test` entry directly to exact `/usr/bin/test` because Bash otherwise
  resolves its builtin before PATH; every sudo test probe is literal `sudo -n /usr/bin/test`, never a bare
  `test` or ambient/`secure_path` lookup. Resolve the final target with `readlink -f`, and require a root-owned
  regular executable with nlink `1`, no group/world write, and no
  setuid/setgid bits. Validate fixed Nginx, systemd, and Certbot with that same rule. The sole exception is
  sudo, which must have lexical and canonical path exact `/usr/bin/sudo`, regular/non-symlink type,
  root:root, nlink `1`, and exact mode `4755`. Owner-setuid `04000` is required for sudo; setgid,
  group/world write, missing setuid, owner/link/path drift, or any other mode is rejected. Never extend this
  exception to another command. Verify each PATH directory or its canonical target is root-owned and not
  group/world writable.

  Add focused metadata cases proving exact `/usr/bin/sudo` mode `4755` passes and each of missing setuid,
  wrong owner, nlink greater than one, lexical/canonical path drift, setgid, group write, and world write
  fails. Independently add setuid and setgid to a non-sudo allowlisted executable and require rejection.

- [ ] **Step 4: Make dry-run truthful and make full first-cutover-only**

  Remove public `dig` from `print_plan`. Replace the false claim:

  ```text
  HOST_NGINX_GRAFANA_PUBLIC_DNS=ABSENT
  ```

  with:

  ```text
  HOST_NGINX_GRAFANA_PUBLIC_DNS=NOT_CHECKED_DRY_RUN
  ```

  Replace the false live-SAN claim:

  ```text
  HOST_NGINX_CERTIFICATE_EXACT_SAN=grafana.learn.hyeonworks.com
  ```

  with the exact two-line dry-run contract:

  ```text
  HOST_NGINX_CERTIFICATE_EXPECTED_SAN=grafana.learn.hyeonworks.com
  HOST_NGINX_CERTIFICATE_SAN=NOT_CHECKED_DRY_RUN
  ```

  Keep public DNS absence in `external_preflight full`.

  Extend the focused dry-run case to require the three new marker lines exactly once, require both legacy
  claims absent, and require the certificate and DNS boundary logs to stay empty. This is a behavioral test,
  not a source-text-only assertion.

  Add a full-only first-cutover residue predicate before `external_preflight`, any ledger preparation, and
  `confirm`. For each literal path below, require both `sudo -n /usr/bin/test ! -e` and
  `sudo -n /usr/bin/test ! -L`:

  ```text
  $ROLLBACK_ROOT/host-nginx
  $ROLLBACK_ROOT/host-nginx/stages.tsv
  $ROLLBACK_ROOT/host-nginx/payloads
  $ROLLBACK_ROOT/host-nginx/payloads/full-prior-0001.conf
  ```

  Check the `host-nginx` root first. If it exists or is a symlink, fail without traversing a child. Only while
  the root is proven absent/non-symlink may the three child paths be checked; recheck root absence after those
  probes to close the prompt-before gate. The normal-ledger and normal-payload fixtures must prove rejection
  before preflight and prompt. This predicate applies only to the first `full` transition; do not change
  metrics, certificate, or deny-only idempotency.

  In `transition`, after computing hashes and validating full evidence, enforce:

  ```bash
  if [[ "$stage" == full && "$active_sha" == "$candidate_sha" ]]; then
    fail 'full cutover is already active without first-cutover provenance'
    return 1
  fi
  ```

  Then retain the existing exact expected-active gate. Do not change idempotency for metrics, certificate, or deny-only modes.

- [ ] **Step 5: Add stale/future proof and rollback-outcome cases**

  Add proof timestamps for exact current, `now-86401`, `now+301`, and invalid calendar values. Add a rollback-failure fake that makes restore/install or reload fail and requires both:

  ```text
  HOST_NGINX_OBSERVABILITY_ROLLBACK=FAIL
  MANUAL_RECOVERY_REQUIRED=YES
  ```

  Retain success assertions for exact prior payload, three stable postchecks, full PASS, and postcheck-failure rollback PASS. Add a fault after ledger/payload or timestamp-backup staging but before `rollback_armed=true`; require nonzero RC, unchanged active deny bytes/reload count, rollback marker absent/N/A, and preserved root-owned staged evidence.

  For every injected directory, ledger, payload, and timestamp staging failure, also require the terminal
  classification `fresh rollback ID + source proof + complete Task 6 required`; same-ID repair, proof-only
  refresh, or Host retry is forbidden.

- [ ] **Step 6: Run GREEN, hostile-boundary checks, and mutation tests**

  Run syntax and the complete focused suite under a 360-second outer bound. Then make private mode-0700 copies and independently inject:

  ```text
  RULES_ACCEPTANCE_SCHEMA=v1
  tuple validation accepts gate name only
  active full returns ALREADY_ACTIVE
  print_plan calls dig
  print_plan emits CERTIFICATE_EXACT_SAN or omits EXPECTED_SAN/NOT_CHECKED
  production retains caller PATH
  active hash uses a prefix comparison
  full first-cutover accepts a normal existing ledger or prior payload
  sudo metadata accepts missing-setuid/setgid/write/owner/link/path drift
  a non-sudo command accepts setuid or setgid
  ```

  For each mutation, require focused RC nonzero. Remove only the identity-checked private mutation root, verify workspace source hashes unchanged, and append results to the Task 1 report.

---

### Task 3: Prove the fresh renderer-root preservation boundary

**Files:**
- Modify: `scripts/validate/test-render-observability-access.sh:796-863`
- Preserve unchanged: `scripts/validate/render-observability-access.sh`
- Create: `.superpowers/sdd/2026-08-15-observability-task7-cutover-correction/task-3-report.md`

**Interfaces:**
- Consumes: a rules-alerts-published handoff and an inventory-only clone.
- Produces: a regression proving the former is immutable input and only the latter accepts complete publication.

- [ ] **Step 1: Freeze renderer/test identities**

  Required starting hashes:

  ```text
  renderer = ae5373b1b20be42e573d58e7384dcc654b1d3cac75a6c269cf684f4a14f3ca1a
  test     = 8736658b802d511776803efc81cf000986864965456d9dbf11e0d3893de87eff
  ```

  Record metadata. Renderer production bytes must remain unchanged in this task.

- [ ] **Step 2: Add the published-root preservation characterization**

  Create an exact seven-file complete publish source. Publish rules-alerts first to a fixture root, fingerprint its root, inventories, and three YAML files, then require `publish_access_outputs ... complete` to fail without changing any fingerprint or adding a temporary/output entry.

  Required first-root topology is:

  ```text
  alertmanager.yaml
  dashboards.yaml
  post-substrate/
  rules.yaml
  target-initial/
  ```

- [ ] **Step 3: Add the inventory-only clone success characterization**

  Copy only the two inventory directories and four inventory files into a new fixture root. Require `validate_access_inventory_root` PASS, complete publication PASS, and exact final top-level set:

  ```text
  alertmanager.yaml
  blackbox.yaml
  dashboards.yaml
  grafana.yaml
  post-substrate/
  private-dns.yaml
  rules.yaml
  target-initial/
  targets.yaml
  ```

  Verify source fingerprints unchanged, all seven destination YAML files regular/owner-exact/mode `0600`/nlink `1`, and no partial publication after injected collision.

- [ ] **Step 4: Run the focused renderer suite**

  Run:

  ```bash
  bash -n scripts/validate/render-observability-access.sh
  bash -n scripts/validate/test-render-observability-access.sh
  timeout --signal=TERM --kill-after=5s 300s \
    bash scripts/validate/test-render-observability-access.sh
  ```

  Expected: syntax RC `0`, suite RC `0`, exact `OBSERVABILITY ACCESS RENDER ASSERTION TEST PASS`, original renderer hash unchanged, and no suite-owned residue. Record evidence in the Task 3 report.

---

### Task 4: Correct the authoritative Task 7 procedure without claiming live completion

**Files:**
- Modify: `bootstrap/manual/phase4-observability-access.md:381-444`
- Modify: `/home/donghyeon/workspace/docs/platform/plans/2026-08-14-observability-slack-risk-acceptance-implementation.md:631-707`
- Append pre-live correction only: `/home/donghyeon/workspace/docs/platform/runbooks/2026-08-13-observability-phase4-resume-worklog.md`
- Preserve: `/home/donghyeon/workspace/docs/platform/plans/2026-08-15-observability-task7-cutover-correction-design.md`
- Create: `.superpowers/sdd/2026-08-15-observability-task7-cutover-correction/task-4-report.md`

**Interfaces:**
- Consumes: Tasks 1–3 GREEN contracts.
- Produces: one unambiguous operator sequence; existing Task 7 boxes remain unchecked.

- [ ] **Step 1: Correct Step 1 and Host execution truth**

  Document that no-arg is source/hash/no-contact only. Live certificate, DNS, Kubernetes, proof age/ID, and NodePort checks occur inside execute before the APPLY prompt.

  Bind exact successful inputs:

  ```text
  PLATFORM_OBSERVABILITY_ROLLBACK_ID=20260814T145009Z
  METRIC_ROOT=/tmp/platform-observability-metrics.dw5gLZ
  active deny SHA-256=dbef6d443bcba58b26a5351ea76f6d09f6da8c2ef07a806e22745cf26c88f518
  full SHA-256=7d2de2a92c3597a0859775da1d2ccbf5a3d72c0b2af5cac2439c82311361f801
  ```

  Add the paste-safe array and immediate RC block from Task 6. Distinguish pre-prompt failure, post-prompt/pre-arm root-owned ledger/payload/backup staging, active-config rollback PASS, and rollback FAIL/manual recovery. Do not call `stages.tsv` a success marker or evidence staging an active Nginx mutation.

  Before execute can display `APPLY`, require name-only, no-follow absence for the `host-nginx` subtree,
  `stages.tsv`, `payloads`, and `payloads/full-prior-0001.conf`. Document that a normal existing
  ledger/payload is residue, not reusable recovery state. Any ledger/payload/timestamp staging failure
  preserves the old ID and requires a fresh rollback ID plus source proof and complete Task 6 before another
  Host attempt.

- [ ] **Step 2: Add the proof-expiry route**

  State that source proof is exact-ID/deny-hash/status/IP/time evidence with maximum age 24 hours. A stale proof stops Task 7 and requires a fresh ID plus complete Task 6 acceptance. Explicitly forbid same-ID proof replacement, timestamp rewriting, Host retry, or reuse of a failed fresh ID.

- [ ] **Step 3: Add exact OIDC membership execution and rollback truth**

  Include interactive username reads, a Bash array, immediate RC capture, exact confirmations:

  ```text
  APPLY default
  RECOVERY KEYCLOAK default
  ```

  Require `GRAFANA_OIDC_TRANSACTION=PASS` and RC `0`. Before `transaction_active=true`, managed Keycloak/Secret/membership mutation is zero and rollback is N/A. Active failures require rollback PASS; rollback FAIL/manual recovery or unknown stage stops. Remove any persistent Task 6 Keycloak reversal-ledger claim.

- [ ] **Step 4: Collapse Steps 3–5 into readiness plus one monolithic smoke**

  Prepare four distinct identities, break-glass access, Slack view, and a genuine outside-LAN/Tailscale client before starting. Keep human OIDC, Slack firing/resolved, and external attestation within exactly one `observability-smoke.sh --execute` call. Require immediate RC plus all five PASS markers. Document RC `2`/external PARTIAL as incomplete Task 7 and forbid server-side substitution.

  In each authority document, bind the execute call with the exact absolute `TASK7_SMOKE` array form from
  Task 7 Step 4; keep any no-arg plan in a separately named `TASK7_SMOKE_DRY` array. This gives the structural
  per-document assertion an unambiguous executable boundary.

- [ ] **Step 5: Correct the regression list and renderer root**

  Remove `scripts/validate/observability-core-smoke.sh --execute`; it does not exist. Remove the second `observability-smoke.sh --execute`. Keep one execution each for core renderer, fresh-root complete access renderer, admin UI, AIStor S3, phase1, phase2, and admin renderer.

  Bind phase1 and phase2 to their absolute two-element Bash arrays and invoke each with the exact inline
  `PLATFORM_HELM_BIN=/home/donghyeon/.local/bin/helm` assignment. The fixed operator `PATH` intentionally
  excludes `/home/donghyeon/.local/bin`, while both scripts otherwise fall back to `command -v helm`; do not
  permit that fallback. Require the same exact Helm assignment for the other three renderer invocations.

  Insert the hardened inventory-only clone procedure from Task 8 below and require preservation of `dw5gLZ` before and after publication.

- [ ] **Step 6: Run focused documentation assertions**

  Against only the central plan and manual, require the simple non-smoke assertions first:

  ```bash
  ! rg -F 'scripts/validate/observability-core-smoke.sh' "$CENTRAL" "$MANUAL"
  rg -Fq 'RECOVERY KEYCLOAK default' "$MANUAL"
  rg -Fq 'HOST_NGINX_FULL_STAGE=PASS' "$MANUAL"
  rg -Fq 'OBSERVABILITY_EXTERNAL_BOUNDARY=PARTIAL' "$MANUAL"
  rg -Fq '/tmp/platform-observability-metrics.dw5gLZ' "$CENTRAL" "$MANUAL"
  ```

  Do not use an aggregate raw-literal count for smoke. Run this structural assertion once per document:

  ```bash
  case $- in *e*) TASK7_DOC_ASSERT_ERREXIT=1 ;; *) TASK7_DOC_ASSERT_ERREXIT=0 ;; esac
  TASK7_DOC_ASSERT_RC=0
  set +e
  for TASK7_DOC in "$CENTRAL" "$MANUAL"; do
    /usr/bin/python3 - "$TASK7_DOC" <<'PY'
  import pathlib
  import re
  import shlex
  import subprocess
  import sys

  path = pathlib.Path(sys.argv[1])
  text = path.read_text(encoding="utf-8")
  blocks = re.findall(r"(?ms)^[ \t]*```bash[^\n]*\n(.*?)^[ \t]*```[ \t]*$", text)
  smoke_path = "/home/donghyeon/workspace/platform/scripts/validate/observability-smoke.sh"
  helm_assignment = "PLATFORM_HELM_BIN=/home/donghyeon/.local/bin/helm"
  phase_specs = {
      "TASK7_PHASE1": "/home/donghyeon/workspace/platform/scripts/validate/render-phase1.sh",
      "TASK7_PHASE2": "/home/donghyeon/workspace/platform/scripts/validate/render-phase2.sh",
  }

  def tokens(block):
      normalized = re.sub(r"\\[ \t]*\n[ \t]*", " ", block)
      lexer = shlex.shlex(
          normalized.replace("\n", " ; "), posix=True, punctuation_chars="();&|",
      )
      lexer.whitespace_split = True
      lexer.commenters = ""
      return list(lexer)

  def materialize_commands(block, arrays):
      stream = tokens(block)
      commands = []
      current = []
      index = 0
      while index < len(stream):
          token = stream[index]
          assignment = re.fullmatch(r"([A-Za-z_][A-Za-z0-9_]*)(\+?)=", token)
          if assignment and index + 1 < len(stream) and stream[index + 1] == "(":
              name, append = assignment.groups()
              index += 2
              depth = 1
              values = []
              while index < len(stream) and depth:
                  item = stream[index]
                  if item == "(":
                      depth += 1
                  elif item == ")":
                      depth -= 1
                      if depth == 0:
                          index += 1
                          break
                  elif item not in {";", "&&", "||", "&", "|"}:
                      values.append(item)
                  index += 1
              if depth != 0:
                  raise SystemExit(f"{path}: unterminated array assignment")
              if append:
                  arrays.setdefault(name, []).extend(values)
              else:
                  arrays[name] = values
              continue
          if token in {";", "&&", "||", "&", "|"}:
              if current:
                  commands.append(current)
                  current = []
              index += 1
              continue
          current.append(token)
          index += 1
      if current:
          commands.append(current)

      materialized = []
      for command in commands:
          argv = []
          for token in command:
              match = re.fullmatch(r"\$\{([A-Za-z_][A-Za-z0-9_]*)\[@\]\}", token)
              if match:
                  if match.group(1) not in arrays:
                      raise SystemExit(f"{path}: invoked undefined array {match.group(1)}")
                  argv.extend(arrays[match.group(1)])
              else:
                  argv.append(token)
          materialized.append(argv)
      return materialized

  def is_smoke_execute(argv):
      has_smoke = any(token.rsplit("/", 1)[-1] == "observability-smoke.sh" for token in argv)
      return has_smoke and "--execute" in argv

  arrays = {}
  smoke_executes = []
  canonical_blocks = []
  phase_executes = {name: [] for name in phase_specs}
  phase_blocks = {name: [] for name in phase_specs}
  for block_index, block in enumerate(blocks):
      completed = subprocess.run(
          ["/usr/bin/bash", "-n"], input=block, text=True,
          stdout=subprocess.DEVNULL, stderr=subprocess.PIPE, check=False,
      )
      if completed.returncode != 0:
          raise SystemExit(f"{path}: fenced Bash block {block_index + 1} is not syntax-valid")
      binding_counts = (
          len(re.findall(r"(?m)^\s*TASK7_SMOKE=\(/usr/bin/bash\)\s*$", block)),
          len(re.findall(
              rf"(?m)^\s*TASK7_SMOKE\+=\({re.escape(smoke_path)}\)\s*$", block,
          )),
          len(re.findall(r"(?m)^\s*TASK7_SMOKE\+=\(--execute\)\s*$", block)),
          len(re.findall(r'''(?m)^\s*"\$\{TASK7_SMOKE\[@\]\}"\s*$''', block)),
      )
      if any(binding_counts):
          if binding_counts != (1, 1, 1, 1):
              raise SystemExit(f"{path}: malformed canonical smoke block")
          canonical_blocks.append(block_index)
      for name, phase_path in phase_specs.items():
          phase_counts = (
              len(re.findall(rf"(?m)^\s*{name}=\(/usr/bin/bash\)\s*$", block)),
              len(re.findall(
                  rf"(?m)^\s*{name}\+=\({re.escape(phase_path)}\)\s*$", block,
              )),
              len(re.findall(
                  rf'''(?m)^\s*{re.escape(helm_assignment)} "\$\{{{name}\[@\]\}}"\s*\n'''
                  rf'''\s*{name}_RC=\$\?\s*$''', block,
              )),
          )
          if any(phase_counts):
              if phase_counts != (1, 1, 1):
                  raise SystemExit(f"{path}: malformed canonical {name} block")
              phase_blocks[name].append(block_index)
      for argv in materialize_commands(block, arrays):
          if is_smoke_execute(argv):
              smoke_executes.append((block_index, argv))
          for name, phase_path in phase_specs.items():
              if any(token.rsplit("/", 1)[-1] == phase_path.rsplit("/", 1)[-1] for token in argv):
                  phase_executes[name].append((block_index, argv))

  canonical_argv = ["/usr/bin/bash", smoke_path, "--execute"]
  if len(canonical_blocks) != 1:
      raise SystemExit(f"{path}: expected one canonical smoke block")
  if smoke_executes != [(canonical_blocks[0], canonical_argv)]:
      raise SystemExit(f"{path}: non-canonical or duplicate smoke execute argv")
  for name, phase_path in phase_specs.items():
      if len(phase_blocks[name]) != 1:
          raise SystemExit(f"{path}: expected one canonical {name} block")
      expected = [helm_assignment, "/usr/bin/bash", phase_path]
      if phase_executes[name] != [(phase_blocks[name][0], expected)]:
          raise SystemExit(f"{path}: non-canonical or duplicate {name} argv")
  PY
    TASK7_ONE_DOC_RC=$?
    printf 'TASK7_DOC_SMOKE_ASSERT_RC=%d DOC=%s\n' "$TASK7_ONE_DOC_RC" "$TASK7_DOC"
    if (( TASK7_ONE_DOC_RC != 0 )); then
      TASK7_DOC_ASSERT_RC=1
    fi
  done
  (( TASK7_DOC_ASSERT_ERREXIT == 0 )) || set -e
  (( TASK7_DOC_ASSERT_RC == 0 ))
  ```

  This requires one absolute smoke argv binding, one exact `--execute`, and one array invocation in the same
  syntax-valid fenced Bash block in each document. The parser materializes every array name and every direct
  command argv across all fenced Bash blocks, so path spelling or wrapper choice cannot bypass the count.
  Test the checker itself with fixtures: canonical execute plus a separate no-arg dry-run must pass; an
  additional relative `bash scripts/validate/observability-smoke.sh --execute`, alternate array name,
  `/usr/bin/env` wrapper, `timeout` wrapper, or line-continued env+timeout wrapper must each fail. Run every
  failing fixture once in the first document position and once in the second to prove per-document RC
  masking is impossible. A duplicate in one document cannot compensate for a missing command in the other
  because every per-document RC contributes to the final nonzero gate. Confirm all Task 7 checkboxes remain
  unchecked and Task 8 remains unstarted.
  The same parser must require one absolute two-element `TASK7_PHASE1` array and one absolute two-element
  `TASK7_PHASE2` array, each invoked exactly once with the exact inline
  `PLATFORM_HELM_BIN=/home/donghyeon/.local/bin/helm` assignment and immediate RC capture in its canonical
  block. Test four private document mutations: remove the assignment from phase1, remove it from phase2,
  replace the phase1 value, and replace the phase2 value. Put each mutation once in the central position and
  once in the manual position while the other document stays canonical; all eight final gates must return
  nonzero, and the unmodified central/manual pair must return zero. Remove only identity-checked private
  fixtures.
  Record per-document structural RCs, hashes, and metadata.

---

### Task 5: Run bounded verification and create the Task 7 non-Git source freeze

**Files:**
- Create: `.superpowers/sdd/2026-08-15-observability-task7-cutover-correction/verify-task7-source-manifest.py`
- Create: `.superpowers/sdd/2026-08-15-observability-task7-cutover-correction/test-verify-task7-source-manifest.py`
- Create: `.superpowers/sdd/2026-08-15-observability-task7-cutover-correction/baseline/task-7-prelive-source-manifest.txt`
- Create: `.superpowers/sdd/2026-08-15-observability-task7-cutover-correction/baseline/task-7-prelive-source-manifest.sha256`
- Create: `.superpowers/sdd/2026-08-15-observability-task7-cutover-correction/baseline/task-7-command-inventory.txt`
- Create: `.superpowers/sdd/2026-08-15-observability-task7-cutover-correction/task-5-report.md`

**Interfaces:**
- Consumes: corrected source/tests/docs and unchanged OIDC/smoke/renderer production.
- Produces: reviewed, sidecar-bound pre-live authority for exactly one Task 7 execution.

- [ ] **Step 1: Run every focused source suite under bounded supervisors**

  Run syntax first, then:

  ```bash
  /usr/bin/timeout --signal=TERM --kill-after=5s 360s \
    /usr/bin/bash /home/donghyeon/workspace/platform/scripts/validate/test-apply-host-nginx-observability.sh
  /usr/bin/timeout --signal=TERM --kill-after=5s 300s \
    /usr/bin/bash /home/donghyeon/workspace/platform/scripts/validate/test-render-observability-access.sh
  /usr/bin/timeout --signal=TERM --kill-after=5s 360s \
    /usr/bin/bash /home/donghyeon/workspace/platform/scripts/validate/test-configure-keycloak-grafana-oidc.sh
  /usr/bin/timeout --signal=TERM --kill-after=5s 360s \
    /usr/bin/bash /home/donghyeon/workspace/platform/scripts/validate/test-observability-smoke.sh
  /usr/bin/bash /home/donghyeon/workspace/platform/scripts/validate/test-host-nginx-observability-source.sh
  ```

  Require every RC `0`, exact suite PASS markers, and zero suite-owned process/temp residue. Do not run live execute modes.

- [ ] **Step 2: Run scanner-focused tests and one authorized full source scan**

  Run the scanner test first. Then run one full scan into a fresh private mode-0700 root with tracing disabled and an immediate RC file. Require exactly one rendered-secret PASS and one sensitive-source PASS, no payload output, and zero new scanner/renderer process or temp residue. Preserve the scan root through review; do not run another full scan in Task 7 unless source changes again.

- [ ] **Step 3: Implement a private strict Task 7 manifest verifier with TDD**

  Adapt the hardened descriptor-retaining algorithm from the existing Task 6 verifier into the new private path without modifying Task 6 authority. The prelive invocation has exactly this argv contract:

  ```text
  argv[0] = /usr/bin/python3
  argv[1] = /home/donghyeon/workspace/platform/.superpowers/sdd/2026-08-15-observability-task7-cutover-correction/verify-task7-source-manifest.py
  argv[2] = literal 64-character lowercase manifest SHA-256 recorded by independent review in task-5-report
  argv[3] = /home/donghyeon/workspace/platform/.superpowers/sdd/2026-08-15-observability-task7-cutover-correction/baseline/task-7-prelive-source-manifest.txt
  argv[4] = /home/donghyeon/workspace/platform/.superpowers/sdd/2026-08-15-observability-task7-cutover-correction/baseline/task-7-prelive-source-manifest.sha256
  ```

  The interpreter, verifier, manifest, and sidecar paths above are literal. The final pair substitutes only
  the two exact `task-7-final-source-manifest` absolute paths and the separately reviewed final digest.
  No argument may come from `pwd`, `readlink`, current file contents, or a shell-computed digest.

  It must enforce: allowlisted prelive/final pair, exact row order and count `21`, canonical absolute paths, unique paths, exact `sha256|mode4|uid:gid|nlink|regular file|size|path` grammar, mode policy, UID:GID `1000:1000`, nlink `1`, retained-descriptor hash/metadata/rebind checks, exact one-line sidecar binding, and no manifest/sidecar/verifier path in rows.

  The verifier is intentionally outside the manifest. Before invocation, independently review and pin its
  literal canonical absolute path, lowercase SHA-256, type, UID:GID, mode, nlink, and size. The launcher
  opens verifier, manifest, and sidecar descriptors before validation, checks descriptor metadata/hash and
  path binding before invocation, invokes only the literal `/usr/bin/python3` + absolute verifier + three
  literal reviewed arguments, and checks the same descriptor/path identities again immediately afterward.
  Do not substitute a PATH-resolved interpreter, relative script, calculated current manifest SHA, or
  self-referential manifest row.

  Write RED tests for wrong pair/SHA/order/count/path/mode/UID/link/type/size/hash, symlink or hardlink manifest/sidecar/row, FIFO, duplicate path, and cross-paired sidecar. Add separate late
  replacement/path-rebind cases for verifier, manifest, and sidecar between precheck/invocation/postcheck;
  each must be caught by retained descriptor/path identity. The test launcher itself must use literal
  `/usr/bin/python3` and absolute verifier/manifest/sidecar arguments. Implement until all tests pass.

- [ ] **Step 4: Freeze the command/launcher inventory and exact 21-row pre-live manifest**

  First create a mode-0600 payload-free `task-7-command-inventory.txt` with schema
  `platform-observability-task7-command-inventory-v1`, the exact production PATH, and one sorted
  `command|name|lexical_path|canonical_path|uid:gid|mode|sha256` row for every Host allowlist command plus
  Nginx, systemd, sudo, and Certbot: exactly `31` command rows (`27` allowlist + `4` fixed executables).
  The Host `test` row must be exact lexical/canonical `/usr/bin/test`; every Host sudo probe is an absolute
  `/usr/bin/test` argv element, never bare `test`. Add safe launcher rows for Host `argc=5`, OIDC `argc=7` with only
  `<admin-stdin>`/`<viewer-stdin>` placeholders, smoke `argc=3`, and complete renderer `argc=6`. Require
  root-owned, non-writable executable targets, safe PATH lineage, exact cardinality, no duplicate/missing row,
  and no username/Secret value. Add a verifier launcher row for literal `/usr/bin/python3`, the absolute
  verifier path and expected metadata, fixed digest sentinel `<reviewed-manifest-sha256>`, and the absolute
  manifest/sidecar pair (`argc=5`, counting the interpreter as argv[0]). The sentinel must occur exactly once
  in that row; the prelive/final manifest's own 64-character lowercase digest may not occupy that argv slot
  or be copied into another manifest-row file. Other independently required file/executable SHA metadata is
  unaffected. Generate a private comparison independently, require `cmp -s`, then remove only that
  comparison. Repeat the comparison immediately before live Host execution.

  This split is mandatory because `task-7-command-inventory.txt` is manifest row 21. Only after all 21 rows
  and the sidecar exist may independent review write the actual lowercase manifest digest to the non-governed
  `task-5-report`/review handoff. The launcher reads no digest from the governed inventory; the controller
  supplies the report's reviewed literal directly as verifier argv[2]. Add tests that replace the governed
  verifier row's sentinel slot with a 64-character manifest digest and require rejection, require sentinel
  exact-once plus exact argc/path/metadata shape, and refuse verifier invocation until the external report
  pin exists and matches the reviewed value.

  The command inventory validator uses the same non-widening metadata policy as Task 2: exact canonical
  `/usr/bin/sudo` is the only row permitted and required to carry owner-setuid and must be exact
  `regular|0:0|4755|1`; every other executable row forbids both setuid and setgid as well as group/world
  write. A private comparison with any sudo or non-sudo bit/path/owner/link drift must fail.

  Use these paths, once each and in this order:

  ```text
  /home/donghyeon/workspace/platform/scripts/bootstrap/apply-host-nginx-observability.sh
  /home/donghyeon/workspace/platform/scripts/validate/test-apply-host-nginx-observability.sh
  /home/donghyeon/workspace/platform/scripts/validate/test-host-nginx-observability-source.sh
  /home/donghyeon/workspace/platform/infrastructure/networking/host-nginx/learn-services-admin.conf
  /home/donghyeon/workspace/platform/infrastructure/networking/host-nginx/learn-services-metrics-guard.conf
  /home/donghyeon/workspace/platform/infrastructure/networking/host-nginx/learn-services-grafana-deny-guard.conf
  /home/donghyeon/workspace/platform/infrastructure/networking/host-nginx/learn-services-observability.conf
  /home/donghyeon/workspace/platform/scripts/bootstrap/apply-observability-access.sh
  /home/donghyeon/workspace/platform/scripts/validate/test-apply-observability-access.sh
  /home/donghyeon/workspace/platform/scripts/validate/render-observability-access.sh
  /home/donghyeon/workspace/platform/scripts/validate/test-render-observability-access.sh
  /home/donghyeon/workspace/platform/scripts/bootstrap/configure-keycloak-grafana-oidc.sh
  /home/donghyeon/workspace/platform/scripts/validate/test-configure-keycloak-grafana-oidc.sh
  /home/donghyeon/workspace/platform/scripts/validate/observability-smoke.sh
  /home/donghyeon/workspace/platform/scripts/validate/test-observability-smoke.sh
  /home/donghyeon/workspace/platform/bootstrap/manual/phase4-observability-access.md
  /home/donghyeon/workspace/docs/platform/plans/2026-08-14-observability-slack-risk-acceptance-implementation.md
  /home/donghyeon/workspace/docs/platform/plans/2026-08-15-observability-task7-cutover-correction-design.md
  /home/donghyeon/workspace/platform/docs/superpowers/plans/2026-08-15-observability-task7-cutover-correction.md
  /home/donghyeon/workspace/docs/platform/runbooks/2026-08-13-observability-phase4-resume-worklog.md
  /home/donghyeon/workspace/platform/.superpowers/sdd/2026-08-15-observability-task7-cutover-correction/baseline/task-7-command-inventory.txt
  ```

  The verifier, manifest, and sidecar themselves are not rows. Create the sidecar with the literal canonical
  manifest path, pin the verifier separately as specified in Step 3, and run the descriptor-retaining strict
  launcher twice with the externally reviewed lowercase manifest SHA and literal absolute paths. Require RC
  `0` both times and unchanged verifier/manifest/sidecar descriptor/path identities after each invocation.

- [ ] **Step 5: Independent pre-live review**

  Review spec coverage, C/I/M, v2 tuple matrix, first-cutover behavior, fixed PATH and command/launcher
  residue rejection, exact 21-row inventory, self-reference-free sentinel row plus external digest pin,
  separately pinned verifier identity and rebind tests, both
  correction authority documents, RED provenance, mutation evidence, original handoff
  preservation, corrected OIDC/smoke sequencing, scanner result, and manifest/sidecar identity. Required
  verdict: Critical `0`, Important `0`, Spec YES, Quality Approved, Ready for Host Task 7 YES. Any other
  verdict returns to the responsible source task and requires a new scan/freeze after correction.

---

### Task 6: Execute the single Host Nginx full cutover

**Files:**
- Preserve: `/var/lib/hyeonworks/platform-rollbacks/observability-20260814T145009Z`
- Preserve: `/tmp/platform-observability-metrics.dw5gLZ`
- Append after outcome: `.superpowers/sdd/2026-08-15-observability-task7-cutover-correction/task-6-report.md`

**Interfaces:**
- Consumes: Task 5 exact reviewed freeze, same Task 6 ID/root/handoff, current source proof, active deny-only config.
- Produces: full Host Nginx PASS or a preserved fail-closed transaction that is never retried under this ID.

- [ ] **Step 1: Bind the successful Task 6 identity in the existing operator shell**

  Before any external preparation command, install the reviewed operator boundary and invalidate Bash's
  command cache:

  ```bash
  TASK7_OPERATOR_PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
  PATH=$TASK7_OPERATOR_PATH
  export PATH
  hash -r
  ```

  Using the exact Task 5 generation procedure and only commands resolved under that boundary, regenerate the
  command/launcher inventory into a fresh private comparison and require byte equality with the reviewed
  inventory. Require every lexical/canonical command row to match before continuing. If this comparison
  cannot be completed first, stop; do not run an ambient-PATH prep command. Then run short assignments only:

  ```bash
  cd /home/donghyeon/workspace/platform
  TASK7_ID=20260814T145009Z
  TASK7_METRIC_ROOT=/tmp/platform-observability-metrics.dw5gLZ
  export PLATFORM_OBSERVABILITY_ROLLBACK_ID="$TASK7_ID"
  [[ "$PLATFORM_OBSERVABILITY_ROLLBACK_ID" == "$TASK7_ID" ]]
  [[ "$(/usr/bin/readlink -f -- "$TASK7_METRIC_ROOT")" == "$TASK7_METRIC_ROOT" ]]
  [[ "$(/usr/bin/stat -c '%F|%u:%g|%a|%h' -- "$TASK7_METRIC_ROOT")" == 'directory|1000:1000|700|4' ]]
  ```

  Use literal `/usr/bin/find`, `/usr/bin/sort`, `/usr/bin/stat`, and `/usr/bin/sha256sum` for the remaining
  preparation checks. Require the root name-only topology to be exactly two phase directories plus
  `dashboards.yaml`, `rules.yaml`, and `alertmanager.yaml`. Verify both known inventory hashes and metadata
  without printing inventory bodies.

- [ ] **Step 2: Revalidate the frozen source and no-contact dry-run**

  Run the Task 5 descriptor-retaining strict launcher twice, exactly as independently reviewed: literal
  `/usr/bin/python3`, literal absolute verifier path, literal reviewed 64-character manifest SHA, and literal
  absolute prelive manifest/sidecar paths. The SHA must come from the non-governed task-5 review handoff,
  never from the command-inventory sentinel or another manifest-row file. Require RC `0` and unchanged separately pinned verifier
  SHA/type/owner/mode/nlink/size/canonical-path identity before and after both invocations.

  Then build `TASK7_HOST_DRY=(/usr/bin/bash
  /home/donghyeon/workspace/platform/scripts/bootstrap/apply-host-nginx-observability.sh)` and invoke it once
  under the same errexit save/`set +e`/immediate RC/restore envelope used by execute. Require:

  ```text
  HOST_NGINX_ACTIVE_SHA256=dbef6d443bcba58b26a5351ea76f6d09f6da8c2ef07a806e22745cf26c88f518
  HOST_NGINX_GRAFANA_DENY_GUARD_SHA256=dbef6d443bcba58b26a5351ea76f6d09f6da8c2ef07a806e22745cf26c88f518
  HOST_NGINX_FULL_SHA256=7d2de2a92c3597a0859775da1d2ccbf5a3d72c0b2af5cac2439c82311361f801
  HOST_NGINX_CERTIFICATE_EXPECTED_SAN=grafana.learn.hyeonworks.com
  HOST_NGINX_CERTIFICATE_SAN=NOT_CHECKED_DRY_RUN
  HOST_NGINX_GRAFANA_PUBLIC_DNS=NOT_CHECKED_DRY_RUN
  HOST_NGINX_OBSERVABILITY_DRY_RUN=PASS
  TASK7_HOST_DRY_RC=0
  ```

  Also require legacy dry-run claims `HOST_NGINX_CERTIFICATE_EXACT_SAN=` and
  `HOST_NGINX_GRAFANA_PUBLIC_DNS=ABSENT` to be absent. Any different/unreadable/already-full active hash stops
  before sudo or execute.

- [ ] **Step 3: Refresh sudo and build a paste-safe argv array**

  Run `/usr/bin/sudo -v`, then require `/usr/bin/sudo -n /usr/bin/true` and print only `SUDO_READY`.

  Build the command without placing the long option in pasted text:

  ```bash
  TASK7_VOD=--verified
  TASK7_VOD+=-output-dir
  TASK7_HOST=(/usr/bin/bash)
  TASK7_HOST+=(/home/donghyeon/workspace/platform/scripts/bootstrap/apply-host-nginx-observability.sh)
  TASK7_HOST+=(--execute)
  TASK7_HOST+=("$TASK7_VOD")
  TASK7_HOST+=("$TASK7_METRIC_ROOT")
  [[ "${#TASK7_HOST[@]}" -eq 5 ]]
  printf 'TASK7_HOST_ARGC=%d\n' "${#TASK7_HOST[@]}"
  ```

  Do not print `%q` for usernames or sensitive data; this Host argv contains only safe paths/options.

- [ ] **Step 4: Execute exactly once and capture immediate RC**

  ```bash
  case $- in *e*) TASK7_HOST_ERREXIT=1 ;; *) TASK7_HOST_ERREXIT=0 ;; esac
  set +e
  "${TASK7_HOST[@]}"
  TASK7_HOST_RC=$?
  printf 'TASK7_HOST_RC=%d\n' "$TASK7_HOST_RC"
  (( TASK7_HOST_ERREXIT == 0 )) || set -e
  ```

  Before entering anything, require exact pre-prompt stage and previous/candidate hashes. Only the operator enters `APPLY`.

  The prompt itself is admissible only after no-follow absence checks have proved all four paths absent under
  both `test -e` and `test -L`:

  ```text
  /var/lib/hyeonworks/platform-rollbacks/observability-20260814T145009Z/host-nginx
  /var/lib/hyeonworks/platform-rollbacks/observability-20260814T145009Z/host-nginx/stages.tsv
  /var/lib/hyeonworks/platform-rollbacks/observability-20260814T145009Z/host-nginx/payloads
  /var/lib/hyeonworks/platform-rollbacks/observability-20260814T145009Z/host-nginx/payloads/full-prior-0001.conf
  ```

  An existing normal path or symlink is a pre-prompt STOP, not resumable ledger state.

  Success requires exact `HOST_NGINX_FULL_STAGE=PASS` and RC `0`. `ALREADY_ACTIVE`, missing PASS, nonzero or ambiguous RC is failure.

- [ ] **Step 5: Classify failure without retry**

  - Pre-prompt failure: active deny unchanged, Host ledger absent, rollback N/A.
  - Post-prompt/pre-arm failure: active deny and reload count unchanged; root-owned ledger/payload or timestamp
    backup may remain and must be preserved; rollback N/A.
  - Armed failure: require `HOST_NGINX_OBSERVABILITY_ROLLBACK=PASS` and active deny hash restored.
  - `ROLLBACK=FAIL`, `MANUAL_RECOVERY_REQUIRED=YES`, unknown active hash, response loss, or missing evidence: STOP for manual recovery.

  If any residue is present before prompt, or any ledger directory/file, payload, or timestamp-backup staging
  step fails, preserve the entire root and start a new Task 6 design/execution with a fresh ID. The same route
  applies when source proof is stale/future. Do not alter this root or proof, recreate only one object, or
  invoke the Host command again under `20260814T145009Z`.

- [ ] **Step 6: Record the Host terminal result**

  Record safe argv cardinality, dry/execute RC, exact stage/hashes, whether APPLY was entered, Nginx active/test/reload outcome, three stable LAN/local postchecks, rollback invocation/outcome, original Task 6 evidence metadata unchanged, and process/temp residue delta. Do not mark later Task 7 steps yet.

---

### Task 7: Reconcile memberships and run the monolithic acceptance once

**Files:**
- Append: `.superpowers/sdd/2026-08-15-observability-task7-cutover-correction/task-7-report.md`
- Preserve unchanged: `scripts/bootstrap/configure-keycloak-grafana-oidc.sh`
- Preserve unchanged: `scripts/validate/observability-smoke.sh`

**Interfaces:**
- Consumes: Host full PASS, prepared human identities, Slack view, and a real external client.
- Produces: OIDC transaction PASS plus one complete machine/OIDC/Slack/external smoke PASS.

- [ ] **Step 1: Prove human/external readiness before mutation**

  Prepare four distinct realm usernames for admin, viewer, no-group, and membership-removal testing; local break-glass access; a Slack channel view; and an outside-LAN/Tailscale client with proxy disabled. If any is unavailable, stop without running membership or smoke.

- [ ] **Step 2: Build and execute the OIDC membership argv once**

  ```bash
  read -r -p 'Grafana organization admin realm username: ' OBS_ADMIN_USER
  read -r -p 'Grafana viewer realm username: ' OBS_VIEWER_USER
  TASK7_OIDC=(/usr/bin/bash /home/donghyeon/workspace/platform/scripts/bootstrap/configure-keycloak-grafana-oidc.sh)
  TASK7_OIDC+=(--execute)
  TASK7_OIDC+=(--admin "$OBS_ADMIN_USER")
  TASK7_OIDC+=(--viewer "$OBS_VIEWER_USER")
  case $- in *e*) TASK7_OIDC_ERREXIT=1 ;; *) TASK7_OIDC_ERREXIT=0 ;; esac
  set +e
  "${TASK7_OIDC[@]}"
  TASK7_OIDC_RC=$?
  unset OBS_ADMIN_USER OBS_VIEWER_USER TASK7_OIDC
  printf 'TASK7_OIDC_RC=%d\n' "$TASK7_OIDC_RC"
  (( TASK7_OIDC_ERREXIT == 0 )) || set -e
  ```

  Enter exact `APPLY default` and `RECOVERY KEYCLOAK default`. Require `GRAFANA_OIDC_TRANSACTION=PASS` and
  RC `0`. A failure before `transaction_active=true` has no managed Keycloak/OIDC Secret/membership mutation
  and rollback marker N/A. An active transaction failure requires `GRAFANA_OIDC_ROLLBACK=PASS`; rollback
  FAIL/manual recovery or unknown stage stops. Do not claim or search for a persistent Task 6 Keycloak ledger.

- [ ] **Step 3: Run the smoke no-arg plan once before execute**

  Build an absolute two-element argv array and use the errexit save/`set +e`/immediate RC/restore envelope:

  ```bash
  TASK7_SMOKE_DRY=(/usr/bin/bash)
  TASK7_SMOKE_DRY+=(/home/donghyeon/workspace/platform/scripts/validate/observability-smoke.sh)
  ```

  Invoke it exactly once. Require no-arg RC `0`, exact `OBSERVABILITY_SMOKE_DRY_RUN=PASS`,
  `HUMAN_EXTERNAL_CLIENT=required`, and `MUTATION=NOT_REQUESTED`. This is not acceptance and does not replace
  execute.

- [ ] **Step 4: Execute the smoke exactly once with immediate RC**

  ```bash
  TASK7_SMOKE=(/usr/bin/bash)
  TASK7_SMOKE+=(/home/donghyeon/workspace/platform/scripts/validate/observability-smoke.sh)
  TASK7_SMOKE+=(--execute)
  case $- in *e*) TASK7_SMOKE_ERREXIT=1 ;; *) TASK7_SMOKE_ERREXIT=0 ;; esac
  set +e
  "${TASK7_SMOKE[@]}"
  TASK7_SMOKE_RC=$?
  printf 'TASK7_SMOKE_RC=%d\n' "$TASK7_SMOKE_RC"
  (( TASK7_SMOKE_ERREXIT == 0 )) || set -e
  ```

  Only the operator supplies the requested identities and exact confirmations. Confirm actual admin/viewer/denied/break-glass/session-revoke/relogin behavior, Slack firing then resolved message, and run the script-rendered `--resolve` command from the true external client before entering its exact dynamic token.

- [ ] **Step 5: Require the full acceptance marker set**

  Success requires all:

  ```text
  OBSERVABILITY_MACHINE_ACCEPTANCE=PASS
  OBSERVABILITY_OIDC_ACCEPTANCE=PASS
  OBSERVABILITY_SLACK_ACCEPTANCE=PASS
  OBSERVABILITY_EXTERNAL_BOUNDARY=PASS
  OBSERVABILITY_SMOKE=PASS
  TASK7_SMOKE_RC=0
  ```

  Machine evidence must include LAN and Tailscale Grafana success, denied local/public boundary, external `403`, public metrics `404`, unknown SNI rejection, LAN NodePorts refused, loopback NodePort exact Traefik identity, exact `30/30` targets, datasource/trace link, persistence, network-policy denial, and Certbot/Nginx health.

  RC `2` or `OBSERVABILITY_EXTERNAL_BOUNDARY=PARTIAL` leaves Task 7 incomplete. Any cleanup ambiguity/manual recovery stops and forbids automatic rerun; review exact owned-object cleanup first.

- [ ] **Step 6: Record payload-free OIDC/smoke evidence**

  Record roles and PASS/FAIL only, never identity values or browser/session data. Record temporary object kinds and cleanup disposition, not payloads. Verify Task 6 root/handoff metadata unchanged and no smoke-owned process/root remains.

---

### Task 8: Run complete rendering from a fresh clone and existing regressions once

**Files:**
- Create: one fresh `/tmp/platform-observability-metrics.XXXXXX` renderer root
- Preserve: `/tmp/platform-observability-metrics.dw5gLZ`
- Append: `.superpowers/sdd/2026-08-15-observability-task7-cutover-correction/task-8-report.md`

**Interfaces:**
- Consumes: Task 7 final acceptance and byte-preserved inventories from the Task 6 handoff.
- Produces: complete renderer and existing platform regression RCs without a second smoke.

- [ ] **Step 1: Fingerprint the original handoff without changing it**

  Reinstall `TASK7_OPERATOR_PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin`, export
  `PATH`, run `hash -r`, and require the independently regenerated command/launcher inventory to remain
  byte-equal before any external preparation command. From this point, the procedure below nevertheless uses
  literal absolute paths for every preparation executable.

  Require exact canonical root, UID:GID `1000:1000`, root mode `0700`, phase modes `0700`, file modes `0600`, nlink `1`, exact root/phase entry sets, and the two known inventory hashes. Record device/inode/type/owner/mode/nlink/size/mtime/ctime/SHA for both inventories and checksums plus the three Task 6 YAML files. Use only literal `/usr/bin/readlink`, `/usr/bin/stat`, `/usr/bin/find`, `/usr/bin/sort`, and `/usr/bin/sha256sum` for these checks.

- [ ] **Step 2: Create the fresh inventory-only root**

  ```bash
  TASK7_RENDER_ROOT="$(/usr/bin/mktemp -d /tmp/platform-observability-metrics.XXXXXX)"
  /usr/bin/chmod 0700 "$TASK7_RENDER_ROOT"
  for phase in target-initial post-substrate; do
    /usr/bin/install -d -m 0700 -- "$TASK7_RENDER_ROOT/$phase"
    for file in inventory.json inventory.sha256; do
      source_file="$TASK7_METRIC_ROOT/$phase/$file"
      destination_file="$TASK7_RENDER_ROOT/$phase/$file"
      [[ -f "$source_file" && ! -L "$source_file" ]]
      /usr/bin/cp --no-dereference --reflink=never -- "$source_file" "$destination_file"
      /usr/bin/chmod 0600 "$destination_file"
      /usr/bin/cmp -s -- "$source_file" "$destination_file"
    done
  done
  unset source_file destination_file
  ```

  Require the exact six-entry topology, current-user ownership, modes, nlinks, byte equality, known hashes, and unchanged original fingerprints. Any failure preserves the new root and stops; do not repair or reuse it.

- [ ] **Step 3: Run core and complete access renderers once**

  Build and invoke the core renderer exactly once with an absolute two-element argv and exact Helm
  environment:

  ```bash
  TASK7_CORE=(/usr/bin/bash)
  TASK7_CORE+=(/home/donghyeon/workspace/platform/scripts/validate/render-observability-core.sh)
  [[ "${#TASK7_CORE[@]}" -eq 2 ]]
  case $- in *e*) TASK7_CORE_ERREXIT=1 ;; *) TASK7_CORE_ERREXIT=0 ;; esac
  set +e
  PLATFORM_HELM_BIN=/home/donghyeon/.local/bin/helm "${TASK7_CORE[@]}"
  TASK7_CORE_RC=$?
  (( TASK7_CORE_ERREXIT == 0 )) || set -e
  printf 'TASK7_CORE_RC=%d\n' "$TASK7_CORE_RC"
  ```

  Require `OBSERVABILITY CORE STATIC RENDER PASS` and `TASK7_CORE_RC=0`. Do not invoke the core renderer in
  Step 4 or through another script.

  Then build and invoke the complete access renderer exactly once:

  ```bash
  TASK7_VOD=--verified
  TASK7_VOD+=-output-dir
  TASK7_COMPLETE=(/usr/bin/bash)
  TASK7_COMPLETE+=(/home/donghyeon/workspace/platform/scripts/validate/render-observability-access.sh)
  TASK7_COMPLETE+=(--component complete)
  TASK7_COMPLETE+=("$TASK7_VOD" "$TASK7_RENDER_ROOT")
  [[ "${#TASK7_COMPLETE[@]}" -eq 6 ]]
  case $- in *e*) TASK7_COMPLETE_ERREXIT=1 ;; *) TASK7_COMPLETE_ERREXIT=0 ;; esac
  set +e
  PLATFORM_HELM_BIN=/home/donghyeon/.local/bin/helm "${TASK7_COMPLETE[@]}"
  TASK7_COMPLETE_RC=$?
  (( TASK7_COMPLETE_ERREXIT == 0 )) || set -e
  printf 'TASK7_COMPLETE_RC=%d\n' "$TASK7_COMPLETE_RC"
  ```

  Require `OBSERVABILITY ACCESS STATIC RENDER PASS` and `TASK7_COMPLETE_RC=0`; verify seven YAML outputs and
  original handoff fingerprints unchanged. Do not invoke the complete renderer again.

- [ ] **Step 4: Run the remaining regressions once each**

  The remaining set has exactly five commands; neither renderer from Step 3 is a member. Build each absolute
  argv and give every invocation its own errexit save/`set +e`/immediate RC/restore envelope:

  ```bash
  TASK7_ADMIN_UI=(/usr/bin/bash)
  TASK7_ADMIN_UI+=(/home/donghyeon/workspace/platform/scripts/validate/admin-ui-smoke.sh)
  [[ "${#TASK7_ADMIN_UI[@]}" -eq 2 ]]
  case $- in *e*) TASK7_ADMIN_UI_ERREXIT=1 ;; *) TASK7_ADMIN_UI_ERREXIT=0 ;; esac
  set +e
  "${TASK7_ADMIN_UI[@]}"
  TASK7_ADMIN_UI_RC=$?
  (( TASK7_ADMIN_UI_ERREXIT == 0 )) || set -e
  printf 'TASK7_ADMIN_UI_RC=%d\n' "$TASK7_ADMIN_UI_RC"

  TASK7_AISTOR_S3=(/usr/bin/bash)
  TASK7_AISTOR_S3+=(/home/donghyeon/workspace/platform/scripts/validate/aistor-s3-smoke.sh)
  TASK7_AISTOR_S3+=(--execute)
  [[ "${#TASK7_AISTOR_S3[@]}" -eq 3 ]]
  case $- in *e*) TASK7_AISTOR_S3_ERREXIT=1 ;; *) TASK7_AISTOR_S3_ERREXIT=0 ;; esac
  set +e
  "${TASK7_AISTOR_S3[@]}"
  TASK7_AISTOR_S3_RC=$?
  (( TASK7_AISTOR_S3_ERREXIT == 0 )) || set -e
  printf 'TASK7_AISTOR_S3_RC=%d\n' "$TASK7_AISTOR_S3_RC"

  TASK7_PHASE1=(/usr/bin/bash)
  TASK7_PHASE1+=(/home/donghyeon/workspace/platform/scripts/validate/render-phase1.sh)
  [[ "${#TASK7_PHASE1[@]}" -eq 2 ]]
  case $- in *e*) TASK7_PHASE1_ERREXIT=1 ;; *) TASK7_PHASE1_ERREXIT=0 ;; esac
  set +e
  PLATFORM_HELM_BIN=/home/donghyeon/.local/bin/helm "${TASK7_PHASE1[@]}"
  TASK7_PHASE1_RC=$?
  (( TASK7_PHASE1_ERREXIT == 0 )) || set -e
  printf 'TASK7_PHASE1_RC=%d\n' "$TASK7_PHASE1_RC"

  TASK7_PHASE2=(/usr/bin/bash)
  TASK7_PHASE2+=(/home/donghyeon/workspace/platform/scripts/validate/render-phase2.sh)
  [[ "${#TASK7_PHASE2[@]}" -eq 2 ]]
  case $- in *e*) TASK7_PHASE2_ERREXIT=1 ;; *) TASK7_PHASE2_ERREXIT=0 ;; esac
  set +e
  PLATFORM_HELM_BIN=/home/donghyeon/.local/bin/helm "${TASK7_PHASE2[@]}"
  TASK7_PHASE2_RC=$?
  (( TASK7_PHASE2_ERREXIT == 0 )) || set -e
  printf 'TASK7_PHASE2_RC=%d\n' "$TASK7_PHASE2_RC"

  TASK7_ADMIN_RENDER=(/usr/bin/bash)
  TASK7_ADMIN_RENDER+=(/home/donghyeon/workspace/platform/scripts/validate/render-admin-services.sh)
  [[ "${#TASK7_ADMIN_RENDER[@]}" -eq 2 ]]
  case $- in *e*) TASK7_ADMIN_RENDER_ERREXIT=1 ;; *) TASK7_ADMIN_RENDER_ERREXIT=0 ;; esac
  set +e
  PLATFORM_HELM_BIN=/home/donghyeon/.local/bin/helm "${TASK7_ADMIN_RENDER[@]}"
  TASK7_ADMIN_RENDER_RC=$?
  (( TASK7_ADMIN_RENDER_ERREXIT == 0 )) || set -e
  printf 'TASK7_ADMIN_RENDER_RC=%d\n' "$TASK7_ADMIN_RENDER_RC"
  ```

  The fixed operator `PATH` excludes `/home/donghyeon/.local/bin`, and both phase renderers fall back to
  `command -v helm` when `PLATFORM_HELM_BIN` is unset. Therefore their exact inline assignments above are
  mandatory and PATH fallback is forbidden. Before execution, run the Task 4 fenced-Bash structural checker
  against both authority documents. Require its current-document RC `0`, then require nonzero RC for four
  identity-checked private mutations that respectively remove the phase1 assignment, remove the phase2
  assignment, replace the phase1 value, or replace the phase2 value. Exercise each mutation in both document
  positions with the other document unchanged; require all eight final gates nonzero. Remove only those
  private copies.

  Do not run nonexistent `observability-core-smoke.sh`. Do not run `observability-smoke.sh --execute` again. Require every RC `0`, expected terminal PASS, and no new residue.

- [ ] **Step 5: Preserve regression evidence through closeout**

  Preserve the original Task 6 handoff, fresh successful Task 7 renderer root, scanner root, rollback root, terminal status/logs, and name-only residue baseline through Task 9 review. Cleanup belongs to the later Task 8 final-closeout plan, not this task.

---

### Task 9: Freeze terminal results, update status truthfully, and review Task 7 completion

**Files:**
- Modify after observed results: `bootstrap/manual/phase4-observability-access.md`
- Modify after observed results: `/home/donghyeon/workspace/docs/platform/plans/2026-08-14-observability-slack-risk-acceptance-implementation.md`
- Append after observed results: `/home/donghyeon/workspace/docs/platform/runbooks/2026-08-13-observability-phase4-resume-worklog.md`
- Create: `.superpowers/sdd/2026-08-15-observability-task7-cutover-correction/baseline/task-7-final-source-manifest.txt`
- Create: `.superpowers/sdd/2026-08-15-observability-task7-cutover-correction/baseline/task-7-final-source-manifest.sha256`
- Create: `.superpowers/sdd/2026-08-15-observability-task7-cutover-correction/task-9-report.md`

**Interfaces:**
- Consumes: exact terminal RCs/markers and preserved pre-live authority.
- Produces: evidence-backed Task 7 completion or an explicit partial/failed state; Task 8 remains separate.

- [ ] **Step 1: Write only observed terminal facts**

  Record Host dry/execute markers and RC, active full hash, rollback invocation state, OIDC transaction/rollback RC, one smoke marker set and RC, external-client result, renderer/platform regression RCs, original handoff preservation, fresh renderer root, temporary-object cleanup, and residue delta. Do not record usernames, external client address beyond the safe public edge already printed by smoke, Secret-derived data, or response bodies.

- [ ] **Step 2: Update Task 7 checkboxes conditionally**

  Mark a Step 1–6 checkbox only when its exact evidence exists. Mark all Task 7 steps complete only if Tasks 6–8 of this plan are RC `0` with every required marker and no unresolved cleanup/recovery. Otherwise leave failed/unrun steps unchecked and state `부분 구현` with the exact blocker. Do not mark any Task 8 checkbox.

- [ ] **Step 3: State the remaining Slack DR exception**

  Even after Task 7 PASS, retain:

  ```text
  Phase 4 operational acceptance: complete with active Slack DR exception
  Slack off-host disaster recovery: deferred / not complete
  ```

  Never convert `RISK_ACCEPTED` into recovery-complete language.

- [ ] **Step 4: Regenerate and strictly verify the final source manifest**

  Regenerate the same exact 21 paths in the same order after terminal documentation changes, bind the final
  sidecar, and independently review the new literal final manifest SHA. Regenerate and compare the command
  inventory first with the same exact-once `<reviewed-manifest-sha256>` sentinel; reject any actual manifest
  digest in its verifier row. Only after the final manifest is complete may the non-governed task-9
  report/review handoff pin its actual digest. Re-pin the still-excluded verifier's canonical absolute path, SHA, type, owner, mode,
  nlink, and size. Run the descriptor-retaining launcher twice using literal `/usr/bin/python3`, the literal
  absolute verifier, the reviewed literal SHA, and the literal absolute final manifest/sidecar pair; require
  RC `0` plus unchanged descriptor/path identities before and after each invocation. Record prelive/final
  manifest and sidecar file hashes separately; do not overwrite or delete the prelive pair.

- [ ] **Step 5: Final independent Task 7 review**

  Verify design/spec coverage, source quality, RED/GREEN/mutation provenance, fixed command boundary, v2 exact tuple, proof age/ID, prompt-before empty Host residue, Host state/rollback classification, OIDC truth, exactly one smoke per token/argv structural document assertion and exactly one live smoke, public/Tailscale/external evidence, `30/30` targets, NodePort boundary, exactly one core plus one complete renderer, exact phase1/phase2 Helm assignments with structural mutation rejection, clone/original preservation, every regression RC, self-reference-free 21-row manifest/sidecar binding and external digest pin, separate verifier pin/rebind defense, mutator/process/temp residue, and accurate docs/check boxes.

  Required terminal verdicts:

  ```text
  Critical=0
  Important=0
  Spec=YES
  Quality=APPROVED
  Task7=COMPLETE
  ReadyToStartTask8=YES
  ```

  Any Critical/Important finding or missing evidence returns to the responsible task; do not claim completion because the day ended or because only server-side checks passed.

---

## Execution Handoff

Plan implementation must use either `superpowers:subagent-driven-development` with a fresh implementer and spec/quality review per task, or `superpowers:executing-plans` with explicit checkpoints. The recommended route is subagent-driven development because Tasks 1–5 are independently reviewable before any live mutation, while Tasks 6–9 require controller/operator coordination.