Files

T

donghyeon-ka 16c337bcc9 Add platform infrastructure configuration

2026-08-28 17:35:41 +09:00

68 KiB

Raw Blame History

Observability Task 7 Cutover Correction Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Correct the fail-closed Host Nginx/Task 6 evidence interface, preserve the successful Task 6 authority, and complete Task 7 through one Host cutover, one OIDC membership transaction, one monolithic smoke, and one bounded regression pass.

Architecture: Change only the Host Nginx production script and its focused test for the v2 evidence, first-cutover, command-boundary, and no-contact defects. Keep OIDC, smoke, and renderer production behavior unchanged; correct their orchestration in the central/manual documents, use a fresh inventory-only renderer root, then freeze exact non-Git identities before live execution.

Tech Stack: Bash 5.2, Python 3.12, Nginx/systemd, Kubernetes/k3s/kubectl, Keycloak Admin API, Prometheus/Alertmanager/Grafana, hardened private /tmp handoffs, SHA-256 non-Git source manifests.

Global Constraints

Do not read, print, hash, copy, or disclose Kubernetes Secret values, Slack webhook data, OAuth codes/state, passwords, cookies, bearer tokens, or private recovery payloads.
Preserve rollback ID 20260814T145009Z and /tmp/platform-observability-metrics.dw5gLZ through Task 7; never use that handoff as complete-render publication output.
Preserve failed IDs 20260814T080303Z, 20260814T140953Z, /tmp/platform-observability-metrics.LNzksC, and all historical evidence without mutation or deletion.
The accepted inventory pins remain 79688d017d38eec9a6f100f8d0f784a5474e79802046ef1c2c11b30d170b0b0c and b1c3049206a1a88165ee672ae9aceac7945673a3bb9c3cf3670b7f0d56c3f291.
The exact Nginx deny/full pins remain dbef6d443bcba58b26a5351ea76f6d09f6da8c2ef07a806e22745cf26c88f518 and 7d2de2a92c3597a0859775da1d2ccbf5a3d72c0b2af5cac2439c82311361f801.
Full cutover accepts only platform-observability-rules-alerts-v2; legacy v1 has no fallback.
First full cutover requires exact deny-only active state. ALREADY_ACTIVE is a STOP condition, not success.
Before the first full-cutover live preflight or APPLY prompt, the host-nginx subtree, ledger, payload directory, and first payload path must each be absent under both existence and symlink tests. Normal existing files/directories are residue and fail closed just like symlinks.
A proof older than 24 hours requires a fresh rollback ID, fresh proof, and fresh Task 6 acceptance. Never recreate only the proof or retry Host under the frozen ID.
Run observability-smoke.sh --execute exactly once, after membership reconciliation and external-client readiness. RC 2 remains partial implementation.
OIDC rollback is in-process automatic rollback; do not claim a persistent Keycloak reversal ledger under the Task 6 root.
Every live command gets immediate RC capture. Interpret rollback by stage: pre-transaction/pre-active-config failures have rollback N/A; active transaction failures require exact rollback PASS; ambiguity/manual recovery stops all later tasks.
Any Host ledger/payload/timestamp staging failure consumes the rollback ID for Host purposes: preserve it, create a fresh ID, and repeat source proof plus complete Task 6. Never repair staging or retry Host in place.
Before any operator-shell preparation command, either install the exact production PATH, export it, run hash -r, and verify the reviewed command inventory, or use literal absolute paths for every preparation executable. Do not mix an ambient-PATH setup command into a fixed-boundary procedure.
Do not mark existing Task 7 checkboxes until the corresponding live result is observed and independently reviewed. Task 8 remains unstarted.
This workspace is non-Git. Do not commit, create a branch, or claim worktree cleanliness; record exact file metadata and SHA-256 instead. After freeze this implementation plan is immutable authority; track progress only in reports.

Task 1: Make the Host acceptance fixture production-faithful

Files:

Modify: scripts/validate/test-apply-host-nginx-observability.sh:102-143
Modify: scripts/bootstrap/apply-host-nginx-observability.sh:34,372-423,517-612
Create: .superpowers/sdd/2026-08-15-observability-task7-cutover-correction/task-1-report.md

Interfaces:

Consumes: Task 6 access-rules-alerts/acceptance.env, the two metric inventory hashes, and the same rollback ID.
Produces: validate_full_evidence() acceptance only for an exact nine-field v2 record and one of two exact Slack gate tuples.
Step 1: Freeze the starting identities

Record SHA-256, mode, UID:GID, nlink, type, and size for the two files. Required starting hashes are:
```
apply = 518119a5e11fe8462c5d41921534351f78a2dd721c1bf69ac638e5bffe83536c
test  = 1c429f7eb2cb73ec25dacd9fe4fdc5e063f62866c6e36732a3e9116fa68b2f53
```
If either hash differs, stop for source-drift review before editing.

Step 2: Change the focused fixture to v2 and add a legacy-v1 rejection test

Replace the acceptance writer with an exact helper whose default is the successful Task 6 risk tuple:

write_acceptance_v2() {
  local path=$1 initial_sha=$2 post_sha=$3
  local gate=${4:-RISK_ACCEPTED} ref uid
  case "$gate" in
    RECOVERY)
      ref=strict-recovery-evidence-v1
      uid=not-applicable
      ;;
    RISK_ACCEPTED)
      ref=2026-08-14-observability-slack-recovery-risk-acceptance-design
      uid=1000
      ;;
    *) return 2 ;;
  esac
  cat >"$path" <<EOF
schema=platform-observability-rules-alerts-v2
rollback_id=$ROLLBACK_ID
target_initial_sha256=$initial_sha
post_substrate_sha256=$post_sha
slack_deployment_gate=$gate
slack_gate_approval_ref=$ref
slack_gate_accepted_by_uid=$uid
accepted_at_utc=$(date -u +%Y-%m-%dT%H:%M:%SZ)
state=accepted
EOF
  chmod 0600 "$path"
}

Add a fixture that removes the three Slack fields and changes schema to platform-observability-rules-alerts-v1. It must expect nonzero RC, exact unchanged deny active bytes, no preflight/commit command, no host-nginx ledger, and no APPLY prompt.

Step 3: Run RED before changing production

Run:
```
bash -n scripts/validate/test-apply-host-nginx-observability.sh
timeout --signal=TERM --kill-after=5s 240s \
  bash scripts/validate/test-apply-host-nginx-observability.sh
```
Expected: syntax RC 0; suite RC nonzero because the default v2 success fixture is rejected by the current v1 parser and/or legacy v1 is still accepted. Record the exact first failing assertion and prove the active fixture remained deny-only.

Step 4: Implement the exact v2 parser and gate tuple validator

Set:

readonly RULES_ACCEPTANCE_SCHEMA=platform-observability-rules-alerts-v2

Read exactly these fields, in any file order but with exact cardinality and no duplicates/extras:

read_exact_env "$acceptance" \
  schema rollback_id target_initial_sha256 post_substrate_sha256 \
  slack_deployment_gate slack_gate_approval_ref slack_gate_accepted_by_uid \
  accepted_at_utc state

Add and call this predicate after schema/hash checks:

validate_slack_acceptance_tuple() {
  local tuple
  tuple="${PARSED_ENV[slack_deployment_gate]}|${PARSED_ENV[slack_gate_approval_ref]}|${PARSED_ENV[slack_gate_accepted_by_uid]}"
  case "$tuple" in
    'RECOVERY|strict-recovery-evidence-v1|not-applicable'|\
    'RISK_ACCEPTED|2026-08-14-observability-slack-recovery-risk-acceptance-design|1000')
      return 0
      ;;
    *)
      fail 'rules-alerts Slack deployment gate tuple mismatch'
      return 1
      ;;
  esac
}

Do not accept v1, ignore the new fields, infer tuple values, or add an environment-controlled fallback.

Step 5: Add the complete evidence adversarial matrix

Add table-driven negatives for:

v1
wrong-schema
missing-slack-gate
extra-field
duplicate-field
recovery-with-risk-ref
recovery-with-uid-1000
risk-with-strict-ref
risk-with-not-applicable
arbitrary-gate
arbitrary-ref
arbitrary-uid
rollback-id
target-initial-hash
post-substrate-hash
state
malformed-accepted-at

Add positive transactions for both exact RECOVERY and RISK_ACCEPTED tuples. Every negative must prove preflight/mutation 0, active bytes unchanged, no Host ledger, no prompt, and nonzero RC.

Step 6: Run GREEN and record evidence

Run:
```
bash -n scripts/bootstrap/apply-host-nginx-observability.sh
bash -n scripts/validate/test-apply-host-nginx-observability.sh
timeout --signal=TERM --kill-after=5s 300s \
  bash scripts/validate/test-apply-host-nginx-observability.sh
```
Expected: both syntax RC 0, focused suite RC 0, exact terminal APPLY HOST NGINX OBSERVABILITY TEST PASS, and no fixture/process residue. Append RED/GREEN commands, RCs, assertion count, final hashes, and metadata to the Task 1 report.

Task 2: Enforce first-cutover state, fixed command boundaries, and no-contact dry-run

Files:

Modify: scripts/bootstrap/apply-host-nginx-observability.sh:1-35,204-265,315-351,883-921
Modify: scripts/validate/test-apply-host-nginx-observability.sh
Append: .superpowers/sdd/2026-08-15-observability-task7-cutover-correction/task-1-report.md

Interfaces:

Consumes: exact deny and full candidate hashes plus the v2 evidence validator from Task 1.
Produces: production execution isolated from ambient PATH, a true no-network dry-run, and a full transition that starts only from deny-only.
Step 1: Add focused RED cases before production changes

Add these cases:
1. Copy the full candidate over the active fixture, invoke full execute, and require failure before preflight/ledger/prompt. Current production emits HOST_NGINX_FULL_STAGE=ALREADY_ACTIVE, so this is RED.
2. Extract print_plan with declare -f and require that it contains none of dig, curl, kubectl, sudo, nginx, or systemctl. Current production contains dig, so this is RED.
3. Put a logging dirname wrapper first in a hostile PATH and run /usr/bin/bash with PLATFORM_HOST_NGINX_TEST_MODE=invalid. Require nonzero RC and an empty hostile log. Current source resolves dirname before rejecting the invalid override, so this is RED.
4. Retain a third-state active fixture and require unchanged bytes, no ledger, no prompt, and nonzero RC.
5. Precreate a normal, non-symlink host-nginx/stages.tsv with the valid header and safe metadata; require rejection before preflight/ledger write/prompt with unchanged deny bytes. Current production appends an attempt to it, so this is RED.
6. In a separate fixture, precreate a normal, non-symlink host-nginx/payloads/full-prior-0001.conf with safe metadata; require the same prompt-before rejection. Retain the existing symlinked parent/ledger/payload and hardlinked-ledger negatives independently.
Run the focused suite once and record all expected RED leaves.
Step 2: Install the production PATH before any external command

Immediately after umask 077, before computing ROOT, install:
```
readonly HOST_NGINX_PRODUCTION_PATH='/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'
PATH=$HOST_NGINX_PRODUCTION_PATH
export PATH
hash -r
```
Compute ROOT without dirname:
```
readonly ROOT="$(cd -- "${BASH_SOURCE[0]%/*}/../.." && pwd -P)"
```
After a test fixture has passed the existing canonical owner/mode/boundary checks, prepend only its validated bin directory for test mode. Production never restores the caller PATH.
Step 3: Validate the exact production resolution set

Add an exact associative allowlist:
```
declare -Ar HOST_NGINX_PRODUCTION_COMMANDS=(
  [awk]=/usr/bin/awk [bash]=/usr/bin/bash [cat]=/usr/bin/cat
  [curl]=/usr/bin/curl [date]=/usr/bin/date [dig]=/usr/bin/dig
  [dirname]=/usr/bin/dirname [grep]=/usr/bin/grep [head]=/usr/bin/head
  [id]=/usr/bin/id [install]=/usr/bin/install [jq]=/usr/bin/jq
  [kubectl]=/usr/local/bin/kubectl [mktemp]=/usr/bin/mktemp
  [openssl]=/usr/bin/openssl [python3]=/usr/bin/python3
  [readlink]=/usr/bin/readlink [rm]=/usr/bin/rm [sed]=/usr/bin/sed
  [sha256sum]=/usr/bin/sha256sum [sleep]=/usr/bin/sleep
  [sort]=/usr/bin/sort [ss]=/usr/bin/ss [stat]=/usr/bin/stat
  [tail]=/usr/bin/tail [test]=/usr/bin/test [tr]=/usr/bin/tr
)
```
The fixed allowlist has exactly 27 entries. For production, require command -v lexical equality for every non-builtin entry, and bind the test entry directly to exact /usr/bin/test because Bash otherwise resolves its builtin before PATH; every sudo test probe is literal sudo -n /usr/bin/test, never a bare test or ambient/secure_path lookup. Resolve the final target with readlink -f, and require a root-owned regular executable with nlink 1, no group/world write, and no setuid/setgid bits. Validate fixed Nginx, systemd, and Certbot with that same rule. The sole exception is sudo, which must have lexical and canonical path exact /usr/bin/sudo, regular/non-symlink type, root:root, nlink 1, and exact mode 4755. Owner-setuid 04000 is required for sudo; setgid, group/world write, missing setuid, owner/link/path drift, or any other mode is rejected. Never extend this exception to another command. Verify each PATH directory or its canonical target is root-owned and not group/world writable.

Add focused metadata cases proving exact /usr/bin/sudo mode 4755 passes and each of missing setuid, wrong owner, nlink greater than one, lexical/canonical path drift, setgid, group write, and world write fails. Independently add setuid and setgid to a non-sudo allowlisted executable and require rejection.
Step 4: Make dry-run truthful and make full first-cutover-only

Remove public dig from print_plan. Replace the false claim:
```
HOST_NGINX_GRAFANA_PUBLIC_DNS=ABSENT
```
with:
```
HOST_NGINX_GRAFANA_PUBLIC_DNS=NOT_CHECKED_DRY_RUN
```
Replace the false live-SAN claim:
```
HOST_NGINX_CERTIFICATE_EXACT_SAN=grafana.learn.hyeonworks.com
```
with the exact two-line dry-run contract:
```
HOST_NGINX_CERTIFICATE_EXPECTED_SAN=grafana.learn.hyeonworks.com
HOST_NGINX_CERTIFICATE_SAN=NOT_CHECKED_DRY_RUN
```
Keep public DNS absence in external_preflight full.

Extend the focused dry-run case to require the three new marker lines exactly once, require both legacy claims absent, and require the certificate and DNS boundary logs to stay empty. This is a behavioral test, not a source-text-only assertion.

Add a full-only first-cutover residue predicate before external_preflight, any ledger preparation, and confirm. For each literal path below, require both sudo -n /usr/bin/test ! -e and sudo -n /usr/bin/test ! -L:
```
$ROLLBACK_ROOT/host-nginx
$ROLLBACK_ROOT/host-nginx/stages.tsv
$ROLLBACK_ROOT/host-nginx/payloads
$ROLLBACK_ROOT/host-nginx/payloads/full-prior-0001.conf
```
Check the host-nginx root first. If it exists or is a symlink, fail without traversing a child. Only while the root is proven absent/non-symlink may the three child paths be checked; recheck root absence after those probes to close the prompt-before gate. The normal-ledger and normal-payload fixtures must prove rejection before preflight and prompt. This predicate applies only to the first full transition; do not change metrics, certificate, or deny-only idempotency.

In transition, after computing hashes and validating full evidence, enforce:
```
if [[ "$stage" == full && "$active_sha" == "$candidate_sha" ]]; then
  fail 'full cutover is already active without first-cutover provenance'
  return 1
fi
```
Then retain the existing exact expected-active gate. Do not change idempotency for metrics, certificate, or deny-only modes.
Step 5: Add stale/future proof and rollback-outcome cases

Add proof timestamps for exact current, now-86401, now+301, and invalid calendar values. Add a rollback-failure fake that makes restore/install or reload fail and requires both:
```
HOST_NGINX_OBSERVABILITY_ROLLBACK=FAIL
MANUAL_RECOVERY_REQUIRED=YES
```
Retain success assertions for exact prior payload, three stable postchecks, full PASS, and postcheck-failure rollback PASS. Add a fault after ledger/payload or timestamp-backup staging but before rollback_armed=true; require nonzero RC, unchanged active deny bytes/reload count, rollback marker absent/N/A, and preserved root-owned staged evidence.

For every injected directory, ledger, payload, and timestamp staging failure, also require the terminal classification fresh rollback ID + source proof + complete Task 6 required; same-ID repair, proof-only refresh, or Host retry is forbidden.

Step 6: Run GREEN, hostile-boundary checks, and mutation tests

Run syntax and the complete focused suite under a 360-second outer bound. Then make private mode-0700 copies and independently inject:

RULES_ACCEPTANCE_SCHEMA=v1
tuple validation accepts gate name only
active full returns ALREADY_ACTIVE
print_plan calls dig
print_plan emits CERTIFICATE_EXACT_SAN or omits EXPECTED_SAN/NOT_CHECKED
production retains caller PATH
active hash uses a prefix comparison
full first-cutover accepts a normal existing ledger or prior payload
sudo metadata accepts missing-setuid/setgid/write/owner/link/path drift
a non-sudo command accepts setuid or setgid

For each mutation, require focused RC nonzero. Remove only the identity-checked private mutation root, verify workspace source hashes unchanged, and append results to the Task 1 report.

Task 3: Prove the fresh renderer-root preservation boundary

Files:

Modify: scripts/validate/test-render-observability-access.sh:796-863
Preserve unchanged: scripts/validate/render-observability-access.sh
Create: .superpowers/sdd/2026-08-15-observability-task7-cutover-correction/task-3-report.md

Interfaces:

Consumes: a rules-alerts-published handoff and an inventory-only clone.
Produces: a regression proving the former is immutable input and only the latter accepts complete publication.

Step 1: Freeze renderer/test identities

Required starting hashes:

renderer = ae5373b1b20be42e573d58e7384dcc654b1d3cac75a6c269cf684f4a14f3ca1a
test     = 8736658b802d511776803efc81cf000986864965456d9dbf11e0d3893de87eff

Record metadata. Renderer production bytes must remain unchanged in this task.

Step 2: Add the published-root preservation characterization

Create an exact seven-file complete publish source. Publish rules-alerts first to a fixture root, fingerprint its root, inventories, and three YAML files, then require publish_access_outputs ... complete to fail without changing any fingerprint or adding a temporary/output entry.

Required first-root topology is:
```
alertmanager.yaml
dashboards.yaml
post-substrate/
rules.yaml
target-initial/
```
Step 3: Add the inventory-only clone success characterization

Copy only the two inventory directories and four inventory files into a new fixture root. Require validate_access_inventory_root PASS, complete publication PASS, and exact final top-level set:
```
alertmanager.yaml
blackbox.yaml
dashboards.yaml
grafana.yaml
post-substrate/
private-dns.yaml
rules.yaml
target-initial/
targets.yaml
```
Verify source fingerprints unchanged, all seven destination YAML files regular/owner-exact/mode 0600/nlink 1, and no partial publication after injected collision.
Step 4: Run the focused renderer suite

Run:
```
bash -n scripts/validate/render-observability-access.sh
bash -n scripts/validate/test-render-observability-access.sh
timeout --signal=TERM --kill-after=5s 300s \
  bash scripts/validate/test-render-observability-access.sh
```
Expected: syntax RC 0, suite RC 0, exact OBSERVABILITY ACCESS RENDER ASSERTION TEST PASS, original renderer hash unchanged, and no suite-owned residue. Record evidence in the Task 3 report.

Task 4: Correct the authoritative Task 7 procedure without claiming live completion

Files:

Modify: bootstrap/manual/phase4-observability-access.md:381-444
Modify: /home/donghyeon/workspace/docs/platform/plans/2026-08-14-observability-slack-risk-acceptance-implementation.md:631-707
Append pre-live correction only: /home/donghyeon/workspace/docs/platform/runbooks/2026-08-13-observability-phase4-resume-worklog.md
Preserve: /home/donghyeon/workspace/docs/platform/plans/2026-08-15-observability-task7-cutover-correction-design.md
Create: .superpowers/sdd/2026-08-15-observability-task7-cutover-correction/task-4-report.md

Interfaces:

Consumes: Tasks 1–3 GREEN contracts.
Produces: one unambiguous operator sequence; existing Task 7 boxes remain unchecked.
Step 1: Correct Step 1 and Host execution truth

Document that no-arg is source/hash/no-contact only. Live certificate, DNS, Kubernetes, proof age/ID, and NodePort checks occur inside execute before the APPLY prompt.

Bind exact successful inputs:
```
PLATFORM_OBSERVABILITY_ROLLBACK_ID=20260814T145009Z
METRIC_ROOT=/tmp/platform-observability-metrics.dw5gLZ
active deny SHA-256=dbef6d443bcba58b26a5351ea76f6d09f6da8c2ef07a806e22745cf26c88f518
full SHA-256=7d2de2a92c3597a0859775da1d2ccbf5a3d72c0b2af5cac2439c82311361f801
```
Add the paste-safe array and immediate RC block from Task 6. Distinguish pre-prompt failure, post-prompt/pre-arm root-owned ledger/payload/backup staging, active-config rollback PASS, and rollback FAIL/manual recovery. Do not call stages.tsv a success marker or evidence staging an active Nginx mutation.

Before execute can display APPLY, require name-only, no-follow absence for the host-nginx subtree, stages.tsv, payloads, and payloads/full-prior-0001.conf. Document that a normal existing ledger/payload is residue, not reusable recovery state. Any ledger/payload/timestamp staging failure preserves the old ID and requires a fresh rollback ID plus source proof and complete Task 6 before another Host attempt.
Step 2: Add the proof-expiry route

State that source proof is exact-ID/deny-hash/status/IP/time evidence with maximum age 24 hours. A stale proof stops Task 7 and requires a fresh ID plus complete Task 6 acceptance. Explicitly forbid same-ID proof replacement, timestamp rewriting, Host retry, or reuse of a failed fresh ID.
Step 3: Add exact OIDC membership execution and rollback truth

Include interactive username reads, a Bash array, immediate RC capture, exact confirmations:
```
APPLY default
RECOVERY KEYCLOAK default
```
Require GRAFANA_OIDC_TRANSACTION=PASS and RC 0. Before transaction_active=true, managed Keycloak/Secret/membership mutation is zero and rollback is N/A. Active failures require rollback PASS; rollback FAIL/manual recovery or unknown stage stops. Remove any persistent Task 6 Keycloak reversal-ledger claim.
Step 4: Collapse Steps 3–5 into readiness plus one monolithic smoke

Prepare four distinct identities, break-glass access, Slack view, and a genuine outside-LAN/Tailscale client before starting. Keep human OIDC, Slack firing/resolved, and external attestation within exactly one observability-smoke.sh --execute call. Require immediate RC plus all five PASS markers. Document RC 2/external PARTIAL as incomplete Task 7 and forbid server-side substitution.

In each authority document, bind the execute call with the exact absolute TASK7_SMOKE array form from Task 7 Step 4; keep any no-arg plan in a separately named TASK7_SMOKE_DRY array. This gives the structural per-document assertion an unambiguous executable boundary.
Step 5: Correct the regression list and renderer root

Remove scripts/validate/observability-core-smoke.sh --execute; it does not exist. Remove the second observability-smoke.sh --execute. Keep one execution each for core renderer, fresh-root complete access renderer, admin UI, AIStor S3, phase1, phase2, and admin renderer.

Bind phase1 and phase2 to their absolute two-element Bash arrays and invoke each with the exact inline PLATFORM_HELM_BIN=/home/donghyeon/.local/bin/helm assignment. The fixed operator PATH intentionally excludes /home/donghyeon/.local/bin, while both scripts otherwise fall back to command -v helm; do not permit that fallback. Require the same exact Helm assignment for the other three renderer invocations.

Insert the hardened inventory-only clone procedure from Task 8 below and require preservation of dw5gLZ before and after publication.

Step 6: Run focused documentation assertions

Against only the central plan and manual, require the simple non-smoke assertions first:

! rg -F 'scripts/validate/observability-core-smoke.sh' "$CENTRAL" "$MANUAL"
rg -Fq 'RECOVERY KEYCLOAK default' "$MANUAL"
rg -Fq 'HOST_NGINX_FULL_STAGE=PASS' "$MANUAL"
rg -Fq 'OBSERVABILITY_EXTERNAL_BOUNDARY=PARTIAL' "$MANUAL"
rg -Fq '/tmp/platform-observability-metrics.dw5gLZ' "$CENTRAL" "$MANUAL"

Do not use an aggregate raw-literal count for smoke. Run this structural assertion once per document:

case $- in *e*) TASK7_DOC_ASSERT_ERREXIT=1 ;; *) TASK7_DOC_ASSERT_ERREXIT=0 ;; esac
TASK7_DOC_ASSERT_RC=0
set +e
for TASK7_DOC in "$CENTRAL" "$MANUAL"; do
  /usr/bin/python3 - "$TASK7_DOC" <<'PY'
import pathlib
import re
import shlex
import subprocess
import sys

path = pathlib.Path(sys.argv[1])
text = path.read_text(encoding="utf-8")
blocks = re.findall(r"(?ms)^[ \t]*```bash[^\n]*\n(.*?)^[ \t]*```[ \t]*$", text)
smoke_path = "/home/donghyeon/workspace/platform/scripts/validate/observability-smoke.sh"
helm_assignment = "PLATFORM_HELM_BIN=/home/donghyeon/.local/bin/helm"
phase_specs = {
    "TASK7_PHASE1": "/home/donghyeon/workspace/platform/scripts/validate/render-phase1.sh",
    "TASK7_PHASE2": "/home/donghyeon/workspace/platform/scripts/validate/render-phase2.sh",
}

def tokens(block):
    normalized = re.sub(r"\\[ \t]*\n[ \t]*", " ", block)
    lexer = shlex.shlex(
        normalized.replace("\n", " ; "), posix=True, punctuation_chars="();&|",
    )
    lexer.whitespace_split = True
    lexer.commenters = ""
    return list(lexer)

def materialize_commands(block, arrays):
    stream = tokens(block)
    commands = []
    current = []
    index = 0
    while index < len(stream):
        token = stream[index]
        assignment = re.fullmatch(r"([A-Za-z_][A-Za-z0-9_]*)(\+?)=", token)
        if assignment and index + 1 < len(stream) and stream[index + 1] == "(":
            name, append = assignment.groups()
            index += 2
            depth = 1
            values = []
            while index < len(stream) and depth:
                item = stream[index]
                if item == "(":
                    depth += 1
                elif item == ")":
                    depth -= 1
                    if depth == 0:
                        index += 1
                        break
                elif item not in {";", "&&", "||", "&", "|"}:
                    values.append(item)
                index += 1
            if depth != 0:
                raise SystemExit(f"{path}: unterminated array assignment")
            if append:
                arrays.setdefault(name, []).extend(values)
            else:
                arrays[name] = values
            continue
        if token in {";", "&&", "||", "&", "|"}:
            if current:
                commands.append(current)
                current = []
            index += 1
            continue
        current.append(token)
        index += 1
    if current:
        commands.append(current)

    materialized = []
    for command in commands:
        argv = []
        for token in command:
            match = re.fullmatch(r"\$\{([A-Za-z_][A-Za-z0-9_]*)\[@\]\}", token)
            if match:
                if match.group(1) not in arrays:
                    raise SystemExit(f"{path}: invoked undefined array {match.group(1)}")
                argv.extend(arrays[match.group(1)])
            else:
                argv.append(token)
        materialized.append(argv)
    return materialized

def is_smoke_execute(argv):
    has_smoke = any(token.rsplit("/", 1)[-1] == "observability-smoke.sh" for token in argv)
    return has_smoke and "--execute" in argv

arrays = {}
smoke_executes = []
canonical_blocks = []
phase_executes = {name: [] for name in phase_specs}
phase_blocks = {name: [] for name in phase_specs}
for block_index, block in enumerate(blocks):
    completed = subprocess.run(
        ["/usr/bin/bash", "-n"], input=block, text=True,
        stdout=subprocess.DEVNULL, stderr=subprocess.PIPE, check=False,
    )
    if completed.returncode != 0:
        raise SystemExit(f"{path}: fenced Bash block {block_index + 1} is not syntax-valid")
    binding_counts = (
        len(re.findall(r"(?m)^\s*TASK7_SMOKE=\(/usr/bin/bash\)\s*$", block)),
        len(re.findall(
            rf"(?m)^\s*TASK7_SMOKE\+=\({re.escape(smoke_path)}\)\s*$", block,
        )),
        len(re.findall(r"(?m)^\s*TASK7_SMOKE\+=\(--execute\)\s*$", block)),
        len(re.findall(r'''(?m)^\s*"\$\{TASK7_SMOKE\[@\]\}"\s*$''', block)),
    )
    if any(binding_counts):
        if binding_counts != (1, 1, 1, 1):
            raise SystemExit(f"{path}: malformed canonical smoke block")
        canonical_blocks.append(block_index)
    for name, phase_path in phase_specs.items():
        phase_counts = (
            len(re.findall(rf"(?m)^\s*{name}=\(/usr/bin/bash\)\s*$", block)),
            len(re.findall(
                rf"(?m)^\s*{name}\+=\({re.escape(phase_path)}\)\s*$", block,
            )),
            len(re.findall(
                rf'''(?m)^\s*{re.escape(helm_assignment)} "\$\{{{name}\[@\]\}}"\s*\n'''
                rf'''\s*{name}_RC=\$\?\s*$''', block,
            )),
        )
        if any(phase_counts):
            if phase_counts != (1, 1, 1):
                raise SystemExit(f"{path}: malformed canonical {name} block")
            phase_blocks[name].append(block_index)
    for argv in materialize_commands(block, arrays):
        if is_smoke_execute(argv):
            smoke_executes.append((block_index, argv))
        for name, phase_path in phase_specs.items():
            if any(token.rsplit("/", 1)[-1] == phase_path.rsplit("/", 1)[-1] for token in argv):
                phase_executes[name].append((block_index, argv))

canonical_argv = ["/usr/bin/bash", smoke_path, "--execute"]
if len(canonical_blocks) != 1:
    raise SystemExit(f"{path}: expected one canonical smoke block")
if smoke_executes != [(canonical_blocks[0], canonical_argv)]:
    raise SystemExit(f"{path}: non-canonical or duplicate smoke execute argv")
for name, phase_path in phase_specs.items():
    if len(phase_blocks[name]) != 1:
        raise SystemExit(f"{path}: expected one canonical {name} block")
    expected = [helm_assignment, "/usr/bin/bash", phase_path]
    if phase_executes[name] != [(phase_blocks[name][0], expected)]:
        raise SystemExit(f"{path}: non-canonical or duplicate {name} argv")
PY
  TASK7_ONE_DOC_RC=$?
  printf 'TASK7_DOC_SMOKE_ASSERT_RC=%d DOC=%s\n' "$TASK7_ONE_DOC_RC" "$TASK7_DOC"
  if (( TASK7_ONE_DOC_RC != 0 )); then
    TASK7_DOC_ASSERT_RC=1
  fi
done
(( TASK7_DOC_ASSERT_ERREXIT == 0 )) || set -e
(( TASK7_DOC_ASSERT_RC == 0 ))

This requires one absolute smoke argv binding, one exact --execute, and one array invocation in the same syntax-valid fenced Bash block in each document. The parser materializes every array name and every direct command argv across all fenced Bash blocks, so path spelling or wrapper choice cannot bypass the count. Test the checker itself with fixtures: canonical execute plus a separate no-arg dry-run must pass; an additional relative bash scripts/validate/observability-smoke.sh --execute, alternate array name, /usr/bin/env wrapper, timeout wrapper, or line-continued env+timeout wrapper must each fail. Run every failing fixture once in the first document position and once in the second to prove per-document RC masking is impossible. A duplicate in one document cannot compensate for a missing command in the other because every per-document RC contributes to the final nonzero gate. Confirm all Task 7 checkboxes remain unchecked and Task 8 remains unstarted. The same parser must require one absolute two-element TASK7_PHASE1 array and one absolute two-element TASK7_PHASE2 array, each invoked exactly once with the exact inline PLATFORM_HELM_BIN=/home/donghyeon/.local/bin/helm assignment and immediate RC capture in its canonical block. Test four private document mutations: remove the assignment from phase1, remove it from phase2, replace the phase1 value, and replace the phase2 value. Put each mutation once in the central position and once in the manual position while the other document stays canonical; all eight final gates must return nonzero, and the unmodified central/manual pair must return zero. Remove only identity-checked private fixtures. Record per-document structural RCs, hashes, and metadata.

Task 5: Run bounded verification and create the Task 7 non-Git source freeze

Files:

Create: .superpowers/sdd/2026-08-15-observability-task7-cutover-correction/verify-task7-source-manifest.py
Create: .superpowers/sdd/2026-08-15-observability-task7-cutover-correction/test-verify-task7-source-manifest.py
Create: .superpowers/sdd/2026-08-15-observability-task7-cutover-correction/baseline/task-7-prelive-source-manifest.txt
Create: .superpowers/sdd/2026-08-15-observability-task7-cutover-correction/baseline/task-7-prelive-source-manifest.sha256
Create: .superpowers/sdd/2026-08-15-observability-task7-cutover-correction/baseline/task-7-command-inventory.txt
Create: .superpowers/sdd/2026-08-15-observability-task7-cutover-correction/task-5-report.md

Interfaces:

Consumes: corrected source/tests/docs and unchanged OIDC/smoke/renderer production.
Produces: reviewed, sidecar-bound pre-live authority for exactly one Task 7 execution.

Step 1: Run every focused source suite under bounded supervisors

Run syntax first, then:

/usr/bin/timeout --signal=TERM --kill-after=5s 360s \
  /usr/bin/bash /home/donghyeon/workspace/platform/scripts/validate/test-apply-host-nginx-observability.sh
/usr/bin/timeout --signal=TERM --kill-after=5s 300s \
  /usr/bin/bash /home/donghyeon/workspace/platform/scripts/validate/test-render-observability-access.sh
/usr/bin/timeout --signal=TERM --kill-after=5s 360s \
  /usr/bin/bash /home/donghyeon/workspace/platform/scripts/validate/test-configure-keycloak-grafana-oidc.sh
/usr/bin/timeout --signal=TERM --kill-after=5s 360s \
  /usr/bin/bash /home/donghyeon/workspace/platform/scripts/validate/test-observability-smoke.sh
/usr/bin/bash /home/donghyeon/workspace/platform/scripts/validate/test-host-nginx-observability-source.sh

Require every RC 0, exact suite PASS markers, and zero suite-owned process/temp residue. Do not run live execute modes.

Step 2: Run scanner-focused tests and one authorized full source scan

Run the scanner test first. Then run one full scan into a fresh private mode-0700 root with tracing disabled and an immediate RC file. Require exactly one rendered-secret PASS and one sensitive-source PASS, no payload output, and zero new scanner/renderer process or temp residue. Preserve the scan root through review; do not run another full scan in Task 7 unless source changes again.
Step 3: Implement a private strict Task 7 manifest verifier with TDD

Adapt the hardened descriptor-retaining algorithm from the existing Task 6 verifier into the new private path without modifying Task 6 authority. The prelive invocation has exactly this argv contract:
```
argv[0] = /usr/bin/python3
argv[1] = /home/donghyeon/workspace/platform/.superpowers/sdd/2026-08-15-observability-task7-cutover-correction/verify-task7-source-manifest.py
argv[2] = literal 64-character lowercase manifest SHA-256 recorded by independent review in task-5-report
argv[3] = /home/donghyeon/workspace/platform/.superpowers/sdd/2026-08-15-observability-task7-cutover-correction/baseline/task-7-prelive-source-manifest.txt
argv[4] = /home/donghyeon/workspace/platform/.superpowers/sdd/2026-08-15-observability-task7-cutover-correction/baseline/task-7-prelive-source-manifest.sha256
```
The interpreter, verifier, manifest, and sidecar paths above are literal. The final pair substitutes only the two exact task-7-final-source-manifest absolute paths and the separately reviewed final digest. No argument may come from pwd, readlink, current file contents, or a shell-computed digest.

It must enforce: allowlisted prelive/final pair, exact row order and count 21, canonical absolute paths, unique paths, exact sha256|mode4|uid:gid|nlink|regular file|size|path grammar, mode policy, UID:GID 1000:1000, nlink 1, retained-descriptor hash/metadata/rebind checks, exact one-line sidecar binding, and no manifest/sidecar/verifier path in rows.

The verifier is intentionally outside the manifest. Before invocation, independently review and pin its literal canonical absolute path, lowercase SHA-256, type, UID:GID, mode, nlink, and size. The launcher opens verifier, manifest, and sidecar descriptors before validation, checks descriptor metadata/hash and path binding before invocation, invokes only the literal /usr/bin/python3 + absolute verifier + three literal reviewed arguments, and checks the same descriptor/path identities again immediately afterward. Do not substitute a PATH-resolved interpreter, relative script, calculated current manifest SHA, or self-referential manifest row.

Write RED tests for wrong pair/SHA/order/count/path/mode/UID/link/type/size/hash, symlink or hardlink manifest/sidecar/row, FIFO, duplicate path, and cross-paired sidecar. Add separate late replacement/path-rebind cases for verifier, manifest, and sidecar between precheck/invocation/postcheck; each must be caught by retained descriptor/path identity. The test launcher itself must use literal /usr/bin/python3 and absolute verifier/manifest/sidecar arguments. Implement until all tests pass.
Step 4: Freeze the command/launcher inventory and exact 21-row pre-live manifest

First create a mode-0600 payload-free task-7-command-inventory.txt with schema platform-observability-task7-command-inventory-v1, the exact production PATH, and one sorted command|name|lexical_path|canonical_path|uid:gid|mode|sha256 row for every Host allowlist command plus Nginx, systemd, sudo, and Certbot: exactly 31 command rows (27 allowlist + 4 fixed executables). The Host test row must be exact lexical/canonical /usr/bin/test; every Host sudo probe is an absolute /usr/bin/test argv element, never bare test. Add safe launcher rows for Host argc=5, OIDC argc=7 with only <admin-stdin>/<viewer-stdin> placeholders, smoke argc=3, and complete renderer argc=6. Require root-owned, non-writable executable targets, safe PATH lineage, exact cardinality, no duplicate/missing row, and no username/Secret value. Add a verifier launcher row for literal /usr/bin/python3, the absolute verifier path and expected metadata, fixed digest sentinel <reviewed-manifest-sha256>, and the absolute manifest/sidecar pair (argc=5, counting the interpreter as argv[0]). The sentinel must occur exactly once in that row; the prelive/final manifest's own 64-character lowercase digest may not occupy that argv slot or be copied into another manifest-row file. Other independently required file/executable SHA metadata is unaffected. Generate a private comparison independently, require cmp -s, then remove only that comparison. Repeat the comparison immediately before live Host execution.

This split is mandatory because task-7-command-inventory.txt is manifest row 21. Only after all 21 rows and the sidecar exist may independent review write the actual lowercase manifest digest to the non-governed task-5-report/review handoff. The launcher reads no digest from the governed inventory; the controller supplies the report's reviewed literal directly as verifier argv[2]. Add tests that replace the governed verifier row's sentinel slot with a 64-character manifest digest and require rejection, require sentinel exact-once plus exact argc/path/metadata shape, and refuse verifier invocation until the external report pin exists and matches the reviewed value.

The command inventory validator uses the same non-widening metadata policy as Task 2: exact canonical /usr/bin/sudo is the only row permitted and required to carry owner-setuid and must be exact regular|0:0|4755|1; every other executable row forbids both setuid and setgid as well as group/world write. A private comparison with any sudo or non-sudo bit/path/owner/link drift must fail.

Use these paths, once each and in this order:
```
/home/donghyeon/workspace/platform/scripts/bootstrap/apply-host-nginx-observability.sh
/home/donghyeon/workspace/platform/scripts/validate/test-apply-host-nginx-observability.sh
/home/donghyeon/workspace/platform/scripts/validate/test-host-nginx-observability-source.sh
/home/donghyeon/workspace/platform/infrastructure/networking/host-nginx/learn-services-admin.conf
/home/donghyeon/workspace/platform/infrastructure/networking/host-nginx/learn-services-metrics-guard.conf
/home/donghyeon/workspace/platform/infrastructure/networking/host-nginx/learn-services-grafana-deny-guard.conf
/home/donghyeon/workspace/platform/infrastructure/networking/host-nginx/learn-services-observability.conf
/home/donghyeon/workspace/platform/scripts/bootstrap/apply-observability-access.sh
/home/donghyeon/workspace/platform/scripts/validate/test-apply-observability-access.sh
/home/donghyeon/workspace/platform/scripts/validate/render-observability-access.sh
/home/donghyeon/workspace/platform/scripts/validate/test-render-observability-access.sh
/home/donghyeon/workspace/platform/scripts/bootstrap/configure-keycloak-grafana-oidc.sh
/home/donghyeon/workspace/platform/scripts/validate/test-configure-keycloak-grafana-oidc.sh
/home/donghyeon/workspace/platform/scripts/validate/observability-smoke.sh
/home/donghyeon/workspace/platform/scripts/validate/test-observability-smoke.sh
/home/donghyeon/workspace/platform/bootstrap/manual/phase4-observability-access.md
/home/donghyeon/workspace/docs/platform/plans/2026-08-14-observability-slack-risk-acceptance-implementation.md
/home/donghyeon/workspace/docs/platform/plans/2026-08-15-observability-task7-cutover-correction-design.md
/home/donghyeon/workspace/platform/docs/superpowers/plans/2026-08-15-observability-task7-cutover-correction.md
/home/donghyeon/workspace/docs/platform/runbooks/2026-08-13-observability-phase4-resume-worklog.md
/home/donghyeon/workspace/platform/.superpowers/sdd/2026-08-15-observability-task7-cutover-correction/baseline/task-7-command-inventory.txt
```
The verifier, manifest, and sidecar themselves are not rows. Create the sidecar with the literal canonical manifest path, pin the verifier separately as specified in Step 3, and run the descriptor-retaining strict launcher twice with the externally reviewed lowercase manifest SHA and literal absolute paths. Require RC 0 both times and unchanged verifier/manifest/sidecar descriptor/path identities after each invocation.
Step 5: Independent pre-live review

Review spec coverage, C/I/M, v2 tuple matrix, first-cutover behavior, fixed PATH and command/launcher residue rejection, exact 21-row inventory, self-reference-free sentinel row plus external digest pin, separately pinned verifier identity and rebind tests, both correction authority documents, RED provenance, mutation evidence, original handoff preservation, corrected OIDC/smoke sequencing, scanner result, and manifest/sidecar identity. Required verdict: Critical 0, Important 0, Spec YES, Quality Approved, Ready for Host Task 7 YES. Any other verdict returns to the responsible source task and requires a new scan/freeze after correction.

Task 6: Execute the single Host Nginx full cutover

Files:

Preserve: /var/lib/hyeonworks/platform-rollbacks/observability-20260814T145009Z
Preserve: /tmp/platform-observability-metrics.dw5gLZ
Append after outcome: .superpowers/sdd/2026-08-15-observability-task7-cutover-correction/task-6-report.md

Interfaces:

Consumes: Task 5 exact reviewed freeze, same Task 6 ID/root/handoff, current source proof, active deny-only config.
Produces: full Host Nginx PASS or a preserved fail-closed transaction that is never retried under this ID.
Step 1: Bind the successful Task 6 identity in the existing operator shell

Before any external preparation command, install the reviewed operator boundary and invalidate Bash's command cache:
```
TASK7_OPERATOR_PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PATH=$TASK7_OPERATOR_PATH
export PATH
hash -r
```
Using the exact Task 5 generation procedure and only commands resolved under that boundary, regenerate the command/launcher inventory into a fresh private comparison and require byte equality with the reviewed inventory. Require every lexical/canonical command row to match before continuing. If this comparison cannot be completed first, stop; do not run an ambient-PATH prep command. Then run short assignments only:
```
cd /home/donghyeon/workspace/platform
TASK7_ID=20260814T145009Z
TASK7_METRIC_ROOT=/tmp/platform-observability-metrics.dw5gLZ
export PLATFORM_OBSERVABILITY_ROLLBACK_ID="$TASK7_ID"
[[ "$PLATFORM_OBSERVABILITY_ROLLBACK_ID" == "$TASK7_ID" ]]
[[ "$(/usr/bin/readlink -f -- "$TASK7_METRIC_ROOT")" == "$TASK7_METRIC_ROOT" ]]
[[ "$(/usr/bin/stat -c '%F|%u:%g|%a|%h' -- "$TASK7_METRIC_ROOT")" == 'directory|1000:1000|700|4' ]]
```
Use literal /usr/bin/find, /usr/bin/sort, /usr/bin/stat, and /usr/bin/sha256sum for the remaining preparation checks. Require the root name-only topology to be exactly two phase directories plus dashboards.yaml, rules.yaml, and alertmanager.yaml. Verify both known inventory hashes and metadata without printing inventory bodies.
Step 2: Revalidate the frozen source and no-contact dry-run

Run the Task 5 descriptor-retaining strict launcher twice, exactly as independently reviewed: literal /usr/bin/python3, literal absolute verifier path, literal reviewed 64-character manifest SHA, and literal absolute prelive manifest/sidecar paths. The SHA must come from the non-governed task-5 review handoff, never from the command-inventory sentinel or another manifest-row file. Require RC 0 and unchanged separately pinned verifier SHA/type/owner/mode/nlink/size/canonical-path identity before and after both invocations.

Then build TASK7_HOST_DRY=(/usr/bin/bash /home/donghyeon/workspace/platform/scripts/bootstrap/apply-host-nginx-observability.sh) and invoke it once under the same errexit save/set +e/immediate RC/restore envelope used by execute. Require:
```
HOST_NGINX_ACTIVE_SHA256=dbef6d443bcba58b26a5351ea76f6d09f6da8c2ef07a806e22745cf26c88f518
HOST_NGINX_GRAFANA_DENY_GUARD_SHA256=dbef6d443bcba58b26a5351ea76f6d09f6da8c2ef07a806e22745cf26c88f518
HOST_NGINX_FULL_SHA256=7d2de2a92c3597a0859775da1d2ccbf5a3d72c0b2af5cac2439c82311361f801
HOST_NGINX_CERTIFICATE_EXPECTED_SAN=grafana.learn.hyeonworks.com
HOST_NGINX_CERTIFICATE_SAN=NOT_CHECKED_DRY_RUN
HOST_NGINX_GRAFANA_PUBLIC_DNS=NOT_CHECKED_DRY_RUN
HOST_NGINX_OBSERVABILITY_DRY_RUN=PASS
TASK7_HOST_DRY_RC=0
```
Also require legacy dry-run claims HOST_NGINX_CERTIFICATE_EXACT_SAN= and HOST_NGINX_GRAFANA_PUBLIC_DNS=ABSENT to be absent. Any different/unreadable/already-full active hash stops before sudo or execute.

Step 3: Refresh sudo and build a paste-safe argv array

Run /usr/bin/sudo -v, then require /usr/bin/sudo -n /usr/bin/true and print only SUDO_READY.

Build the command without placing the long option in pasted text:

TASK7_VOD=--verified
TASK7_VOD+=-output-dir
TASK7_HOST=(/usr/bin/bash)
TASK7_HOST+=(/home/donghyeon/workspace/platform/scripts/bootstrap/apply-host-nginx-observability.sh)
TASK7_HOST+=(--execute)
TASK7_HOST+=("$TASK7_VOD")
TASK7_HOST+=("$TASK7_METRIC_ROOT")
[[ "${#TASK7_HOST[@]}" -eq 5 ]]
printf 'TASK7_HOST_ARGC=%d\n' "${#TASK7_HOST[@]}"

Do not print %q for usernames or sensitive data; this Host argv contains only safe paths/options.

Step 4: Execute exactly once and capture immediate RC

case $- in *e*) TASK7_HOST_ERREXIT=1 ;; *) TASK7_HOST_ERREXIT=0 ;; esac
set +e
"${TASK7_HOST[@]}"
TASK7_HOST_RC=$?
printf 'TASK7_HOST_RC=%d\n' "$TASK7_HOST_RC"
(( TASK7_HOST_ERREXIT == 0 )) || set -e

Before entering anything, require exact pre-prompt stage and previous/candidate hashes. Only the operator enters APPLY.

The prompt itself is admissible only after no-follow absence checks have proved all four paths absent under both test -e and test -L:

/var/lib/hyeonworks/platform-rollbacks/observability-20260814T145009Z/host-nginx
/var/lib/hyeonworks/platform-rollbacks/observability-20260814T145009Z/host-nginx/stages.tsv
/var/lib/hyeonworks/platform-rollbacks/observability-20260814T145009Z/host-nginx/payloads
/var/lib/hyeonworks/platform-rollbacks/observability-20260814T145009Z/host-nginx/payloads/full-prior-0001.conf

An existing normal path or symlink is a pre-prompt STOP, not resumable ledger state.

Success requires exact HOST_NGINX_FULL_STAGE=PASS and RC 0. ALREADY_ACTIVE, missing PASS, nonzero or ambiguous RC is failure.

Step 5: Classify failure without retry
- Pre-prompt failure: active deny unchanged, Host ledger absent, rollback N/A.
- Post-prompt/pre-arm failure: active deny and reload count unchanged; root-owned ledger/payload or timestamp backup may remain and must be preserved; rollback N/A.
- Armed failure: require HOST_NGINX_OBSERVABILITY_ROLLBACK=PASS and active deny hash restored.
- ROLLBACK=FAIL, MANUAL_RECOVERY_REQUIRED=YES, unknown active hash, response loss, or missing evidence: STOP for manual recovery.
If any residue is present before prompt, or any ledger directory/file, payload, or timestamp-backup staging step fails, preserve the entire root and start a new Task 6 design/execution with a fresh ID. The same route applies when source proof is stale/future. Do not alter this root or proof, recreate only one object, or invoke the Host command again under 20260814T145009Z.
Step 6: Record the Host terminal result

Record safe argv cardinality, dry/execute RC, exact stage/hashes, whether APPLY was entered, Nginx active/test/reload outcome, three stable LAN/local postchecks, rollback invocation/outcome, original Task 6 evidence metadata unchanged, and process/temp residue delta. Do not mark later Task 7 steps yet.

Task 7: Reconcile memberships and run the monolithic acceptance once

Files:

Append: .superpowers/sdd/2026-08-15-observability-task7-cutover-correction/task-7-report.md
Preserve unchanged: scripts/bootstrap/configure-keycloak-grafana-oidc.sh
Preserve unchanged: scripts/validate/observability-smoke.sh

Interfaces:

Consumes: Host full PASS, prepared human identities, Slack view, and a real external client.
Produces: OIDC transaction PASS plus one complete machine/OIDC/Slack/external smoke PASS.
Step 1: Prove human/external readiness before mutation

Prepare four distinct realm usernames for admin, viewer, no-group, and membership-removal testing; local break-glass access; a Slack channel view; and an outside-LAN/Tailscale client with proxy disabled. If any is unavailable, stop without running membership or smoke.

Step 2: Build and execute the OIDC membership argv once

read -r -p 'Grafana organization admin realm username: ' OBS_ADMIN_USER
read -r -p 'Grafana viewer realm username: ' OBS_VIEWER_USER
TASK7_OIDC=(/usr/bin/bash /home/donghyeon/workspace/platform/scripts/bootstrap/configure-keycloak-grafana-oidc.sh)
TASK7_OIDC+=(--execute)
TASK7_OIDC+=(--admin "$OBS_ADMIN_USER")
TASK7_OIDC+=(--viewer "$OBS_VIEWER_USER")
case $- in *e*) TASK7_OIDC_ERREXIT=1 ;; *) TASK7_OIDC_ERREXIT=0 ;; esac
set +e
"${TASK7_OIDC[@]}"
TASK7_OIDC_RC=$?
unset OBS_ADMIN_USER OBS_VIEWER_USER TASK7_OIDC
printf 'TASK7_OIDC_RC=%d\n' "$TASK7_OIDC_RC"
(( TASK7_OIDC_ERREXIT == 0 )) || set -e

Enter exact APPLY default and RECOVERY KEYCLOAK default. Require GRAFANA_OIDC_TRANSACTION=PASS and RC 0. A failure before transaction_active=true has no managed Keycloak/OIDC Secret/membership mutation and rollback marker N/A. An active transaction failure requires GRAFANA_OIDC_ROLLBACK=PASS; rollback FAIL/manual recovery or unknown stage stops. Do not claim or search for a persistent Task 6 Keycloak ledger.

Step 3: Run the smoke no-arg plan once before execute

Build an absolute two-element argv array and use the errexit save/set +e/immediate RC/restore envelope:
```
TASK7_SMOKE_DRY=(/usr/bin/bash)
TASK7_SMOKE_DRY+=(/home/donghyeon/workspace/platform/scripts/validate/observability-smoke.sh)
```
Invoke it exactly once. Require no-arg RC 0, exact OBSERVABILITY_SMOKE_DRY_RUN=PASS, HUMAN_EXTERNAL_CLIENT=required, and MUTATION=NOT_REQUESTED. This is not acceptance and does not replace execute.

Step 4: Execute the smoke exactly once with immediate RC

TASK7_SMOKE=(/usr/bin/bash)
TASK7_SMOKE+=(/home/donghyeon/workspace/platform/scripts/validate/observability-smoke.sh)
TASK7_SMOKE+=(--execute)
case $- in *e*) TASK7_SMOKE_ERREXIT=1 ;; *) TASK7_SMOKE_ERREXIT=0 ;; esac
set +e
"${TASK7_SMOKE[@]}"
TASK7_SMOKE_RC=$?
printf 'TASK7_SMOKE_RC=%d\n' "$TASK7_SMOKE_RC"
(( TASK7_SMOKE_ERREXIT == 0 )) || set -e

Only the operator supplies the requested identities and exact confirmations. Confirm actual admin/viewer/denied/break-glass/session-revoke/relogin behavior, Slack firing then resolved message, and run the script-rendered --resolve command from the true external client before entering its exact dynamic token.

Step 5: Require the full acceptance marker set

Success requires all:
```
OBSERVABILITY_MACHINE_ACCEPTANCE=PASS
OBSERVABILITY_OIDC_ACCEPTANCE=PASS
OBSERVABILITY_SLACK_ACCEPTANCE=PASS
OBSERVABILITY_EXTERNAL_BOUNDARY=PASS
OBSERVABILITY_SMOKE=PASS
TASK7_SMOKE_RC=0
```
Machine evidence must include LAN and Tailscale Grafana success, denied local/public boundary, external 403, public metrics 404, unknown SNI rejection, LAN NodePorts refused, loopback NodePort exact Traefik identity, exact 30/30 targets, datasource/trace link, persistence, network-policy denial, and Certbot/Nginx health.

RC 2 or OBSERVABILITY_EXTERNAL_BOUNDARY=PARTIAL leaves Task 7 incomplete. Any cleanup ambiguity/manual recovery stops and forbids automatic rerun; review exact owned-object cleanup first.
Step 6: Record payload-free OIDC/smoke evidence

Record roles and PASS/FAIL only, never identity values or browser/session data. Record temporary object kinds and cleanup disposition, not payloads. Verify Task 6 root/handoff metadata unchanged and no smoke-owned process/root remains.

Task 8: Run complete rendering from a fresh clone and existing regressions once

Files:

Create: one fresh /tmp/platform-observability-metrics.XXXXXX renderer root
Preserve: /tmp/platform-observability-metrics.dw5gLZ
Append: .superpowers/sdd/2026-08-15-observability-task7-cutover-correction/task-8-report.md

Interfaces:

Consumes: Task 7 final acceptance and byte-preserved inventories from the Task 6 handoff.
Produces: complete renderer and existing platform regression RCs without a second smoke.
Step 1: Fingerprint the original handoff without changing it

Reinstall TASK7_OPERATOR_PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin, export PATH, run hash -r, and require the independently regenerated command/launcher inventory to remain byte-equal before any external preparation command. From this point, the procedure below nevertheless uses literal absolute paths for every preparation executable.

Require exact canonical root, UID:GID 1000:1000, root mode 0700, phase modes 0700, file modes 0600, nlink 1, exact root/phase entry sets, and the two known inventory hashes. Record device/inode/type/owner/mode/nlink/size/mtime/ctime/SHA for both inventories and checksums plus the three Task 6 YAML files. Use only literal /usr/bin/readlink, /usr/bin/stat, /usr/bin/find, /usr/bin/sort, and /usr/bin/sha256sum for these checks.

Step 2: Create the fresh inventory-only root

TASK7_RENDER_ROOT="$(/usr/bin/mktemp -d /tmp/platform-observability-metrics.XXXXXX)"
/usr/bin/chmod 0700 "$TASK7_RENDER_ROOT"
for phase in target-initial post-substrate; do
  /usr/bin/install -d -m 0700 -- "$TASK7_RENDER_ROOT/$phase"
  for file in inventory.json inventory.sha256; do
    source_file="$TASK7_METRIC_ROOT/$phase/$file"
    destination_file="$TASK7_RENDER_ROOT/$phase/$file"
    [[ -f "$source_file" && ! -L "$source_file" ]]
    /usr/bin/cp --no-dereference --reflink=never -- "$source_file" "$destination_file"
    /usr/bin/chmod 0600 "$destination_file"
    /usr/bin/cmp -s -- "$source_file" "$destination_file"
  done
done
unset source_file destination_file

Require the exact six-entry topology, current-user ownership, modes, nlinks, byte equality, known hashes, and unchanged original fingerprints. Any failure preserves the new root and stops; do not repair or reuse it.

Step 3: Run core and complete access renderers once

Build and invoke the core renderer exactly once with an absolute two-element argv and exact Helm environment:

TASK7_CORE=(/usr/bin/bash)
TASK7_CORE+=(/home/donghyeon/workspace/platform/scripts/validate/render-observability-core.sh)
[[ "${#TASK7_CORE[@]}" -eq 2 ]]
case $- in *e*) TASK7_CORE_ERREXIT=1 ;; *) TASK7_CORE_ERREXIT=0 ;; esac
set +e
PLATFORM_HELM_BIN=/home/donghyeon/.local/bin/helm "${TASK7_CORE[@]}"
TASK7_CORE_RC=$?
(( TASK7_CORE_ERREXIT == 0 )) || set -e
printf 'TASK7_CORE_RC=%d\n' "$TASK7_CORE_RC"

Require OBSERVABILITY CORE STATIC RENDER PASS and TASK7_CORE_RC=0. Do not invoke the core renderer in Step 4 or through another script.

Then build and invoke the complete access renderer exactly once:

TASK7_VOD=--verified
TASK7_VOD+=-output-dir
TASK7_COMPLETE=(/usr/bin/bash)
TASK7_COMPLETE+=(/home/donghyeon/workspace/platform/scripts/validate/render-observability-access.sh)
TASK7_COMPLETE+=(--component complete)
TASK7_COMPLETE+=("$TASK7_VOD" "$TASK7_RENDER_ROOT")
[[ "${#TASK7_COMPLETE[@]}" -eq 6 ]]
case $- in *e*) TASK7_COMPLETE_ERREXIT=1 ;; *) TASK7_COMPLETE_ERREXIT=0 ;; esac
set +e
PLATFORM_HELM_BIN=/home/donghyeon/.local/bin/helm "${TASK7_COMPLETE[@]}"
TASK7_COMPLETE_RC=$?
(( TASK7_COMPLETE_ERREXIT == 0 )) || set -e
printf 'TASK7_COMPLETE_RC=%d\n' "$TASK7_COMPLETE_RC"

Require OBSERVABILITY ACCESS STATIC RENDER PASS and TASK7_COMPLETE_RC=0; verify seven YAML outputs and original handoff fingerprints unchanged. Do not invoke the complete renderer again.

Step 4: Run the remaining regressions once each

The remaining set has exactly five commands; neither renderer from Step 3 is a member. Build each absolute argv and give every invocation its own errexit save/set +e/immediate RC/restore envelope:

TASK7_ADMIN_UI=(/usr/bin/bash)
TASK7_ADMIN_UI+=(/home/donghyeon/workspace/platform/scripts/validate/admin-ui-smoke.sh)
[[ "${#TASK7_ADMIN_UI[@]}" -eq 2 ]]
case $- in *e*) TASK7_ADMIN_UI_ERREXIT=1 ;; *) TASK7_ADMIN_UI_ERREXIT=0 ;; esac
set +e
"${TASK7_ADMIN_UI[@]}"
TASK7_ADMIN_UI_RC=$?
(( TASK7_ADMIN_UI_ERREXIT == 0 )) || set -e
printf 'TASK7_ADMIN_UI_RC=%d\n' "$TASK7_ADMIN_UI_RC"

TASK7_AISTOR_S3=(/usr/bin/bash)
TASK7_AISTOR_S3+=(/home/donghyeon/workspace/platform/scripts/validate/aistor-s3-smoke.sh)
TASK7_AISTOR_S3+=(--execute)
[[ "${#TASK7_AISTOR_S3[@]}" -eq 3 ]]
case $- in *e*) TASK7_AISTOR_S3_ERREXIT=1 ;; *) TASK7_AISTOR_S3_ERREXIT=0 ;; esac
set +e
"${TASK7_AISTOR_S3[@]}"
TASK7_AISTOR_S3_RC=$?
(( TASK7_AISTOR_S3_ERREXIT == 0 )) || set -e
printf 'TASK7_AISTOR_S3_RC=%d\n' "$TASK7_AISTOR_S3_RC"

TASK7_PHASE1=(/usr/bin/bash)
TASK7_PHASE1+=(/home/donghyeon/workspace/platform/scripts/validate/render-phase1.sh)
[[ "${#TASK7_PHASE1[@]}" -eq 2 ]]
case $- in *e*) TASK7_PHASE1_ERREXIT=1 ;; *) TASK7_PHASE1_ERREXIT=0 ;; esac
set +e
PLATFORM_HELM_BIN=/home/donghyeon/.local/bin/helm "${TASK7_PHASE1[@]}"
TASK7_PHASE1_RC=$?
(( TASK7_PHASE1_ERREXIT == 0 )) || set -e
printf 'TASK7_PHASE1_RC=%d\n' "$TASK7_PHASE1_RC"

TASK7_PHASE2=(/usr/bin/bash)
TASK7_PHASE2+=(/home/donghyeon/workspace/platform/scripts/validate/render-phase2.sh)
[[ "${#TASK7_PHASE2[@]}" -eq 2 ]]
case $- in *e*) TASK7_PHASE2_ERREXIT=1 ;; *) TASK7_PHASE2_ERREXIT=0 ;; esac
set +e
PLATFORM_HELM_BIN=/home/donghyeon/.local/bin/helm "${TASK7_PHASE2[@]}"
TASK7_PHASE2_RC=$?
(( TASK7_PHASE2_ERREXIT == 0 )) || set -e
printf 'TASK7_PHASE2_RC=%d\n' "$TASK7_PHASE2_RC"

TASK7_ADMIN_RENDER=(/usr/bin/bash)
TASK7_ADMIN_RENDER+=(/home/donghyeon/workspace/platform/scripts/validate/render-admin-services.sh)
[[ "${#TASK7_ADMIN_RENDER[@]}" -eq 2 ]]
case $- in *e*) TASK7_ADMIN_RENDER_ERREXIT=1 ;; *) TASK7_ADMIN_RENDER_ERREXIT=0 ;; esac
set +e
PLATFORM_HELM_BIN=/home/donghyeon/.local/bin/helm "${TASK7_ADMIN_RENDER[@]}"
TASK7_ADMIN_RENDER_RC=$?
(( TASK7_ADMIN_RENDER_ERREXIT == 0 )) || set -e
printf 'TASK7_ADMIN_RENDER_RC=%d\n' "$TASK7_ADMIN_RENDER_RC"

The fixed operator PATH excludes /home/donghyeon/.local/bin, and both phase renderers fall back to command -v helm when PLATFORM_HELM_BIN is unset. Therefore their exact inline assignments above are mandatory and PATH fallback is forbidden. Before execution, run the Task 4 fenced-Bash structural checker against both authority documents. Require its current-document RC 0, then require nonzero RC for four identity-checked private mutations that respectively remove the phase1 assignment, remove the phase2 assignment, replace the phase1 value, or replace the phase2 value. Exercise each mutation in both document positions with the other document unchanged; require all eight final gates nonzero. Remove only those private copies.

Do not run nonexistent observability-core-smoke.sh. Do not run observability-smoke.sh --execute again. Require every RC 0, expected terminal PASS, and no new residue.

Step 5: Preserve regression evidence through closeout

Preserve the original Task 6 handoff, fresh successful Task 7 renderer root, scanner root, rollback root, terminal status/logs, and name-only residue baseline through Task 9 review. Cleanup belongs to the later Task 8 final-closeout plan, not this task.

Task 9: Freeze terminal results, update status truthfully, and review Task 7 completion

Files:

Modify after observed results: bootstrap/manual/phase4-observability-access.md
Modify after observed results: /home/donghyeon/workspace/docs/platform/plans/2026-08-14-observability-slack-risk-acceptance-implementation.md
Append after observed results: /home/donghyeon/workspace/docs/platform/runbooks/2026-08-13-observability-phase4-resume-worklog.md
Create: .superpowers/sdd/2026-08-15-observability-task7-cutover-correction/baseline/task-7-final-source-manifest.txt
Create: .superpowers/sdd/2026-08-15-observability-task7-cutover-correction/baseline/task-7-final-source-manifest.sha256
Create: .superpowers/sdd/2026-08-15-observability-task7-cutover-correction/task-9-report.md

Interfaces:

Consumes: exact terminal RCs/markers and preserved pre-live authority.
Produces: evidence-backed Task 7 completion or an explicit partial/failed state; Task 8 remains separate.
Step 1: Write only observed terminal facts

Record Host dry/execute markers and RC, active full hash, rollback invocation state, OIDC transaction/rollback RC, one smoke marker set and RC, external-client result, renderer/platform regression RCs, original handoff preservation, fresh renderer root, temporary-object cleanup, and residue delta. Do not record usernames, external client address beyond the safe public edge already printed by smoke, Secret-derived data, or response bodies.
Step 2: Update Task 7 checkboxes conditionally

Mark a Step 1–6 checkbox only when its exact evidence exists. Mark all Task 7 steps complete only if Tasks 6–8 of this plan are RC 0 with every required marker and no unresolved cleanup/recovery. Otherwise leave failed/unrun steps unchecked and state 부분 구현 with the exact blocker. Do not mark any Task 8 checkbox.
Step 3: State the remaining Slack DR exception

Even after Task 7 PASS, retain:
```
Phase 4 operational acceptance: complete with active Slack DR exception
Slack off-host disaster recovery: deferred / not complete
```
Never convert RISK_ACCEPTED into recovery-complete language.
Step 4: Regenerate and strictly verify the final source manifest

Regenerate the same exact 21 paths in the same order after terminal documentation changes, bind the final sidecar, and independently review the new literal final manifest SHA. Regenerate and compare the command inventory first with the same exact-once <reviewed-manifest-sha256> sentinel; reject any actual manifest digest in its verifier row. Only after the final manifest is complete may the non-governed task-9 report/review handoff pin its actual digest. Re-pin the still-excluded verifier's canonical absolute path, SHA, type, owner, mode, nlink, and size. Run the descriptor-retaining launcher twice using literal /usr/bin/python3, the literal absolute verifier, the reviewed literal SHA, and the literal absolute final manifest/sidecar pair; require RC 0 plus unchanged descriptor/path identities before and after each invocation. Record prelive/final manifest and sidecar file hashes separately; do not overwrite or delete the prelive pair.
Step 5: Final independent Task 7 review

Verify design/spec coverage, source quality, RED/GREEN/mutation provenance, fixed command boundary, v2 exact tuple, proof age/ID, prompt-before empty Host residue, Host state/rollback classification, OIDC truth, exactly one smoke per token/argv structural document assertion and exactly one live smoke, public/Tailscale/external evidence, 30/30 targets, NodePort boundary, exactly one core plus one complete renderer, exact phase1/phase2 Helm assignments with structural mutation rejection, clone/original preservation, every regression RC, self-reference-free 21-row manifest/sidecar binding and external digest pin, separate verifier pin/rebind defense, mutator/process/temp residue, and accurate docs/check boxes.

Required terminal verdicts:
```
Critical=0
Important=0
Spec=YES
Quality=APPROVED
Task7=COMPLETE
ReadyToStartTask8=YES
```
Any Critical/Important finding or missing evidence returns to the responsible task; do not claim completion because the day ended or because only server-side checks passed.

Execution Handoff

Plan implementation must use either superpowers:subagent-driven-development with a fresh implementer and spec/quality review per task, or superpowers:executing-plans with explicit checkpoints. The recommended route is subagent-driven development because Tasks 1–5 are independently reviewable before any live mutation, while Tasks 6–9 require controller/operator coordination.

68 KiB Raw Blame History Unescape Escape

Observability Task 7 Cutover Correction Implementation Plan

Global Constraints

Task 1: Make the Host acceptance fixture production-faithful

Task 2: Enforce first-cutover state, fixed command boundaries, and no-contact dry-run

Task 3: Prove the fresh renderer-root preservation boundary

Task 4: Correct the authoritative Task 7 procedure without claiming live completion

Task 5: Run bounded verification and create the Task 7 non-Git source freeze

Task 6: Execute the single Host Nginx full cutover

Task 7: Reconcile memberships and run the monolithic acceptance once

Task 8: Run complete rendering from a fresh clone and existing regressions once

Task 9: Freeze terminal results, update status truthfully, and review Task 7 completion

Execution Handoff

68 KiB

Raw Blame History