project-infra/docs/superpowers/plans/2026-08-02-project-infra-phase-1-dev-gitops.md

# Project Infra Phase 1 dev GitOps Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use `superpowers:subagent-driven-development` (recommended) or `superpowers:executing-plans` to execute this plan task by task. Use `superpowers:test-driven-development` for behavior changes and `superpowers:verification-before-completion` before reporting success.

**Goal:** Rename the active rollout from `lab` to `dev`, make operator desired state render from Git, and correct the ForwardAuth/Keycloak ownership defects without changing a live cluster.

**Architecture:** The catalog remains ownership-first and environment-neutral in `base`, while `overlays/dev` contains dev-only values. `gitops/clusters/dev/main` is the only current deployable Kubernetes root. A shared render function always enables Kustomize Helm support. Operator charts live in independent platform units and are selected by a dedicated ordered stage. This plan only changes Git-owned source and local tests; it must not invoke `kubectl apply`, `kubectl delete`, or Helm mutation commands.

**Tech Stack:** Bash, Kustomize 5.8.1, Helm 3.19.4, Kubernetes/K3s manifests, repository shell test harness, `rg`.

## Global constraints

- Read `/AGENTS.md`, the nearest nested `AGENTS.md`, `docs/standards/infra/STYLE.md`, and the directly relevant standards before each manifest task.
- Preserve `mnt`, `minio-operator`, and `vault-secrets-operator-system` namespace names in Phase 1.
- Preserve `.local.test` hostnames as dev-only values.
- Never apply `gitops/clusters/dev/main/all`; it is render/audit-only.
- Do not run a live-cluster mutation command while executing this plan.
- Do not edit CI workflow definitions or the sibling `cicd-platform` repository.
- Commit only the files listed by the current task and inspect `git status --short` before each commit.

## Dependency and interface contract

This plan is executed before the automation-safety and validation-structure plans.

- `tests/run.sh [name-fragment]` discovers executable `tests/contracts/*-test.sh` files and optionally filters by basename.
- `tests/lib/assert.sh` provides `fail`, `assert_eq`, `assert_file_exists`, `assert_file_absent`, `assert_contains`, and `assert_not_contains`.
- `scripts/lib/kustomize.sh` exports `kustomize_render ENTRYPOINT`; rendered YAML is written only to stdout and diagnostics only to stderr.
- Every later render path must call `kustomize_render`; no later script may reimplement `kustomize build` or `kubectl kustomize`.
- The ordered dev stages are `00-platform`, `05-operators`, `10-vault`, `20-secrets`, `30-data`, `35-registry`, `40-operations`, and `50-apps`.

---

## Task 1: Add the contract-test harness and rename the active environment

**Files:**

- Create: `tests/lib/assert.sh`
- Create: `tests/run.sh`
- Create: `tests/contracts/dev-layout-test.sh`
- Move: `gitops/clusters/lab/main` → `gitops/clusters/dev/main`
- Move: `gitops/apps/auth-migration/overlays/lab` → `gitops/apps/auth-migration/overlays/dev`
- Move: `gitops/apps/auth-server/overlays/lab` → `gitops/apps/auth-server/overlays/dev`
- Move: `gitops/apps/identity-postgres/overlays/lab` → `gitops/apps/identity-postgres/overlays/dev`
- Move: `gitops/apps/keycloak-realm-import/overlays/lab` → `gitops/apps/keycloak-realm-import/overlays/dev`
- Move: `gitops/platform/cert-manager/overlays/lab` → `gitops/platform/cert-manager/overlays/dev`
- Move: `gitops/platform/forward-auth/overlays/lab` → `gitops/platform/forward-auth/overlays/dev`
- Move: `gitops/platform/keycloak-operator/overlays/lab` → `gitops/platform/keycloak-operator/overlays/dev`
- Move: `gitops/platform/keycloak/overlays/lab` → `gitops/platform/keycloak/overlays/dev`
- Move: `gitops/platform/minio/overlays/lab` → `gitops/platform/minio/overlays/dev`
- Move: `gitops/platform/registry/overlays/lab` → `gitops/platform/registry/overlays/dev`
- Move: `gitops/platform/secret-delivery/overlays/lab` → `gitops/platform/secret-delivery/overlays/dev`
- Move: `gitops/platform/traefik/overlays/lab` → `gitops/platform/traefik/overlays/dev`
- Move: `gitops/platform/vault/overlays/lab` → `gitops/platform/vault/overlays/dev`
- Move: `gitops/policies/baseline/overlays/lab` → `gitops/policies/baseline/overlays/dev`
- Modify: moved Kustomization files, `tests/kustomize-entrypoints.txt`
- Modify: `AGENTS.md`, `README.md`, `guide.md`, `gitops/AGENTS.md`, `gitops/PROJECT.md`, `gitops/clusters/AGENTS.md`, `gitops/clusters/README.md`, `gitops/clusters/dev/main/README.md`, `docs/architecture.md`, `docs/architecture/repository-structure.md`, `docs/guides/getting-started.md`, `docs/ingress-traefik.md`, `docs/operations.md`, `docs/security-hardening.md`, `docs/troubleshooting.md`, `docs/vault-vso.md`, `scripts/bin/bootstrap.sh`, `scripts/bin/teardown.sh`, `scripts/tasks/minio-provision-registry.sh`, `scripts/ci/validate-docs.sh`, `scripts/README.md`, `tests/README.md`

**Step 1: Write the failing layout contract**

Use strict mode in every new script. The runner must execute each selected test in a fresh Bash process and return non-zero if any test fails:

```bash
#!/usr/bin/env bash
set -Eeuo pipefail
IFS=$'\n\t'

ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
filter="${1:-}"
failed=0

while IFS= read -r test_file; do
  [[ -z "$filter" || "$(basename "$test_file")" == *"$filter"* ]] || continue
  bash "$test_file" || failed=$((failed + 1))
done < <(find "$ROOT_DIR/tests/contracts" -type f -name '*-test.sh' -print | sort)

((failed == 0))
```

`dev-layout-test.sh` must assert all of these conditions:

```text
gitops/clusters/dev/main exists
gitops/clusters/lab does not exist
every selected unit has overlays/dev and no overlays/lab
tests/kustomize-entrypoints.txt contains gitops/clusters/dev/main
tests/kustomize-entrypoints.txt does not contain gitops/clusters/lab
no deployable kustomization contains example.com/environment: lab
the active docs listed above do not instruct users to deploy lab
```

**Step 2: Run the contract and verify that it fails for the current tree**

Run:

```bash
bash tests/run.sh dev-layout
```

Expected: non-zero, with the first failure reporting the missing `gitops/clusters/dev/main` path.

**Step 3: Move the directories with Git-aware renames**

Run one `git mv` per source directory. Do not use recursive copy/delete. After the moves, replace only active environment references:

```bash
rg -l 'clusters/lab|overlays/lab|environment: lab|\blab\b' \
  AGENTS.md README.md guide.md gitops scripts docs tests \
  --glob '!docs/standards/**' \
  --glob '!docs/examples/**' \
  --glob '!docs/superpowers/**'
```

Review every hit and change current deployment semantics from `lab` to `dev`, including bootstrap/teardown accepted environment names and MinIO's dev-only HTTP default. Preserve historical text in approved design/spec documents and normative examples that intentionally compare environments. Change `gitops/AGENTS.md` base reuse wording to name `dev`, `staging`, and `prod`; it may still describe `lab` only as an optional disposable environment.

**Step 4: Update the moved cluster root**

All resource paths under `gitops/clusters/dev/main/stages/*/kustomization.yaml` must select `overlays/dev`. Every stage label must be:

```yaml
labels:
  - pairs:
      example.com/environment: dev
      example.com/owner-team: platform
    includeSelectors: false
    includeTemplates: true
```

Remove `../../../../../tenants/mnt/base` from `stages/00-platform`; namespace ownership belongs only to the separate `namespaces` entrypoint.

Update `tests/kustomize-entrypoints.txt` to contain only dev cluster entrypoints. Keep `all/` in the render inventory, but mark it audit-only in the cluster README and tests README.

**Step 5: Run the test and documentation gate**

Run:

```bash
bash tests/run.sh dev-layout
bash scripts/ci/validate-docs.sh
git diff --check
```

Expected: all commands exit 0.

**Step 6: Commit**

```bash
git add tests gitops AGENTS.md README.md guide.md docs scripts/bin/bootstrap.sh scripts/bin/teardown.sh scripts/tasks/minio-provision-registry.sh scripts/ci/validate-docs.sh scripts/README.md
git commit -m "refactor: dev 환경 경로로 전환"
```

---

## Task 2: Introduce one Helm-aware Kustomize render interface

**Files:**

- Create: `scripts/lib/kustomize.sh`
- Create: `tests/contracts/kustomize-render-test.sh`
- Modify: `scripts/ci/validate.sh`
- Modify: `scripts/lib/common.sh` only if it is needed to share `REPO_ROOT` without changing cluster behavior

**Step 1: Write the failing fake-command test**

The test creates a temporary `PATH` with fake `kustomize` and `kubectl` executables. It must prove:

1. when `kustomize` exists, the helper calls exactly `kustomize build --enable-helm ENTRYPOINT`;
2. when only `kubectl` exists, it calls exactly `kubectl kustomize ENTRYPOINT --enable-helm`;
3. when neither exists, it exits non-zero;
4. rendered YAML remains on stdout and the command trace remains outside stdout.

Run:

```bash
bash tests/run.sh kustomize-render
```

Expected: non-zero because `scripts/lib/kustomize.sh` does not yet exist.

**Step 2: Implement the shared helper**

Use this public interface:

```bash
#!/usr/bin/env bash
set -Eeuo pipefail
IFS=$'\n\t'

kustomize_render() {
  local entrypoint="${1:?entrypoint is required}"
  if command -v kustomize >/dev/null 2>&1; then
    kustomize build --enable-helm "$entrypoint"
  elif command -v kubectl >/dev/null 2>&1; then
    kubectl kustomize "$entrypoint" --enable-helm
  else
    printf 'ERROR: kustomize 또는 kubectl이 필요합니다.\n' >&2
    return 1
  fi
}
```

Source this file from `scripts/ci/validate.sh` and delete its local `render_kustomization` implementation. Change the validation call to:

```bash
kustomize_render "$entrypoint" >"$rendered"
```

**Step 3: Verify the helper and current dev renders**

Run:

```bash
bash tests/run.sh kustomize-render
VALIDATION_PROFILE=local bash scripts/ci/validate.sh
git diff --check
```

Expected: all commands exit 0; render log lines identify dev entrypoints.

**Step 4: Commit**

```bash
git add scripts/lib/kustomize.sh scripts/ci/validate.sh tests/contracts/kustomize-render-test.sh
git commit -m "refactor: Helm 지원 렌더 경로 통합"
```

---

## Task 3: Add declarative operator namespaces and catalog units

**Files:**

- Create: `gitops/tenants/minio-operator/base/kustomization.yaml`
- Create: `gitops/tenants/minio-operator/base/namespace.yaml`
- Create: `gitops/tenants/vault-secrets-operator-system/base/kustomization.yaml`
- Create: `gitops/tenants/vault-secrets-operator-system/base/namespace.yaml`
- Create: `gitops/platform/minio-operator/base/kustomization.yaml`
- Create: `gitops/platform/minio-operator/overlays/dev/kustomization.yaml`
- Create: `gitops/platform/minio-operator/overlays/dev/values.yaml`
- Create: `gitops/platform/vault-secrets-operator/base/kustomization.yaml`
- Create: `gitops/platform/vault-secrets-operator/overlays/dev/kustomization.yaml`
- Move: `gitops/platform/secret-delivery/base/helm/values.yaml` → `gitops/platform/vault-secrets-operator/overlays/dev/values.yaml`
- Create: `gitops/clusters/dev/main/stages/05-operators/kustomization.yaml`
- Modify: `gitops/clusters/dev/main/namespaces/kustomization.yaml`
- Modify: `gitops/clusters/dev/main/all/kustomization.yaml`
- Modify: `tests/kustomize-entrypoints.txt`
- Create: `tests/contracts/operator-render-test.sh`
- Modify: `gitops/platform/README.md`, `gitops/tenants/README.md`, `gitops/clusters/dev/main/README.md`

**Step 1: Write the failing operator render contract**

Render `gitops/clusters/dev/main/stages/05-operators` through `kustomize_render`. Assert:

```text
the render exits 0
at least two Deployment documents exist
tenants.minio.min.io exists as a CustomResourceDefinition name
vaultstaticsecrets.secrets.hashicorp.com exists as a CustomResourceDefinition name
the namespaces entrypoint renders Namespace/minio-operator and Namespace/vault-secrets-operator-system
every namespaced resource in the operator stage targets minio-operator or vault-secrets-operator-system
no HelmRelease custom resource is present
chart versions 7.0.0 and 0.9.0 are pinned in source
includeCRDs: true is present in both dev overlay kustomizations
```

Run:

```bash
bash tests/run.sh operator-render
```

Expected: non-zero because the operator stage does not exist.

**Step 2: Add namespace ownership units**

Each Namespace must use the standard labels from `gitops/tenants/mnt/base/namespace.yaml`, with `app.kubernetes.io/name` and `app.kubernetes.io/instance` set to its namespace. Apply the `restricted` Pod Security labels. Each base Kustomization contains only its `namespace.yaml`.

Add both bases to `gitops/clusters/dev/main/namespaces/kustomization.yaml` after `mnt`.

**Step 3: Add the MinIO Operator unit**

The dev overlay Kustomization must be exactly shaped as follows:

```yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - ../../base

helmCharts:
  - name: operator
    repo: https://operator.min.io
    version: 7.0.0
    releaseName: minio-operator
    namespace: minio-operator
    includeCRDs: true
    valuesFile: values.yaml
```

Use only chart-supported keys from the pinned MinIO Operator 7.0.0 values schema. `values.yaml` must be:

```yaml
operator:
  replicaCount: 1
  securityContext:
    runAsUser: 1000
    runAsGroup: 1000
    runAsNonRoot: true
    fsGroup: 1000
    seccompProfile:
      type: RuntimeDefault
  containerSecurityContext:
    runAsUser: 1000
    runAsGroup: 1000
    runAsNonRoot: true
    allowPrivilegeEscalation: false
    readOnlyRootFilesystem: true
    capabilities:
      drop:
        - ALL
    seccompProfile:
      type: RuntimeDefault
  resources:
    requests:
      cpu: 200m
      memory: 256Mi
      ephemeral-storage: 500Mi
    limits:
      memory: 384Mi
      ephemeral-storage: 1Gi
```

The single replica is a dev overlay decision. A later staging/prod overlay must reassess replicas, PDB, and topology from measured capacity.

**Step 4: Add the Vault Secrets Operator unit**

The dev overlay Kustomization must pin:

```yaml
helmCharts:
  - name: vault-secrets-operator
    repo: https://helm.releases.hashicorp.com
    version: 0.9.0
    releaseName: vault-secrets-operator
    namespace: vault-secrets-operator-system
    includeCRDs: true
    valuesFile: values.yaml
```

Move the existing hardened VSO values file unchanged first. Remove the now-empty `gitops/platform/secret-delivery/base/helm` directory from Git. Keep secret-delivery CRs in their existing unit; only the controller chart moves.

**Step 5: Add the ordered stage and inventory**

`gitops/clusters/dev/main/stages/05-operators/kustomization.yaml` contains exactly the two dev overlays and the normal dev/platform labels. Insert it between `00-platform` and `10-vault` in `all/kustomization.yaml` and in `tests/kustomize-entrypoints.txt`.

Do not add these operator overlays to `00-platform`.

**Step 6: Verify render, schema, and policy**

Run:

```bash
bash tests/run.sh operator-render
VALIDATION_PROFILE=full bash scripts/ci/validate.sh
git diff --check
```

Expected: operator test exits 0; full validation reports no invalid schema or policy result. If chart download is unavailable, stop and report the external availability failure rather than committing an unrendered chart definition.

**Step 7: Commit**

```bash
git add gitops/tenants gitops/platform/minio-operator gitops/platform/vault-secrets-operator gitops/platform/secret-delivery gitops/clusters/dev/main tests
git commit -m "feat: operator desired state를 GitOps로 선언"
```

---

## Task 4: Put ForwardAuth resources and patches under their real owners

**Files:**

- Move: `gitops/platform/forward-auth/component/oauth2-proxy-config.yaml` → `gitops/platform/forward-auth/overlays/dev/oauth2-proxy-config.yaml`
- Move: `gitops/platform/forward-auth/component/oauth2-proxy-ingress.yaml` → `gitops/platform/forward-auth/overlays/dev/oauth2-proxy-ingress.yaml`
- Move: `gitops/platform/forward-auth/component/oauth2-proxy-middleware.yaml` → `gitops/platform/forward-auth/overlays/dev/oauth2-proxy-middleware.yaml`
- Move: `gitops/platform/forward-auth/component/oauth2-proxy-networkpolicy.yaml` → `gitops/platform/forward-auth/overlays/dev/oauth2-proxy-networkpolicy.yaml`
- Delete: `gitops/platform/forward-auth/component/kustomization.yaml`
- Modify: `gitops/platform/forward-auth/overlays/dev/kustomization.yaml`
- Modify: `gitops/apps/auth-server/base/configmap.yaml`
- Modify: `gitops/apps/auth-server/overlays/dev/kustomization.yaml`
- Modify: `gitops/platform/keycloak/overlays/dev/kustomization.yaml`
- Delete: `gitops/platform/keycloak/overlays/dev/ingress-admin.yaml`
- Create: `tests/contracts/dev-ingress-render-test.sh`

**Step 1: Write the failing rendered-output assertions**

Render `gitops/clusters/dev/main/stages/50-apps` and `00-platform`. Extract resources by `kind` and `metadata.name` without assuming document order. Assert:

```text
Ingress/auth-server annotation is exactly:
mnt-oauth2-proxy-errors@kubernetescrd,mnt-oauth2-proxy-auth@kubernetescrd,kube-system-security-headers@kubernetescrd
Ingress/keycloak exists
Ingress/keycloak-admin does not exist
ConfigMap/auth-server-config in the app stage contains SPRING_PROFILES_ACTIVE=dev
ConfigMap/auth-server-config rendered from gitops/apps/auth-server/base does not contain SPRING_PROFILES_ACTIVE
the 00-platform render contains oauth2-proxy ConfigMap, Ingress, Middleware resources, and NetworkPolicy
```

Run:

```bash
bash tests/run.sh dev-ingress-render
```

Expected: non-zero because the current auth-server annotation has only the security middleware.

**Step 2: Refactor the ForwardAuth overlay**

Change the dev overlay from a Component reference to normal ownership:

```yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: mnt

resources:
  - ../../base
  - oauth2-proxy-config.yaml
  - oauth2-proxy-ingress.yaml
  - oauth2-proxy-middleware.yaml
  - oauth2-proxy-networkpolicy.yaml
```

Delete the component directory after all resources are moved. The platform stage continues to select only `forward-auth/overlays/dev`.

**Step 3: Move the auth-server patch to the auth-server owner**

Add this patch to `gitops/apps/auth-server/overlays/dev/kustomization.yaml`:

```yaml
  - target:
      kind: Ingress
      name: auth-server
    patch: |-
      - op: replace
        path: /metadata/annotations/traefik.ingress.kubernetes.io~1router.middlewares
        value: mnt-oauth2-proxy-errors@kubernetescrd,mnt-oauth2-proxy-auth@kubernetescrd,kube-system-security-headers@kubernetescrd
```

Remove `SPRING_PROFILES_ACTIVE` from the base ConfigMap and add it through the existing dev ConfigMap patch.

**Step 4: Remove the dev admin Ingress from desired state**

Remove `ingress-admin.yaml` from the Keycloak dev Kustomization and delete the file. Do not run `kubectl delete`; live cleanup belongs to Phase 2.

**Step 5: Verify exact rendered behavior**

Run:

```bash
bash tests/run.sh dev-ingress-render
VALIDATION_PROFILE=full bash scripts/ci/validate.sh
git diff --check
```

Expected: all commands exit 0.

**Step 6: Commit**

```bash
git add gitops/platform/forward-auth gitops/apps/auth-server gitops/platform/keycloak tests/contracts/dev-ingress-render-test.sh
git commit -m "fix: dev ForwardAuth 소유 경계 수정"
```

---

## Task 5: Record the environment and CI ownership decisions

**Files:**

- Create: `docs/adr/0002-dev-environment-and-external-ci-ownership.md`
- Modify: `docs/architecture/repository-structure.md`
- Modify: `README.md`
- Modify: `docs/superpowers/specs/2026-08-02-project-infra-dev-structure-refactor-design.md`
- Modify: `scripts/ci/validate-docs.sh`

**Step 1: Extend the docs contract so it fails first**

Add assertions to `scripts/ci/validate-docs.sh` requiring the ADR to contain all of these literals:

```text
Status: Accepted
gitops/clusters/dev/main
cicd-platform
delivery-platform.yaml
all/ is render/audit-only
```

Run:

```bash
bash scripts/ci/validate-docs.sh
```

Expected: non-zero because ADR 0002 does not exist.

**Step 2: Write the ADR**

Use sections `Context`, `Decision`, `Consequences`, and `Phase 2 follow-up`. Record:

- current rollout environment is `dev`, not `lab`;
- dev is controller-less and uses ordered bootstrap stages;
- `all/ is render/audit-only` and is never an apply target;
- project-infra owns local validation contracts;
- `cicd-platform` owns workflow execution, authoritative tools, evidence, and the future `delivery-platform.yaml` consumer onboarding;
- existing workflows remain transitional and are not changed in Phase 1.

Verify that the approved design document remains marked `Status: approved`.

**Step 3: Verify documentation**

Run:

```bash
bash scripts/ci/validate-docs.sh
git diff --check
```

Expected: both commands exit 0.

**Step 4: Commit**

```bash
git add docs/adr docs/architecture/repository-structure.md README.md docs/superpowers/specs scripts/ci/validate-docs.sh
git commit -m "docs: dev 및 CI 소유권 결정 기록"
```

---

## Task 6: Plan-level verification checkpoint

**Files:** No source changes expected.

**Step 1: Prove no live mutation was introduced by this plan**

Run:

```bash
rg -n 'kubectl[[:space:]]+(apply|delete|replace|patch)|helm[[:space:]]+(install|upgrade|uninstall)' tests scripts/ci scripts/lib
```

Expected: no new invocation in files created or modified by this plan. Existing task/bootstrap matches are handled by the next plan.

**Step 2: Run the complete plan contract**

Run:

```bash
bash tests/run.sh
VALIDATION_PROFILE=full bash scripts/ci/validate.sh
bash scripts/ci/validate-docs.sh
git diff --check
git status --short
```

Expected:

- tests and validation exit 0;
- every inventory entrypoint renders with Helm enabled;
- no active `lab` cluster or overlay path remains;
- `git diff --check` emits nothing;
- status contains no unrelated files.

**Step 3: Inspect commits**

```bash
git log --oneline --decorate -7
```

Confirm that each task is independently reviewable. Do not squash before the user reviews the phase.