project-infra/docs/superpowers/specs/2026-08-02-project-infra-dev-structure-refactor-design.md

# Project Infra dev structure refactor design

Date: 2026-08-02

Status: proposed for written-spec review

Target repository: `project-infra`

## Context

The repository review found that the directory model is mostly coherent, but
several live paths and helper scripts do not follow the ownership rules declared
by `AGENTS.md` and `docs/standards/infra/`.

The most important correction is environmental: the currently deployed tree is
development, not a disposable lab. The existing `gitops/clusters/lab/main` and
referenced `overlays/lab` paths therefore misrepresent the intended environment.

The work is intentionally divided into two phases:

1. Phase 1 changes Git-owned source, local validation, and helper-script safety.
   It does not mutate a live cluster, migrate stateful data, or change the Vault
   runtime architecture.
2. Phase 2 performs runtime adoption and stateful architecture changes with
   explicit diff, rollback, and data-preservation procedures.

CI execution is owned by the sibling `cicd-platform` repository. This repository
owns only its project-specific validation contract and tests.

## Goals

- Make `dev` the only current Kubernetes rollout environment.
- Restore catalog, overlay, cluster-entrypoint, and script ownership boundaries.
- Move MinIO Operator and Vault Secrets Operator desired state from installation
  scripts into Kustomize-rendered Git source.
- Correct the ineffective ForwardAuth composition and remove the unprotected
  Keycloak admin Ingress from dev desired state.
- Make bootstrap, Vault initialization, and teardown fail closed.
- Add local behavioral and rendered-output contracts that prevent regressions.
- Remove misleading placeholders and relocate helper/document assets to their
  declared directory classes.

## Non-goals

Phase 1 does not:

- run `kubectl apply`, delete live resources, or uninstall existing Helm releases;
- create a real dev IaC root or choose Terraform versus OpenTofu;
- rename the live `mnt`, `minio-operator`, or
  `vault-secrets-operator-system` namespaces;
- replace the current Vault workload with the official chart;
- enable Vault TLS, Raft HA, auto-unseal, audit devices, or snapshot automation;
- edit the sibling `cicd-platform` repository or claim that its Kubernetes
  delivery capability is active;
- invent organization identifiers for `CODEOWNERS`.

## Ownership model

### Repository-local ownership

`project-infra` owns:

- Kubernetes and IaC desired state;
- the ordered dev deployment entrypoints;
- `make validate` and `make check` as project-specific validation commands;
- render contracts, script tests, and documentation checks;
- thin bootstrap, readiness, backup, restore, and teardown orchestration.

### CI platform ownership

`cicd-platform` owns:

- workflow triggers and the required consumer workflow;
- runners, toolchain supply and authoritative version locks;
- caching, required statuses, evidence, and result publication;
- the `delivery-platform.yaml` consumer contract and CI/CD capabilities.

The current `project-infra` GitHub and Gitea workflow files are treated as a
transitional integration. Phase 1 neither expands nor deletes them. The local
`.mise.toml` is a developer convenience, not the authoritative CI toolchain.
Central onboarding is a separate `cicd-platform` task because the inspected
`cd-gitops-kubernetes` capability is currently P0 and disabled, and no
`project-infra` consumer manifest exists there.

## Phase 1 design

### 1. Rename the current rollout from lab to dev

Move the active cluster root:

```text
gitops/clusters/lab/main
  -> gitops/clusters/dev/main
```

Move every overlay selected by that root from `overlays/lab` to `overlays/dev`.
Update cluster references, entrypoint inventory, operational commands,
environment labels, AGENTS files, README files, and runbooks accordingly.

The development hostnames ending in `.local.test` remain dev-only values. The
current `mnt` namespace remains unchanged until the Phase 2 namespace migration.
The existing `traefik-dev` instance label is correct once the environment is
properly named dev.

Dev retains the current controller-less exception: `scripts/bin/bootstrap.sh`
applies `namespaces` and ordered `stages/*`. The `all/` tree remains render/audit
only and must never be applied.

### 2. Represent operator desired state with Kustomize Helm rendering

Create independent catalog units for:

- MinIO Operator chart `7.0.0`;
- Vault Secrets Operator chart `0.9.0`.

Each dev overlay pins the chart repository, chart version, release name,
namespace, values file, and CRD inclusion through Kustomize `helmCharts`.
Existing namespace names are retained in Phase 1. Namespace objects are declared
under `gitops/tenants/` and selected by the dev namespaces entrypoint.

All repository render paths use a shared helper that invokes Kustomize with
`--enable-helm`. The catalog becomes the desired-state source; the existing
`minio-operator-install.sh` and `vso-install.sh` installation/cleanup logic is
removed.

Operator resources are isolated in an ordered operator stage rather than mixed
with unrelated platform resources. This is a safety refinement to the initial
`00-platform` proposal: an existing dev cluster currently has Helm-managed
operator resources, so normal bootstrap must not silently seize them.

The operator stage follows this transition contract:

- a new cluster with no matching operator resources may apply the Git-rendered
  stage after normal render, server dry-run, diff, and confirmation;
- a cluster with legacy Helm-managed operator resources is detected and left
  untouched unless an explicit adoption gate is supplied;
- Phase 1 tests the gate but does not exercise it against a live cluster;
- the actual Helm-to-Kustomize adoption and legacy release cleanup happen in
  Phase 2 under a dedicated runbook;
- after adoption, bootstrap applies the operator stage normally and waits for
  Deployments and required CRDs before continuing.

No dirty Helm release is automatically uninstalled.

### 3. Correct ForwardAuth and ingress ownership

The current ForwardAuth Kustomize Component tries to patch the sibling
`Ingress/auth-server`. Kustomize cannot apply a child component patch to a
resource assembled beside it by the cluster stage, so the rendered Ingress lacks
the oauth2-proxy middleware.

Refactor as follows:

- retain the environment-neutral oauth2-proxy Deployment, Service, and
  ServiceAccount in `forward-auth/base`;
- move the dev ConfigMap, Ingress, Middleware, and NetworkPolicy from the
  cross-unit component into `forward-auth/overlays/dev`;
- move the auth-server Ingress middleware patch into
  `auth-server/overlays/dev`, where the target Ingress is owned;
- assert the exact middleware chain in the rendered dev apps stage.

`auth.local.test`, internal Keycloak addresses, `cookie_secure=false`, and the
trusted proxy CIDR are permitted only in the dev overlay.

Remove `ingress-admin.yaml` from the Keycloak dev overlay. The public Ingress
remains. Removing the file from desired state does not delete an already-live
Ingress because Phase 1 performs no cluster mutation; live removal is a Phase 2
operation.

Move `SPRING_PROFILES_ACTIVE=dev` out of the auth-server base and into its dev
overlay.

### 4. Fail closed on Kubernetes target selection

Every cluster-mutating entrypoint requires `KUBE_CONTEXT` or an explicit
environment mapping such as `KUBE_CONTEXT_DEV`. Remove the interactive fallback
that accepts the current context after retyping it.

After validation, wrappers always pass the locked context explicitly to
`kubectl`. No command silently falls back to whichever context happens to be
current. Context mapping and the chosen environment remain separate inputs so
tests can prove both the missing-input and wrong-target cases.

### 5. Make Vault initialization recoverable and non-overwriting

The initialization material contract becomes:

- reject symlinks;
- require a parent directory that already exists;
- require mode `0600` for an existing file;
- validate required JSON fields before use;
- if Vault is uninitialized and the target file already exists, abort without
  changing either side;
- if Vault is initialized but required key material is missing or invalid, abort
  with recovery guidance;
- provide no automatic overwrite escape hatch.

The current `vault login` flow writes a token cache inside the Vault Pod. Replace
it with an authenticated execution wrapper that reads credentials through stdin,
exports them only to the child Vault process, and does not write `.vault-token`.

First initialization uses the root token only long enough to enable Kubernetes
auth and create a scoped `vault-bootstrap` policy and role. A dedicated
ServiceAccount then obtains a short-lived audience-bound TokenRequest JWT and
exchanges it for a short-lived Vault token. Repeatable policy, role, and seed
tasks use this path, so the root token may be moved offline and removed from the
JSON file without breaking later reconciliation. Ephemeral tokens are revoked or
allowed to expire after a short TTL and are never logged.

Move Vault policy bodies and role definitions out of shell heredocs into
versioned HCL/JSON under `bootstrap/foundation/vault/`. Scripts inject only the
explicit environment parameters and orchestrate application. Secret generation
and seeding remain stateful tasks rather than Kubernetes desired state.

### 6. Tighten bootstrap and teardown behavior

Every applied stage follows:

```text
render --enable-helm
  -> server-side dry-run
  -> diff
  -> confirmation
  -> server-side apply
  -> readiness boundary
```

`SKIP_DIFF=yes` is accepted only for dev. Staging and prod fail when it is set.

Teardown defaults to deleting `50-apps` only. One-shot operations require
`DELETE_OPERATIONS=yes`. Data, Vault, namespaces, platform controllers, and CRDs
remain progressively gated. CRD deletion requires an additional explicit gate
because deleting a CRD can delete every corresponding custom resource.

After operator desired state moves to Kustomize, teardown no longer performs
implicit Helm uninstalls. A legacy Helm release is handled only by the Phase 2
adoption/rollback runbook.

### 7. Strengthen repository-local validation

`make validate` remains the developer-friendly profile. `make check` runs the
repository's full local contract and fails when required validators are absent.
This distinction does not define how CI provisions tools; that is a
`cicd-platform` responsibility.

Use one Kustomize render helper everywhere and add rendered-output assertions:

- inventory and actual dev entrypoints agree;
- `_template` is not imported by a cluster root;
- `all/` is not referenced by apply code;
- deployable output has no `example.com/environment: lab` label;
- auth-server has the exact oauth2-proxy and security middleware chain;
- Keycloak admin Ingress is absent;
- the operator stage contains the expected Deployments and CRDs;
- no unresolved placeholder appears in deployable output.

Add Bash behavioral tests with fake commands and fixtures. They cover explicit
context requirements, key-file collision refusal, Vault state/file mismatch,
no token-cache writes, diff-skip gates, teardown scopes, CRD gates, operator
adoption gates, and the absence of direct Helm install/uninstall calls.

Documentation validation checks every YAML fence for syntax. Complete Kubernetes
examples outside documented bad-example sections also pass kubeconform and
kube-linter. Explicitly abbreviated snippets and headings named `나쁜 예시`, `❌`,
or `bad example` are excluded from schema/policy checks. Operational path checks,
credential-literal checks, and executable-bit checks remain deterministic.

The GitHub and Gitea workflow definitions are not changed in this phase.

### 8. Normalize repository structure and file modes

Move entrypoints to their declared classes:

```text
scripts/doctor.sh
  -> scripts/bin/doctor.sh

scripts/validate.sh
  -> scripts/ci/validate-structure.sh
```

Update the Makefile and docs. Entrypoints use strict mode, safe `IFS`, functions,
and `main "$@"`; user-facing commands have usage text. Kubernetes YAML and Helm
values files lose accidental executable bits and are stored as regular `0644`
source.

Remove non-runnable placeholder leaves:

- `infrastructure/components/{compute,database,networking,storage}`;
- `infrastructure/live/{dev,staging,prod}/cluster`.

Retain the parent documentation and `_template` roots. Correct copy examples so
an existing destination does not create a nested `_template` directory.

Move presentation material under `docs/presentation/`. Compute paths from the
script location, add build instructions, and pin the directly used Python
dependencies. Keep Draw.io sources and PNG inputs required to reproduce the
deck. Remove the generated `build/deck.pptx` from Git and ignore the build
directory.

Record the dev environment decision and the external CI ownership boundary in
an ADR. Keep `CODEOWNERS.example` until a real organization/team identifier is
provided.

## Phase 2 design boundary

Phase 2 starts only after Phase 1 source and tests are accepted. It requires a
new implementation plan because it changes live ownership and state.

It includes:

1. select Terraform or OpenTofu, provider, backend, locking, and credentials;
2. create the first real `infrastructure/live/dev` root;
3. migrate `mnt` and operator namespaces to the approved dev-prefixed model;
4. adopt existing Helm-managed MinIO/VSO resources into Kustomize ownership and
   remove legacy releases with a tested rollback path;
5. delete the existing live Keycloak admin Ingress after diff and confirmation;
6. replace the handmade Vault workload with the official chart;
7. implement TLS, Raft HA, auto-unseal, audit, backup, restore, and rollback;
8. coordinate central `project-infra` onboarding in `cicd-platform` before
   retiring transitional consumer workflows.

Phase 2 must define data backup, restore rehearsal, PVC retention, Vault seal
compatibility, resource adoption, and abort criteria before any apply.

## Error handling and recovery

- A missing explicit context is an input error, not an interactive prompt.
- A Vault state/key-file mismatch is a recovery condition, never permission to
  initialize or overwrite.
- Render, schema, policy, server dry-run, or diff errors stop before apply.
- Operator legacy ownership stops automatic adoption and points to the Phase 2
  runbook.
- Teardown gates are independent and additive; a broad flag does not silently
  imply CRD deletion.
- Phase 1 changes are recoverable through Git because it performs no external
  mutation.

## Verification and acceptance criteria

Phase 1 is complete only when:

- the worktree contains no active cluster or overlay path named `lab`;
- every declared dev entrypoint renders with Helm support;
- schema and policy validation passes for the aggregate and ordered stages;
- the rendered auth-server middleware and Keycloak ingress assertions pass;
- operator resources and CRDs render from Git-pinned chart definitions;
- all script behavioral tests pass without a real cluster;
- shell syntax, ShellCheck, shfmt, secret scanning, docs checks, and file-mode
  checks pass under the full local profile;
- no live cluster command was executed during Phase 1 implementation;
- documentation distinguishes repository-local validation from centrally owned
  CI execution;
- `git diff --check` is clean and the final diff contains no unrelated changes.

## Known risks

- Kustomize Helm rendering needs Helm and chart-repository access; pinned chart
  versions reduce but do not eliminate upstream availability risk.
- Existing Helm-managed resources cannot be assumed safe for SSA adoption; live
  ownership inspection is deferred to Phase 2.
- Renaming an overlay path does not remove old live resources. The Phase 2 diff
  and prune/delete procedure must identify leftovers explicitly.
- Vault bootstrap access is privileged. Its policy must be reviewed against the
  minimum API paths required by the checked-in definitions and seed tasks.
- Central CI coverage remains transitional until `cicd-platform` provides and
  activates an appropriate infrastructure validation capability.