project-infra/docs/superpowers/specs/2026-08-02-project-infra-dev-structure-refactor-design.md

Project Infra dev structure refactor design

Date: 2026-08-02

Status: approved

Target repository: project-infra

Context

The repository review found that the directory model is mostly coherent, but several live paths and helper scripts do not follow the ownership rules declared by AGENTS.md and docs/standards/infra/.

The most important correction is environmental: the currently deployed tree is development, not a disposable lab. The existing gitops/clusters/lab/main and referenced overlays/lab paths therefore misrepresent the intended environment.

The work is intentionally divided into two phases:

1. Phase 1 changes Git-owned source, local validation, and helper-script safety. It does not mutate a live cluster, migrate stateful data, or change the Vault runtime architecture.
2. Phase 2 performs runtime adoption and stateful architecture changes with explicit diff, rollback, and data-preservation procedures.

CI execution is owned by the sibling cicd-platform repository. This repository owns only its project-specific validation contract and tests.

Goals

• Make dev the only current Kubernetes rollout environment.
• Restore catalog, overlay, cluster-entrypoint, and script ownership boundaries.
• Move MinIO Operator and Vault Secrets Operator desired state from installation scripts into Kustomize-rendered Git source.
• Correct the ineffective ForwardAuth composition and remove the unprotected Keycloak admin Ingress from dev desired state.
• Make bootstrap, Vault initialization, and teardown fail closed.
• Add local behavioral and rendered-output contracts that prevent regressions.
• Remove misleading placeholders and relocate helper/document assets to their declared directory classes.

Non-goals

Phase 1 does not:

• run kubectl apply, delete live resources, or uninstall existing Helm releases;
• create a real dev IaC root or choose Terraform versus OpenTofu;
• rename the live mnt, minio-operator, or vault-secrets-operator-system namespaces;
• replace the current Vault workload with the official chart;
• enable Vault TLS, Raft HA, auto-unseal, audit devices, or snapshot automation;
• edit the sibling cicd-platform repository or claim that its Kubernetes delivery capability is active;
• invent organization identifiers for CODEOWNERS.

Ownership model

Repository-local ownership

project-infra owns:

• Kubernetes and IaC desired state;
• the ordered dev deployment entrypoints;
• make validate and make check as project-specific validation commands;
• render contracts, script tests, and documentation checks;
• thin bootstrap, readiness, backup, restore, and teardown orchestration.

CI platform ownership

cicd-platform owns:

• workflow triggers and the required consumer workflow;
• runners, toolchain supply and authoritative version locks;
• caching, required statuses, evidence, and result publication;
• the delivery-platform.yaml consumer contract and CI/CD capabilities.

The current project-infra GitHub and Gitea workflow files are treated as a transitional integration. Phase 1 neither expands nor deletes them. The local .mise.toml is a developer convenience, not the authoritative CI toolchain. Central onboarding is a separate cicd-platform task because the inspected cd-gitops-kubernetes capability is currently P0 and disabled, and no project-infra consumer manifest exists there.

Phase 1 design

1. Rename the current rollout from lab to dev

Move the active cluster root:

gitops/clusters/lab/main
  -> gitops/clusters/dev/main

Move every overlay selected by that root from overlays/lab to overlays/dev. Update cluster references, entrypoint inventory, operational commands, environment labels, AGENTS files, README files, and runbooks accordingly.

The development hostnames ending in .local.test remain dev-only values. The current mnt namespace remains unchanged until the Phase 2 namespace migration. The existing traefik-dev instance label is correct once the environment is properly named dev.

Dev retains the current controller-less exception: scripts/bin/bootstrap.sh applies namespaces and ordered stages/*. The all/ tree remains render/audit only and must never be applied.

2. Represent operator desired state with Kustomize Helm rendering

Create independent catalog units for:

• MinIO Operator chart 7.0.0;
• Vault Secrets Operator chart 0.9.0.

Each dev overlay pins the chart repository, chart version, release name, namespace, values file, and CRD inclusion through Kustomize helmCharts. Existing namespace names are retained in Phase 1. Namespace objects are declared under gitops/tenants/ and selected by the dev namespaces entrypoint.

All repository render paths use a shared helper that invokes Kustomize with --enable-helm. The catalog becomes the desired-state source; the existing minio-operator-install.sh and vso-install.sh installation/cleanup logic is removed.

Operator resources are isolated in an ordered operator stage rather than mixed with unrelated platform resources. This is a safety refinement to the initial 00-platform proposal: an existing dev cluster currently has Helm-managed operator resources, so normal bootstrap must not silently seize them.

The operator stage follows this transition contract:

• a new cluster with no matching operator resources may apply the Git-rendered stage after normal render, server dry-run, diff, and confirmation;
• a cluster with legacy Helm-managed operator resources is detected and left untouched unless an explicit adoption gate is supplied;
• Phase 1 tests the gate but does not exercise it against a live cluster;
• the actual Helm-to-Kustomize adoption and legacy release cleanup happen in Phase 2 under a dedicated runbook;
• after adoption, bootstrap applies the operator stage normally and waits for Deployments and required CRDs before continuing.

No dirty Helm release is automatically uninstalled.

3. Correct ForwardAuth and ingress ownership

The current ForwardAuth Kustomize Component tries to patch the sibling Ingress/auth-server. Kustomize cannot apply a child component patch to a resource assembled beside it by the cluster stage, so the rendered Ingress lacks the oauth2-proxy middleware.

Refactor as follows:

• retain the environment-neutral oauth2-proxy Deployment, Service, and ServiceAccount in forward-auth/base;
• move the dev ConfigMap, Ingress, Middleware, and NetworkPolicy from the cross-unit component into forward-auth/overlays/dev;
• move the auth-server Ingress middleware patch into auth-server/overlays/dev, where the target Ingress is owned;
• assert the exact middleware chain in the rendered dev apps stage.

auth.local.test, internal Keycloak addresses, cookie_secure=false, and the trusted proxy CIDR are permitted only in the dev overlay.

Remove ingress-admin.yaml from the Keycloak dev overlay. The public Ingress remains. Removing the file from desired state does not delete an already-live Ingress because Phase 1 performs no cluster mutation; live removal is a Phase 2 operation.

Move SPRING_PROFILES_ACTIVE=dev out of the auth-server base and into its dev overlay.

4. Fail closed on Kubernetes target selection

Every cluster-mutating entrypoint requires KUBE_CONTEXT or an explicit environment mapping such as KUBE_CONTEXT_DEV. Remove the interactive fallback that accepts the current context after retyping it.

After validation, wrappers always pass the locked context explicitly to kubectl. No command silently falls back to whichever context happens to be current. Context mapping and the chosen environment remain separate inputs so tests can prove both the missing-input and wrong-target cases.

5. Make Vault initialization recoverable and non-overwriting

The initialization material contract becomes:

• reject symlinks;
• require a parent directory that already exists;
• require mode 0600 for an existing file;
• validate required JSON fields before use;
• if Vault is uninitialized and the target file already exists, abort without changing either side;
• if Vault is initialized but required key material is missing or invalid, abort with recovery guidance;
• provide no automatic overwrite escape hatch.

The current vault login flow writes a token cache inside the Vault Pod. Replace it with an authenticated execution wrapper that reads credentials through stdin, exports them only to the child Vault process, and does not write .vault-token.

First initialization uses the root token only long enough to enable Kubernetes auth and create a scoped vault-bootstrap policy and role. A dedicated ServiceAccount then obtains a short-lived audience-bound TokenRequest JWT and exchanges it for a short-lived Vault token. Repeatable policy, role, and seed tasks use this path, so the root token may be moved offline and removed from the JSON file without breaking later reconciliation. Ephemeral tokens are revoked or allowed to expire after a short TTL and are never logged.

Move Vault policy bodies and role definitions out of shell heredocs into versioned HCL/JSON under bootstrap/foundation/vault/. Scripts inject only the explicit environment parameters and orchestrate application. Secret generation and seeding remain stateful tasks rather than Kubernetes desired state.

6. Tighten bootstrap and teardown behavior

Every applied stage follows:

render --enable-helm
  -> server-side dry-run
  -> diff
  -> confirmation
  -> server-side apply
  -> readiness boundary

SKIP_DIFF=yes is accepted only for dev. Staging and prod fail when it is set.

Teardown defaults to deleting 50-apps only. One-shot operations require DELETE_OPERATIONS=yes. Data, Vault, namespaces, platform controllers, and CRDs remain progressively gated. CRD deletion requires an additional explicit gate because deleting a CRD can delete every corresponding custom resource.

After operator desired state moves to Kustomize, teardown no longer performs implicit Helm uninstalls. A legacy Helm release is handled only by the Phase 2 adoption/rollback runbook.

7. Strengthen repository-local validation

make validate remains the developer-friendly profile. make check runs the repository's full local contract and fails when required validators are absent. This distinction does not define how CI provisions tools; that is a cicd-platform responsibility.

Use one Kustomize render helper everywhere and add rendered-output assertions:

• inventory and actual dev entrypoints agree;
• _template is not imported by a cluster root;
• all/ is not referenced by apply code;
• deployable output has no example.com/environment: lab label;
• auth-server has the exact oauth2-proxy and security middleware chain;
• Keycloak admin Ingress is absent;
• the operator stage contains the expected Deployments and CRDs;
• no unresolved placeholder appears in deployable output.

Add Bash behavioral tests with fake commands and fixtures. They cover explicit context requirements, key-file collision refusal, Vault state/file mismatch, no token-cache writes, diff-skip gates, teardown scopes, CRD gates, operator adoption gates, and the absence of direct Helm install/uninstall calls.

Documentation validation checks every YAML fence for syntax. Complete Kubernetes examples outside documented bad-example sections also pass kubeconform and kube-linter. Explicitly abbreviated snippets and headings named 나쁜 예시, ❌, or bad example are excluded from schema/policy checks. Operational path checks, credential-literal checks, and executable-bit checks remain deterministic.

The GitHub and Gitea workflow definitions are not changed in this phase.

8. Normalize repository structure and file modes

Move entrypoints to their declared classes:

scripts/doctor.sh
  -> scripts/bin/doctor.sh

scripts/validate.sh
  -> scripts/ci/validate-structure.sh

Update the Makefile and docs. Entrypoints use strict mode, safe IFS, functions, and main "$@"; user-facing commands have usage text. Kubernetes YAML and Helm values files lose accidental executable bits and are stored as regular 0644 source.

Remove non-runnable placeholder leaves:

• infrastructure/components/{compute,database,networking,storage};
• infrastructure/live/{dev,staging,prod}/cluster.

Retain the parent documentation and _template roots. Correct copy examples so an existing destination does not create a nested _template directory.

Move presentation material under docs/presentation/. Compute paths from the script location, add build instructions, and pin the directly used Python dependencies. Keep Draw.io sources and PNG inputs required to reproduce the deck. Remove the generated build/deck.pptx from Git and ignore the build directory.

Record the dev environment decision and the external CI ownership boundary in an ADR. Keep CODEOWNERS.example until a real organization/team identifier is provided.

Phase 2 design boundary

Phase 2 starts only after Phase 1 source and tests are accepted. It requires a new implementation plan because it changes live ownership and state.

It includes:

1. select Terraform or OpenTofu, provider, backend, locking, and credentials;
2. create the first real infrastructure/live/dev root;
3. migrate mnt and operator namespaces to the approved dev-prefixed model;
4. adopt existing Helm-managed MinIO/VSO resources into Kustomize ownership and remove legacy releases with a tested rollback path;
5. delete the existing live Keycloak admin Ingress after diff and confirmation;
6. replace the handmade Vault workload with the official chart;
7. implement TLS, Raft HA, auto-unseal, audit, backup, restore, and rollback;
8. coordinate central project-infra onboarding in cicd-platform before retiring transitional consumer workflows.

Phase 2 must define data backup, restore rehearsal, PVC retention, Vault seal compatibility, resource adoption, and abort criteria before any apply.

Error handling and recovery

• A missing explicit context is an input error, not an interactive prompt.
• A Vault state/key-file mismatch is a recovery condition, never permission to initialize or overwrite.
• Render, schema, policy, server dry-run, or diff errors stop before apply.
• Operator legacy ownership stops automatic adoption and points to the Phase 2 runbook.
• Teardown gates are independent and additive; a broad flag does not silently imply CRD deletion.
• Phase 1 changes are recoverable through Git because it performs no external mutation.

Verification and acceptance criteria

Phase 1 is complete only when:

• the worktree contains no active cluster or overlay path named lab;
• every declared dev entrypoint renders with Helm support;
• schema and policy validation passes for the aggregate and ordered stages;
• the rendered auth-server middleware and Keycloak ingress assertions pass;
• operator resources and CRDs render from Git-pinned chart definitions;
• all script behavioral tests pass without a real cluster;
• shell syntax, ShellCheck, shfmt, secret scanning, docs checks, and file-mode checks pass under the full local profile;
• no live cluster command was executed during Phase 1 implementation;
• documentation distinguishes repository-local validation from centrally owned CI execution;
• git diff --check is clean and the final diff contains no unrelated changes.

Known risks

• Kustomize Helm rendering needs Helm and chart-repository access; pinned chart versions reduce but do not eliminate upstream availability risk.
• Existing Helm-managed resources cannot be assumed safe for SSA adoption; live ownership inspection is deferred to Phase 2.
• Renaming an overlay path does not remove old live resources. The Phase 2 diff and prune/delete procedure must identify leftovers explicitly.
• Vault bootstrap access is privileged. Its policy must be reviewed against the minimum API paths required by the checked-in definitions and seed tasks.
• Central CI coverage remains transitional until cicd-platform provides and activates an appropriate infrastructure validation capability.