788 lines
29 KiB
Markdown
788 lines
29 KiB
Markdown
# Project Infra Phase 1 automation and Vault safety Implementation Plan
|
|
|
|
> **For agentic workers:** REQUIRED SUB-SKILL: Use `superpowers:subagent-driven-development` (recommended) or `superpowers:executing-plans` to execute this plan task by task. Use `superpowers:test-driven-development` for every script behavior change, `superpowers:systematic-debugging` for unexpected failures, and `superpowers:verification-before-completion` before reporting success.
|
|
|
|
**Goal:** Make cluster targeting, Vault initialization/authentication, bootstrap, and teardown fail closed while keeping all tests disconnected from a real cluster.
|
|
|
|
**Architecture:** Kubernetes clients are unusable until an explicit environment-to-context mapping is validated and locked. Vault credentials enter the Pod only through stdin, no CLI token cache is created, and repeatable tasks use a short-lived Kubernetes-auth token issued to a dedicated ServiceAccount. Bootstrap and teardown render through the shared Helm-aware Kustomize helper and expose narrow, additive safety gates. Legacy Helm ownership is detected read-only and never removed automatically.
|
|
|
|
**Tech Stack:** Bash, fake-command contract tests, `kubectl`, Kustomize, Helm, Vault CLI 1.17.2, Vault HCL/JSON policies, `jq`, Kubernetes TokenRequest.
|
|
|
|
## Global constraints
|
|
|
|
- Complete the dev GitOps plan first; this plan depends on `tests/run.sh`, `tests/lib/assert.sh`, `scripts/lib/kustomize.sh`, and `gitops/clusters/dev/main/stages/05-operators`.
|
|
- Read `docs/standards/infra/STYLE.md`, `scripts.md`, `config-and-secrets.md`, `security-hardening.md`, `operations-runbook-upgrade-rollback.md`, `vault.md`, and relevant approved examples before editing.
|
|
- All tests must run against temporary fake `kubectl`, `helm`, and Vault responses. Never point a test at the current kubeconfig.
|
|
- Never print, trace, or place a root token, unseal key, Kubernetes JWT, Vault token, password, or MinIO secret in a process argument.
|
|
- Do not add an overwrite override for Vault initialization material.
|
|
- Do not run `kubectl apply`, `kubectl delete`, `helm upgrade`, or `helm uninstall` against a live cluster while implementing this plan.
|
|
- Do not edit workflow files or the sibling `cicd-platform` repository.
|
|
|
|
## Dependency and public interfaces
|
|
|
|
- `require_kube_context ENV` accepts only `KUBE_CONTEXT` or `KUBE_CONTEXT_<ENV_UPPER>`, verifies that context exists, locks it, and never reads the interactive current context as approval.
|
|
- `kubectl` and `helm` wrappers fail if `KUBE_CONTEXT_LOCKED` is empty; after locking, they always add the explicit context flag.
|
|
- `vault_exec ARGS` is for unauthenticated Vault operations only: status, init, and unseal.
|
|
- `vault_exec_with_token TOKEN ARGS` sends only the token through stdin, exports it only for the child Vault process, and is used by commands with no payload.
|
|
- `vault_exec_with_token_input TOKEN ARGS` prepends the token to the caller's stdin payload for commands that read HCL/JSON from stdin.
|
|
- `vault_root_session_begin KEYS_FILE` is called only by `vault-init.sh` when first-bootstrap access must be created or repaired.
|
|
- `vault_try_session_begin` attempts only Kubernetes auth and returns non-zero without exiting so `vault-init.sh` can decide whether first-bootstrap repair is possible.
|
|
- `vault_session_begin` wraps `vault_try_session_begin` and aborts with recovery guidance when the role is unavailable; repeatable seed/admin/provision tasks never fall back to root.
|
|
- `vault_exec_authenticated ARGS` and `vault_exec_authenticated_input ARGS` require an active in-memory session and delegate to their corresponding transport.
|
|
- `vault_session_end` revokes a Kubernetes-auth token with `vault token revoke -self`, then unsets it; a root bootstrap token is only unset.
|
|
- `apply_entrypoint LABEL PATH` uses render → server dry-run → diff → confirmation → server-side apply.
|
|
- Destructive flags are additive. No broad flag implies CRD deletion.
|
|
|
|
---
|
|
|
|
## Task 1: Require an explicit Kubernetes context and lock every client call
|
|
|
|
**Files:**
|
|
|
|
- Create: `tests/contracts/kube-context-test.sh`
|
|
- Modify: `scripts/lib/common.sh`
|
|
- Modify: `scripts/README.md`
|
|
|
|
**Step 1: Write the failing context contract**
|
|
|
|
Use fake `kubectl` and `helm` programs that append NUL-safe argument records to a temporary log. Cover these cases in separate subprocesses:
|
|
|
|
```text
|
|
require_kube_context dev fails when KUBE_CONTEXT and KUBE_CONTEXT_DEV are both empty
|
|
CONFIRM=yes does not bypass the missing mapping
|
|
KUBE_CONTEXT_DEV=dev-k3s succeeds when kubectl config get-contexts dev-k3s succeeds
|
|
an unknown mapped context fails
|
|
KUBE_CONTEXT takes precedence over KUBE_CONTEXT_DEV
|
|
after success every kubectl call contains --context dev-k3s
|
|
after success every helm call contains --kube-context dev-k3s
|
|
calling either wrapper before the lock fails and never invokes the fake binary
|
|
no call to kubectl config current-context occurs
|
|
```
|
|
|
|
Run:
|
|
|
|
```bash
|
|
bash tests/run.sh kube-context
|
|
```
|
|
|
|
Expected: non-zero because the current wrapper falls back to an unlocked client and the current context.
|
|
|
|
**Step 2: Make wrappers fail closed**
|
|
|
|
Replace the wrapper behavior with this invariant:
|
|
|
|
```bash
|
|
kubectl() {
|
|
[[ -n "${KUBE_CONTEXT_LOCKED:-}" ]] \
|
|
|| die "kubectl 호출 전에 require_kube_context로 context를 잠가야 합니다."
|
|
command kubectl --context "$KUBE_CONTEXT_LOCKED" "$@"
|
|
}
|
|
|
|
helm() {
|
|
[[ -n "${KUBE_CONTEXT_LOCKED:-}" ]] \
|
|
|| die "helm 호출 전에 require_kube_context로 context를 잠가야 합니다."
|
|
command helm --kube-context "$KUBE_CONTEXT_LOCKED" "$@"
|
|
}
|
|
```
|
|
|
|
`require_kube_context` must use `command kubectl` while establishing the lock so it does not recurse into the guarded wrapper:
|
|
|
|
```bash
|
|
expected="${KUBE_CONTEXT:-${!mapped_var:-}}"
|
|
[[ -n "$expected" ]] \
|
|
|| die "KUBE_CONTEXT 또는 ${mapped_var}가 필수입니다."
|
|
command kubectl config get-contexts "$expected" --no-headers >/dev/null 2>&1 \
|
|
|| die "kube-context를 찾을 수 없습니다: $expected"
|
|
_lock_kube_context "$expected"
|
|
```
|
|
|
|
Do not require the selected context to equal `current-context`; explicit wrapper flags make current context irrelevant.
|
|
|
|
**Step 3: Verify behavior and static call discipline**
|
|
|
|
Run:
|
|
|
|
```bash
|
|
bash tests/run.sh kube-context
|
|
rg -n 'command (kubectl|helm)' scripts --glob '*.sh'
|
|
git diff --check
|
|
```
|
|
|
|
Expected: tests exit 0. Direct `command kubectl` appears only inside `require_kube_context` and the locked wrapper; direct `command helm` appears only inside the locked wrapper.
|
|
|
|
**Step 4: Commit**
|
|
|
|
```bash
|
|
git add scripts/lib/common.sh scripts/README.md tests/contracts/kube-context-test.sh
|
|
git commit -m "fix: Kubernetes context를 명시적으로 잠금"
|
|
```
|
|
|
|
---
|
|
|
|
## Task 2: Refuse unsafe Vault initialization material states
|
|
|
|
**Files:**
|
|
|
|
- Create: `tests/contracts/vault-key-material-test.sh`
|
|
- Modify: `scripts/tasks/vault-init.sh`
|
|
- Modify: `.gitignore` only if the existing Vault key pattern is incomplete
|
|
|
|
**Step 1: Write the failing state-matrix test**
|
|
|
|
Source `vault-init.sh` in subprocesses after overriding `vault_status_json` and `vault_exec`. Use a temporary parent directory and assert this matrix:
|
|
|
|
| Vault state | File state | Result |
|
|
|---|---|---|
|
|
| uninitialized | absent | initialize once, atomically create mode `0600` |
|
|
| uninitialized | existing valid | abort; preserve checksum and mode |
|
|
| uninitialized | existing invalid | abort; preserve checksum and mode |
|
|
| initialized | absent | abort with recovery guidance |
|
|
| initialized | invalid JSON | abort; preserve file |
|
|
| initialized | wrong mode | abort; preserve file |
|
|
| initialized | valid keys, root token present | continue |
|
|
| initialized | valid keys, root token absent | continue to Kubernetes-auth path |
|
|
| any | symlink target | abort without reading or writing target |
|
|
| any | missing parent | abort without creating parent |
|
|
|
|
Required JSON fields are:
|
|
|
|
```jq
|
|
(.unseal_threshold | type == "number" and . > 0) and
|
|
(.unseal_threshold as $threshold |
|
|
.unseal_keys_b64 | type == "array" and length >= $threshold) and
|
|
([.unseal_keys_b64[] | type == "string" and length > 0] | all) and
|
|
((.root_token == null) or (.root_token | type == "string" and length > 0))
|
|
```
|
|
|
|
Run:
|
|
|
|
```bash
|
|
bash tests/run.sh vault-key-material
|
|
```
|
|
|
|
Expected: non-zero; at minimum the uninitialized/existing case overwrites under the current implementation.
|
|
|
|
**Step 2: Split resolution, validation, and creation**
|
|
|
|
Create private functions in `vault-init.sh`:
|
|
|
|
```text
|
|
resolve_keys_file
|
|
validate_existing_keys_file
|
|
validate_vault_file_state
|
|
initialize_to_new_keys_file
|
|
```
|
|
|
|
`validate_existing_keys_file` must use `lstat` semantics through `[[ -L ]]`, `stat -c '%a'`, and `jq -e`. It must require exact mode `600`.
|
|
|
|
`initialize_to_new_keys_file` must:
|
|
|
|
1. verify the target is absent immediately before init;
|
|
2. create a temp file inside the existing parent with mode `0600` using `umask 077`;
|
|
3. write and validate init JSON in the temp file;
|
|
4. install without replacement by creating an atomic hard link from the same-directory temporary file to the absent target;
|
|
5. abort if another process created the target;
|
|
6. never expose an environment variable that allows replacement.
|
|
|
|
Use `ln -- "$tmp_out" "$KEYS_FILE"`; because both paths are in the same directory, link creation is atomic and fails if the target exists. Remove the temporary link only after success. The test must exercise the race/collision branch with a fake creation between validation and installation.
|
|
|
|
**Step 3: Verify the complete matrix**
|
|
|
|
Run:
|
|
|
|
```bash
|
|
bash tests/run.sh vault-key-material
|
|
bash -n scripts/tasks/vault-init.sh
|
|
git diff --check
|
|
```
|
|
|
|
Expected: all commands exit 0; test output confirms that every abort preserves the original checksum.
|
|
|
|
**Step 4: Commit**
|
|
|
|
```bash
|
|
git add scripts/tasks/vault-init.sh tests/contracts/vault-key-material-test.sh .gitignore
|
|
git commit -m "fix: Vault 초기화 자료 덮어쓰기 차단"
|
|
```
|
|
|
|
---
|
|
|
|
## Task 3: Add versioned Vault definitions and stdin-only authenticated execution
|
|
|
|
**Files:**
|
|
|
|
- Create: `bootstrap/foundation/vault/README.md`
|
|
- Create: `bootstrap/foundation/vault/policies/vault-bootstrap.hcl`
|
|
- Create: `bootstrap/foundation/vault/policies/vso-auth-platform.hcl`
|
|
- Create: `bootstrap/foundation/vault/policies/vso-storage.hcl`
|
|
- Create: `bootstrap/foundation/vault/policies/vault-admin.hcl`
|
|
- Create: `bootstrap/foundation/vault/roles/vault-bootstrap.json`
|
|
- Create: `bootstrap/foundation/vault/roles/vso-auth-platform.json`
|
|
- Create: `bootstrap/foundation/vault/roles/vso-storage.json`
|
|
- Create: `gitops/platform/vault/base/vault-bootstrap-serviceaccount.yaml`
|
|
- Modify: `gitops/platform/vault/base/kustomization.yaml`
|
|
- Create: `tests/contracts/vault-auth-session-test.sh`
|
|
- Create: `tests/contracts/vault-definition-test.sh`
|
|
- Modify: `scripts/lib/vault.sh`
|
|
|
|
**Step 1: Write failing authentication and definition tests**
|
|
|
|
`vault-auth-session-test.sh` must use a fake `kubectl` that records arguments and stdin. Prove:
|
|
|
|
```text
|
|
vault_exec_with_token never includes the token in kubectl or sh argv
|
|
the remote shell reads the first stdin line as VAULT_TOKEN
|
|
JSON payload following the token line reaches `vault write PATH -` unchanged
|
|
no command contains `vault login`
|
|
no command or source contains `.vault-token`
|
|
TokenRequest uses serviceaccount/vault-bootstrap, namespace mnt, audience vault, duration 10m
|
|
successful Kubernetes login sets a session token and session end revokes self
|
|
vault_root_session_begin succeeds only when root_token exists
|
|
vault_try_session_begin and vault_session_begin never fall back to root
|
|
missing Kubernetes login aborts with recovery guidance
|
|
```
|
|
|
|
`vault-definition-test.sh` must assert that every HCL/JSON file exists, JSON parses, role audience is `vault`, role TTL is at most `15m` for bootstrap, and no policy body remains in a shell heredoc.
|
|
|
|
Run:
|
|
|
|
```bash
|
|
bash tests/run.sh vault-auth-session
|
|
bash tests/run.sh vault-definition
|
|
```
|
|
|
|
Expected: both tests fail because the definitions and wrapper do not exist.
|
|
|
|
**Step 2: Add the dedicated ServiceAccount**
|
|
|
|
Create a standard-labeled ServiceAccount named `vault-bootstrap` with `automountServiceAccountToken: false`; TokenRequest does not require an automatically mounted token. Add it to the Vault base Kustomization. Do not bind this ServiceAccount to `system:auth-delegator`; the Vault server's existing ServiceAccount performs token review.
|
|
|
|
**Step 3: Add the checked-in policies**
|
|
|
|
`vault-bootstrap.hcl` grants only the paths required by checked-in reconciliation tasks:
|
|
|
|
```hcl
|
|
path "sys/mounts" {
|
|
capabilities = ["read", "list"]
|
|
}
|
|
path "sys/mounts/secret" {
|
|
capabilities = ["create", "read", "update"]
|
|
}
|
|
path "sys/auth" {
|
|
capabilities = ["read", "list"]
|
|
}
|
|
path "sys/auth/kubernetes" {
|
|
capabilities = ["create", "read", "update"]
|
|
}
|
|
path "sys/auth/userpass" {
|
|
capabilities = ["create", "read", "update"]
|
|
}
|
|
path "auth/kubernetes/config" {
|
|
capabilities = ["create", "read", "update"]
|
|
}
|
|
path "auth/kubernetes/role/*" {
|
|
capabilities = ["create", "read", "update", "delete", "list"]
|
|
}
|
|
path "auth/userpass/users/*" {
|
|
capabilities = ["create", "read", "update", "delete", "list"]
|
|
}
|
|
path "sys/policies/acl/*" {
|
|
capabilities = ["create", "read", "update", "delete", "list"]
|
|
}
|
|
path "secret/data/*" {
|
|
capabilities = ["create", "read", "update", "list"]
|
|
}
|
|
path "secret/metadata/*" {
|
|
capabilities = ["read", "list"]
|
|
}
|
|
path "auth/token/revoke-self" {
|
|
capabilities = ["update"]
|
|
}
|
|
```
|
|
|
|
Move the existing VSO and admin policy bodies byte-for-byte into their named HCL files, then delete the shell heredocs in the caller tasks during Task 4.
|
|
|
|
**Step 4: Add role JSON**
|
|
|
|
`vault-bootstrap.json` is:
|
|
|
|
```json
|
|
{
|
|
"bound_service_account_names": ["vault-bootstrap"],
|
|
"bound_service_account_namespaces": ["mnt"],
|
|
"audience": "vault",
|
|
"token_policies": ["vault-bootstrap"],
|
|
"token_ttl": "10m",
|
|
"token_max_ttl": "15m"
|
|
}
|
|
```
|
|
|
|
The VSO role files preserve the existing ServiceAccount, namespace, audience, policy, and one-hour TTL values. Keep one JSON file per role so drift is reviewable.
|
|
|
|
`vso-auth-platform.json` is:
|
|
|
|
```json
|
|
{
|
|
"bound_service_account_names": ["vault-secrets-operator"],
|
|
"bound_service_account_namespaces": ["mnt"],
|
|
"audience": "vault",
|
|
"token_policies": ["vso-auth-platform"],
|
|
"token_ttl": "1h",
|
|
"token_max_ttl": "1h"
|
|
}
|
|
```
|
|
|
|
`vso-storage.json` has the same shape and replaces both occurrences of `vso-auth-platform` with `vso-storage`.
|
|
|
|
**Step 5: Implement authenticated stdin transport**
|
|
|
|
In `scripts/lib/vault.sh`, retain unauthenticated `vault_exec` for status/init/unseal and add:
|
|
|
|
```bash
|
|
vault_exec_with_token() {
|
|
local token="${1:?token is required}"
|
|
shift
|
|
{
|
|
printf '%s\n' "$token"
|
|
} | kubectl exec -i -n "$VAULT_NAMESPACE" "$VAULT_POD" -- \
|
|
sh -c 'IFS= read -r VAULT_TOKEN; export VAULT_TOKEN; exec vault "$@"' vault "$@"
|
|
}
|
|
|
|
vault_exec_with_token_input() {
|
|
local token="${1:?token is required}"
|
|
shift
|
|
{
|
|
printf '%s\n' "$token"
|
|
cat
|
|
} | kubectl exec -i -n "$VAULT_NAMESPACE" "$VAULT_POD" -- \
|
|
sh -c 'IFS= read -r VAULT_TOKEN; export VAULT_TOKEN; exec vault "$@"' vault "$@"
|
|
}
|
|
|
|
vault_exec_authenticated() {
|
|
[[ -n "${VAULT_SESSION_TOKEN:-}" ]] || die "Vault 인증 session이 없습니다."
|
|
vault_exec_with_token "$VAULT_SESSION_TOKEN" "$@"
|
|
}
|
|
|
|
vault_exec_authenticated_input() {
|
|
[[ -n "${VAULT_SESSION_TOKEN:-}" ]] || die "Vault 인증 session이 없습니다."
|
|
vault_exec_with_token_input "$VAULT_SESSION_TOKEN" "$@"
|
|
}
|
|
```
|
|
|
|
Build the Kubernetes login JSON without putting the JWT in argv:
|
|
|
|
```bash
|
|
request_vault_bootstrap_token() {
|
|
local jwt response
|
|
jwt="$(kubectl create token vault-bootstrap \
|
|
-n "$VAULT_NAMESPACE" --audience=vault --duration=10m)"
|
|
response="$(printf '%s' "$jwt" \
|
|
| jq -Rs '{role:"vault-bootstrap", jwt:.}' \
|
|
| vault_exec write -format=json auth/kubernetes/login -)"
|
|
unset jwt
|
|
jq -er '.auth.client_token' <<<"$response"
|
|
}
|
|
```
|
|
|
|
Register `vault_session_end` with `trap_cleanup_fn` only once per process. `vault_root_session_begin` reads the optional root token from the validated key file and marks the session kind as root; it is not called from `vault_session_begin`. Do not revoke the first-init root token; unset it after bootstrap access has been written. Never invoke `vault login`.
|
|
|
|
**Step 6: Verify policy and transport contracts**
|
|
|
|
Run:
|
|
|
|
```bash
|
|
bash tests/run.sh vault-auth-session
|
|
bash tests/run.sh vault-definition
|
|
VALIDATION_PROFILE=local bash scripts/ci/validate.sh
|
|
git diff --check
|
|
```
|
|
|
|
Expected: all commands exit 0.
|
|
|
|
**Step 7: Commit**
|
|
|
|
```bash
|
|
git add bootstrap/foundation/vault gitops/platform/vault scripts/lib/vault.sh tests/contracts/vault-auth-session-test.sh tests/contracts/vault-definition-test.sh
|
|
git commit -m "feat: 단기 Vault bootstrap 인증 추가"
|
|
```
|
|
|
|
---
|
|
|
|
## Task 4: Migrate Vault tasks away from root login and shell heredocs
|
|
|
|
**Files:**
|
|
|
|
- Modify: `scripts/tasks/vault-init.sh`
|
|
- Modify: `scripts/tasks/vault-seed-apps.sh`
|
|
- Modify: `scripts/tasks/vault-setup-admin.sh`
|
|
- Modify: `scripts/tasks/minio-provision-registry.sh`
|
|
- Create: `tests/contracts/vault-task-auth-test.sh`
|
|
- Modify: `bootstrap/foundation/vault/README.md`
|
|
|
|
**Step 1: Write the failing caller contract**
|
|
|
|
The contract must statically and behaviorally prove:
|
|
|
|
```text
|
|
vault_login_root_from_keyfile is absent
|
|
vault_exec_sh is absent
|
|
every authenticated read/write calls vault_exec_authenticated
|
|
all policy write commands read an HCL file from bootstrap/foundation/vault/policies
|
|
all Kubernetes role writes read a JSON file from bootstrap/foundation/vault/roles
|
|
seed JSON and admin password payloads remain on stdin after the token prefix
|
|
minio provisioning uses dev HTTP defaults and requires an explicit override for insecure non-dev TLS
|
|
root_token may be absent from a valid post-bootstrap key file
|
|
```
|
|
|
|
Run:
|
|
|
|
```bash
|
|
bash tests/run.sh vault-task-auth
|
|
```
|
|
|
|
Expected: non-zero because current tasks call `vault_login_root_from_keyfile` and use cached authentication.
|
|
|
|
**Step 2: Make `vault-init.sh` establish bootstrap access safely**
|
|
|
|
Execution order must be:
|
|
|
|
```text
|
|
resolve and validate state/file
|
|
initialize only when both Vault and file are absent
|
|
unseal with stdin keys
|
|
attempt `vault_try_session_begin`
|
|
if unavailable, require root_token and use stdin-authenticated root session
|
|
enable KV and Kubernetes auth
|
|
write vault-bootstrap policy and role from files
|
|
end root session without revoke
|
|
start Kubernetes-auth session
|
|
write VSO policies and roles from files
|
|
remove legacy definitions
|
|
revoke the short-lived session
|
|
```
|
|
|
|
When the root token was needed, pass it to `vault_exec_with_token`; do not run a login command. If the bootstrap login still fails after role reconciliation, abort before application policy changes.
|
|
|
|
**Step 3: Convert all repeatable tasks to short-lived sessions**
|
|
|
|
At the beginning of each of these task `main` functions, call `vault_session_begin` after unseal/readiness prerequisites:
|
|
|
|
- `vault-seed-apps.sh`
|
|
- `vault-setup-admin.sh`
|
|
- `minio-provision-registry.sh`
|
|
|
|
All Vault reads and writes then call `vault_exec_authenticated`. For a JSON payload, the caller pipeline remains:
|
|
|
|
```bash
|
|
printf '%s\0%s\0' "$first" "$second" \
|
|
| jq -Rs 'split("\u0000") | {first: .[0], second: .[1]}' \
|
|
| vault_exec_authenticated_input kv put secret/example -
|
|
```
|
|
|
|
Because `vault_exec_with_token_input` prepends one token line and then copies stdin, the JSON body reaches the Vault CLI unchanged. Reads and argument-only writes use `vault_exec_authenticated`, so they never consume unrelated process stdin.
|
|
|
|
**Step 4: Apply checked-in definitions**
|
|
|
|
Replace all HCL heredocs and role key/value argv with:
|
|
|
|
```bash
|
|
vault_exec_authenticated_input policy write vso-auth-platform - \
|
|
<"$REPO_ROOT/bootstrap/foundation/vault/policies/vso-auth-platform.hcl"
|
|
vault_exec_authenticated_input write auth/kubernetes/role/vso-auth-platform - \
|
|
<"$REPO_ROOT/bootstrap/foundation/vault/roles/vso-auth-platform.json"
|
|
```
|
|
|
|
Use the corresponding file for each policy and role. Keep runtime secrets out of these definition files.
|
|
|
|
**Step 5: Preserve the dev-only MinIO transport semantics**
|
|
|
|
Verify the environment rename from the dev GitOps plan: HTTP is the default only for dev. Staging/prod default to verified HTTPS, and `ALLOW_INSECURE_MINIO_TLS=yes` remains an additional explicit gate outside dev. Do not reintroduce a `lab` branch while migrating Vault calls.
|
|
|
|
**Step 6: Verify callers**
|
|
|
|
Run:
|
|
|
|
```bash
|
|
bash tests/run.sh vault-key-material
|
|
bash tests/run.sh vault-auth-session
|
|
bash tests/run.sh vault-definition
|
|
bash tests/run.sh vault-task-auth
|
|
rg -n 'vault login|\.vault-token|vault_exec_sh|vault_login_root_from_keyfile' scripts bootstrap
|
|
git diff --check
|
|
```
|
|
|
|
Expected: tests exit 0 and the final `rg` returns no matches.
|
|
|
|
**Step 7: Commit**
|
|
|
|
```bash
|
|
git add scripts/tasks bootstrap/foundation/vault/README.md tests/contracts/vault-task-auth-test.sh
|
|
git commit -m "refactor: Vault 작업을 단기 인증으로 전환"
|
|
```
|
|
|
|
---
|
|
|
|
## Task 5: Rework bootstrap around ordered Git-rendered stages
|
|
|
|
**Files:**
|
|
|
|
- Modify: `scripts/bin/bootstrap.sh`
|
|
- Delete: `scripts/tasks/minio-operator-install.sh`
|
|
- Delete: `scripts/tasks/vso-install.sh`
|
|
- Create: `tests/contracts/bootstrap-flow-test.sh`
|
|
- Modify: `scripts/README.md`, `docs/operations.md`, `docs/vault-vso.md`, `gitops/clusters/dev/main/README.md`
|
|
|
|
**Step 1: Write the failing bootstrap contract**
|
|
|
|
Use fake clients and source/call individual functions. Prove:
|
|
|
|
```text
|
|
only dev, staging, and prod are accepted environment names
|
|
the shipped environment documented by usage is dev
|
|
every render calls kustomize_render and therefore --enable-helm
|
|
each stage order is namespaces, 00-platform, 05-operators, 10-vault, 20-secrets, 30-data, 35-registry, 40-operations, 50-apps
|
|
each apply path is render, server dry-run, diff, confirm, SSA apply, readiness
|
|
SKIP_DIFF=yes succeeds only for dev and fails before render for staging/prod
|
|
legacy Helm releases make 05-operators skip by default
|
|
ADOPT_LEGACY_OPERATORS=yes allows 05-operators only after a dedicated confirmation
|
|
no legacy release is uninstalled
|
|
new clusters with no legacy release apply 05-operators normally
|
|
operator readiness waits for both required CRDs and both controller Deployments
|
|
all/ is never passed to apply_entrypoint
|
|
```
|
|
|
|
Run:
|
|
|
|
```bash
|
|
bash tests/run.sh bootstrap-flow
|
|
```
|
|
|
|
Expected: non-zero because current bootstrap installs operators through task scripts and omits the operator stage.
|
|
|
|
**Step 2: Use the shared renderer and validate the diff gate**
|
|
|
|
Source `scripts/lib/kustomize.sh`, delete `render_entrypoint`, and redirect `kustomize_render` output into the temp file.
|
|
|
|
Add:
|
|
|
|
```bash
|
|
validate_diff_policy() {
|
|
if [[ "${SKIP_DIFF:-}" == "yes" && "$ENV_NAME" != "dev" ]]; then
|
|
die "SKIP_DIFF=yes 는 dev에서만 허용됩니다."
|
|
fi
|
|
}
|
|
```
|
|
|
|
Call it immediately after environment parsing and before any render or client mutation.
|
|
|
|
**Step 3: Add read-only legacy ownership detection**
|
|
|
|
Detect either known release without changing it:
|
|
|
|
```bash
|
|
legacy_operator_release_exists() {
|
|
helm -n minio-operator status minio-operator >/dev/null 2>&1 \
|
|
|| helm -n vault-secrets-operator-system status vault-secrets-operator >/dev/null 2>&1
|
|
}
|
|
```
|
|
|
|
Behavior:
|
|
|
|
- no legacy release: apply `05-operators` normally;
|
|
- legacy release and no gate: log a warning, skip stage apply, then verify readiness;
|
|
- legacy release and `ADOPT_LEGACY_OPERATORS=yes`: require a dedicated confirmation naming both releases before `apply_entrypoint`;
|
|
- never run Helm install, upgrade, or uninstall.
|
|
|
|
**Step 4: Add operator readiness boundaries**
|
|
|
|
Wait for:
|
|
|
|
```text
|
|
crd/tenants.minio.min.io Established
|
|
crd/vaultstaticsecrets.secrets.hashicorp.com Established
|
|
MinIO operator Deployment selected by app.kubernetes.io/name=operator Available
|
|
VSO Deployment selected by app.kubernetes.io/name=vault-secrets-operator Available
|
|
```
|
|
|
|
Use namespace-scoped rollout or condition waits and finite timeouts. Continue to stage `10-vault` only after all four checks pass.
|
|
|
|
**Step 5: Remove imperative installers and update task flow**
|
|
|
|
Delete `install_helm_operators` and both installer scripts. Keep Helm as a required local command because Kustomize Helm rendering needs it. Update usage to `dev|staging|prod`, mark dev as the currently shipped root, and document `ADOPT_LEGACY_OPERATORS=yes` as a Phase 2 transition-only gate.
|
|
|
|
**Step 6: Verify bootstrap behavior**
|
|
|
|
Run:
|
|
|
|
```bash
|
|
bash tests/run.sh bootstrap-flow
|
|
bash -n scripts/bin/bootstrap.sh
|
|
rg -n 'helm (upgrade|install|uninstall)|minio-operator-install|vso-install' scripts
|
|
git diff --check
|
|
```
|
|
|
|
Expected: tests and syntax exit 0; `rg` returns no mutation/install-script matches.
|
|
|
|
**Step 7: Commit**
|
|
|
|
```bash
|
|
git add scripts/bin/bootstrap.sh scripts/tasks scripts/README.md docs/operations.md docs/vault-vso.md gitops/clusters/dev/main/README.md tests/contracts/bootstrap-flow-test.sh
|
|
git commit -m "refactor: bootstrap을 Git 렌더 단계로 전환"
|
|
```
|
|
|
|
---
|
|
|
|
## Task 6: Make teardown scope narrow and gates additive
|
|
|
|
**Files:**
|
|
|
|
- Modify: `scripts/bin/teardown.sh`
|
|
- Create: `tests/contracts/teardown-scope-test.sh`
|
|
- Modify: `scripts/README.md`, `docs/operations.md`, `gitops/clusters/dev/main/README.md`
|
|
|
|
**Step 1: Write the failing teardown matrix**
|
|
|
|
Fake `kustomize_render`, `kubectl`, and `helm`, then assert:
|
|
|
|
| Flags | Permitted deletion |
|
|
|---|---|
|
|
| none | `50-apps` only |
|
|
| `DELETE_OPERATIONS=yes` | plus `40-operations` |
|
|
| `DELETE_DATA=yes` without operations | abort before deletion |
|
|
| operations + data | plus `35-registry`, `30-data`, `20-secrets` |
|
|
| `DELETE_VAULT=yes` without data | abort |
|
|
| operations + data + Vault | plus `10-vault` |
|
|
| `DELETE_NAMESPACE=yes` without Vault | abort |
|
|
| through namespace | plus namespace `mnt` |
|
|
| `TEARDOWN_PLATFORM=yes` without namespace | abort |
|
|
| through platform | plus `00-platform`, but not `05-operators` |
|
|
| `DELETE_OPERATORS=yes` without platform | abort |
|
|
| operators without `DELETE_CRDS=yes` | abort before operator deletion |
|
|
| all additive flags | plus `05-operators` and both operator namespaces |
|
|
|
|
Also prove:
|
|
|
|
```text
|
|
delete rendering uses kustomize_render, not kubectl delete -k
|
|
no helm uninstall command runs under any matrix row
|
|
FORCE_FINALIZERS=yes requires DELETE_NAMESPACE=yes
|
|
the confirmation summary names every enabled scope
|
|
```
|
|
|
|
Run:
|
|
|
|
```bash
|
|
bash tests/run.sh teardown-scope
|
|
```
|
|
|
|
Expected: non-zero because current default deletes `40-operations` and platform teardown uninstalls Helm releases.
|
|
|
|
**Step 2: Replace delete-by-kustomization with delete-by-rendered-file**
|
|
|
|
Source `scripts/lib/kustomize.sh`. `delete_entrypoint` must render to a temporary file, show the exact resource count, require confirmation, and then call:
|
|
|
|
```bash
|
|
kubectl delete -f "$rendered" \
|
|
--ignore-not-found \
|
|
--wait=true \
|
|
--timeout=300s
|
|
```
|
|
|
|
This is required because the operator stage uses Helm rendering and `kubectl delete -k` does not honor the shared `--enable-helm` interface.
|
|
|
|
**Step 3: Implement additive gate validation before any deletion**
|
|
|
|
Add one `validate_teardown_gates` function that checks all dependencies before the initial confirmation:
|
|
|
|
```text
|
|
DELETE_DATA requires DELETE_OPERATIONS
|
|
DELETE_VAULT requires DELETE_DATA
|
|
DELETE_NAMESPACE requires DELETE_VAULT
|
|
TEARDOWN_PLATFORM requires DELETE_NAMESPACE
|
|
DELETE_OPERATORS requires TEARDOWN_PLATFORM and DELETE_CRDS
|
|
DELETE_CRDS requires DELETE_OPERATORS
|
|
FORCE_FINALIZERS requires DELETE_NAMESPACE
|
|
```
|
|
|
|
Unknown non-empty flag values must fail; accepted values are empty or `yes`.
|
|
|
|
**Step 4: Implement reverse-order scopes**
|
|
|
|
Deletion order when every gate is enabled:
|
|
|
|
```text
|
|
50-apps
|
|
40-operations
|
|
35-registry
|
|
30-data
|
|
20-secrets
|
|
10-vault
|
|
namespace/mnt
|
|
05-operators
|
|
namespace/vault-secrets-operator-system and namespace/minio-operator
|
|
00-platform
|
|
```
|
|
|
|
The operator stage deletion is allowed only when both `DELETE_OPERATORS=yes` and `DELETE_CRDS=yes`. Explain in usage that deleting the rendered chart stage deletes CRDs and can delete every custom resource. Do not call `helm uninstall`.
|
|
|
|
**Step 5: Verify all scopes**
|
|
|
|
Run:
|
|
|
|
```bash
|
|
bash tests/run.sh teardown-scope
|
|
bash -n scripts/bin/teardown.sh
|
|
rg -n 'helm[[:space:]]+.*uninstall|kubectl[[:space:]]+delete[[:space:]]+-k' scripts/bin/teardown.sh
|
|
git diff --check
|
|
```
|
|
|
|
Expected: tests and syntax exit 0; `rg` returns no matches.
|
|
|
|
**Step 6: Commit**
|
|
|
|
```bash
|
|
git add scripts/bin/teardown.sh scripts/README.md docs/operations.md gitops/clusters/dev/main/README.md tests/contracts/teardown-scope-test.sh
|
|
git commit -m "fix: teardown 삭제 범위를 단계별로 제한"
|
|
```
|
|
|
|
---
|
|
|
|
## Task 7: Automation plan verification checkpoint
|
|
|
|
**Files:** No source changes expected.
|
|
|
|
**Step 1: Run focused contracts**
|
|
|
|
```bash
|
|
bash tests/run.sh kube-context
|
|
bash tests/run.sh vault-key-material
|
|
bash tests/run.sh vault-auth-session
|
|
bash tests/run.sh vault-definition
|
|
bash tests/run.sh vault-task-auth
|
|
bash tests/run.sh bootstrap-flow
|
|
bash tests/run.sh teardown-scope
|
|
```
|
|
|
|
Expected: every command exits 0 without accessing a real kubeconfig.
|
|
|
|
**Step 2: Run static secret and mutation checks**
|
|
|
|
```bash
|
|
rg -n 'vault login|\.vault-token|vault_exec_sh|vault_login_root_from_keyfile' scripts bootstrap
|
|
rg -n 'helm[[:space:]]+(install|upgrade|uninstall)' scripts
|
|
rg -n 'Usage:.*lab|\blab default\b|ENV_NAME.*lab' scripts
|
|
git diff --check
|
|
```
|
|
|
|
Expected: all three `rg` commands return no matches and `git diff --check` emits nothing.
|
|
|
|
**Step 3: Run repository validation available at this checkpoint**
|
|
|
|
```bash
|
|
bash tests/run.sh
|
|
VALIDATION_PROFILE=full bash scripts/ci/validate.sh
|
|
git status --short
|
|
```
|
|
|
|
Expected: tests and validation exit 0; status contains no unrelated files. The validation-structure plan may add stricter file-mode and documentation-example gates later, but this checkpoint must pass all currently defined gates.
|