feat(traefik): re-apply the trust overlay when trust is already live

apply-trust.sh is the observe -> trust transition. When the live
Traefik is already in the trust phase and only another value changed
(PR #3 resources), it stops at "observation phase must not trust
forwarded headers" before applying, so the change had to be applied
with a bare kubectl apply outside the guarded path.

reapply-trust.sh checks that the runtime is in the trust phase with
the reviewed CIDR and that forwarded-header trust works (a request
through Host Nginx is logged with the forwarded client, not the proxy
address), saves the live HelmChartConfig, shows the change with
kubectl diff and does nothing when there is none, asks for
"APPLY <context> RETRUST <cidr>", applies, and repeats the checks plus
the NodePort, Gitea health and site-manifest checks. Any failure after
the apply re-applies the saved HelmChartConfig, so the trusted CIDR is
never dropped.

Risk: the apply path was not run against the cluster (the live values
already match). The restore path reuses the same wait and health
helpers as apply-trust.sh.

Verification (desktop):
- Live (already trust, values match): "Forwarded-header trust in
  effect (ClientHost 127.0.0.1, proxy 10.42.0.1)", then "already
  matches the trust overlay. Nothing to do.", exit 0.
- Copy with memory 64Mi changed to 65Mi, answering "no": the diff
  shows only that line, "ERROR: cancelled", exit 1; live still 64Mi.
- The ClientHost check of apply-trust.sh would have failed here
  (127.0.0.1 is expected in the trust phase), which is why the check
  is inverted.
This commit is contained in:
DongHyeonka
2026-11-12 02:00:20 +09:00
parent 8b019d26a0
commit cc5decc538
2 changed files with 141 additions and 0 deletions
@@ -94,6 +94,7 @@ infrastructure/networking/traefik/
├── apply-observe.sh
├── observe-client-host.sh
├── apply-trust.sh
├── reapply-trust.sh
├── rollback-to-observe.sh
└── validate.sh
```
@@ -238,6 +239,20 @@ bash infrastructure/networking/traefik/scripts/observe-client-host.sh
`apply-trust.sh`를 실행한다. Chart, 이미지, context, API server, NodePort 경계 또는
재관측 값이 기대와 다르면 스크립트가 적용을 중단한다.
이미 trust인 상태에서 trust overlay의 다른 값(예: `resources`)만 바꿨을 때는 `apply-trust.sh`가
「observation phase must not trust forwarded headers」로 멈춘다(observe → trust 전환 전용이다). 그때는
`reapply-trust.sh`를 쓴다.
```sh
bash infrastructure/networking/traefik/scripts/reapply-trust.sh --observed-client-host 10.42.0.1 --execute
# 승인: APPLY <현재-context> RETRUST 10.42.0.1/32
```
- 적용 전에 runtime이 trust인지, 전달 헤더 신뢰가 실제로 작동하는지(Host Nginx 경유 요청의 ClientHost가
프록시 주소가 아님)를 본다. 바꿀 것을 `kubectl diff`로 보여 주고, 차이가 없으면 아무것도 하지 않는다.
- 적용 뒤 같은 검사를 다시 하고, 하나라도 실패하면 직전에 라이브였던 HelmChartConfig를 다시 적용한다.
trusted CIDR이 빠지는 일은 없다.
### 롤백
trust만 제거하고 JSON access log를 남기는 표준 롤백은 observe overlay를 적용한다.
+126
View File
@@ -0,0 +1,126 @@
#!/usr/bin/env bash
# Re-apply the trust overlay when the live Traefik is already in the trust phase,
# for example after a values change such as resources. apply-trust.sh is only for
# the observe -> trust transition and stops when the runtime already trusts the
# CIDR ("observation phase must not trust forwarded headers").
#
# bash infrastructure/networking/traefik/scripts/reapply-trust.sh \
# --observed-client-host 10.42.0.1 --execute
#
# On any failure after the apply it restores the HelmChartConfig that was live
# before, so the trusted CIDR is never dropped.
set -Eeuo pipefail
set +x
umask 077
readonly SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd -P)"
# shellcheck source=common.sh
source "${SCRIPT_DIR}/common.sh"
[[ "$#" -eq 3 && "$1" == "--observed-client-host" && "$3" == "--execute" ]] || {
sed -n '2,11p' "${BASH_SOURCE[0]}" >&2
exit 2
}
readonly REVIEWED_CLIENT_HOST="$2"
require_commands kubectl jq rg curl nc python3 awk sort date cmp find bash
reviewed_cidr="$(host_to_exact_cidr "$REVIEWED_CLIENT_HOST")"
declared_cidr="$(normalize_exact_host_cidr "$(source_trusted_proxy_cidr)")"
[[ "$declared_cidr" != "$SENTINEL_TRUSTED_PROXY_CIDR" ]] || \
fail "trust overlay still contains the non-routable sentinel CIDR"
[[ "$declared_cidr" == "$reviewed_cidr" ]] || \
fail "declared CIDR ${declared_cidr} does not match ClientHost ${REVIEWED_CLIENT_HOST}"
bash "${SCRIPT_DIR}/validate.sh"
work_dir="$(mktemp -d "${TMPDIR:-/tmp}/platform-traefik-retrust.XXXXXX")"
restore_required=false
selected_context=""
cleanup() {
local exit_code=$?
trap - EXIT
if [[ "$exit_code" -ne 0 && "$restore_required" == "true" ]]; then
printf '\nRESTORE: re-applying the HelmChartConfig that was live before.\n' >&2
set +e
kubectl --context "$selected_context" apply --filename "${work_dir}/previous.yaml"
wait_for_runtime trust "$declared_cidr"
kubectl --context "$selected_context" --namespace kube-system \
rollout status deployment/traefik --timeout=5m
wait_for_nodeport_boundary_and_health
printf 'RESTORE complete. The previous trust values are live again.\n' >&2
set -e
fi
case "$work_dir" in
/tmp/platform-traefik-retrust.*|"${TMPDIR:-/tmp}"/platform-traefik-retrust.*) rm -rf -- "$work_dir" ;;
esac
exit "$exit_code"
}
trap cleanup EXIT
trap 'exit 130' INT
trap 'exit 143' TERM
trust_render="${work_dir}/trust.yaml"
render_overlay "$TRUST_OVERLAY" "$trust_render"
assert_live_baseline
assert_runtime trust "$declared_cidr"
assert_nodeport_boundary_and_health
# In the trust phase Traefik logs the forwarded client, not the proxy. Seeing the
# proxy address (the reviewed ClientHost) would mean the trust is not working.
assert_trust_effective() {
local seen
seen="$(observe_host_nginx_client_host)"
[[ "$seen" != "$REVIEWED_CLIENT_HOST" ]] || \
fail "Traefik logged the proxy ${seen} as ClientHost; forwarded-header trust is not in effect"
printf 'Forwarded-header trust in effect (ClientHost %s, proxy %s)\n' "$seen" "$REVIEWED_CLIENT_HOST"
}
assert_trust_effective
kubectl --namespace kube-system get helmchartconfig traefik --output=json |
python3 -I -c '
import json, sys
o = json.load(sys.stdin)
md = o["metadata"]
keep = {k: md[k] for k in ("name", "namespace", "labels", "annotations") if k in md}
keep.get("annotations", {}).pop("kubectl.kubernetes.io/last-applied-configuration", None)
print(json.dumps({"apiVersion": o["apiVersion"], "kind": o["kind"], "metadata": keep, "spec": o["spec"]}))
' >"${work_dir}/previous.yaml"
printf '\nChange to apply (kubectl diff, server dry-run):\n'
set +e
KUBECTL_EXTERNAL_DIFF="diff -u -N" kubectl diff --filename "$trust_render" | grep -E '^[-+] ' | grep -v -E '^[-+] generation:'
diff_rc=${PIPESTATUS[0]}
set -e
if [[ "$diff_rc" -eq 0 ]]; then
printf 'Live HelmChartConfig already matches the trust overlay. Nothing to do.\n'
exit 0
fi
(( diff_rc == 1 )) || fail "kubectl diff failed"
selected_context="$(current_context)"
selected_api_server="$(current_api_server)"
printf '\nKubernetes context: %s\nAPI server: %s\nTrusted CIDR (unchanged): %s\n' \
"$selected_context" "$selected_api_server" "$declared_cidr"
printf 'Type APPLY %s RETRUST %s to continue: ' "$selected_context" "$declared_cidr"
read -r confirmation
[[ "$confirmation" == "APPLY ${selected_context} RETRUST ${declared_cidr}" ]] || fail "cancelled"
[[ "$(current_context)" == "$selected_context" ]] || fail "kubectl context changed after confirmation"
[[ "$(current_api_server)" == "$selected_api_server" ]] || fail "Kubernetes API server changed after confirmation"
assert_runtime trust "$declared_cidr"
restore_required=true
kubectl apply --filename "$trust_render"
wait_for_runtime trust "$declared_cidr"
kubectl --namespace kube-system rollout status deployment/traefik --timeout=5m
assert_runtime trust "$declared_cidr"
assert_live_hcc_matches_overlay "$TRUST_OVERLAY" "${work_dir}/post-apply-trust.yaml"
assert_live_baseline
wait_for_nodeport_boundary_and_health
assert_manifest_https
assert_trust_effective
restore_required=false
printf '\nTRUST PHASE RE-APPLIED\n'
printf 'Traefik web trusted CIDR: %s\n' "$declared_cidr"
printf 'Gitea health and HTTPS site-manifest checks: PASS\n'