The previous commit took the tech-log pg_hba change out of this PR
because W1's CNPG backup work owns the same Cluster manifest. Update the
review so it no longer says the fix is in this PR: section 6.1 now gives
W1 the exact change (commit 1c4ef9a to cherry-pick), its render
invariants and the RED/GREEN and kubectl diff evidence, and the W1
overlap table lists it.
Verification: grep finds no remaining "fixed in this PR" claim for
pg_hba; git diff origin/main...HEAD lists only docs/ and Traefik paths.
This reverts commit 1c4ef9a.
W1 owns the CNPG backup work, which changes the same Cluster manifest
(services/platform-postgres/base/cluster.yaml). The program rule is that
another session does not edit files in W1's scope; overlaps are recorded
as findings and handed over. The drift itself is unchanged: the live
Cluster still has two techlog pg_hba rules the repository does not
declare, so applying the current render would cut the production
backend off its database. The exact change, its render invariants and
the RED/GREEN evidence are in the maturity review, section 6.
Verification: git diff origin/main...HEAD no longer lists any
services/platform-postgres or scripts/validate/render-phase1.sh path.
The desktop k3s runs production and all CI on one node, and three
incidents this week came from that. Record where the configuration is
weak before the next one.
The review scores 16 axes from 0 to 5 against the k3s hardening guide,
the k3s CIS self-assessment, the NSA/CISA hardening guide and the
Kubernetes security checklist, lists the top ten risks, compares the
rendered repository with the live cluster, and summarizes kube-bench,
kube-linter, Polaris and kubescape runs. Items W1 is already fixing are
marked and not repeated. docs/README.md is the new docs map that the
docs-system convention asks for when a docs folder is added.
Risk: documentation only. Tool results were checked finding by finding;
21 kube-bench FAILs and several Polaris and kubescape items are false
positives and are reported as such.
Verification: every count and value in the review comes from read-only
commands run on 2026-11-11 (kubectl get/diff, configz, journal, lsblk,
openssl). The scanners ran with pinned digests and their images were
removed afterwards.
The k3s-bundled Traefik runs with no requests or limits, so it is a
BestEffort pod. The kubelet on this node has no memory eviction
threshold (the k3s default evictionHard only covers nodefs and
imagefs), so under memory pressure the kernel OOM killer chooses, and
BestEffort containers have oom_score_adj 1000. Every public route goes
through this one pod.
Set requests cpu 50m / memory 64Mi and a 256Mi memory limit in the base
and in both overlay patches, because the baseline and trust patches
replace valuesContent as a whole. No CPU limit, to avoid throttling.
Measured use on 2026-11-11 was about 5m CPU and 42Mi memory.
Rejected: raising Traefik to Guaranteed. It would reserve CPU the
ingress does not use, and the real fix for memory pressure is kubelet
reservation and eviction thresholds, which need a k3s restart.
Risk: applying the trust overlay makes the k3s helm-controller roll
Traefik once (RollingUpdate, a few seconds). Not applied here.
Verification (desktop):
- infrastructure/networking/traefik/scripts/validate.sh passes.
- kubectl diff of the trust overlay against the live HelmChartConfig
adds only the resources block.
- baseline and observe renders both contain the same resources.
The live platform-postgres Cluster has two pg_hba rules for the techlog
role that the repository does not declare. They were added outside the
repository for hyeonworks.com. Applying the current render would remove
them and the production backend could no longer reach its database.
Declare the same two rules, in the same position after the Keycloak
pair, and pin them with render invariants like the Gitea and Keycloak
pairs. The techlog DatabaseRole, Database, Secret and NetworkPolicies
stay outside this build root; where they belong is an owner decision
recorded in the 2026-11-11 maturity review.
Risk: none for the live cluster. The rendered Cluster now matches the
live object. A fresh install without the tech-log tenant gets two
rules for a role that does not exist, which PostgreSQL ignores.
Verification (desktop, Helm 3.19.4, Kustomize 5.8.1):
- render-phase1.sh with the old cluster.yaml and the new invariants
fails: "the tech-log role must authenticate only to the tech-log
database"; with the new cluster.yaml it passes.
- kubectl diff of the new render against the live cluster shows no
Cluster change; only the pre-existing DatabaseRole default fields.
On 2026-11-11 the desktop booted with the RTC read as UTC while it held
KST, so the clock ran 9 hours ahead until NTP pulled it back. Prometheus
had written one hour of samples stamped 18:36-19:36Z into its TSDB head
during that window. With outOfOrderTimeWindow at the chart default of
0s, every later scrape was older than the head's valid range and was
dropped as "out of bounds". From 11:21Z no metric was stored, no rule
could fire on real data, and KubeAPIDown (an absent() rule) fired
falsely. It would have stayed blind until the wall clock passed the
bogus head time, about 7 hours later.
Set prometheusSpec.tsdb.outOfOrderTimeWindow to 10h. Samples up to 10
hours behind the head are now accepted, which covers the +9h KST-as-UTC
misread. The README records why.
Rejected: deleting the head WAL from the Retain PV. It works at once
but removes persisted data and needs the pod stopped. Rejected: waiting
for the clock to catch up. It leaves monitoring blind for hours during
the period when the cause of the incident is still being found.
Risk: out-of-order samples use an extra in-memory head and vertical
compaction only while they arrive; in normal operation none arrive. A
genuinely late exporter could now backfill up to 10h, which is harmless
on this single-node Prometheus. Preventing the clock jump itself is the
host change in infrastructure/host, not this commit.
Verification: the values file parses and holds
prometheus.prometheusSpec.tsdb.outOfOrderTimeWindow=10h. The live
Prometheus CR already carried tsdb.outOfOrderTimeWindow=0s from this
chart key, so the key maps to spec.tsdb. The same value was patched
onto the live CR: the operator wrote out_of_order_time_window: 10h, the
config reloaded at 13:04:37Z, "too old" warnings dropped to 0, all 34
targets report up, and KubeAPIDown resolved. The pinned renderer
(scripts/validate/render-observability-core.sh) was not run: it requires
Helm v3.19.4 and kustomize v5.8.1, which are not installed on the
laptop or the desktop.
The Gitea README still said Actions and the package registry are
disabled on first deployment, and had no guidance for either.
Replace that decision with the 2026-11-07 state, add where packages
are stored and how much space they really have (a local PV does not
enforce the 50Gi request), the cleanup rules to use for Maven
SNAPSHOTs, and who may publish and read Maven packages. Record the
applied manifest SHA-256 and link the runner guide from the root
README.
Verification: statements checked against the Gitea 1.27 docs and
source (routers/api/packages/maven, modules/web/router.go), the live
app.ini, and API responses after the rollout.
The template repositories need a real runner for their CI, and the
owner chose one container on the desktop host (the k3s node).
Add compose.yaml and config.yaml for the official Gitea Runner 5.4.1
dind image, pinned by index digest, plus the operating guide. The
runner has capacity 1, one ubuntu-latest label pinned to
runner-images ubuntu-24.04-v26.11.03 by digest, restart: always, and
a 6 CPU / 10g cap (no more than half of the 12 cores and 31GiB).
Jobs use the dind daemon through the runner's Docker proxy for
Testcontainers; the host Docker socket is never mounted, job
containers stay unprivileged, valid_volumes is empty, and sweep
removes what a job created.
Dropped alternatives: the deprecated gitea/act_runner 0.6.1 (moved
to gitea/runner), the basic image with the host docker.sock (jobs
would own the host daemon), and dind-rootless (Testcontainers
compatibility needs separate checks).
Risk: the runner container is privileged; a job can still start a
privileged container on the inner daemon, so only trusted repository
workflows should run here. Runner jobs do not take the desktop's
heavy-work lock; the CPU and memory caps are the only guard.
Verification: config keys checked with `gitea-runner config get`,
`docker compose config` passes, the runner registered at user scope
and the Gitea API shows desktop-runner online and idle; it came back
online after `docker restart` without registering again.
The backend template is moving its messaging and gRPC libraries into
separate repositories and will consume them as Maven versions from
Gitea Packages. [packages] ENABLED=false and repo.packages in
DISABLED_REPO_UNITS make the registry unavailable.
Turn on [packages] and leave DISABLED_REPO_UNITS as an explicit empty
value. The chart init container runs `gitea config edit-ini
--apply-env` against the app.ini on the data volume, so a removed key
would keep the old disabled list; an empty value overwrites it.
SSH and LFS stay off. This ships with the Actions change in the same
rollout, so Gitea restarts once.
Risk: packages are stored under /data/packages on the root SSD local
PV. The 50Gi PVC size is not enforced by a local PV; the real bound is
free space on the host root disk, shared with the host and the CI
runner. No per-owner size limit is set yet.
Verification: render-phase1.sh invariants pass; kubectl diff against
live shows only the inline-config keys actions, packages, repository
and the Deployment checksum/config annotation.
Refs: Gitea 1.27 config cheat sheet [packages], usage/packages
The template repositories cannot run CI: [actions] ENABLED=false and
repo.actions in DISABLED_REPO_UNITS make Gitea ignore has_actions=true
on the repository API (200 response, value unchanged).
Turn on [actions] and drop only repo.actions from DISABLED_REPO_UNITS.
Packages stay disabled (packages.ENABLED=false, repo.packages kept).
Repositories created while the unit was disabled have no Actions unit
row, so every existing repository keeps has_actions=false until its
owner enables it. The values file is shared by the baseline and OIDC
build roots, so both renders change the same way.
Risk: new repositories get repo.actions by default (Gitea default
DEFAULT_REPO_UNITS). They run nothing without a runner in scope.
Gitea itself needs no new egress; runners connect inbound over the
existing Host Nginx -> Traefik HTTPS path.
Refs: Gitea 1.27 config cheat sheet [actions], [repository]