The kubelet starts deleting unused images at 85% root use and evicts
pods at 90%. The root disk warning fired at 85% and the critical at
90%, so both arrived when the kubelet was already acting. On
2026-11-09 the warning fired for 33 hours, then an eviction stopped
the tech-log pods, the image GC deleted their imagePullPolicy: Never
images, and production went down.
Move the warning to 80% for 10m and the critical to 85% for 5m, and
add PlatformRootFilesystemFillingFast (critical, for 15m): the 6h
linear trend of free bytes falls below 10% of the disk, the eviction
point, within 6 hours. On 2026-11-07 the disk lost 17G in 9 hours;
this forecast would have fired hours before the 80% warning. The
info alert at 75% stays.
Add scripts/validate/test-platform-disk-alerts.sh, promtool unit
tests for the usage bands and the forecast with the pinned Prometheus
image. The existing rules test needs live metric inventories, so this
one runs offline.
Risk: the live rules change only when the observability access apply
runs; this commit does not touch the cluster. The forecast can fire
on a large build that the BuildKit GC removes minutes later; the 15m
hold and the 6h window damp that.
Verification: the new test failed against the old rules (no alert at
81%, 86% or for a 1-per-minute fill) and passes after the change;
promtool check rules finds 9 rules. test-observability-alerting.sh
passes its static contract, then stops at the helm render because
helm v3.19.4 is not installed on the desktop.
In a long-lived checkout the observability renderer finds an ignored
.helm/charts/<chart>-<version> cache left by an earlier render and
refuses ("exact chart cache already exists"), so live-drift-check.sh
fails with exit 2 in /home/donghyeon/workspace/platform, the path the
runbooks use.
When run inside a git working tree, copy the tracked and untracked
(not ignored, not deleted) files to the drift work directory and
render from there. Local edits are still checked; ignored caches are
not carried over. Outside a git tree (the daily timer renders a
git archive of origin/main) nothing changes.
Also accept the one drift this exposed: the kube-prometheus-stack
PrometheusRule still has the two Kubelet certificate alerts that PR #6
disabled in values; it goes away when the observability core is
re-applied in the restart window (until 2026-11-30).
Risk: read-only, as before. The copy is removed with the work
directory.
Verification (desktop, fresh clone of main 8b019d2 with an ignored
services/observability/tempo/.helm/charts/tempo-2.2.3 created):
- old script: "REJECT: exact chart cache already exists", exit 2.
- new script as an uncommitted edit in that clone: SUMMARY drift=1
accepted=12 (the PrometheusRule above), exit 1.
- new script from a git archive export, as the timer runs it: same
summary.
apply-trust.sh is the observe -> trust transition. When the live
Traefik is already in the trust phase and only another value changed
(PR #3 resources), it stops at "observation phase must not trust
forwarded headers" before applying, so the change had to be applied
with a bare kubectl apply outside the guarded path.
reapply-trust.sh checks that the runtime is in the trust phase with
the reviewed CIDR and that forwarded-header trust works (a request
through Host Nginx is logged with the forwarded client, not the proxy
address), saves the live HelmChartConfig, shows the change with
kubectl diff and does nothing when there is none, asks for
"APPLY <context> RETRUST <cidr>", applies, and repeats the checks plus
the NodePort, Gitea health and site-manifest checks. Any failure after
the apply re-applies the saved HelmChartConfig, so the trusted CIDR is
never dropped.
Risk: the apply path was not run against the cluster (the live values
already match). The restore path reuses the same wait and health
helpers as apply-trust.sh.
Verification (desktop):
- Live (already trust, values match): "Forwarded-header trust in
effect (ClientHost 127.0.0.1, proxy 10.42.0.1)", then "already
matches the trust overlay. Nothing to do.", exit 0.
- Copy with memory 64Mi changed to 65Mi, answering "no": the diff
shows only that line, "ERROR: cancelled", exit 1; live still 64Mi.
- The ClientHost check of apply-trust.sh would have failed here
(127.0.0.1 is expected in the trust phase), which is why the check
is inverted.
The owner decisions in section 8 were settled on 2026-11-12 and the
section 7 recommendations went into separate PRs. Readers of the
review need to know where each one landed and that one recommendation
was withdrawn.
Add a table from each 7.x item to its PR, withdraw protect-kernel-
defaults (CIS dropped it in v1.7 and its kernel.panic_on_oops=1 would
reboot this dual-boot desktop on any oops), and note that the API
server certificate is measured with openssl on the host instead of a
blackbox probe.
Risk: documentation only.
Verification: the PR numbers match the open and merged platform-core
PRs on 2026-11-12.
All SSD Local PVs share the root filesystem with the OS, container
images and build caches; review A12 and risk 2. Owner decision 3 asks
for a plan only.
Compare four options (shrink the Windows SSD, a new SSD, the HDD, do
nothing), recommend a new SSD and otherwise the Windows SSD, and give
the migration: mount the new filesystem at the same /srv/k3s/ssd path
so no PV changes, two rsync passes around a short stop, rollback. Note
the scripts whose root-SSD findmnt checks must change in the same PR.
Risk: documentation only; status 제안.
Verification: the PV sizes and paths come from the 2026-11-11 cluster
inventory; the findmnt checks were found with git grep.
The review found no procedure for losing the host, restoring the k3s
datastore or Gitea, upgrading k3s, or checking the platform after a
reboot, and the existing runbooks live outside version control.
Add five runbooks and a restart-window procedure in docs/runbooks:
after-reboot-checks (clock, k3s and certificates, node, pods in
dependency order, public paths, Prometheus, alerts, runner),
k3s-datastore-restore, gitea-restore (file-only vs full restore, with
the warning that a PITR of the shared database also rewinds Keycloak
and tech-log), k3s-upgrade (patch within a minor), host-rebuild (order
and the easy-to-miss parts: host name for PV node affinity, Tailscale
address in the DNS config, time sync before the first k3s start), and
k3s-restart-window (what to apply in the one restart window, with the
PriorityClasses first). The docs map gains runbooks/ and design/.
Each runbook starts with its preconditions and the commands that check
them; on 2026-11-12 the backups they need are not installed yet, and
they say so.
Risk: documentation only. Restores and the rebuild were not executed.
Verification: a fresh subagent with only these files ran every command
in after-reboot-checks on the desktop and walked the others read-only;
its findings (about 60, e.g. a READY check that missed 0/1 Running
pods, `stable` pointing to another minor, a missing pgAdmin directory,
time sync after the first k3s start) are fixed here. The revised
after-reboot commands were rerun on the desktop and match the text.
Every repository path the runbooks name exists on main or in the
named PR.
/etc/rancher/k3s/config.yaml.d/30-nodeport-loopback.yaml keeps Traefik's
NodePorts on 127.0.0.0/8 so the LAN cannot bypass Host Nginx. The
Traefik README and validator depend on it, but the file itself was only
on the host, so a rebuilt host would not get it.
Add the live content as is (kube-proxy-arg nodeport-addresses). The
host-rebuild and datastore-restore runbooks install it.
Risk: none; nothing installs it automatically.
Verification: the file is byte-for-byte the live drop-in
(ssh desktop cat ...); 40-secrets-encryption.yaml in the repository
already matches the host.
A Gitea dump must be restored with the Gitea version that wrote it, and
a restore checks the repository count against the dump. Neither was
recorded outside the archive, so the restore runbook had nothing to
compare with.
Upload a small .meta file with each dump: creation time, the
Deployment image, the gitea --version output and the repository count.
docs/runbooks/gitea-restore.md reads it.
Risk: none for the backup itself; the file holds no secret.
Verification: bash -n; kubectl -n gitea exec deploy/gitea -c gitea --
gitea --version prints "gitea version 1.27.0 built with ..." on the
desktop, so the third field is the version.
The PriorityClasses exist in the repository but no pod uses them, so
every platform pod still evicts at priority 0.
Attach platform-critical to Keycloak (CR spec.scheduling) and Gitea
(chart priorityClassName), and platform-low to Loki (singleBinary) and
Tempo. The Keycloak CR also gets spec.image at the running 26.7.0
digest with startOptimized: false: with spec.image the operator
otherwise assumes a pre-built optimized image, and this is the stock
one. Grafana and pgAdmin have no top-level priorityClassName in their
charts and are left for later; platform-postgres is W1's file and
tech-log belongs to its deploy/ directory.
Risk: applying restarts Keycloak, Gitea, Loki and Tempo once, so it
belongs in the k3s restart window, after the PriorityClasses exist (a
pod that names a missing class is rejected). The accepted-drift list
holds these objects until 2026-11-30.
Verification (desktop): render-phase1/2 and render-observability-core
pass with the four priorityClassName values; kubectl diff (server
dry-run) against live shows only these fields.
The Keycloak Operator remote resource is pinned to the tag 26.7.0, and
that tag's content changed once (live build-timestamp 2026-07-09,
later renders 2026-07-23). Both images use the 26.7.0 tag with
imagePullPolicy Always, so every restart depends on quay.io and could
pull different bits.
Pin the remote resource to the tag's commit, the operator image and
RELATED_IMAGE_KEYCLOAK to the digests that were running on 2026-11-12
(pod imageID) and set the operator to IfNotPresent. The env value is
set with a strategic merge patch keyed by name; a JSON patch by list
index hit KUBERNETES_NAMESPACE instead. render-phase2.sh and
apply-keycloak.sh check the commit and the digest.
Risk: applying restarts the Keycloak operator once (restart window).
Keycloak itself keeps running.
Verification (desktop): render-phase2.sh passes; the rendered
Deployment has the pinned image, IfNotPresent and the digest in
RELATED_IMAGE_KEYCLOAK with the other env entries unchanged.
AIStor runs inside the k3s cluster that the datastore archive restores.
After losing the whole cluster the AIStor copy cannot be read until k3s
is running again, which is what the archive is for.
Also keep the seven newest archives in
/srv/k3s/aistor/host-backups/k3s-datastore (root 0700) on the same HDD
filesystem (sdb3). The directory is next to AIStor's drive directory
data/, so AIStor does not list it as a bucket, and it is readable by
mounting sdb3 alone. A metric reports how many local copies exist.
Rejected: the NTFS archive partition (sdb2). It has no Unix
permissions and Windows on the same machine can read it; the archive
holds the server token.
Risk: the files live inside the AIStor PersistentVolume root, which is
mounted into the MinIO pod (uid 1000); mode 0700 root keeps the pod
from reading them.
Verification: bash -n; /srv/k3s/aistor contains only data/ (2026-11-12).
A backup nobody has restored is not known to work. The 2026-08 k3s
recovery bundle never got its isolated restore drill.
k3s-datastore-restore-drill.sh downloads the latest archive, checks its
sha256 and integrity, starts the same k3s version in a network-less
throwaway container on the copy, and checks the API comes up, the
Secret count is close to live and every Secret decrypts with the
token. gitea-backup-restore-drill.sh extracts the latest dump and
checks app.ini (INTERNAL_TOKEN, SECRET_KEY), git fsck on every
repository and the repository count against live Gitea. Both accept a
local copy and write a drill timestamp metric.
Risk: read-only for the cluster. The k3s drill runs the restored
control plane with --network none, so its controllers reach nothing.
Verification (desktop):
- test-k3s-datastore-restore-drill.sh: a fresh k3s with three test
Secrets (stored encrypted, k8s:enc:) restores with all Secrets
decrypted in 7s; before folding the WAL the test saw 2 of 3 rows,
which is why the real backup uses sqlite3 .backup.
- gitea-backup-restore-drill.sh on a live repository dump: 29
repositories, 0 fsck failures, 29 live; the dump was deleted after.
Backups, the daily drift check and the k3s serving certificates have no
alert. k3s renews certificates only when it starts within 120 days of
expiry, and the two kube-prometheus-stack kubelet certificate alerts
can never fire because the k3s kubelet exports no
kubelet_certificate_manager_* metric.
Add a PrometheusRule (platform-host-checks) for backup age, failed
runs, missing metrics, drift left for 6h, a stale drift check, and
serving certificate expiry (warning under 100 days, critical under 14).
Add the two remaining host timers (certificate expiry via openssl on
6443 and 10250; the drift check against origin/main) and an installer
that copies the scripts to a root-owned path, creates the textfile and
work directories and enables all four timers. Disable the two dead
kubelet certificate rules in kube-prometheus-stack.
Risk: the metrics need the node-exporter textfile collector, a one-line
change in W1's node-exporter values that was asked of W1. Until then
PlatformHostBackupMetricsAbsent fires after a day, which is the point.
Verification (desktop):
- test-platform-host-checks-alerts.sh: promtool check and unit tests
pass; with the k3s threshold changed from 2 to 5 days they fail.
- render-observability-core.sh passes; the render has no Kubelet*
certificate alerts and still has both KubeClientCertificateExpiration.
- k3s-cert-expiry.sh as the desktop user reads both certificates.
- install-host-checks.sh --check: systemd-analyze verify is clean.
Nothing backs up the k3s datastore or Gitea's repositories, packages and
app.ini. W1 covers platform-postgres only. One SSD failure loses the
source, CI artefacts, published libraries and every cluster Secret.
Add two root systemd timers that write to a new AIStor bucket on the
HDD (platform-host-backups, same AIStor as W1's postgres backups):
- k3s datastore, daily: sqlite3 .backup of the live state.db plus the
server token (k3s backup/restore docs: the token is needed to read
it back), integrity_check, tar.gz, sha256, upload. Off-host copy is
prepared behind two variables (gpg recipient file, rclone target) and
stays off until the owner chooses a place.
- Gitea, weekly: gitea dump --skip-db streamed from the live container,
listing checked for app.ini and repositories, sha256, upload. The DB
is in the CNPG backup.
Each run writes textfile metrics for the alerts in the next commit.
create-host-backup-credentials.sh makes the bucket (50GiB quota,
lifecycle 56 days for gitea/, 30 days for k3s-datastore/), a
bucket-scoped policy and user, and a root-only env file; no key is
printed and mc gets keys only through a 0600 env file.
Rejected: in-cluster CronJobs. The datastore is a root-owned host file
and the dump needs the Gitea DB connection; a hostPath root pod or a
second Gitea-labelled pod is a larger privilege than a root timer.
Risk: not installed here (owner sudo steps). The k3s archive holds the
token; together with the datastore it decrypts every Secret. It lives
in a bucket only the backup user can read.
Verification (desktop): a dump streamed with --file - is a valid
tar.gz with app.ini; repositories appear as repos/<owner>/<name>.git
(29, matching the live count). The scripts parse with bash -n.
The restart bundle needs root, touches k3s, kubelet and systemd files,
and must not leave the cluster half-configured. The W1 time-sync
drop-ins need a restart window too.
apply-k3s-restart-bundle.sh --check parses the files and prints the
install plan without root. --execute checks the cluster is Ready,
backs up every file it replaces plus a sqlite .backup of the datastore
and the token, installs the W6 files and W1's two time-sync drop-ins
as they are, enables systemd-time-wait-sync, restarts k3s once with
--no-block (the unit has TimeoutStartSec=0), then verifies the kubelet
configz values, the audit log, that the default namespace rejects a
privileged pod, three health URLs and the time-sync unit. Any failure
restores the backup and restarts again. Last, it creates the
local-storage .skip file and clears the local-path default annotation.
Risk: --execute was not run (owner sudo step). The API is unavailable
for one to two minutes; containers keep running (KillMode=process). The
script refuses to run until W1's host files are on main.
Verification (desktop, non-root):
- --check without the W1 files: MISSING ... (W1), exit 1.
- --check with the W1 files overlaid: parse and invariants pass, plan
printed, exit 0.
- --execute as non-root: "--execute needs root (sudo)".
The kubelet has no memory eviction threshold and no reservations, so
under memory pressure the kernel OOM killer picks a victim and
Postgres (oom_score_adj 985) is near the front. The API server writes
no audit log. Namespaces without labels have no Pod Security level, and
graceful node shutdown is off.
Add the files for one k3s restart: a kubelet drop-in (system 1 CPU/2Gi,
kube 0.5 CPU/1Gi, hard eviction memory 500Mi plus the two k3s disk
defaults, soft eviction 1Gi for 1m, shutdown 60s/20s), an API server
drop-in that appends the audit and admission-config flags, a Metadata
audit policy and a cluster Pod Security default (baseline, warn and
audit restricted, kube-system exempt).
Rejected: protect-kernel-defaults, recommended in review 7.3. CIS
dropped it in v1.7, and the sysctls it requires include
kernel.panic_on_oops=1, which reboots this dual-boot desktop on any
kernel oops; a reboot started the 2026-11-11 clock incident. The README
records the reason.
Risk: nothing is installed by this commit. The kubelet merges the
drop-in after k3s's 00-k3s-defaults.conf (k3s passes only --config-dir);
evictionHard lists all three keys because setting the map zeroes the
others.
Verification: the apply script's --check parses every file and asserts
the invariants (next commit).
The 2026-11-11 review found a pg_hba rule that existed only in the
cluster by rendering every root by hand and running kubectl diff.
Nothing does that regularly, so drift is found by accident.
live-drift-check.sh renders all live roots (Phase 1 and 2, admin
services, observability core, the Traefik trust overlay, the tenant,
PriorityClass and keycloak policy roots), strips Secrets and runs
kubectl diff, which is a server-side dry run. It prints one line per
drifted object, honours a dated accepted-drift list and can write
Prometheus textfile metrics for a later alert.
Risk: read-only. It consumes the render scripts' verified-output-dir
handoff and deletes those directories on exit; the README notes this
second consumer. Secret values are not compared.
Verification (desktop, before this series is applied): 17 drifted
objects, matching the review plus the new roots; exit 1; the textfile
holds platform_live_drift_objects 17.
The tech-log tenant needs objects inside platform namespaces: its
database and role, ingress to Postgres and AIStor, blackbox probes,
alert rules, a dashboard and two observability NetworkPolicies. They
were only ever applied with kubectl and no repository holds them, so a
deleted object cannot be recreated and a platform re-apply can drop
tech-log access without anyone noticing.
Declare them in services/tenants/techlog, copied from each object's
last-applied configuration. The dashboard follows the repository's
dashboard pattern (JSON file + configMapGenerator, same name, same
labels). The hyeonworks-prod Namespace moves here as the tenant
boundary and gets PSA enforce=baseline, warn/audit=restricted.
Rejected: enforce=restricted now. A server dry-run shows all three
tech-log pods would violate it; baseline has zero violations.
Risk: the render matches the live objects exactly except the added PSA
labels. The pg_hba rules and the Secret stay out of this root (Cluster
owner and Git policy).
Verification (desktop):
- render-tenants-and-policies.sh passes.
- kubectl diff of the render shows only the Namespace PSA labels.
- kubectl label --dry-run=server enforce=baseline: no warnings.
The keycloak namespace only has the ingress policy the Keycloak
Operator creates. Neither the Keycloak pod nor the Operator pod has any
egress restriction, unlike every other platform namespace.
Add a namespace-wide default deny and allow only what is used: DNS,
platform-postgres 5432 (server and realm-import jobs), JGroups between
Keycloak pods, and for the Operator the API server and the Keycloak
management ports. Keycloak ingress stays owned by the Operator policy.
Rejected: allowing public 443 like Gitea does. Neither the repository
nor the Keycloak CR configures SMTP or an external identity provider,
and the only open TCP connections from keycloak-0 on 2026-11-12 were to
PostgreSQL. Adding SMTP or an IdP later needs a new egress rule; the
README says so.
Risk: a missed dependency would break logins for every service. The
README lists the post-apply checks (OIDC discovery, a new TCP
connection to Postgres from keycloak-0, Operator logs) and the one-line
rollback.
Verification: render and kubectl diff only add the five policies.
Every platform pod runs at priority 0. During the 2026-11-09 disk
eviction the tech-log production pods were ranked with Grafana and
kube-state-metrics. The kubelet ranks evictions by usage over requests
and then by priority, so without classes the database and the
production app have no head start.
Add two classes and attach nothing yet: platform-critical (1000000,
may preempt) for platform-postgres, Gitea, Keycloak and tech-log, and
platform-low (-1000, never preempts) for Grafana, pgAdmin, Loki and
Tempo. Neither is the global default, so no pod changes silently.
Risk: creating the classes changes no pod. Attaching them restarts the
pods, so that is left for the k3s restart window.
Verification: render-tenants-and-policies.sh (added later in this
series) checks the values, the preemption policy and globalDefault.
The previous commit took the tech-log pg_hba change out of this PR
because W1's CNPG backup work owns the same Cluster manifest. Update the
review so it no longer says the fix is in this PR: section 6.1 now gives
W1 the exact change (commit 1c4ef9a to cherry-pick), its render
invariants and the RED/GREEN and kubectl diff evidence, and the W1
overlap table lists it.
Verification: grep finds no remaining "fixed in this PR" claim for
pg_hba; git diff origin/main...HEAD lists only docs/ and Traefik paths.
This reverts commit 1c4ef9a.
W1 owns the CNPG backup work, which changes the same Cluster manifest
(services/platform-postgres/base/cluster.yaml). The program rule is that
another session does not edit files in W1's scope; overlaps are recorded
as findings and handed over. The drift itself is unchanged: the live
Cluster still has two techlog pg_hba rules the repository does not
declare, so applying the current render would cut the production
backend off its database. The exact change, its render invariants and
the RED/GREEN evidence are in the maturity review, section 6.
Verification: git diff origin/main...HEAD no longer lists any
services/platform-postgres or scripts/validate/render-phase1.sh path.
The desktop k3s runs production and all CI on one node, and three
incidents this week came from that. Record where the configuration is
weak before the next one.
The review scores 16 axes from 0 to 5 against the k3s hardening guide,
the k3s CIS self-assessment, the NSA/CISA hardening guide and the
Kubernetes security checklist, lists the top ten risks, compares the
rendered repository with the live cluster, and summarizes kube-bench,
kube-linter, Polaris and kubescape runs. Items W1 is already fixing are
marked and not repeated. docs/README.md is the new docs map that the
docs-system convention asks for when a docs folder is added.
Risk: documentation only. Tool results were checked finding by finding;
21 kube-bench FAILs and several Polaris and kubescape items are false
positives and are reported as such.
Verification: every count and value in the review comes from read-only
commands run on 2026-11-11 (kubectl get/diff, configz, journal, lsblk,
openssl). The scanners ran with pinned digests and their images were
removed afterwards.
The k3s-bundled Traefik runs with no requests or limits, so it is a
BestEffort pod. The kubelet on this node has no memory eviction
threshold (the k3s default evictionHard only covers nodefs and
imagefs), so under memory pressure the kernel OOM killer chooses, and
BestEffort containers have oom_score_adj 1000. Every public route goes
through this one pod.
Set requests cpu 50m / memory 64Mi and a 256Mi memory limit in the base
and in both overlay patches, because the baseline and trust patches
replace valuesContent as a whole. No CPU limit, to avoid throttling.
Measured use on 2026-11-11 was about 5m CPU and 42Mi memory.
Rejected: raising Traefik to Guaranteed. It would reserve CPU the
ingress does not use, and the real fix for memory pressure is kubelet
reservation and eviction thresholds, which need a k3s restart.
Risk: applying the trust overlay makes the k3s helm-controller roll
Traefik once (RollingUpdate, a few seconds). Not applied here.
Verification (desktop):
- infrastructure/networking/traefik/scripts/validate.sh passes.
- kubectl diff of the trust overlay against the live HelmChartConfig
adds only the resources block.
- baseline and observe renders both contain the same resources.
The live platform-postgres Cluster has two pg_hba rules for the techlog
role that the repository does not declare. They were added outside the
repository for hyeonworks.com. Applying the current render would remove
them and the production backend could no longer reach its database.
Declare the same two rules, in the same position after the Keycloak
pair, and pin them with render invariants like the Gitea and Keycloak
pairs. The techlog DatabaseRole, Database, Secret and NetworkPolicies
stay outside this build root; where they belong is an owner decision
recorded in the 2026-11-11 maturity review.
Risk: none for the live cluster. The rendered Cluster now matches the
live object. A fresh install without the tech-log tenant gets two
rules for a role that does not exist, which PostgreSQL ignores.
Verification (desktop, Helm 3.19.4, Kustomize 5.8.1):
- render-phase1.sh with the old cluster.yaml and the new invariants
fails: "the tech-log role must authenticate only to the tech-log
database"; with the new cluster.yaml it passes.
- kubectl diff of the new render against the live cluster shows no
Cluster change; only the pre-existing DatabaseRole default fields.
On 2026-11-11 the desktop booted with the RTC read as UTC while it held
KST, so the clock ran 9 hours ahead until NTP pulled it back. Prometheus
had written one hour of samples stamped 18:36-19:36Z into its TSDB head
during that window. With outOfOrderTimeWindow at the chart default of
0s, every later scrape was older than the head's valid range and was
dropped as "out of bounds". From 11:21Z no metric was stored, no rule
could fire on real data, and KubeAPIDown (an absent() rule) fired
falsely. It would have stayed blind until the wall clock passed the
bogus head time, about 7 hours later.
Set prometheusSpec.tsdb.outOfOrderTimeWindow to 10h. Samples up to 10
hours behind the head are now accepted, which covers the +9h KST-as-UTC
misread. The README records why.
Rejected: deleting the head WAL from the Retain PV. It works at once
but removes persisted data and needs the pod stopped. Rejected: waiting
for the clock to catch up. It leaves monitoring blind for hours during
the period when the cause of the incident is still being found.
Risk: out-of-order samples use an extra in-memory head and vertical
compaction only while they arrive; in normal operation none arrive. A
genuinely late exporter could now backfill up to 10h, which is harmless
on this single-node Prometheus. Preventing the clock jump itself is the
host change in infrastructure/host, not this commit.
Verification: the values file parses and holds
prometheus.prometheusSpec.tsdb.outOfOrderTimeWindow=10h. The live
Prometheus CR already carried tsdb.outOfOrderTimeWindow=0s from this
chart key, so the key maps to spec.tsdb. The same value was patched
onto the live CR: the operator wrote out_of_order_time_window: 10h, the
config reloaded at 13:04:37Z, "too old" warnings dropped to 0, all 34
targets report up, and KubeAPIDown resolved. The pinned renderer
(scripts/validate/render-observability-core.sh) was not run: it requires
Helm v3.19.4 and kustomize v5.8.1, which are not installed on the
laptop or the desktop.
The Gitea README still said Actions and the package registry are
disabled on first deployment, and had no guidance for either.
Replace that decision with the 2026-11-07 state, add where packages
are stored and how much space they really have (a local PV does not
enforce the 50Gi request), the cleanup rules to use for Maven
SNAPSHOTs, and who may publish and read Maven packages. Record the
applied manifest SHA-256 and link the runner guide from the root
README.
Verification: statements checked against the Gitea 1.27 docs and
source (routers/api/packages/maven, modules/web/router.go), the live
app.ini, and API responses after the rollout.
The template repositories need a real runner for their CI, and the
owner chose one container on the desktop host (the k3s node).
Add compose.yaml and config.yaml for the official Gitea Runner 5.4.1
dind image, pinned by index digest, plus the operating guide. The
runner has capacity 1, one ubuntu-latest label pinned to
runner-images ubuntu-24.04-v26.11.03 by digest, restart: always, and
a 6 CPU / 10g cap (no more than half of the 12 cores and 31GiB).
Jobs use the dind daemon through the runner's Docker proxy for
Testcontainers; the host Docker socket is never mounted, job
containers stay unprivileged, valid_volumes is empty, and sweep
removes what a job created.
Dropped alternatives: the deprecated gitea/act_runner 0.6.1 (moved
to gitea/runner), the basic image with the host docker.sock (jobs
would own the host daemon), and dind-rootless (Testcontainers
compatibility needs separate checks).
Risk: the runner container is privileged; a job can still start a
privileged container on the inner daemon, so only trusted repository
workflows should run here. Runner jobs do not take the desktop's
heavy-work lock; the CPU and memory caps are the only guard.
Verification: config keys checked with `gitea-runner config get`,
`docker compose config` passes, the runner registered at user scope
and the Gitea API shows desktop-runner online and idle; it came back
online after `docker restart` without registering again.
The backend template is moving its messaging and gRPC libraries into
separate repositories and will consume them as Maven versions from
Gitea Packages. [packages] ENABLED=false and repo.packages in
DISABLED_REPO_UNITS make the registry unavailable.
Turn on [packages] and leave DISABLED_REPO_UNITS as an explicit empty
value. The chart init container runs `gitea config edit-ini
--apply-env` against the app.ini on the data volume, so a removed key
would keep the old disabled list; an empty value overwrites it.
SSH and LFS stay off. This ships with the Actions change in the same
rollout, so Gitea restarts once.
Risk: packages are stored under /data/packages on the root SSD local
PV. The 50Gi PVC size is not enforced by a local PV; the real bound is
free space on the host root disk, shared with the host and the CI
runner. No per-owner size limit is set yet.
Verification: render-phase1.sh invariants pass; kubectl diff against
live shows only the inline-config keys actions, packages, repository
and the Deployment checksum/config annotation.
Refs: Gitea 1.27 config cheat sheet [packages], usage/packages
The template repositories cannot run CI: [actions] ENABLED=false and
repo.actions in DISABLED_REPO_UNITS make Gitea ignore has_actions=true
on the repository API (200 response, value unchanged).
Turn on [actions] and drop only repo.actions from DISABLED_REPO_UNITS.
Packages stay disabled (packages.ENABLED=false, repo.packages kept).
Repositories created while the unit was disabled have no Actions unit
row, so every existing repository keeps has_actions=false until its
owner enables it. The values file is shared by the baseline and OIDC
build roots, so both renders change the same way.
Risk: new repositories get repo.actions by default (Gitea default
DEFAULT_REPO_UNITS). They run nothing without a runner in scope.
Gitea itself needs no new egress; runners connect inbound over the
existing Host Nginx -> Traefik HTTPS path.
Refs: Gitea 1.27 config cheat sheet [actions], [repository]