The Docker daemon inside the gitea-runner container keeps every job
image and all build cache in the gitea-runner_docker volume on the
root SSD that production k3s also uses. It grew to 16G in the two days
after the runner was installed and held 12.43G of images and 8.44G of
build cache on 2026-11-11. The README only said to look once a week
and prune by hand.
Mount daemon.json into the runner so the inner BuildKit GC keeps 2GB
to 10GB of build cache, and shrinks to 2GB while the disk has less
than 35GB free. Add gitea-runner-prune.service and .timer, a host
systemd timer for Sunday 04:30 KST that removes unused images created
more than 7 days ago and build cache older than 3 days. The service
runs as donghyeon in the docker group, so it needs root only to
install. A user timer was rejected because lingering is off for
donghyeon and user units stop with the last login session.
Risk: image prune filters on image creation time, so the pinned job
image is deleted weekly and the next job pulls about 543MB again.
Recreating the runner to pick up daemon.json cancels a running job,
so it is done only while the runner is idle.
Verification: systemd-analyze verify passes for both units, and the
calendar expression's next run is Sun 2026-11-15 04:30 KST. docker
compose config passes; dockerd --validate inside the runner (Docker
29.9.3) accepts daemon.json.
The host Docker daemon had no daemon.json. The default BuildKit policy
on this 228GiB disk keeps 21.42GiB of build cache untouched and allows
169.5GiB (docker buildx inspect default), so agent gate builds piled
up about 23G in two days before the 2026-11-09 disk incident.
Add infrastructure/host/docker/daemon.json with builder.gc
defaultReservedSpace 5GB, defaultMaxUsedSpace 15GB and
defaultMinFreeSpace 35GB. In Docker 29 the old defaultKeepStorage key
means defaultReservedSpace, a floor and not a cap, so the cap is
defaultMaxUsedSpace. 35GiB free is about 85% use, where the kubelet
starts deleting images; below it the cache shrinks to the floor first.
Moving the Docker data root to the HDD was rejected by the owner
because builds get slower.
Risk: applying needs sudo and a Docker restart, which restarts the
runner, workmachine and cloudflared and fails running CI jobs and
gates. The host apply guide added in this series says when that is
safe. dockerd --validate does not check keys inside builder.gc, so
the guide checks the live policy with docker buildx inspect.
Verification: dockerd --validate passes for the file on Docker
29.6.1; it also passes a typo inside builder.gc and rejects an unknown
top-level key, which is why the live check is in the guide.
The kubelet starts deleting unused images at 85% root use and evicts
pods at 90%. The root disk warning fired at 85% and the critical at
90%, so both arrived when the kubelet was already acting. On
2026-11-09 the warning fired for 33 hours, then an eviction stopped
the tech-log pods, the image GC deleted their imagePullPolicy: Never
images, and production went down.
Move the warning to 80% for 10m and the critical to 85% for 5m, and
add PlatformRootFilesystemFillingFast (critical, for 15m): the 6h
linear trend of free bytes falls below 10% of the disk, the eviction
point, within 6 hours. On 2026-11-07 the disk lost 17G in 9 hours;
this forecast would have fired hours before the 80% warning. The
info alert at 75% stays.
Add scripts/validate/test-platform-disk-alerts.sh, promtool unit
tests for the usage bands and the forecast with the pinned Prometheus
image. The existing rules test needs live metric inventories, so this
one runs offline.
Risk: the live rules change only when the observability access apply
runs; this commit does not touch the cluster. The forecast can fire
on a large build that the BuildKit GC removes minutes later; the 15m
hold and the 6h window damp that.
Verification: the new test failed against the old rules (no alert at
81%, 86% or for a 1-per-minute fill) and passes after the change;
promtool check rules finds 9 rules. test-observability-alerting.sh
passes its static contract, then stops at the helm render because
helm v3.19.4 is not installed on the desktop.
The Gitea README still said Actions and the package registry are
disabled on first deployment, and had no guidance for either.
Replace that decision with the 2026-11-07 state, add where packages
are stored and how much space they really have (a local PV does not
enforce the 50Gi request), the cleanup rules to use for Maven
SNAPSHOTs, and who may publish and read Maven packages. Record the
applied manifest SHA-256 and link the runner guide from the root
README.
Verification: statements checked against the Gitea 1.27 docs and
source (routers/api/packages/maven, modules/web/router.go), the live
app.ini, and API responses after the rollout.
The template repositories need a real runner for their CI, and the
owner chose one container on the desktop host (the k3s node).
Add compose.yaml and config.yaml for the official Gitea Runner 5.4.1
dind image, pinned by index digest, plus the operating guide. The
runner has capacity 1, one ubuntu-latest label pinned to
runner-images ubuntu-24.04-v26.11.03 by digest, restart: always, and
a 6 CPU / 10g cap (no more than half of the 12 cores and 31GiB).
Jobs use the dind daemon through the runner's Docker proxy for
Testcontainers; the host Docker socket is never mounted, job
containers stay unprivileged, valid_volumes is empty, and sweep
removes what a job created.
Dropped alternatives: the deprecated gitea/act_runner 0.6.1 (moved
to gitea/runner), the basic image with the host docker.sock (jobs
would own the host daemon), and dind-rootless (Testcontainers
compatibility needs separate checks).
Risk: the runner container is privileged; a job can still start a
privileged container on the inner daemon, so only trusted repository
workflows should run here. Runner jobs do not take the desktop's
heavy-work lock; the CPU and memory caps are the only guard.
Verification: config keys checked with `gitea-runner config get`,
`docker compose config` passes, the runner registered at user scope
and the Gitea API shows desktop-runner online and idle; it came back
online after `docker restart` without registering again.
The backend template is moving its messaging and gRPC libraries into
separate repositories and will consume them as Maven versions from
Gitea Packages. [packages] ENABLED=false and repo.packages in
DISABLED_REPO_UNITS make the registry unavailable.
Turn on [packages] and leave DISABLED_REPO_UNITS as an explicit empty
value. The chart init container runs `gitea config edit-ini
--apply-env` against the app.ini on the data volume, so a removed key
would keep the old disabled list; an empty value overwrites it.
SSH and LFS stay off. This ships with the Actions change in the same
rollout, so Gitea restarts once.
Risk: packages are stored under /data/packages on the root SSD local
PV. The 50Gi PVC size is not enforced by a local PV; the real bound is
free space on the host root disk, shared with the host and the CI
runner. No per-owner size limit is set yet.
Verification: render-phase1.sh invariants pass; kubectl diff against
live shows only the inline-config keys actions, packages, repository
and the Deployment checksum/config annotation.
Refs: Gitea 1.27 config cheat sheet [packages], usage/packages
The template repositories cannot run CI: [actions] ENABLED=false and
repo.actions in DISABLED_REPO_UNITS make Gitea ignore has_actions=true
on the repository API (200 response, value unchanged).
Turn on [actions] and drop only repo.actions from DISABLED_REPO_UNITS.
Packages stay disabled (packages.ENABLED=false, repo.packages kept).
Repositories created while the unit was disabled have no Actions unit
row, so every existing repository keeps has_actions=false until its
owner enables it. The values file is shared by the baseline and OIDC
build roots, so both renders change the same way.
Risk: new repositories get repo.actions by default (Gitea default
DEFAULT_REPO_UNITS). They run nothing without a runner in scope.
Gitea itself needs no new egress; runners connect inbound over the
existing Host Nginx -> Traefik HTTPS path.
Refs: Gitea 1.27 config cheat sheet [actions], [repository]