21 Commits
Author SHA1 Message Date
DongHyeonka 0d703ff249 docs(postgres): record why the backups go to the HDD
The owner asked on 2026-11-12 that production database backups stay
off the SSD. State the evidence that the AIStor volume is the HDD
(aistor-data-local-pv -> /srv/k3s/aistor -> /dev/sdb3, WD20EZAZ,
ROTA=1), why the NTFS archive on sdb2 was not used (no POSIX
permissions, no S3 API, a daily dump gives no point-in-time restore),
and that the restore drill uses a short-lived local-path volume on the
SSD because the HDD has no dynamic StorageClass.
2026-11-12 07:41:43 +09:00
DongHyeonka d92dc5d163 feat(observability): read node-exporter textfile metrics
The host timers from the k3s maturity work (last Gitea dump and k3s
datastore backup, live drift count, API server and kubelet certificate
expiry) write their results as *.prom files. node-exporter does not
read any textfile directory, so Prometheus never sees them.

Pass --collector.textfile.directory pointing at
/var/lib/node_exporter/textfile_collector under the read-only host
root mount the chart already has. No volume is added. The host
installer creates the directory; until then the collector only
reports an error metric.

Verification: helm template of prometheus-node-exporter 4.56.1 with
these values renders the argument next to --path.rootfs=/host/root.
2026-11-12 07:29:48 +09:00
DongHyeonka 4551a2ece6 feat(platform-postgres): run the shared database as Guaranteed QoS
The one PostgreSQL instance holds the Gitea, Keycloak and tech-log
databases. It is Burstable at priority 0 with oom_score_adj 985, so
under memory pressure the kernel kills it before almost anything else,
and an eviction treats it like Grafana (k3s maturity review
2026-11-11, 7.2 items 2 and 3).

Set priorityClassName platform-critical (services/priority-classes)
and raise the requests to the existing limits, CPU 2 and memory 4Gi,
so the pod is Guaranteed and its oom_score_adj becomes -997. The
backup README now gives the restart, not pg_hba, as the reason to add
spec.backup with kubectl patch.

Risk: applying this changes the pod spec, so CloudNativePG restarts
the single instance and the three databases are down for that time.
It belongs in the k3s restart window. Node requests grow by 1.75 CPU
and 3.5Gi; peak use over the last 6 hours was 0.025 CPU and 195MB.

Verification: kubectl diff of the rendered build root against the live
cluster shows only priorityClassName, the two requests and the
DatabaseRole defaults that were already there; no pg_hba line differs.
render-phase1.sh passes.
2026-11-12 07:29:48 +09:00
DongHyeonka d47d57aa14 fix(platform-postgres): keep the tech-log pg_hba rules the cluster uses
The live platform-postgres Cluster has two pg_hba rules for the techlog
role that the repository does not declare. They were added outside the
repository for hyeonworks.com. Applying the current render would remove
them and the production backend could no longer reach its database.

Declare the same two rules, in the same position after the Keycloak
pair, and pin them with render invariants like the Gitea and Keycloak
pairs. The techlog DatabaseRole, Database, Secret and NetworkPolicies
stay outside this build root; where they belong is an owner decision
recorded in the 2026-11-11 maturity review.

Risk: none for the live cluster. The rendered Cluster now matches the
live object. A fresh install without the tech-log tenant gets two
rules for a role that does not exist, which PostgreSQL ignores.

Verification (desktop, Helm 3.19.4, Kustomize 5.8.1):
- render-phase1.sh with the old cluster.yaml and the new invariants
  fails: "the tech-log role must authenticate only to the tech-log
  database"; with the new cluster.yaml it passes.
- kubectl diff of the new render against the live cluster shows no
  Cluster change; only the pre-existing DatabaseRole default fields.
2026-11-12 07:28:31 +09:00
DongHyeonka d7c845ca5d refactor(postgres): keep the backup alerts with the backup
The observability core renderer rejects any alert or record whose
name contains "backup" (no_unscoped_monitors_rules): backup
monitoring is outside the observability core by design. f19ce54 put
PlatformPostgresBackupMissing and PlatformPostgresWALArchiveBacklog in
core-rules.yaml, so the pinned renderer failed with "out-of-scope
monitor or rule is present".

Move both alerts unchanged into
services/platform-postgres-backup/prometheus-rule.yaml in
platform-data, which Prometheus reads because the namespace has
observability.hyeonworks.com/enabled=true and the rule has
observability.hyeonworks.com/instance=home. They now point their
runbook_url at the backup README and are applied with the
ScheduledBackup. Renaming them to dodge the filter was rejected: it
would keep backup rules inside a scope that excludes them on purpose.

Verification: test-platform-disk-alerts.sh loads both rule files and
passes; promtool checks 13 rules. render-observability-core.sh and
test-render-observability-core.sh pass, and kubectl apply
--dry-run=server accepts the backup kustomization with the rule.
scan-platform-sensitive-source.sh stops at its observability-core
renderer with RC=23 here and on an origin/main snapshot alike, so it
says nothing about this change.

Fixes: f19ce54740 ("feat(postgres): back up platform-postgres to AIStor every day")
2026-11-12 07:28:31 +09:00
DongHyeonka bed20b456b fix(observability): stop throttling node-exporter
node-exporter averages about 2m CPU, but each scrape runs its
collectors in parallel. With a 250m limit (25ms per 100ms CFS period)
26% of its CFS periods were throttled over 30 minutes on 2026-11-11
(container_cpu_cfs_throttled_periods_total / cfs_periods_total), which
delays scrapes and can make them time out.

Raise the CPU limit to one core. The request stays 100m, so
scheduling does not change, and a runaway collector is still capped.
Removing the limit was rejected because every other workload in this
repository sets one.

Risk: none beyond one core of headroom on a 12-core node. The live
DaemonSet changes only when the observability core is applied again.

Verification: the throttling ratio and usage came from live
Prometheus queries. helm template of prometheus-node-exporter 4.56.1
with these values renders limits.cpu "1". The effect after rollout is
not measured yet.
2026-11-12 07:28:03 +09:00
DongHyeonka 619b90552f feat(observability): alert when Prometheus itself stops working
Nothing tells the owner when monitoring itself fails. On 2026-11-11
Prometheus stored samples 9 hours ahead, then rejected every new
sample as out of bounds: rules kept running on no data, and the only
sign was a false KubeAPIDown. If Prometheus, Alertmanager or the node
went down, no rule could fire at all.

Add two rules and one route. PlatformMetricsIngestionStopped fires
when no up sample from node-exporter is stored for 5 minutes;
absent() still fires while the TSDB rejects samples. PlatformHeartbeat
always fires, and a new platform-heartbeat route sends it every 5
minutes to an external dead man's switch through a webhook whose URL
comes from the Secret alertmanager-heartbeat. When the pings stop,
that service alerts, which covers Prometheus, Alertmanager, the node
and the network being down. The chart Watchdog stays disabled; the
platform rule keeps the route inside this repository.

Rejected: an in-cluster CronJob that queries Prometheus and posts to
Alertmanager. It still depends on Alertmanager and the node, and
needs a new workload in the exact observability NetworkPolicy matrix.

Risk: the AlertmanagerConfig now references alertmanager-heartbeat;
applying it before that Secret exists keeps the operator from
rendering the new Alertmanager config. The alerting README orders the
steps: create the external check, create the Secret, then apply.
The heartbeat needs an account on an external service, which only
the owner can create.

Verification: promtool tests for both rules failed against the old
rules and pass now. test-observability-alerting.sh pins the new route
and receiver in its exact contract and passes the static contract,
the global Alertmanager render with Helm v3.19.4 and the server
dry run (--server-dry-run). kubectl apply --dry-run=server accepts
the updated PrometheusRule.
2026-11-12 07:28:03 +09:00
DongHyeonka f19ce54740 feat(postgres): back up platform-postgres to AIStor every day
platform-postgres holds the Gitea, Keycloak and tech-log databases and
had no backup: no spec.backup, no ScheduledBackup, and its only copy
is a local PV on the root SSD. A lost SSD or a bad migration had no
way back.

Add services/platform-postgres-backup: a merge patch that gives the
live Cluster barman-cloud WAL archiving and base backups to
s3://platform-postgres-backups on the AIStor ObjectStore (HDD sdb3)
with 30-day retention and gzip, a daily 03:00 KST ScheduledBackup that
also runs once at creation, the object-storage ingress for the
PostgreSQL pods, the bucket policy, and a restore drill Cluster that
only reads the bucket. scripts/bootstrap/create-postgres-backup-
credentials.sh creates the bucket (100GiB quota), policy, user and
the platform-data Secret through a restricted AIStor client Pod,
proves the user can write its bucket and cannot list loki, and prints
no key. Add PlatformPostgresBackupMissing (no base backup in 26h) and
PlatformPostgresWALArchiveBacklog (more than 10 WAL files waiting for
15m; a stuck archive fills the SSD volume).

The native barmanObjectStore is deprecated since CNPG 1.26 and the
operator warns it is removed in 1.31.0. The Barman Cloud Plugin needs
cert-manager, which this cluster does not have; installing two
operators before the first backup is slower and riskier. The bucket
layout is the same, so the plugin can take over before CNPG 1.31.
Volume snapshots do not work on the static local PV.

Risk: the live Cluster has pg_hba lines for techlog that the Phase 1
source lacks, so re-applying services/platform-postgres would cut the
tech-log backend off. The README applies only spec.backup with
kubectl patch. Backups stay on the same desktop; an off-machine copy
is not part of this change. Nothing was applied to the cluster.

Verification: kubectl apply --dry-run=server accepts the
ScheduledBackup, the NetworkPolicy and the drill Cluster; kubectl
patch --dry-run=server keeps both techlog pg_hba lines and adds
spec.backup. The row-count comparison from the drill steps ran
read-only on production (keycloak 100, techlog 44, gitea 116 tables).
mc command syntax was checked with the pinned AIStor client offline.
The two new alert tests failed against the old rules and pass now.
bash -n passes for the script; shellcheck is not installed on the
laptop or the desktop.
2026-11-12 07:28:03 +09:00
DongHyeonka 7fe38428e3 feat(observability): alert on the root disk before kubelet acts
The kubelet starts deleting unused images at 85% root use and evicts
pods at 90%. The root disk warning fired at 85% and the critical at
90%, so both arrived when the kubelet was already acting. On
2026-11-09 the warning fired for 33 hours, then an eviction stopped
the tech-log pods, the image GC deleted their imagePullPolicy: Never
images, and production went down.

Move the warning to 80% for 10m and the critical to 85% for 5m, and
add PlatformRootFilesystemFillingFast (critical, for 15m): the 6h
linear trend of free bytes falls below 10% of the disk, the eviction
point, within 6 hours. On 2026-11-07 the disk lost 17G in 9 hours;
this forecast would have fired hours before the 80% warning. The
info alert at 75% stays.

Add scripts/validate/test-platform-disk-alerts.sh, promtool unit
tests for the usage bands and the forecast with the pinned Prometheus
image. The existing rules test needs live metric inventories, so this
one runs offline.

Risk: the live rules change only when the observability access apply
runs; this commit does not touch the cluster. The forecast can fire
on a large build that the BuildKit GC removes minutes later; the 15m
hold and the 6h window damp that.

Verification: the new test failed against the old rules (no alert at
81%, 86% or for a 1-per-minute fill) and passes after the change;
promtool check rules finds 9 rules. test-observability-alerting.sh
passes its static contract, then stops at the helm render because
helm v3.19.4 is not installed on the desktop.
2026-11-12 07:28:03 +09:00
donghyeon.kang 28595c47e2 Merge pull request 'feat(keycloak): pin images and attach PriorityClasses for the restart window' (#7) from donghyeon.kang/w6-restart-window-pins into main 2026-11-11 16:48:42 +00:00
DongHyeonka de600a0bb1 feat(priority): attach PriorityClasses to Keycloak, Gitea, Loki, Tempo
The PriorityClasses exist in the repository but no pod uses them, so
every platform pod still evicts at priority 0.

Attach platform-critical to Keycloak (CR spec.scheduling) and Gitea
(chart priorityClassName), and platform-low to Loki (singleBinary) and
Tempo. The Keycloak CR also gets spec.image at the running 26.7.0
digest with startOptimized: false: with spec.image the operator
otherwise assumes a pre-built optimized image, and this is the stock
one. Grafana and pgAdmin have no top-level priorityClassName in their
charts and are left for later; platform-postgres is W1's file and
tech-log belongs to its deploy/ directory.

Risk: applying restarts Keycloak, Gitea, Loki and Tempo once, so it
belongs in the k3s restart window, after the PriorityClasses exist (a
pod that names a missing class is rejected). The accepted-drift list
holds these objects until 2026-11-30.

Verification (desktop): render-phase1/2 and render-observability-core
pass with the four priorityClassName values; kubectl diff (server
dry-run) against live shows only these fields.
2026-11-12 01:25:24 +09:00
DongHyeonka 9e9ea94461 feat(backup): keep k3s datastore archives outside AIStor on the HDD
AIStor runs inside the k3s cluster that the datastore archive restores.
After losing the whole cluster the AIStor copy cannot be read until k3s
is running again, which is what the archive is for.

Also keep the seven newest archives in
/srv/k3s/aistor/host-backups/k3s-datastore (root 0700) on the same HDD
filesystem (sdb3). The directory is next to AIStor's drive directory
data/, so AIStor does not list it as a bucket, and it is readable by
mounting sdb3 alone. A metric reports how many local copies exist.

Rejected: the NTFS archive partition (sdb2). It has no Unix
permissions and Windows on the same machine can read it; the archive
holds the server token.

Risk: the files live inside the AIStor PersistentVolume root, which is
mounted into the MinIO pod (uid 1000); mode 0700 root keeps the pod
from reading them.

Verification: bash -n; /srv/k3s/aistor contains only data/ (2026-11-12).
2026-11-12 01:14:45 +09:00
DongHyeonka 012452cf78 feat(observability): alert on host backups, drift and k3s serving certs
Backups, the daily drift check and the k3s serving certificates have no
alert. k3s renews certificates only when it starts within 120 days of
expiry, and the two kube-prometheus-stack kubelet certificate alerts
can never fire because the k3s kubelet exports no
kubelet_certificate_manager_* metric.

Add a PrometheusRule (platform-host-checks) for backup age, failed
runs, missing metrics, drift left for 6h, a stale drift check, and
serving certificate expiry (warning under 100 days, critical under 14).
Add the two remaining host timers (certificate expiry via openssl on
6443 and 10250; the drift check against origin/main) and an installer
that copies the scripts to a root-owned path, creates the textfile and
work directories and enables all four timers. Disable the two dead
kubelet certificate rules in kube-prometheus-stack.

Risk: the metrics need the node-exporter textfile collector, a one-line
change in W1's node-exporter values that was asked of W1. Until then
PlatformHostBackupMetricsAbsent fires after a day, which is the point.

Verification (desktop):
- test-platform-host-checks-alerts.sh: promtool check and unit tests
  pass; with the k3s threshold changed from 2 to 5 days they fail.
- render-observability-core.sh passes; the render has no Kubelet*
  certificate alerts and still has both KubeClientCertificateExpiration.
- k3s-cert-expiry.sh as the desktop user reads both certificates.
- install-host-checks.sh --check: systemd-analyze verify is clean.
2026-11-12 01:13:02 +09:00
DongHyeonka 77daa39a41 feat(backup): back up the k3s datastore and Gitea to AIStor
Nothing backs up the k3s datastore or Gitea's repositories, packages and
app.ini. W1 covers platform-postgres only. One SSD failure loses the
source, CI artefacts, published libraries and every cluster Secret.

Add two root systemd timers that write to a new AIStor bucket on the
HDD (platform-host-backups, same AIStor as W1's postgres backups):
- k3s datastore, daily: sqlite3 .backup of the live state.db plus the
  server token (k3s backup/restore docs: the token is needed to read
  it back), integrity_check, tar.gz, sha256, upload. Off-host copy is
  prepared behind two variables (gpg recipient file, rclone target) and
  stays off until the owner chooses a place.
- Gitea, weekly: gitea dump --skip-db streamed from the live container,
  listing checked for app.ini and repositories, sha256, upload. The DB
  is in the CNPG backup.
Each run writes textfile metrics for the alerts in the next commit.
create-host-backup-credentials.sh makes the bucket (50GiB quota,
lifecycle 56 days for gitea/, 30 days for k3s-datastore/), a
bucket-scoped policy and user, and a root-only env file; no key is
printed and mc gets keys only through a 0600 env file.

Rejected: in-cluster CronJobs. The datastore is a root-owned host file
and the dump needs the Gitea DB connection; a hostPath root pod or a
second Gitea-labelled pod is a larger privilege than a root timer.

Risk: not installed here (owner sudo steps). The k3s archive holds the
token; together with the datastore it decrypts every Secret. It lives
in a bucket only the backup user can read.

Verification (desktop): a dump streamed with --file - is a valid
tar.gz with app.ini; repositories appear as repos/<owner>/<name>.git
(29, matching the live count). The scripts parse with bash -n.
2026-11-12 01:13:02 +09:00
DongHyeonka ebdf33f072 feat(tenants): declare the tech-log objects inside platform namespaces
The tech-log tenant needs objects inside platform namespaces: its
database and role, ingress to Postgres and AIStor, blackbox probes,
alert rules, a dashboard and two observability NetworkPolicies. They
were only ever applied with kubectl and no repository holds them, so a
deleted object cannot be recreated and a platform re-apply can drop
tech-log access without anyone noticing.

Declare them in services/tenants/techlog, copied from each object's
last-applied configuration. The dashboard follows the repository's
dashboard pattern (JSON file + configMapGenerator, same name, same
labels). The hyeonworks-prod Namespace moves here as the tenant
boundary and gets PSA enforce=baseline, warn/audit=restricted.

Rejected: enforce=restricted now. A server dry-run shows all three
tech-log pods would violate it; baseline has zero violations.

Risk: the render matches the live objects exactly except the added PSA
labels. The pg_hba rules and the Secret stay out of this root (Cluster
owner and Git policy).

Verification (desktop):
- render-tenants-and-policies.sh passes.
- kubectl diff of the render shows only the Namespace PSA labels.
- kubectl label --dry-run=server enforce=baseline: no warnings.
2026-11-12 00:29:18 +09:00
DongHyeonka c2a3f75bc3 feat(priority): add platform-critical and platform-low PriorityClasses
Every platform pod runs at priority 0. During the 2026-11-09 disk
eviction the tech-log production pods were ranked with Grafana and
kube-state-metrics. The kubelet ranks evictions by usage over requests
and then by priority, so without classes the database and the
production app have no head start.

Add two classes and attach nothing yet: platform-critical (1000000,
may preempt) for platform-postgres, Gitea, Keycloak and tech-log, and
platform-low (-1000, never preempts) for Grafana, pgAdmin, Loki and
Tempo. Neither is the global default, so no pod changes silently.

Risk: creating the classes changes no pod. Attaching them restarts the
pods, so that is left for the k3s restart window.

Verification: render-tenants-and-policies.sh (added later in this
series) checks the values, the preemption policy and globalDefault.
2026-11-12 00:28:57 +09:00
DongHyeonka af5a90bef8 fix(observability): let Prometheus accept samples after a clock jump
On 2026-11-11 the desktop booted with the RTC read as UTC while it held
KST, so the clock ran 9 hours ahead until NTP pulled it back. Prometheus
had written one hour of samples stamped 18:36-19:36Z into its TSDB head
during that window. With outOfOrderTimeWindow at the chart default of
0s, every later scrape was older than the head's valid range and was
dropped as "out of bounds". From 11:21Z no metric was stored, no rule
could fire on real data, and KubeAPIDown (an absent() rule) fired
falsely. It would have stayed blind until the wall clock passed the
bogus head time, about 7 hours later.

Set prometheusSpec.tsdb.outOfOrderTimeWindow to 10h. Samples up to 10
hours behind the head are now accepted, which covers the +9h KST-as-UTC
misread. The README records why.

Rejected: deleting the head WAL from the Retain PV. It works at once
but removes persisted data and needs the pod stopped. Rejected: waiting
for the clock to catch up. It leaves monitoring blind for hours during
the period when the cause of the incident is still being found.

Risk: out-of-order samples use an extra in-memory head and vertical
compaction only while they arrive; in normal operation none arrive. A
genuinely late exporter could now backfill up to 10h, which is harmless
on this single-node Prometheus. Preventing the clock jump itself is the
host change in infrastructure/host, not this commit.

Verification: the values file parses and holds
prometheus.prometheusSpec.tsdb.outOfOrderTimeWindow=10h. The live
Prometheus CR already carried tsdb.outOfOrderTimeWindow=0s from this
chart key, so the key maps to spec.tsdb. The same value was patched
onto the live CR: the operator wrote out_of_order_time_window: 10h, the
config reloaded at 13:04:37Z, "too old" warnings dropped to 0, all 34
targets report up, and KubeAPIDown resolved. The pinned renderer
(scripts/validate/render-observability-core.sh) was not run: it requires
Helm v3.19.4 and kustomize v5.8.1, which are not installed on the
laptop or the desktop.
2026-11-11 22:09:23 +09:00
DongHyeonka a92e6b854b docs(gitea): record the Actions and package registry rollout
The Gitea README still said Actions and the package registry are
disabled on first deployment, and had no guidance for either.

Replace that decision with the 2026-11-07 state, add where packages
are stored and how much space they really have (a local PV does not
enforce the 50Gi request), the cleanup rules to use for Maven
SNAPSHOTs, and who may publish and read Maven packages. Record the
applied manifest SHA-256 and link the runner guide from the root
README.

Verification: statements checked against the Gitea 1.27 docs and
source (routers/api/packages/maven, modules/web/router.go), the live
app.ini, and API responses after the rollout.
2026-11-07 18:36:55 +09:00
DongHyeonka 87dee1f749 feat(gitea): enable the package registry
The backend template is moving its messaging and gRPC libraries into
separate repositories and will consume them as Maven versions from
Gitea Packages. [packages] ENABLED=false and repo.packages in
DISABLED_REPO_UNITS make the registry unavailable.

Turn on [packages] and leave DISABLED_REPO_UNITS as an explicit empty
value. The chart init container runs `gitea config edit-ini
--apply-env` against the app.ini on the data volume, so a removed key
would keep the old disabled list; an empty value overwrites it.
SSH and LFS stay off. This ships with the Actions change in the same
rollout, so Gitea restarts once.

Risk: packages are stored under /data/packages on the root SSD local
PV. The 50Gi PVC size is not enforced by a local PV; the real bound is
free space on the host root disk, shared with the host and the CI
runner. No per-owner size limit is set yet.

Verification: render-phase1.sh invariants pass; kubectl diff against
live shows only the inline-config keys actions, packages, repository
and the Deployment checksum/config annotation.

Refs: Gitea 1.27 config cheat sheet [packages], usage/packages
2026-11-07 18:22:23 +09:00
DongHyeonka 7cd4a419fd feat(gitea): enable Actions at the server level
The template repositories cannot run CI: [actions] ENABLED=false and
repo.actions in DISABLED_REPO_UNITS make Gitea ignore has_actions=true
on the repository API (200 response, value unchanged).

Turn on [actions] and drop only repo.actions from DISABLED_REPO_UNITS.
Packages stay disabled (packages.ENABLED=false, repo.packages kept).
Repositories created while the unit was disabled have no Actions unit
row, so every existing repository keeps has_actions=false until its
owner enables it. The values file is shared by the baseline and OIDC
build roots, so both renders change the same way.

Risk: new repositories get repo.actions by default (Gitea default
DEFAULT_REPO_UNITS). They run nothing without a runner in scope.
Gitea itself needs no new egress; runners connect inbound over the
existing Host Nginx -> Traefik HTTPS path.

Refs: Gitea 1.27 config cheat sheet [actions], [repository]
2026-11-07 17:37:52 +09:00
donghyeon-ka 16c337bcc9 Add platform infrastructure configuration 2026-08-28 17:35:41 +09:00