Author SHA1 Message Date
donghyeon.kang d0875bfe44 Merge pull request 'feat(ci): add a read-only CI metrics script for the desktop runner' (#12) from claude/w2-ci-metrics into main 2026-11-13 10:43:35 +00:00
DongHyeonka dd8e9140b7 feat(ci): add a read-only CI metrics script for the desktop runner
There is no number for how long CI runs wait, how busy the one runner is,
how many runs a pull request costs or how long main stays red. Run times
were added up by hand from single runs, so whether a change helped could
not be told apart from load.

infrastructure/ci/metrics/ci_metrics.py reads the Gitea Actions API (runs,
their jobs, closed pull requests) for the repositories the runner serves
and prints the CI manual's metrics 1-5 per repository and stage: run
execution p50/p90, queue wait p50/p90, runner utilisation per KST day,
pull request runs per merged pull request, and main red time. It only
reads; credentials come from GITEA_TOKEN or `git credential fill` and are
never printed.

The arithmetic follows what our Gitea 1.27 returns, which differs from the
API reference: a run has no created_at (the earliest job created_at is
used), unstarted runs and jobs report 1970-01-01 times (treated as
absent), and a job's created_at is its run's, so queue wait is measured
per run (first job start minus run creation) instead of per job.

Verification: python3 -m unittest test_ci_metrics (11 offline tests) and
one run against the server for 2026-11-07..13 (two repositories, 30 s,
rc=0). The sensitive-source scan needs rg and ran on the desktop host.

Refs: design-docs ci/03-time-reduction-plan.md M-0
2026-11-13 19:34:38 +09:00
donghyeon.kang 5d27096ed0 Merge pull request 'fix(validate): run the sensitive-source scan from any checkout' (#11) from donghyeon.kang/w6-scanner-fix into main 2026-11-11 23:46:20 +00:00
DongHyeonka ae4e256c95 fix(validate): accept CloudNativePG s3Credentials Secret references
PR #10 added the platform-postgres backup patch and its restore-drill
Cluster. Both hold barmanObjectStore.s3Credentials, where accessKeyId
and secretAccessKey are SecretKeySelectors (name and key). The
structured scan treats any non-string value under a key ending in
"access key" as a credential literal, so the source scan reported
CREDENTIAL_LITERAL for both files on main.

An entry directly under s3Credentials, named accessKeyId or
secretAccessKey, whose value is a mapping of exactly name and key
with non-empty strings, is now a reference. This mirrors the existing
exact rule for the Grafana client-secret secretKeyRef.

Rejected: a general "name and key is always a reference" rule. It
would also pass a literal hidden under some other credential key that
happens to have those two fields.

Risk: none beyond the two key names under that one parent. A literal
string, an extra child, or the same mapping under another parent is
still rejected (new test cases).

Verification: on desktop, the source scan of main 205927f reported the
two files before this change and passes after it. The unit test
passes in a non-canonical clone.
2026-11-12 08:36:38 +09:00
DongHyeonka 27eb8c1799 fix(validate): render the sensitive-source scan from a clean copy
The scanner ran the five renderers inside the scanned checkout. The
desktop checkout holds eight .helm chart caches left on 2026-08-11
(ignored by git), and render-observability-core.sh refuses an existing
exact chart cache. The scan stopped with "renderer failed:
observability-core RC=23" before it checked any rendered Secret, on
main and on every branch.

In a git checkout the renderers now run from a private 0700 copy in
/tmp of the tracked and not-ignored files, minus files deleted from
the working tree (git ls-files, then tar). The copy is removed with
the other renderer temp paths. A tree that is not a git checkout, such
as the test fixtures, still renders in place.

Rejected: deleting the stale caches only. That fixes one host until a
cache is left again, and the scanner is not the place to delete files
in the tree it scans. The renderer's refusal of an existing cache
stays; it guards the chart digest checks.

Risk: an ignored file is not in the copy. No renderer reads one today
(no renderer references .env, local-recovery.env, .superpowers or
secrets/). If one ever does, its render fails closed rather than
scanning less.

Verification: on desktop, real renderers against a clone of main
205927f with the eight stale caches copied in. Before: phase1, phase2
and admin pass, then "renderer failed: observability-core RC=23".
With this branch: all five renderers pass, "PLATFORM RENDERED SECRET
SCAN PASS", the eight caches in the scanned clone are untouched and no
render copy is left in /tmp. The new test cases fail before this
change and pass after it.
2026-11-12 08:36:38 +09:00
DongHyeonka 149b885e35 fix(validate): scan the checkout that holds the sensitive-source scanner
scan-platform-sensitive-source.sh always scanned the fixed path
/home/donghyeon/workspace/platform, and its list of shell files with
embedded Python held the same fixed paths. Run from a temporary copy
or a CI runner, it scanned the desktop checkout instead of itself, or
nothing at all. Its own test failed at the first assertion in any
other checkout: the test file lost its embedded-Python context there
and was reported as a credential literal.

The platform root is now the checkout that contains the scanner
(dirname of BASH_SOURCE, physical path), and the embedded-Python list
is resolved under that root. The central documentation root stays the
desktop default. --docs-root DIR replaces it and --platform-only skips
it, for a copy or runner that has no central docs. A missing docs root
stops with exit 2 and names both options instead of scanning less.

Rejected: a --platform-root option. Scanning a tree with a scanner
from another tree would let a scanned tree differ from the rules that
scan it.

Risk: on the desktop host the default run is unchanged. A caller that
passed arguments before got usage and exit 2. It now gets the new
options.

Verification: on desktop, a non-canonical clone of main 205927f with
this change: test-scan-platform-sensitive-source.sh passes. Before the
change the same test in a worktree copy failed with "canonical scanner
test content leaf found a credential literal".
2026-11-12 08:02:05 +09:00
donghyeon.kang 205927f4e7 Merge pull request 'feat(platform): guard the disk, clock, backups and monitoring after this week's incidents' (#10) from claude/w1-infra-hardening into main 2026-11-11 22:43:49 +00:00
DongHyeonka 0d703ff249 docs(postgres): record why the backups go to the HDD
The owner asked on 2026-11-12 that production database backups stay
off the SSD. State the evidence that the AIStor volume is the HDD
(aistor-data-local-pv -> /srv/k3s/aistor -> /dev/sdb3, WD20EZAZ,
ROTA=1), why the NTFS archive on sdb2 was not used (no POSIX
permissions, no S3 API, a daily dump gives no point-in-time restore),
and that the restore drill uses a short-lived local-path volume on the
SSD because the HDD has no dynamic StorageClass.
2026-11-12 07:41:43 +09:00
DongHyeonka c04d79481c docs(host): point the time-sync step at the k3s restart bundle
scripts/bootstrap/apply-k3s-restart-bundle.sh now installs the two
time-sync drop-ins and enables systemd-time-wait-sync together with the
kubelet drop-ins, so the owner restarts k3s once. Say so in step 2 and
keep the manual commands for installing only these two files.
2026-11-12 07:36:11 +09:00
DongHyeonka d92dc5d163 feat(observability): read node-exporter textfile metrics
The host timers from the k3s maturity work (last Gitea dump and k3s
datastore backup, live drift count, API server and kubelet certificate
expiry) write their results as *.prom files. node-exporter does not
read any textfile directory, so Prometheus never sees them.

Pass --collector.textfile.directory pointing at
/var/lib/node_exporter/textfile_collector under the read-only host
root mount the chart already has. No volume is added. The host
installer creates the directory; until then the collector only
reports an error metric.

Verification: helm template of prometheus-node-exporter 4.56.1 with
these values renders the argument next to --path.rootfs=/host/root.
2026-11-12 07:29:48 +09:00
DongHyeonka 4551a2ece6 feat(platform-postgres): run the shared database as Guaranteed QoS
The one PostgreSQL instance holds the Gitea, Keycloak and tech-log
databases. It is Burstable at priority 0 with oom_score_adj 985, so
under memory pressure the kernel kills it before almost anything else,
and an eviction treats it like Grafana (k3s maturity review
2026-11-11, 7.2 items 2 and 3).

Set priorityClassName platform-critical (services/priority-classes)
and raise the requests to the existing limits, CPU 2 and memory 4Gi,
so the pod is Guaranteed and its oom_score_adj becomes -997. The
backup README now gives the restart, not pg_hba, as the reason to add
spec.backup with kubectl patch.

Risk: applying this changes the pod spec, so CloudNativePG restarts
the single instance and the three databases are down for that time.
It belongs in the k3s restart window. Node requests grow by 1.75 CPU
and 3.5Gi; peak use over the last 6 hours was 0.025 CPU and 195MB.

Verification: kubectl diff of the rendered build root against the live
cluster shows only priorityClassName, the two requests and the
DatabaseRole defaults that were already there; no pg_hba line differs.
render-phase1.sh passes.
2026-11-12 07:29:48 +09:00
DongHyeonka d47d57aa14 fix(platform-postgres): keep the tech-log pg_hba rules the cluster uses
The live platform-postgres Cluster has two pg_hba rules for the techlog
role that the repository does not declare. They were added outside the
repository for hyeonworks.com. Applying the current render would remove
them and the production backend could no longer reach its database.

Declare the same two rules, in the same position after the Keycloak
pair, and pin them with render invariants like the Gitea and Keycloak
pairs. The techlog DatabaseRole, Database, Secret and NetworkPolicies
stay outside this build root; where they belong is an owner decision
recorded in the 2026-11-11 maturity review.

Risk: none for the live cluster. The rendered Cluster now matches the
live object. A fresh install without the tech-log tenant gets two
rules for a role that does not exist, which PostgreSQL ignores.

Verification (desktop, Helm 3.19.4, Kustomize 5.8.1):
- render-phase1.sh with the old cluster.yaml and the new invariants
  fails: "the tech-log role must authenticate only to the tech-log
  database"; with the new cluster.yaml it passes.
- kubectl diff of the new render against the live cluster shows no
  Cluster change; only the pre-existing DatabaseRole default fields.
2026-11-12 07:28:31 +09:00
DongHyeonka d7c845ca5d refactor(postgres): keep the backup alerts with the backup
The observability core renderer rejects any alert or record whose
name contains "backup" (no_unscoped_monitors_rules): backup
monitoring is outside the observability core by design. f19ce54 put
PlatformPostgresBackupMissing and PlatformPostgresWALArchiveBacklog in
core-rules.yaml, so the pinned renderer failed with "out-of-scope
monitor or rule is present".

Move both alerts unchanged into
services/platform-postgres-backup/prometheus-rule.yaml in
platform-data, which Prometheus reads because the namespace has
observability.hyeonworks.com/enabled=true and the rule has
observability.hyeonworks.com/instance=home. They now point their
runbook_url at the backup README and are applied with the
ScheduledBackup. Renaming them to dodge the filter was rejected: it
would keep backup rules inside a scope that excludes them on purpose.

Verification: test-platform-disk-alerts.sh loads both rule files and
passes; promtool checks 13 rules. render-observability-core.sh and
test-render-observability-core.sh pass, and kubectl apply
--dry-run=server accepts the backup kustomization with the rule.
scan-platform-sensitive-source.sh stops at its observability-core
renderer with RC=23 here and on an origin/main snapshot alike, so it
says nothing about this change.

Fixes: f19ce54740 ("feat(postgres): back up platform-postgres to AIStor every day")
2026-11-12 07:28:31 +09:00
DongHyeonka 2b1310972c fix(observability): pin the heartbeat route in the core renderer
render-observability-core.sh keeps its own exact copy of the
AlertmanagerConfig, and test-render-observability-core.sh feeds it the
same copy as a fixture. 619b905 added the platform-heartbeat route and
receiver to the source and to test-observability-alerting.sh but not
to these two, so the pinned renderer rejected the source with
"platform AlertmanagerConfig is not the exact global Slack contract".

Add the route and receiver to both copies and name the heartbeat in
the reject message.

Verification: render-observability-core.sh (Helm v3.19.4, kustomize
v5.8.1) prints OBSERVABILITY CORE STATIC RENDER PASS, also with
--verified-output-dir; test-render-observability-core.sh passes.

Fixes: 619b90552f ("feat(observability): alert when Prometheus itself stops working")
2026-11-12 07:28:31 +09:00
DongHyeonka bed20b456b fix(observability): stop throttling node-exporter
node-exporter averages about 2m CPU, but each scrape runs its
collectors in parallel. With a 250m limit (25ms per 100ms CFS period)
26% of its CFS periods were throttled over 30 minutes on 2026-11-11
(container_cpu_cfs_throttled_periods_total / cfs_periods_total), which
delays scrapes and can make them time out.

Raise the CPU limit to one core. The request stays 100m, so
scheduling does not change, and a runaway collector is still capped.
Removing the limit was rejected because every other workload in this
repository sets one.

Risk: none beyond one core of headroom on a 12-core node. The live
DaemonSet changes only when the observability core is applied again.

Verification: the throttling ratio and usage came from live
Prometheus queries. helm template of prometheus-node-exporter 4.56.1
with these values renders limits.cpu "1". The effect after rollout is
not measured yet.
2026-11-12 07:28:03 +09:00
DongHyeonka 619b90552f feat(observability): alert when Prometheus itself stops working
Nothing tells the owner when monitoring itself fails. On 2026-11-11
Prometheus stored samples 9 hours ahead, then rejected every new
sample as out of bounds: rules kept running on no data, and the only
sign was a false KubeAPIDown. If Prometheus, Alertmanager or the node
went down, no rule could fire at all.

Add two rules and one route. PlatformMetricsIngestionStopped fires
when no up sample from node-exporter is stored for 5 minutes;
absent() still fires while the TSDB rejects samples. PlatformHeartbeat
always fires, and a new platform-heartbeat route sends it every 5
minutes to an external dead man's switch through a webhook whose URL
comes from the Secret alertmanager-heartbeat. When the pings stop,
that service alerts, which covers Prometheus, Alertmanager, the node
and the network being down. The chart Watchdog stays disabled; the
platform rule keeps the route inside this repository.

Rejected: an in-cluster CronJob that queries Prometheus and posts to
Alertmanager. It still depends on Alertmanager and the node, and
needs a new workload in the exact observability NetworkPolicy matrix.

Risk: the AlertmanagerConfig now references alertmanager-heartbeat;
applying it before that Secret exists keeps the operator from
rendering the new Alertmanager config. The alerting README orders the
steps: create the external check, create the Secret, then apply.
The heartbeat needs an account on an external service, which only
the owner can create.

Verification: promtool tests for both rules failed against the old
rules and pass now. test-observability-alerting.sh pins the new route
and receiver in its exact contract and passes the static contract,
the global Alertmanager render with Helm v3.19.4 and the server
dry run (--server-dry-run). kubectl apply --dry-run=server accepts
the updated PrometheusRule.
2026-11-12 07:28:03 +09:00
DongHyeonka f19ce54740 feat(postgres): back up platform-postgres to AIStor every day
platform-postgres holds the Gitea, Keycloak and tech-log databases and
had no backup: no spec.backup, no ScheduledBackup, and its only copy
is a local PV on the root SSD. A lost SSD or a bad migration had no
way back.

Add services/platform-postgres-backup: a merge patch that gives the
live Cluster barman-cloud WAL archiving and base backups to
s3://platform-postgres-backups on the AIStor ObjectStore (HDD sdb3)
with 30-day retention and gzip, a daily 03:00 KST ScheduledBackup that
also runs once at creation, the object-storage ingress for the
PostgreSQL pods, the bucket policy, and a restore drill Cluster that
only reads the bucket. scripts/bootstrap/create-postgres-backup-
credentials.sh creates the bucket (100GiB quota), policy, user and
the platform-data Secret through a restricted AIStor client Pod,
proves the user can write its bucket and cannot list loki, and prints
no key. Add PlatformPostgresBackupMissing (no base backup in 26h) and
PlatformPostgresWALArchiveBacklog (more than 10 WAL files waiting for
15m; a stuck archive fills the SSD volume).

The native barmanObjectStore is deprecated since CNPG 1.26 and the
operator warns it is removed in 1.31.0. The Barman Cloud Plugin needs
cert-manager, which this cluster does not have; installing two
operators before the first backup is slower and riskier. The bucket
layout is the same, so the plugin can take over before CNPG 1.31.
Volume snapshots do not work on the static local PV.

Risk: the live Cluster has pg_hba lines for techlog that the Phase 1
source lacks, so re-applying services/platform-postgres would cut the
tech-log backend off. The README applies only spec.backup with
kubectl patch. Backups stay on the same desktop; an off-machine copy
is not part of this change. Nothing was applied to the cluster.

Verification: kubectl apply --dry-run=server accepts the
ScheduledBackup, the NetworkPolicy and the drill Cluster; kubectl
patch --dry-run=server keeps both techlog pg_hba lines and adds
spec.backup. The row-count comparison from the drill steps ran
read-only on production (keycloak 100, techlog 44, gitea 116 tables).
mc command syntax was checked with the pinned AIStor client offline.
The two new alert tests failed against the old rules and pass now.
bash -n passes for the script; shellcheck is not installed on the
laptop or the desktop.
2026-11-12 07:28:03 +09:00
DongHyeonka 1349ee2f25 docs(ci): list daemon.json in the runner runtime folder 2026-11-12 07:28:03 +09:00
DongHyeonka fb5535aeb6 docs(host): add the owner steps for the desktop host settings
The host files need root, so agent sessions do not install them. The
owner needs one ordered list with checks and rollback for each step,
including when a Docker restart is safe.

Add infrastructure/host/README.md: why the disk and clock settings
exist, the BuildKit key meanings in Docker 29, and four steps. Pin the
tech-log images against kubelet image GC until they move to the Gitea
registry, install the k3s time-sync drop-ins, set the RTC to UTC on
the Windows side, and install the host daemon.json during a window
with no CI job, no agent gate holding the heavy lock, and workmachine
not in use.

Risk: the image pin is a manual label on the current tags; a new
deployment tag needs the label again until the registry move lands.

Verification: the commands were checked on the desktop where they do
not need root (systemctl show, timedatectl, docker buildx inspect,
docker exec gitea-runner docker ps); the sudo steps were not run.
2026-11-12 07:28:03 +09:00
DongHyeonka 03ed6719b0 feat(host): start k3s only after the clock is synchronized
The desktop dual-boots Windows, which writes local time (KST) to the
RTC, while Ubuntu reads the RTC as UTC. On 2026-11-11 the boot started
9 hours ahead; k3s issued its certificates at that time, timesyncd
then set the clock back, the certificates were not yet valid, and
every Ingress returned 502 until k3s was restarted by hand.

Add a k3s.service drop-in with Wants= and After=time-sync.target, and
a drop-in that caps systemd-time-wait-sync.service at 10 minutes.
Enabling systemd-time-wait-sync makes time-sync.target wait for the
first NTP sync; its packaged TimeoutStartSec=infinity would keep k3s
down forever when NTP is unreachable, so after 10 minutes it fails,
the target is reached anyway and k3s starts as it does today.

Risk: the order only applies at boot, so nothing changes until the
next reboot. If NTP is down at boot, k3s starts 10 minutes later. The
Windows side still writes local time until RealTimeIsUniversal is set
there; the host guide covers that step.

Verification: systemd-analyze verify passes for k3s.service and
systemd-time-wait-sync.service with the drop-ins, and an unknown key
added to each drop-in is reported, so both files are read.
2026-11-12 07:28:03 +09:00
DongHyeonka a5f0bff26e feat(ci): cap the runner Docker and prune it weekly
The Docker daemon inside the gitea-runner container keeps every job
image and all build cache in the gitea-runner_docker volume on the
root SSD that production k3s also uses. It grew to 16G in the two days
after the runner was installed and held 12.43G of images and 8.44G of
build cache on 2026-11-11. The README only said to look once a week
and prune by hand.

Mount daemon.json into the runner so the inner BuildKit GC keeps 2GB
to 10GB of build cache, and shrinks to 2GB while the disk has less
than 35GB free. Add gitea-runner-prune.service and .timer, a host
systemd timer for Sunday 04:30 KST that removes unused images created
more than 7 days ago and build cache older than 3 days. The service
runs as donghyeon in the docker group, so it needs root only to
install. A user timer was rejected because lingering is off for
donghyeon and user units stop with the last login session.

Risk: image prune filters on image creation time, so the pinned job
image is deleted weekly and the next job pulls about 543MB again.
Recreating the runner to pick up daemon.json cancels a running job,
so it is done only while the runner is idle.

Verification: systemd-analyze verify passes for both units, and the
calendar expression's next run is Sun 2026-11-15 04:30 KST. docker
compose config passes; dockerd --validate inside the runner (Docker
29.9.3) accepts daemon.json.
2026-11-12 07:28:03 +09:00
DongHyeonka 3bca2cf841 feat(host): cap the host Docker build cache
The host Docker daemon had no daemon.json. The default BuildKit policy
on this 228GiB disk keeps 21.42GiB of build cache untouched and allows
169.5GiB (docker buildx inspect default), so agent gate builds piled
up about 23G in two days before the 2026-11-09 disk incident.

Add infrastructure/host/docker/daemon.json with builder.gc
defaultReservedSpace 5GB, defaultMaxUsedSpace 15GB and
defaultMinFreeSpace 35GB. In Docker 29 the old defaultKeepStorage key
means defaultReservedSpace, a floor and not a cap, so the cap is
defaultMaxUsedSpace. 35GiB free is about 85% use, where the kubelet
starts deleting images; below it the cache shrinks to the floor first.
Moving the Docker data root to the HDD was rejected by the owner
because builds get slower.

Risk: applying needs sudo and a Docker restart, which restarts the
runner, workmachine and cloudflared and fails running CI jobs and
gates. The host apply guide added in this series says when that is
safe. dockerd --validate does not check keys inside builder.gc, so
the guide checks the live policy with docker buildx inspect.

Verification: dockerd --validate passes for the file on Docker
29.6.1; it also passes a typo inside builder.gc and rejects an unknown
top-level key, which is why the live check is in the guide.
2026-11-12 07:28:03 +09:00
DongHyeonka 7fe38428e3 feat(observability): alert on the root disk before kubelet acts
The kubelet starts deleting unused images at 85% root use and evicts
pods at 90%. The root disk warning fired at 85% and the critical at
90%, so both arrived when the kubelet was already acting. On
2026-11-09 the warning fired for 33 hours, then an eviction stopped
the tech-log pods, the image GC deleted their imagePullPolicy: Never
images, and production went down.

Move the warning to 80% for 10m and the critical to 85% for 5m, and
add PlatformRootFilesystemFillingFast (critical, for 15m): the 6h
linear trend of free bytes falls below 10% of the disk, the eviction
point, within 6 hours. On 2026-11-07 the disk lost 17G in 9 hours;
this forecast would have fired hours before the 80% warning. The
info alert at 75% stays.

Add scripts/validate/test-platform-disk-alerts.sh, promtool unit
tests for the usage bands and the forecast with the pinned Prometheus
image. The existing rules test needs live metric inventories, so this
one runs offline.

Risk: the live rules change only when the observability access apply
runs; this commit does not touch the cluster. The forecast can fire
on a large build that the BuildKit GC removes minutes later; the 15m
hold and the 6h window damp that.

Verification: the new test failed against the old rules (no alert at
81%, 86% or for a 1-per-minute fill) and passes after the change;
promtool check rules finds 9 rules. test-observability-alerting.sh
passes its static contract, then stops at the helm render because
helm v3.19.4 is not installed on the desktop.
2026-11-12 07:28:03 +09:00
donghyeon.kang 68a5f7d621 Merge pull request 'fix(validate): re-apply Traefik trust and render drift from a clean copy' (#9) from donghyeon.kang/w6-followups into main 2026-11-11 17:14:02 +00:00
DongHyeonka 050a4de235 fix(validate): render the drift check from a clean working-tree copy
In a long-lived checkout the observability renderer finds an ignored
.helm/charts/<chart>-<version> cache left by an earlier render and
refuses ("exact chart cache already exists"), so live-drift-check.sh
fails with exit 2 in /home/donghyeon/workspace/platform, the path the
runbooks use.

When run inside a git working tree, copy the tracked and untracked
(not ignored, not deleted) files to the drift work directory and
render from there. Local edits are still checked; ignored caches are
not carried over. Outside a git tree (the daily timer renders a
git archive of origin/main) nothing changes.

Also accept the one drift this exposed: the kube-prometheus-stack
PrometheusRule still has the two Kubelet certificate alerts that PR #6
disabled in values; it goes away when the observability core is
re-applied in the restart window (until 2026-11-30).

Risk: read-only, as before. The copy is removed with the work
directory.

Verification (desktop, fresh clone of main 8b019d2 with an ignored
services/observability/tempo/.helm/charts/tempo-2.2.3 created):
- old script: "REJECT: exact chart cache already exists", exit 2.
- new script as an uncommitted edit in that clone: SUMMARY drift=1
  accepted=12 (the PrometheusRule above), exit 1.
- new script from a git archive export, as the timer runs it: same
  summary.
2026-11-12 02:00:20 +09:00
DongHyeonka cc5decc538 feat(traefik): re-apply the trust overlay when trust is already live
apply-trust.sh is the observe -> trust transition. When the live
Traefik is already in the trust phase and only another value changed
(PR #3 resources), it stops at "observation phase must not trust
forwarded headers" before applying, so the change had to be applied
with a bare kubectl apply outside the guarded path.

reapply-trust.sh checks that the runtime is in the trust phase with
the reviewed CIDR and that forwarded-header trust works (a request
through Host Nginx is logged with the forwarded client, not the proxy
address), saves the live HelmChartConfig, shows the change with
kubectl diff and does nothing when there is none, asks for
"APPLY <context> RETRUST <cidr>", applies, and repeats the checks plus
the NodePort, Gitea health and site-manifest checks. Any failure after
the apply re-applies the saved HelmChartConfig, so the trusted CIDR is
never dropped.

Risk: the apply path was not run against the cluster (the live values
already match). The restore path reuses the same wait and health
helpers as apply-trust.sh.

Verification (desktop):
- Live (already trust, values match): "Forwarded-header trust in
  effect (ClientHost 127.0.0.1, proxy 10.42.0.1)", then "already
  matches the trust overlay. Nothing to do.", exit 0.
- Copy with memory 64Mi changed to 65Mi, answering "no": the diff
  shows only that line, "ERROR: cancelled", exit 1; live still 64Mi.
- The ClientHost check of apply-trust.sh would have failed here
  (127.0.0.1 is expected in the trust phase), which is why the check
  is inverted.
2026-11-12 02:00:20 +09:00
donghyeon.kang 8b019d26a0 Merge pull request 'docs(runbooks): add restore, rebuild, upgrade and reboot runbooks' (#8) from donghyeon.kang/w6-runbooks into main 2026-11-11 16:48:53 +00:00
donghyeon.kang 28595c47e2 Merge pull request 'feat(keycloak): pin images and attach PriorityClasses for the restart window' (#7) from donghyeon.kang/w6-restart-window-pins into main 2026-11-11 16:48:42 +00:00
donghyeon.kang 00a6680b76 Merge pull request 'feat(backup): back up k3s and Gitea to AIStor with alerts and drills' (#6) from donghyeon.kang/w6-host-backups into main 2026-11-11 16:48:31 +00:00
donghyeon.kang 4ab37ff18c Merge pull request 'feat(host): add the k3s restart bundle and its apply script' (#5) from donghyeon.kang/w6-k3s-restart-bundle into main 2026-11-11 16:48:20 +00:00
DongHyeonka cb78ccdb0b feat(backup): store the Gitea version next to each dump
A Gitea dump must be restored with the Gitea version that wrote it, and
a restore checks the repository count against the dump. Neither was
recorded outside the archive, so the restore runbook had nothing to
compare with.

Upload a small .meta file with each dump: creation time, the
Deployment image, the gitea --version output and the repository count.
docs/runbooks/gitea-restore.md reads it.

Risk: none for the backup itself; the file holds no secret.

Verification: bash -n; kubectl -n gitea exec deploy/gitea -c gitea --
gitea --version prints "gitea version 1.27.0 built with ..." on the
desktop, so the third field is the version.
2026-11-12 01:38:24 +09:00
DongHyeonka de600a0bb1 feat(priority): attach PriorityClasses to Keycloak, Gitea, Loki, Tempo
The PriorityClasses exist in the repository but no pod uses them, so
every platform pod still evicts at priority 0.

Attach platform-critical to Keycloak (CR spec.scheduling) and Gitea
(chart priorityClassName), and platform-low to Loki (singleBinary) and
Tempo. The Keycloak CR also gets spec.image at the running 26.7.0
digest with startOptimized: false: with spec.image the operator
otherwise assumes a pre-built optimized image, and this is the stock
one. Grafana and pgAdmin have no top-level priorityClassName in their
charts and are left for later; platform-postgres is W1's file and
tech-log belongs to its deploy/ directory.

Risk: applying restarts Keycloak, Gitea, Loki and Tempo once, so it
belongs in the k3s restart window, after the PriorityClasses exist (a
pod that names a missing class is rejected). The accepted-drift list
holds these objects until 2026-11-30.

Verification (desktop): render-phase1/2 and render-observability-core
pass with the four priorityClassName values; kubectl diff (server
dry-run) against live shows only these fields.
2026-11-12 01:25:24 +09:00
DongHyeonka 9751db0e97 feat(keycloak): pin the operator source and both images by digest
The Keycloak Operator remote resource is pinned to the tag 26.7.0, and
that tag's content changed once (live build-timestamp 2026-07-09,
later renders 2026-07-23). Both images use the 26.7.0 tag with
imagePullPolicy Always, so every restart depends on quay.io and could
pull different bits.

Pin the remote resource to the tag's commit, the operator image and
RELATED_IMAGE_KEYCLOAK to the digests that were running on 2026-11-12
(pod imageID) and set the operator to IfNotPresent. The env value is
set with a strategic merge patch keyed by name; a JSON patch by list
index hit KUBERNETES_NAMESPACE instead. render-phase2.sh and
apply-keycloak.sh check the commit and the digest.

Risk: applying restarts the Keycloak operator once (restart window).
Keycloak itself keeps running.

Verification (desktop): render-phase2.sh passes; the rendered
Deployment has the pinned image, IfNotPresent and the digest in
RELATED_IMAGE_KEYCLOAK with the other env entries unchanged.
2026-11-12 01:25:24 +09:00
DongHyeonka 9e9ea94461 feat(backup): keep k3s datastore archives outside AIStor on the HDD
AIStor runs inside the k3s cluster that the datastore archive restores.
After losing the whole cluster the AIStor copy cannot be read until k3s
is running again, which is what the archive is for.

Also keep the seven newest archives in
/srv/k3s/aistor/host-backups/k3s-datastore (root 0700) on the same HDD
filesystem (sdb3). The directory is next to AIStor's drive directory
data/, so AIStor does not list it as a bucket, and it is readable by
mounting sdb3 alone. A metric reports how many local copies exist.

Rejected: the NTFS archive partition (sdb2). It has no Unix
permissions and Windows on the same machine can read it; the archive
holds the server token.

Risk: the files live inside the AIStor PersistentVolume root, which is
mounted into the MinIO pod (uid 1000); mode 0700 root keeps the pod
from reading them.

Verification: bash -n; /srv/k3s/aistor contains only data/ (2026-11-12).
2026-11-12 01:14:45 +09:00
DongHyeonka ba3dddc139 feat(validate): add restore drills for the k3s and Gitea backups
A backup nobody has restored is not known to work. The 2026-08 k3s
recovery bundle never got its isolated restore drill.

k3s-datastore-restore-drill.sh downloads the latest archive, checks its
sha256 and integrity, starts the same k3s version in a network-less
throwaway container on the copy, and checks the API comes up, the
Secret count is close to live and every Secret decrypts with the
token. gitea-backup-restore-drill.sh extracts the latest dump and
checks app.ini (INTERNAL_TOKEN, SECRET_KEY), git fsck on every
repository and the repository count against live Gitea. Both accept a
local copy and write a drill timestamp metric.

Risk: read-only for the cluster. The k3s drill runs the restored
control plane with --network none, so its controllers reach nothing.

Verification (desktop):
- test-k3s-datastore-restore-drill.sh: a fresh k3s with three test
  Secrets (stored encrypted, k8s:enc:) restores with all Secrets
  decrypted in 7s; before folding the WAL the test saw 2 of 3 rows,
  which is why the real backup uses sqlite3 .backup.
- gitea-backup-restore-drill.sh on a live repository dump: 29
  repositories, 0 fsck failures, 29 live; the dump was deleted after.
2026-11-12 01:13:02 +09:00
DongHyeonka 012452cf78 feat(observability): alert on host backups, drift and k3s serving certs
Backups, the daily drift check and the k3s serving certificates have no
alert. k3s renews certificates only when it starts within 120 days of
expiry, and the two kube-prometheus-stack kubelet certificate alerts
can never fire because the k3s kubelet exports no
kubelet_certificate_manager_* metric.

Add a PrometheusRule (platform-host-checks) for backup age, failed
runs, missing metrics, drift left for 6h, a stale drift check, and
serving certificate expiry (warning under 100 days, critical under 14).
Add the two remaining host timers (certificate expiry via openssl on
6443 and 10250; the drift check against origin/main) and an installer
that copies the scripts to a root-owned path, creates the textfile and
work directories and enables all four timers. Disable the two dead
kubelet certificate rules in kube-prometheus-stack.

Risk: the metrics need the node-exporter textfile collector, a one-line
change in W1's node-exporter values that was asked of W1. Until then
PlatformHostBackupMetricsAbsent fires after a day, which is the point.

Verification (desktop):
- test-platform-host-checks-alerts.sh: promtool check and unit tests
  pass; with the k3s threshold changed from 2 to 5 days they fail.
- render-observability-core.sh passes; the render has no Kubelet*
  certificate alerts and still has both KubeClientCertificateExpiration.
- k3s-cert-expiry.sh as the desktop user reads both certificates.
- install-host-checks.sh --check: systemd-analyze verify is clean.
2026-11-12 01:13:02 +09:00
DongHyeonka 77daa39a41 feat(backup): back up the k3s datastore and Gitea to AIStor
Nothing backs up the k3s datastore or Gitea's repositories, packages and
app.ini. W1 covers platform-postgres only. One SSD failure loses the
source, CI artefacts, published libraries and every cluster Secret.

Add two root systemd timers that write to a new AIStor bucket on the
HDD (platform-host-backups, same AIStor as W1's postgres backups):
- k3s datastore, daily: sqlite3 .backup of the live state.db plus the
  server token (k3s backup/restore docs: the token is needed to read
  it back), integrity_check, tar.gz, sha256, upload. Off-host copy is
  prepared behind two variables (gpg recipient file, rclone target) and
  stays off until the owner chooses a place.
- Gitea, weekly: gitea dump --skip-db streamed from the live container,
  listing checked for app.ini and repositories, sha256, upload. The DB
  is in the CNPG backup.
Each run writes textfile metrics for the alerts in the next commit.
create-host-backup-credentials.sh makes the bucket (50GiB quota,
lifecycle 56 days for gitea/, 30 days for k3s-datastore/), a
bucket-scoped policy and user, and a root-only env file; no key is
printed and mc gets keys only through a 0600 env file.

Rejected: in-cluster CronJobs. The datastore is a root-owned host file
and the dump needs the Gitea DB connection; a hostPath root pod or a
second Gitea-labelled pod is a larger privilege than a root timer.

Risk: not installed here (owner sudo steps). The k3s archive holds the
token; together with the datastore it decrypts every Secret. It lives
in a bucket only the backup user can read.

Verification (desktop): a dump streamed with --file - is a valid
tar.gz with app.ini; repositories appear as repos/<owner>/<name>.git
(29, matching the live count). The scripts parse with bash -n.
2026-11-12 01:13:02 +09:00
DongHyeonka 457d88e293 feat(bootstrap): apply the k3s restart bundle with one restart
The restart bundle needs root, touches k3s, kubelet and systemd files,
and must not leave the cluster half-configured. The W1 time-sync
drop-ins need a restart window too.

apply-k3s-restart-bundle.sh --check parses the files and prints the
install plan without root. --execute checks the cluster is Ready,
backs up every file it replaces plus a sqlite .backup of the datastore
and the token, installs the W6 files and W1's two time-sync drop-ins
as they are, enables systemd-time-wait-sync, restarts k3s once with
--no-block (the unit has TimeoutStartSec=0), then verifies the kubelet
configz values, the audit log, that the default namespace rejects a
privileged pod, three health URLs and the time-sync unit. Any failure
restores the backup and restarts again. Last, it creates the
local-storage .skip file and clears the local-path default annotation.

Risk: --execute was not run (owner sudo step). The API is unavailable
for one to two minutes; containers keep running (KillMode=process). The
script refuses to run until W1's host files are on main.

Verification (desktop, non-root):
- --check without the W1 files: MISSING ... (W1), exit 1.
- --check with the W1 files overlaid: parse and invariants pass, plan
  printed, exit 0.
- --execute as non-root: "--execute needs root (sudo)".
2026-11-12 00:49:58 +09:00
DongHyeonka e88725aab8 feat(host): add the k3s kubelet, audit and pod security drop-ins
The kubelet has no memory eviction threshold and no reservations, so
under memory pressure the kernel OOM killer picks a victim and
Postgres (oom_score_adj 985) is near the front. The API server writes
no audit log. Namespaces without labels have no Pod Security level, and
graceful node shutdown is off.

Add the files for one k3s restart: a kubelet drop-in (system 1 CPU/2Gi,
kube 0.5 CPU/1Gi, hard eviction memory 500Mi plus the two k3s disk
defaults, soft eviction 1Gi for 1m, shutdown 60s/20s), an API server
drop-in that appends the audit and admission-config flags, a Metadata
audit policy and a cluster Pod Security default (baseline, warn and
audit restricted, kube-system exempt).

Rejected: protect-kernel-defaults, recommended in review 7.3. CIS
dropped it in v1.7, and the sysctls it requires include
kernel.panic_on_oops=1, which reboots this dual-boot desktop on any
kernel oops; a reboot started the 2026-11-11 clock incident. The README
records the reason.

Risk: nothing is installed by this commit. The kubelet merges the
drop-in after k3s's 00-k3s-defaults.conf (k3s passes only --config-dir);
evictionHard lists all three keys because setting the map zeroes the
others.

Verification: the apply script's --check parses every file and asserts
the invariants (next commit).
2026-11-12 00:49:58 +09:00
78 changed files with 3843 additions and 92 deletions
+54 -29
View File
@@ -76,44 +76,67 @@ Gradle·Docker 게이트는 `flock /tmp/claude-gates/heavy.lock`으로 서로
## 디스크와 정리
안쪽 daemon의 데이터(job 이미지, Testcontainers 이미지, 빌드 cache)는 이름 있는
볼륨 `gitea-runner_docker`에 쌓인다. desktop root 디스크는 228G 중 182G를 쓰고
있다(2026-11-07, 여유 34G). 예상 사용량은 다음과 같다.
볼륨 `gitea-runner_docker`에 쌓인다. 이 볼륨은 desktop 루트 SSD에 있고, 운영 k3s와
같은 디스크다. 설치 이틀 만에 16G가 됐다(2026-11-09: 이미지 12.35G, 빌드 cache 3.4G).
2026-11-11에는 이미지 12.43G, 빌드 cache 8.44G였다.
| 항목 | 크기 |
| --- | --- |
| runner 이미지(`gitea/runner:5.4.1-dind`) | 약 133MB |
| job 이미지(`runner-images:ubuntu-24.04`) | amd64 압축 543MB, 안쪽 daemon에서 2.325GB(2026-11-07 실측) |
| Testcontainers 이미지(PostgreSQL, Ryuk 등) | 수백 MB |
| actions/cache(`/data/cache`) | 상한 5GB |
상한은 두 가지로 건다.
runner는 job이 끝나면 job 컨테이너와 proxy로 만든 컨테이너·네트워크·볼륨을
지우지만(`container.sweep: true`) 이미지는 지우지 않는다. 정리 주기는 다음과 같다.
| 무엇 | 방법 | 값 |
| --- | --- | --- |
| 빌드 cache | `daemon.json`의 BuildKit GC. 안쪽 daemon이 스스로 지운다 | 최소 2GB, 상한 10GB. 디스크 여유가 35GB 아래면 2GB까지 |
| job 이미지 | 호스트 systemd timer `gitea-runner-prune.timer`. 주 1회 | 만든 지 7일 지난 안 쓰는 이미지, 3일 지난 빌드 cache |
| actions/cache | `config.yaml`의 `cache.size_limit` | 5GB |
- 매주 한 번 사용량을 본다.
- BuildKit 키의 뜻과 값을 고른 이유는 [호스트 설정](../../host/README.md)의
「BuildKit 상한 값」에 있다. 호스트와 같은 키에 더 작은 값을 쓴다.
- Docker daemon은 이미지를 스스로 지우지 않는다. 그래서 이미지는 timer로 지운다.
- `image prune --filter until=168h`는 이미지를 **만든** 시각으로 거른다. 마지막으로 쓴
시각이 아니다. 그래서 고정한 job 이미지(`runner-images:ubuntu-24.04`, 2026-11-03 생성)도
지워지고, 다음 job이 압축 약 543MB를 다시 받는다. 실행 중인 job 컨테이너가 쓰는
이미지는 지워지지 않는다.
- timer는 일요일 04:30(KST)에 돈다. 그 시각에 꺼져 있었으면 다음 부팅 때 돈다
(`Persistent=true`). service는 `donghyeon` 사용자(docker group)로 돌아 root가 필요 없다.
```bash
docker system df -v | grep gitea-runner
docker exec gitea-runner docker system df
```
설치(소유자, sudo 한 번):
- 안쪽 사용량이 15GB를 넘거나 root 디스크 여유가 20G 아래로 내려가면 job이 없을 때
안쪽 이미지를 정리한다. 고정된 job 이미지도 지워지므로 다음 job이 압축 약 543MB를
다시 받는다.
```bash
sudo install -m 0644 infrastructure/ci/gitea-runner/gitea-runner-prune.service \
infrastructure/ci/gitea-runner/gitea-runner-prune.timer /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now gitea-runner-prune.timer
systemctl list-timers gitea-runner-prune.timer
```
```bash
docker exec gitea-runner docker image prune -a -f
docker exec gitea-runner docker builder prune -f
```
바로 한 번 돌리려면 job이 없을 때 `sudo systemctl start gitea-runner-prune.service`를
실행하고 `journalctl -u gitea-runner-prune.service -n 30`으로 결과를 본다.
- 볼륨 `gitea-runner_docker` 자체를 지우는 것은 runner를 완전히 지울 때만 한다
(「되돌리기」). 등록 파일이 있는 `gitea-runner_data`와 섞지 않는다.
되돌리기:
```bash
sudo systemctl disable --now gitea-runner-prune.timer
sudo rm /etc/systemd/system/gitea-runner-prune.service /etc/systemd/system/gitea-runner-prune.timer
sudo systemctl daemon-reload
```
사용량 보기:
```bash
docker system df -v | grep gitea-runner
docker exec gitea-runner docker system df
docker exec gitea-runner docker buildx inspect default | sed -n '/GC Policy rule#1/,/Min Free/p'
```
볼륨 `gitea-runner_docker` 자체를 지우는 것은 runner를 완전히 지울 때만 한다
(「되돌리기」). 등록 파일이 있는 `gitea-runner_data`와 섞지 않는다.
## 파일과 위치
| 무엇 | 위치 |
| --- | --- |
| 원본 설정 | 이 디렉터리의 `compose.yaml`, `config.yaml` |
| desktop 실행 폴더 | `/home/donghyeon/.local/share/gitea-runner/` (`compose.yaml`, `config.yaml`, `secrets/`) |
| 원본 설정 | 이 디렉터리의 `compose.yaml`, `config.yaml`, `daemon.json` |
| 주 1회 정리 | 이 디렉터리의 `gitea-runner-prune.service`·`.timer` → `/etc/systemd/system/` |
| desktop 실행 폴더 | `/home/donghyeon/.local/share/gitea-runner/` (`compose.yaml`, `config.yaml`, `daemon.json`, `secrets/`) |
| 등록 파일 `.runner` | Docker 볼륨 `gitea-runner_data`의 `/data/.runner` (runner 자격 증명, Secret으로 취급) |
| job 이미지·안쪽 daemon 데이터 | Docker 볼륨 `gitea-runner_docker` (`/var/lib/docker`) |
| actions/cache 저장소 | `gitea-runner_data`의 `/data/cache`, 상한 5GB |
@@ -133,8 +156,10 @@ runner는 job이 끝나면 job 컨테이너와 proxy로 만든 컨테이너·네
```bash
install -d -m 0700 ~/.local/share/gitea-runner ~/.local/share/gitea-runner/secrets
install -m 0644 infrastructure/ci/gitea-runner/compose.yaml infrastructure/ci/gitea-runner/config.yaml ~/.local/share/gitea-runner/
sha256sum infrastructure/ci/gitea-runner/*.yaml ~/.local/share/gitea-runner/*.yaml
install -m 0644 infrastructure/ci/gitea-runner/compose.yaml infrastructure/ci/gitea-runner/config.yaml \
infrastructure/ci/gitea-runner/daemon.json ~/.local/share/gitea-runner/
sha256sum infrastructure/ci/gitea-runner/{compose.yaml,config.yaml,daemon.json} \
~/.local/share/gitea-runner/{compose.yaml,config.yaml,daemon.json}
```
2. 등록 token을 편집기로 `~/.local/share/gitea-runner/secrets/registration-token`에
@@ -170,7 +195,7 @@ runner는 job이 끝나면 job 컨테이너와 proxy로 만든 컨테이너·네
재생성은 실행 중인 job을 취소하므로 job이 없을 때 한다.
- 이미지 올리기: `compose.yaml`의 tag와 digest를 함께 바꾼다. 등록 파일은 그대로 유효하다.
major가 바뀌면 [Upgrading](https://docs.gitea.com/runner/upgrade/)의 호환 변경을 먼저 읽는다.
- 디스크: 「디스크와 정리」의 주기를 따른다.
- 디스크: 「디스크와 정리」의 상한과 timer가 맡는다. 매주 `list-timers`로 마지막 실행을 본다.
## 되돌리기
@@ -22,6 +22,8 @@ services:
CONFIG_FILE: /config.yaml
volumes:
- ./config.yaml:/config.yaml:ro
# Caps the build cache of the inner Docker daemon. See README.md.
- ./daemon.json:/etc/docker/daemon.json:ro
- ./secrets:/run/gitea-runner:ro
- data:/data
- docker:/var/lib/docker
@@ -0,0 +1,10 @@
{
"builder": {
"gc": {
"enabled": true,
"defaultReservedSpace": "2GB",
"defaultMaxUsedSpace": "10GB",
"defaultMinFreeSpace": "35GB"
}
}
}
@@ -0,0 +1,17 @@
# Weekly cleanup of the Docker daemon inside the gitea-runner container.
# Installed by infrastructure/ci/gitea-runner/README.md ("디스크와 정리").
[Unit]
Description=Prune old images and build cache in the Gitea runner Docker
Requires=docker.service
After=docker.service
[Service]
Type=oneshot
# The docker group is enough for docker exec. No root needed at run time.
User=donghyeon
Group=docker
# image prune keeps images used by a running job container; until= filters
# on image creation time, so the pinned job image is pulled again after a week.
ExecStart=/usr/bin/docker exec gitea-runner docker image prune --all --force --filter until=168h
ExecStart=/usr/bin/docker exec gitea-runner docker builder prune --all --force --filter until=72h
ExecStartPost=/usr/bin/docker exec gitea-runner docker system df
@@ -0,0 +1,12 @@
[Unit]
Description=Weekly prune of the Gitea runner Docker
[Timer]
# Sunday 04:30 local time (Asia/Seoul), when CI and agent sessions are quiet.
OnCalendar=Sun *-*-* 04:30:00
RandomizedDelaySec=30min
# Run at the next boot if the machine was off at that time.
Persistent=true
[Install]
WantedBy=timers.target
+46
View File
@@ -0,0 +1,46 @@
# CI 지표
desktop runner 하나가 받는 repo들의 CI 지표를 Gitea Actions API에서 뽑는다. 읽기만 하고 서버에 아무것도 쓰지 않는다. 지표의 정의와 목표는 design-docs `ci/manual.md` 10절이고, 주 1회 결과를 design-docs `ci/metrics.md`에 한 줄 남긴다.
## 돌리기
```bash
python3 infrastructure/ci/metrics/ci_metrics.py \
--repo clean-architecture-backend-template \
--repo clean-architecture-frontend-template \
--since 2026-11-07 --until 2026-11-13
```
- 날짜는 KST 하루 단위이고 `--until`도 그날을 포함한다(`--utc-offset-hours`로 바꾼다).
- 인증: `GITEA_TOKEN`(읽기 전용 토큰)이 있으면 그것을, 없으면 `git credential fill`의 그 호스트 자격 증명을 쓴다. 어느 쪽도 출력하지 않는다.
- `--json`은 표 대신 숫자를 JSON으로 낸다.
- 두 repo, 지난 7일이면 30초 안팎이다(run마다 jobs API를 한 번 부른다).
## 내는 것
| # | 지표 | 계산 |
| --- | --- | --- |
| 1 | 실행 시간 p50·p90 | 단계(`workflow:event`)별, 끝난 run(success·failure)의 job 실행 시간 합. cancelled run은 run 수에만 센다 |
| 2 | 대기 시간 p50·p90 | run의 첫 job 시작 − run 생성 |
| 3 | runner 사용률 | 날짜(job 시작 시각, KST)별 job 실행 시간 합 ÷ 1,440분. 준 repo를 모두 더한다(runner가 하나라서) |
| 4 | PR당 run 수 | 기간 안에 병합된 PR마다 그 PR의 `pull_request` run 수. 병합 전 7일의 run까지 센다 |
| 5 | main 빨강 시간 | workflow별 main push run이 처음 실패한 run의 생성부터 다음 성공 run의 끝까지. 기간 끝까지 빨강이면 그때까지 |
지표 6(흔들림)은 각 repo `docs/ledger/flaky-tests.jsonl`, 지표 7(캐시 적중)은 PR run 로그에서 나온다. 이 스크립트는 보지 않는다.
## 우리 서버(Gitea 1.27)가 API 명세와 다른 곳
2026-11-13 응답 한 건씩으로 확인했다. 계산은 서버를 따른다.
- run에 `created_at`이 없다. run의 job은 run과 함께 만들어지므로, job `created_at`의 가장 이른 값을 run 생성 시각으로 쓴다.
- 시작하지 않은 run·job의 `started_at`·`completed_at`은 `1970-01-01T00:00:00Z`다. 없는 값으로 본다.
- job의 `created_at`은 그 job이 실행할 수 있게 된 시각이 아니라 run의 생성 시각이다. runner 하나가 run의 job을 차례로 돌리므로 job마다 `started_at − created_at`을 대기로 세면 앞 job의 실행이 대기에 들어간다. 그래서 대기는 run 단위로 센다. `needs`가 있는 job이 늦게 풀려 대기열 뒤로 가는 시간은 이 지표에 안 보이고 실행 시간에도 안 들어간다.
- run 목록의 run에는 `head_branch`가 없다. 단계와 ref는 `path`(`<workflow>@<ref>`)에서 읽고, PR 번호는 `pull_requests`에서 읽는다.
## 시험
```bash
cd infrastructure/ci/metrics && python3 -m unittest -v test_ci_metrics
```
서버 없이 계산만 본다. 만든 run·job 기록으로 대기·실행·빨강 시간·사용률을 확인한다.
+384
View File
@@ -0,0 +1,384 @@
#!/usr/bin/env python3
"""CI metrics for the repositories one Gitea Actions runner serves.
Reads the Gitea Actions API (runs, their jobs, closed pull requests) and prints, per repository and
stage, the CI manual's metrics 1-5: run execution time, queue wait, runner utilisation, pull
request runs per merged pull request and time main stayed red. Read-only; it never writes to the
server.
What our Gitea 1.27 returns differs from the gitea.com API reference in three places, and the
arithmetic below follows the server:
* A run carries no ``created_at``. A run's jobs are all created with the run, so the earliest job
``created_at`` is the run's creation time.
* A run or job that has not started reports ``started_at`` and ``completed_at`` as
``1970-01-01T00:00:00Z``. Those are treated as absent.
* A job's ``created_at`` is the run's, not the moment it became runnable. One runner runs a run's
jobs one after another, so ``started_at - created_at`` of a later job counts the earlier jobs'
execution as waiting. Queue wait is therefore measured per run: first job start minus run
creation.
Credentials: ``GITEA_TOKEN`` (a read-only token) if set, otherwise ``git credential fill`` for the
server host. Neither is printed.
"""
from __future__ import annotations
import argparse
import base64
import json
import math
import os
import subprocess
import sys
import urllib.parse
import urllib.request
from dataclasses import dataclass, field
from datetime import date, datetime, timedelta, timezone
EPOCH_PREFIXES = ("1970-01-01", "0001-01-01")
FINISHED = {"success", "failure"}
def parse_time(value: str | None) -> datetime | None:
"""An API timestamp, or None for a missing or zero value."""
if not value or value.startswith(EPOCH_PREFIXES):
return None
return datetime.fromisoformat(value.replace("Z", "+00:00"))
def percentile(values: list[float], fraction: float) -> float | None:
"""Nearest-rank percentile; None for no values."""
if not values:
return None
ordered = sorted(values)
rank = max(1, math.ceil(fraction * len(ordered)))
return ordered[rank - 1]
@dataclass
class Job:
name: str
created: datetime | None
started: datetime | None
completed: datetime | None
conclusion: str | None
@property
def minutes(self) -> float:
if self.started is None or self.completed is None:
return 0.0
return max(0.0, (self.completed - self.started).total_seconds() / 60)
@dataclass
class Run:
repo: str
id: int
workflow: str
ref: str
event: str
status: str
conclusion: str | None
pull_request: int | None
jobs: list[Job] = field(default_factory=list)
@property
def stage(self) -> str:
return f"{self.workflow}:{self.event}"
@property
def created(self) -> datetime | None:
times = [job.created for job in self.jobs if job.created is not None]
return min(times) if times else None
@property
def first_start(self) -> datetime | None:
times = [job.started for job in self.jobs if job.started is not None]
return min(times) if times else None
@property
def completed(self) -> datetime | None:
times = [job.completed for job in self.jobs if job.completed is not None]
return max(times) if times and self.status == "completed" else None
@property
def execution_minutes(self) -> float:
return sum(job.minutes for job in self.jobs)
@property
def queue_minutes(self) -> float | None:
if self.created is None or self.first_start is None:
return None
return max(0.0, (self.first_start - self.created).total_seconds() / 60)
@property
def main_push(self) -> bool:
return self.event == "push" and self.ref == "refs/heads/main"
def run_from_api(repo: str, raw: dict, jobs: list[dict]) -> Run:
workflow, _, ref = raw.get("path", "").partition("@")
pulls = raw.get("pull_requests") or []
return Run(
repo=repo,
id=raw["id"],
workflow=workflow,
ref=ref,
event=raw.get("event", ""),
status=raw.get("status", ""),
conclusion=raw.get("conclusion"),
pull_request=pulls[0]["number"] if pulls else None,
jobs=[
Job(
name=job.get("name", ""),
created=parse_time(job.get("created_at")),
started=parse_time(job.get("started_at")),
completed=parse_time(job.get("completed_at")),
conclusion=job.get("conclusion"),
)
for job in jobs
],
)
class Gitea:
"""The few read-only API calls this script makes."""
def __init__(self, server: str):
self.server = server.rstrip("/")
self.header = self._authorization()
def _authorization(self) -> str:
token = os.environ.get("GITEA_TOKEN")
if token:
return f"token {token}"
host = urllib.parse.urlparse(self.server).netloc
answer = subprocess.run(
["git", "credential", "fill"],
input=f"protocol=https\nhost={host}\n\n",
capture_output=True,
text=True,
check=True,
).stdout
fields = dict(line.split("=", 1) for line in answer.splitlines() if "=" in line)
pair = f"{fields['username']}:{fields['password']}".encode()
return "Basic " + base64.b64encode(pair).decode()
def get(self, path: str) -> object:
request = urllib.request.Request(
f"{self.server}/api/v1{path}",
headers={"Authorization": self.header, "Accept": "application/json"},
)
with urllib.request.urlopen(request, timeout=30) as response:
return json.load(response)
def fetch_runs(api: Gitea, owner: str, repo: str, since: datetime) -> list[Run]:
"""Runs created at or after ``since``, newest first. Run ids grow with creation time."""
runs: list[Run] = []
page = 1
while True:
body = api.get(f"/repos/{owner}/{repo}/actions/runs?limit=50&page={page}")
batch = body.get("workflow_runs") or []
if not batch:
return runs
for raw in batch:
jobs = api.get(f"/repos/{owner}/{repo}/actions/runs/{raw['id']}/jobs").get("jobs") or []
run = run_from_api(repo, raw, jobs)
if run.created is not None and run.created < since:
return runs
runs.append(run)
page += 1
def fetch_merged_pulls(api: Gitea, owner: str, repo: str, since: datetime, until: datetime) -> list[int]:
numbers: list[int] = []
page = 1
while True:
batch = api.get(f"/repos/{owner}/{repo}/pulls?state=closed&sort=recentupdate&limit=50&page={page}")
if not batch:
return numbers
for pull in batch:
merged = parse_time(pull.get("merged_at"))
if pull.get("merged") and merged is not None and since <= merged < until:
numbers.append(pull["number"])
updated = [parse_time(pull.get("updated_at")) for pull in batch]
if all(time is not None and time < since for time in updated):
return numbers
page += 1
@dataclass
class RedEpisode:
workflow: str
first_failed_run: int
fixed_by_run: int | None
hours: float
def red_episodes(runs: list[Run], now: datetime) -> list[RedEpisode]:
"""Main stays red from the first failing main push run's creation to the next green one's end."""
episodes: list[RedEpisode] = []
by_workflow: dict[str, list[Run]] = {}
for run in runs:
if run.main_push and run.status == "completed" and run.conclusion in FINISHED:
by_workflow.setdefault(run.workflow, []).append(run)
for workflow, history in by_workflow.items():
history.sort(key=lambda run: run.created or now)
start: Run | None = None
for run in history:
if run.conclusion == "failure" and start is None:
start = run
elif run.conclusion == "success" and start is not None:
end = run.completed or now
episodes.append(
RedEpisode(workflow, start.id, run.id, (end - start.created).total_seconds() / 3600)
)
start = None
if start is not None:
episodes.append(RedEpisode(workflow, start.id, None, (now - start.created).total_seconds() / 3600))
return episodes
def stage_rows(runs: list[Run]) -> list[dict]:
rows = []
for stage in sorted({run.stage for run in runs}):
finished = [run for run in runs if run.stage == stage and run.conclusion in FINISHED]
waited = [run.queue_minutes for run in runs if run.stage == stage and run.queue_minutes is not None]
executed = [run.execution_minutes for run in finished]
rows.append(
{
"stage": stage,
"runs": len([run for run in runs if run.stage == stage]),
"finished": len(finished),
"red": len([run for run in finished if run.conclusion == "failure"]),
"exec_p50": percentile(executed, 0.5),
"exec_p90": percentile(executed, 0.9),
"wait_p50": percentile(waited, 0.5),
"wait_p90": percentile(waited, 0.9),
}
)
return rows
def utilisation(runs: list[Run], offset: timedelta) -> dict[date, float]:
"""Job execution minutes per local day (job start day) across every repository given."""
minutes: dict[date, float] = {}
for run in runs:
for job in run.jobs:
if job.started is not None and job.minutes > 0:
day = (job.started + offset).date()
minutes[day] = minutes.get(day, 0.0) + job.minutes
return minutes
def runs_per_merged_pull(runs: list[Run], merged: list[int]) -> list[int]:
return [len([run for run in runs if run.pull_request == number]) for number in merged]
def fmt(value: float | None, digits: int = 1) -> str:
return "-" if value is None else f"{value:.{digits}f}"
def report(window: str, per_repo: dict[str, dict], days: dict[date, float]) -> str:
lines = [f"# CI 지표 ({window})", ""]
for repo, data in per_repo.items():
lines += [
f"## {repo}",
"",
"| 단계 | run | 끝난 run | 빨강 | 실행 p50(분) | 실행 p90(분) | 대기 p50(분) | 대기 p90(분) |",
"|---|---|---|---|---|---|---|---|",
]
for row in data["stages"]:
lines.append(
f"| {row['stage']} | {row['runs']} | {row['finished']} | {row['red']} | {fmt(row['exec_p50'])} | "
f"{fmt(row['exec_p90'])} | {fmt(row['wait_p50'])} | {fmt(row['wait_p90'])} |"
)
counts = data["runs_per_pull"]
lines += [
"",
f"- 병합된 PR {len(counts)}개, PR당 run 수 p50 {fmt(percentile(counts, 0.5), 0)}, "
f"p90 {fmt(percentile(counts, 0.9), 0)}, 최대 {max(counts) if counts else '-'}",
]
episodes = data["red"]
if episodes:
longest = max(episodes, key=lambda episode: episode.hours)
lines.append(
f"- main 빨강 {len(episodes)}회, 합 {sum(e.hours for e in episodes):.1f}시간, p90 "
f"{fmt(percentile([e.hours for e in episodes], 0.9))}시간, 가장 긴 것 {longest.hours:.1f}시간"
f"({longest.workflow} run {longest.first_failed_run} → "
f"{longest.fixed_by_run if longest.fixed_by_run else '아직 빨강'})"
)
else:
lines.append("- main 빨강 0회")
lines.append("")
lines += ["## runner 사용률 (위 repo 합, job 시작 날짜 기준)", "", "| 날짜 | 실행 합(분) | 사용률 |", "|---|---|---|"]
for day in sorted(days):
lines.append(f"| {day} | {days[day]:.0f} | {days[day] / 1440:.0%} |")
if days:
busiest = max(days, key=days.get)
lines += ["", f"- 바쁜 날 {busiest}: {days[busiest] / 1440:.0%}"]
return "\n".join(lines) + "\n"
def main(argv: list[str]) -> int:
parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
parser.add_argument("--server", default="https://git.learn.hyeonworks.com")
parser.add_argument("--owner", default="donghyeon.kang")
parser.add_argument("--repo", action="append", required=True, help="repeat for each repository")
parser.add_argument("--since", required=True, type=date.fromisoformat, help="first local day, YYYY-MM-DD")
parser.add_argument("--until", required=True, type=date.fromisoformat, help="last local day, inclusive")
parser.add_argument("--utc-offset-hours", type=int, default=9, help="local day boundary (default KST)")
parser.add_argument("--json", action="store_true", help="print the numbers as JSON instead of tables")
args = parser.parse_args(argv)
offset = timedelta(hours=args.utc_offset_hours)
since = datetime.combine(args.since, datetime.min.time(), timezone.utc) - offset
until = datetime.combine(args.until + timedelta(days=1), datetime.min.time(), timezone.utc) - offset
# A pull request merged in the window may have had runs in the week before it.
lookback = since - timedelta(days=7)
now = min(datetime.now(timezone.utc), until)
api = Gitea(args.server)
per_repo: dict[str, dict] = {}
window_runs: list[Run] = []
for repo in args.repo:
fetched = fetch_runs(api, args.owner, repo, lookback)
in_window = [run for run in fetched if run.created is not None and since <= run.created < until]
window_runs += in_window
merged = fetch_merged_pulls(api, args.owner, repo, since, until)
per_repo[repo] = {
"stages": stage_rows(in_window),
"runs_per_pull": runs_per_merged_pull(fetched, merged),
"red": red_episodes(in_window, now),
}
days = utilisation(window_runs, offset)
window = f"{args.since} ~ {args.until}, UTC{args.utc_offset_hours:+d}"
if args.json:
print(
json.dumps(
{
"window": window,
"repos": {
repo: {
"stages": data["stages"],
"runs_per_pull": data["runs_per_pull"],
"red": [episode.__dict__ for episode in data["red"]],
}
for repo, data in per_repo.items()
},
"utilisation_minutes": {str(day): round(value, 1) for day, value in sorted(days.items())},
},
ensure_ascii=False,
indent=2,
)
)
else:
print(report(window, per_repo, days), end="")
return 0
if __name__ == "__main__":
sys.exit(main(sys.argv[1:]))
@@ -0,0 +1,124 @@
"""Offline tests for ci_metrics: the arithmetic on run and job records, no server."""
import unittest
from datetime import datetime, timedelta, timezone
import ci_metrics as m
T0 = datetime(2026, 11, 13, 0, 0, tzinfo=timezone.utc)
def at(minutes: float) -> str:
return (T0 + timedelta(minutes=minutes)).isoformat().replace("+00:00", "Z")
def job(name, created, started=None, completed=None, conclusion="success"):
return {
"name": name,
"created_at": at(created),
"started_at": at(started) if started is not None else "1970-01-01T00:00:00Z",
"completed_at": at(completed) if completed is not None else "1970-01-01T00:00:00Z",
"conclusion": conclusion,
}
def run(run_id, path, event, conclusion, jobs, status="completed", pull=None):
raw = {"id": run_id, "path": path, "event": event, "status": status, "conclusion": conclusion}
if pull is not None:
raw["pull_requests"] = [{"number": pull}]
return m.run_from_api("backend", raw, jobs)
class ParsingTest(unittest.TestCase):
def test_a_zero_timestamp_from_a_job_that_has_not_started_is_absent(self):
self.assertIsNone(m.parse_time("1970-01-01T00:00:00Z"))
def test_the_run_is_created_when_its_earliest_job_is(self):
r = run(1, "pr.yml@refs/pull/7/head", "pull_request", "success", [job("done", 5, 20, 32)], pull=7)
self.assertEqual((r.created, r.stage, r.pull_request), (T0 + timedelta(minutes=5), "pr.yml:pull_request", 7))
class TimingTest(unittest.TestCase):
def test_queue_wait_is_the_first_job_start_minus_run_creation_not_each_jobs_own_gap(self):
# given: one runner runs the jobs one after another, so the second job's own gap is 40 minutes
r = run(1, "main.yml@refs/heads/main", "push", "success", [job("a", 0, 10, 30), job("b", 0, 30, 40)])
# when
waited = r.queue_minutes
# then
self.assertEqual(waited, 10)
def test_execution_is_the_sum_of_job_run_times_and_a_skipped_job_adds_nothing(self):
r = run(1, "main.yml@refs/heads/main", "push", "success",
[job("a", 0, 10, 30), job("b", 0, 30, 40), job("c", 0, 40, 40, "skipped")])
self.assertEqual(r.execution_minutes, 30)
def test_a_queued_run_has_no_wait_and_no_execution_yet(self):
r = run(1, "pr.yml@refs/pull/7/head", "pull_request", None, [job("done", 0)], status="queued", pull=7)
self.assertEqual((r.queue_minutes, r.execution_minutes), (None, 0.0))
def test_percentiles_use_the_nearest_rank(self):
self.assertEqual((m.percentile([3, 1, 2, 10], 0.5), m.percentile([3, 1, 2, 10], 0.9)), (2, 10))
class StageTest(unittest.TestCase):
def test_a_cancelled_run_counts_as_a_run_but_not_in_execution_percentiles(self):
# given
runs = [
run(1, "pr.yml@refs/pull/7/head", "pull_request", "success", [job("done", 0, 1, 13)], pull=7),
run(2, "pr.yml@refs/pull/7/head", "pull_request", "cancelled", [job("done", 0, 1, 2)], pull=7),
]
# when
row = m.stage_rows(runs)[0]
# then
self.assertEqual((row["runs"], row["finished"], row["exec_p50"], row["wait_p50"]), (2, 1, 12, 1))
def test_runs_per_merged_pull_request_count_every_attempt_of_that_pull_request(self):
runs = [
run(1, "pr.yml@refs/pull/7/head", "pull_request", "cancelled", [job("done", 0)], pull=7),
run(2, "pr.yml@refs/pull/7/head", "pull_request", "success", [job("done", 1, 2, 14)], pull=7),
run(3, "pr.yml@refs/pull/8/head", "pull_request", "success", [job("done", 1, 2, 14)], pull=8),
]
self.assertEqual(m.runs_per_merged_pull(runs, [7, 8]), [2, 1])
class RedTest(unittest.TestCase):
def test_main_is_red_from_the_first_failing_runs_creation_to_the_next_green_runs_end(self):
# given: two failures in a row, then a green run that ends at minute 210
runs = [
run(1, "main.yml@refs/heads/main", "push", "failure", [job("main", 0, 5, 30, "failure")]),
run(2, "main.yml@refs/heads/main", "push", "failure", [job("main", 60, 65, 95, "failure")]),
run(3, "main.yml@refs/heads/main", "push", "success", [job("main", 120, 125, 210)]),
run(4, "pr.yml@refs/pull/7/head", "pull_request", "failure", [job("done", 0, 1, 2, "failure")], pull=7),
]
# when
episodes = m.red_episodes(runs, now=T0 + timedelta(hours=10))
# then
self.assertEqual([(e.first_failed_run, e.fixed_by_run, e.hours) for e in episodes], [(1, 3, 3.5)])
def test_main_still_red_at_the_end_of_the_window_counts_up_to_now(self):
runs = [run(1, "main.yml@refs/heads/main", "push", "failure", [job("main", 0, 5, 30, "failure")])]
episodes = m.red_episodes(runs, now=T0 + timedelta(hours=2))
self.assertEqual([(e.fixed_by_run, e.hours) for e in episodes], [(None, 2.0)])
class UtilisationTest(unittest.TestCase):
def test_job_minutes_count_on_the_local_day_the_job_started(self):
# given: 14:30 UTC is 23:30 KST on the 13th, 15:30 UTC is 00:30 KST on the 14th
r = run(1, "main.yml@refs/heads/main", "push", "success",
[job("a", 0, 14 * 60 + 30, 14 * 60 + 50), job("b", 0, 15 * 60 + 30, 15 * 60 + 40)])
# when
days = m.utilisation([r], timedelta(hours=9))
# then
self.assertEqual({str(day): value for day, value in days.items()}, {"2026-11-13": 20, "2026-11-14": 10})
if __name__ == "__main__":
unittest.main()
@@ -14,6 +14,11 @@ Kubernetes 리소스를 직접 참조하며 업스트림 매니페스트를 복
- 저장소: <https://github.com/keycloak/keycloak-k8s-resources>
- 리소스 경로: `kubernetes`
- Keycloak 및 Operator 버전: `26.7.0`
- 원격 ref: 태그가 아니라 그 태그의 커밋 `38fd7a0c84e15d4b66b37d1c2ea728ca2f5416a4`(2026-11-12). 같은 태그의 내용이
한 번 바뀌었다(build-timestamp 07-09 → 07-23, 성숙도 점검 A5).
- 이미지: Operator·Keycloak 모두 2026-11-12에 돌던 digest로 고정, Operator `imagePullPolicy: IfNotPresent`.
Keycloak CR의 `spec.image`를 정하면 Operator가 최적화 빌드된 이미지로 보므로 `startOptimized: false`를 같이 둔다.
올릴 때는 태그·커밋·두 digest와 `scripts/validate/render-phase2.sh`·`scripts/bootstrap/apply-keycloak.sh`의 검사 문자열을 함께 바꾼다.
- 설치 모드: `keycloak` namespace 범위
업스트림 namespace 범위 Deployment는 각 컨트롤러의 감시 namespace를
@@ -3,8 +3,38 @@ kind: Kustomization
namespace: keycloak
# keycloak-k8s-resources tag 26.7.0 pointed at this commit on 2026-11-12. The tag
# content changed once (live build-timestamp 2026-07-09, later renders 2026-07-23),
# so the commit is pinned instead of the tag.
resources:
- github.com/keycloak/keycloak-k8s-resources/kubernetes?ref=26.7.0
- github.com/keycloak/keycloak-k8s-resources/kubernetes?ref=38fd7a0c84e15d4b66b37d1c2ea728ca2f5416a4
# The 26.7.0 images that were running on 2026-11-12 (pod imageID), by digest.
images:
- name: quay.io/keycloak/keycloak-operator
digest: sha256:257ee324da2c701b7fc1ed1b2ce0e03979d1bf149952383bdcf6be904ae5b4f3
patches:
# Merged by container and env name (strategic merge), not by list index.
- target:
group: apps
version: v1
kind: Deployment
name: keycloak-operator
patch: |-
apiVersion: apps/v1
kind: Deployment
metadata:
name: keycloak-operator
spec:
template:
spec:
containers:
- name: keycloak-operator
imagePullPolicy: IfNotPresent
env:
- name: RELATED_IMAGE_KEYCLOAK
value: quay.io/keycloak/keycloak@sha256:0f198be292568439d700cdbfb893e69a6009bb43a94a06a945b1d3d506c76b13
labels:
- pairs:
+188
View File
@@ -0,0 +1,188 @@
# desktop 호스트 설정
desktop 한 대(k3s 노드 `donghyeon-system-product-name`)는 운영 k3s, Gitea
Actions runner, 에이전트 세션의 Docker 게이트를 같이 돌린다. 루트 SSD 하나와
시계 하나가 무너지면 운영까지 같이 내려간다. 이 폴더는 그 두 가지를 막는 호스트
설정 조각이다. Kubernetes 리소스가 아니므로 `kustomization.yaml`이 없고, root
권한으로 아래 경로에 설치한다.
| 파일 | 설치 경로 | 하는 일 |
| --- | --- | --- |
| `docker/daemon.json` | `/etc/docker/daemon.json` | 호스트 Docker 빌드 캐시 상한 |
| `systemd/k3s.service.d/10-after-time-sync.conf` | `/etc/systemd/system/k3s.service.d/` | k3s를 시간 동기화 뒤에 시작 |
| `systemd/systemd-time-wait-sync.service.d/10-timeout.conf` | `/etc/systemd/system/systemd-time-wait-sync.service.d/` | 동기화를 10분까지만 기다림 |
runner 안쪽 Docker 상한과 주 1회 정리는
[`../ci/gitea-runner/README.md`](../ci/gitea-runner/README.md)의 「디스크와 정리」에 있다.
## 왜 바꾸나
### 디스크 (2026-11-09 사고)
- 11-07~08 이틀 동안 에이전트 게이트의 Docker 빌드 캐시가 약 23G 쌓였다. 상한이 없었다.
- 루트 사용률 85%에서 kubelet이 안 쓰는 이미지를 지우기 시작했고, 90%(여유 10% 미만)에서
파드를 내보냈다. 내보내진 tech-log 파드의 이미지가 「안 쓰는 이미지」가 되어 지워졌고,
`imagePullPolicy: Never`라 다시 받지 못했다.
- 디스크 경보(85% 경고)는 사고 전 33시간 동안 울렸지만, 이미 kubelet 이미지 GC가 시작되는
시점이었다.
### BuildKit 상한 값
Docker 29에서 `builder.gc` 키의 뜻은 다음과 같다(moby `daemon/config/builder.go`).
| 키 | 뜻 | 호스트 값 |
| --- | --- | --- |
| `defaultReservedSpace` | 이 양 아래로는 지우지 않는다 | 5GB |
| `defaultMaxUsedSpace` | 캐시가 이 양을 넘으면 지운다(상한) | 15GB |
| `defaultMinFreeSpace` | 디스크 여유가 이 양 아래면 `reservedSpace`까지 지운다 | 35GB |
- 예전 키 `defaultKeepStorage`는 이제 `defaultReservedSpace`와 같은 뜻이다. 상한이 아니라
「최소 보존량」이라 쓰지 않는다.
- 설정 전 기본 정책은 `docker buildx inspect default`에서 Reserved 21.42GiB,
Max Used 169.5GiB, Min Free 42.84GiB였다(2026-11-11). 캐시가 21GB까지는 전혀 지워지지
않았다.
- `35GB`는 1024 단위다. 루트 228GiB에서 여유 35GiB는 사용률 약 85%, 즉 kubelet이 이미지를
지우기 시작하는 지점이다. 그 전에 빌드 캐시부터 줄인다.
- Docker 저장소를 HDD(sdb)로 옮기는 안은 빌드가 느려져 하지 않는다(소유자 결정, 2026-11-11).
### 시계 (2026-11-11 사고)
- desktop은 Windows와 이중 부팅한다(`efibootmgr`: Windows Boot Manager, Ubuntu).
- Linux는 RTC를 UTC로 읽는다(`timedatectl`: `RTC in local TZ: no`). Windows는 기본값으로
RTC에 현지 시각(KST)을 쓴다. 그래서 Windows를 쓴 뒤 Linux로 부팅하면 시계가 9시간 앞선다.
- 11-11 부팅이 그 상태로 시작했다(`journalctl --list-boots`의 시작 시각 11-12 04:33 KST).
k3s가 그 시각으로 인증서를 만든 뒤 timesyncd가 시계를 되돌렸고, 인증서가 「not yet valid」가
되어 전부 502가 됐다. `sudo systemctl restart k3s`로 복구했다.
- Prometheus도 그 사이 9시간 앞선 시각의 샘플을 저장했다. 시계가 돌아온 뒤 새 샘플이 전부
「out of bounds」로 버려져, 미래 시각을 따라잡을 때까지 지표와 경보가 멈춘다.
고치는 방법은 둘이다.
1. **k3s가 시간 동기화 뒤에 시작한다.** `systemd-time-wait-sync.service`는 커널 시계가
NTP로 맞춰질 때까지 `time-sync.target`을 막는다. k3s는 `After=time-sync.target`으로 그
뒤에 시작한다. 이 서비스는 기본으로 꺼져 있고 끝없이 기다리므로, 켜고 10분 상한을 둔다.
10분 안에 NTP에 닿지 못하면 k3s는 지금처럼 그냥 시작한다.
2. **Windows도 RTC를 UTC로 쓰게 한다.** 그러면 9시간 차이가 처음부터 생기지 않는다.
## 소유자가 실행할 명령
sudo가 필요해 에이전트 세션은 실행하지 않는다. platform-core checkout(이 PR이 병합된
main)에서 위에서부터 순서대로 실행한다. 단계마다 확인 명령과 되돌리는 법이 있다.
### 1. 운영 이미지를 kubelet 이미지 GC에서 지킨다 (재시작 없음, 바로)
tech-log 이미지를 Gitea 레지스트리로 옮기기 전까지만 쓴다. pinned 라벨이 있는 이미지는
kubelet이 지우지 않는다.
```bash
sudo k3s ctr -n k8s.io images label docker.io/library/tech-log-backend:46b1cc7 io.cri-containerd.pinned=pinned
sudo k3s ctr -n k8s.io images label docker.io/library/tech-log-frontend:7a4e201 io.cri-containerd.pinned=pinned
sudo k3s ctr -n k8s.io images ls | grep -E 'tech-log-(backend|frontend)'
```
마지막 줄의 LABELS 칸에 `io.cri-containerd.pinned=pinned`가 있어야 한다.
배포 이미지가 바뀌면 새 tag에 다시 붙인다.
되돌리기: 라벨 값을 비운다.
```bash
sudo k3s ctr -n k8s.io images label docker.io/library/tech-log-backend:46b1cc7 io.cri-containerd.pinned=
sudo k3s ctr -n k8s.io images label docker.io/library/tech-log-frontend:7a4e201 io.cri-containerd.pinned=
```
### 2. k3s를 시간 동기화 뒤에 시작한다 (재시작 없음, 다음 부팅부터)
k3s 재시작 창을 열 때는 `scripts/bootstrap/apply-k3s-restart-bundle.sh`(`infrastructure/host/k3s`의
kubelet drop-in과 함께)가 아래 두 파일을 그대로 설치하고 `systemd-time-wait-sync`를 켠다. 그 묶음을
쓰면 이 단계는 건너뛴다. 묶음 없이 이것만 할 때는 아래 명령을 쓴다.
```bash
sudo install -D -m 0644 infrastructure/host/systemd/k3s.service.d/10-after-time-sync.conf \
/etc/systemd/system/k3s.service.d/10-after-time-sync.conf
sudo install -D -m 0644 infrastructure/host/systemd/systemd-time-wait-sync.service.d/10-timeout.conf \
/etc/systemd/system/systemd-time-wait-sync.service.d/10-timeout.conf
sudo systemctl daemon-reload
sudo systemctl enable systemd-time-wait-sync.service
```
확인:
```bash
systemctl show k3s -p After -p Wants | grep -o time-sync.target
systemctl is-enabled systemd-time-wait-sync.service
systemctl show systemd-time-wait-sync.service -p TimeoutStartUSec
```
`time-sync.target` 두 줄, `enabled`, `TimeoutStartUSec=10min`이 나와야 한다. 순서는 부팅할
때만 쓰이므로 지금 k3s를 다시 시작하지 않는다. 다음 재부팅 뒤
`journalctl -b -u systemd-time-wait-sync -u k3s | head`에서 동기화가 k3s 시작보다 앞서는지 본다.
되돌리기:
```bash
sudo systemctl disable systemd-time-wait-sync.service
sudo rm /etc/systemd/system/k3s.service.d/10-after-time-sync.conf \
/etc/systemd/system/systemd-time-wait-sync.service.d/10-timeout.conf
sudo systemctl daemon-reload
```
### 3. RTC를 UTC로 맞춘다
Linux 쪽은 이미 UTC다. 확인만 한다.
```bash
timedatectl | grep -e 'RTC in local TZ' -e 'System clock synchronized'
```
`RTC in local TZ: no`가 아니면 `sudo timedatectl set-local-rtc 0`을 실행한다.
Windows 쪽(다음에 Windows로 부팅했을 때, 관리자 명령 프롬프트):
```bat
reg add "HKLM\SYSTEM\CurrentControlSet\Control\TimeZoneInformation" /v RealTimeIsUniversal /t REG_DWORD /d 1 /f
```
그 뒤 Windows 설정 → 시간 및 언어 → 날짜 및 시간 → 「지금 동기화」를 누른다. Windows가
RTC를 UTC로 다시 쓴다. Linux로 돌아와 `timedatectl`의 `RTC time`이 `Universal time`과
같은지 본다.
되돌리기(Windows): `reg delete "HKLM\SYSTEM\CurrentControlSet\Control\TimeZoneInformation" /v RealTimeIsUniversal /f`
### 4. 호스트 Docker 빌드 캐시 상한 (Docker 재시작 필요)
`systemctl restart docker`는 호스트 Docker의 컨테이너를 모두 다시 시작한다. 운영 k3s는
자기 containerd를 쓰므로 영향이 없다. 영향을 받는 것:
| 컨테이너 | 영향 |
| --- | --- |
| `gitea-runner` | 실행 중인 CI job이 실패한다. `restart: always`로 다시 뜬다 |
| `workmachine` | cokacremote 원격 작업이 끊긴다. `unless-stopped`로 다시 뜬다 |
| `chatgpt-cloudflared` | 터널이 잠깐 끊긴다. `unless-stopped`로 다시 뜬다 |
| 에이전트 세션 게이트 | 돌고 있던 Testcontainers·이미지 빌드가 실패한다 |
**안전한 때:** 아래 세 가지가 모두 참일 때.
```bash
# 1) CI job이 없다: job 컨테이너가 0개
docker exec gitea-runner docker ps --format '{{.Names}}' | wc -l
# 2) 에이전트 세션의 무거운 게이트가 없다: lock을 바로 잡을 수 있다
flock -n /tmp/claude-gates/heavy.lock true && echo free
# 3) workmachine을 지금 쓰고 있지 않다 (소유자 판단)
```
적용은 lock을 잡은 채로 한다. 재시작하는 동안 새 게이트가 시작되지 않는다.
```bash
sudo install -m 0644 infrastructure/host/docker/daemon.json /etc/docker/daemon.json
sudo dockerd --validate --config-file /etc/docker/daemon.json
flock /tmp/claude-gates/heavy.lock sudo systemctl restart docker
docker ps --format '{{.Names}} {{.Status}}'
docker buildx inspect default | sed -n '/GC Policy rule#1/,/Min Free/p'
```
`dockerd --validate`는 맨 위 키만 검사하고 `builder.gc` 안쪽 키 오타는 잡지 못한다. 그래서
마지막 줄로 실제 정책을 본다. `Reserved Space: 5GiB`, `Max Used Space: 15GiB`,
`Min Free Space: 35GiB`가 나와야 한다. 세 컨테이너가 `Up`이어야 한다.
되돌리기: `sudo rm /etc/docker/daemon.json` 뒤 같은 조건에서 Docker를 다시 시작한다(설치 전에는
이 파일이 없었다).
@@ -0,0 +1,12 @@
[Unit]
Description=Back up Gitea (gitea dump --skip-db) to AIStor (platform-host-backups)
Documentation=file:///usr/local/lib/platform-backup/README.md
After=k3s.service docker.service
Requires=docker.service
[Service]
Type=oneshot
ExecStart=/bin/bash /usr/local/lib/platform-backup/backup/gitea-backup.sh
Nice=10
IOSchedulingClass=idle
TimeoutStartSec=1h
@@ -0,0 +1,10 @@
[Unit]
Description=Weekly Gitea backup
[Timer]
OnCalendar=Sun *-*-* 04:00:00
RandomizedDelaySec=10min
Persistent=true
[Install]
WantedBy=timers.target
@@ -0,0 +1,8 @@
[Unit]
Description=Record the k3s API server and kubelet serving certificate expiry
After=k3s.service
[Service]
Type=oneshot
User=donghyeon
ExecStart=/bin/bash /usr/local/lib/platform-backup/validate/k3s-cert-expiry.sh
@@ -0,0 +1,9 @@
[Unit]
Description=Daily k3s certificate expiry check
[Timer]
OnCalendar=*-*-* 05:00:00
Persistent=true
[Install]
WantedBy=timers.target
@@ -0,0 +1,11 @@
[Unit]
Description=Back up the k3s datastore and token to AIStor (platform-host-backups)
Documentation=file:///usr/local/lib/platform-backup/README.md
After=k3s.service docker.service
Requires=docker.service
[Service]
Type=oneshot
ExecStart=/bin/bash /usr/local/lib/platform-backup/backup/k3s-datastore-backup.sh
Nice=10
IOSchedulingClass=idle
@@ -0,0 +1,10 @@
[Unit]
Description=Daily k3s datastore backup
[Timer]
OnCalendar=*-*-* 03:30:00
RandomizedDelaySec=10min
Persistent=true
[Install]
WantedBy=timers.target
@@ -0,0 +1,16 @@
[Unit]
Description=Compare the cluster with platform-core origin/main (kubectl diff, server dry-run)
After=k3s.service network-online.target
Wants=network-online.target
[Service]
Type=oneshot
User=donghyeon
Environment=PLATFORM_HELM_BIN=/home/donghyeon/.local/bin/helm
Environment=PLATFORM_CHECKOUT=/home/donghyeon/workspace/platform
ExecStart=/bin/bash /usr/local/lib/platform-backup/validate/live-drift-check-main.sh
# Exit 1 means "drift found"; the metric and its alert report that. Only 2 (the
# check itself failed) marks the unit failed.
SuccessExitStatus=1
Nice=10
TimeoutStartSec=30min
@@ -0,0 +1,10 @@
[Unit]
Description=Daily live drift check
[Timer]
OnCalendar=*-*-* 05:15:00
RandomizedDelaySec=10min
Persistent=true
[Install]
WantedBy=timers.target
+10
View File
@@ -0,0 +1,10 @@
{
"builder": {
"gc": {
"enabled": true,
"defaultReservedSpace": "5GB",
"defaultMaxUsedSpace": "15GB",
"defaultMinFreeSpace": "35GB"
}
}
}
+92
View File
@@ -0,0 +1,92 @@
# k3s 재시작 묶음
k3s를 **한 번만** 재시작해서 다음을 같이 넣는다. 근거는 `docs/reviews/2026-11-11-k3s-maturity-review.md`
A3·A9·A10·7.2·7.3·7.5절, 결정은 같은 문서 8절 결정 5(2026-11-12 총괄 확정)다.
| 파일 | 설치 위치 | 하는 일 |
| --- | --- | --- |
| `kubelet.conf.d/50-platform-resources.conf` | `/var/lib/rancher/k3s/agent/etc/kubelet.conf.d/` | 예약(system 1 CPU·2Gi, kube 0.5 CPU·1Gi), 메모리 축출(hard 500Mi, soft 1Gi/1분), graceful node shutdown(60초, critical 20초) |
| `config.yaml.d/60-platform-apiserver.yaml` | `/etc/rancher/k3s/config.yaml.d/` | API 서버 감사 로그와 PodSecurity admission 설정 파일 지정 |
| `server/audit.yaml` | `/var/lib/rancher/k3s/server/audit.yaml` | 감사 정책: Metadata 수준(본문을 남기지 않아 Secret 값이 로그에 없다), 잦고 값 없는 요청은 뺌 |
| `server/psa.yaml` | `/var/lib/rancher/k3s/server/psa.yaml` | 라벨 없는 namespace의 기본값: enforce `baseline`, warn·audit `restricted`, `kube-system` 예외 |
| (W1) `../systemd/k3s.service.d/10-after-time-sync.conf` | `/etc/systemd/system/k3s.service.d/` | k3s를 시간 동기화 뒤에 시작 |
| (W1) `../systemd/systemd-time-wait-sync.service.d/10-timeout.conf` | `/etc/systemd/system/systemd-time-wait-sync.service.d/` | 동기화를 10분까지만 기다림 |
그리고 `/var/lib/rancher/k3s/server/manifests/local-storage.yaml.skip`을 만들어 k3s가 local-path
addon을 다시 덮어쓰지 않게 한 뒤, `local-path` StorageClass의 기본 표시를 끈다. 쓰는 PVC가 없고
(2026-11-11), 기본으로 남아 있으면 `storageClassName`을 빠뜨린 PVC가 루트 SSD에 Delete 정책으로 생긴다.
k3s 문서의 `.skip` 방식이다(packaged components). 그 뒤 local-path-provisioner는 k3s 업그레이드로
갱신되지 않는다.
## 값을 이렇게 정한 이유
- 예약: 노드 32GiB, 12 CPU. 파드 requests 합이 메모리 11Gi(34%)라 예약 3Gi를 빼도 여유가 크다.
예약이 없으면 Allocatable = Capacity라 파드가 노드 전부를 잡을 수 있고, 같은 호스트의 Docker
빌드·CI·데스크톱 앱과 겹치면 커널 OOM killer가 고른다.
- `evictionHard`: 메모리·디스크 셋을 모두 적는다. 이 맵을 정하면 정하지 않은 항목의 기본값이 0이 된다
(`mergeDefaultEvictionSettings: false`). 디스크 5%는 k3s 기본값 그대로다.
- graceful shutdown: 두 값이 0이면 기능이 꺼진다. 재부팅 때 Postgres 같은 파드가 정상 종료할 시간을
준다. 호스트 종료가 최대 60초 늦어진다. kubelet이 logind `InhibitDelayMaxSec`(지금 30초)가 모자라면
스스로 늘린다. 적용 뒤 `systemd-analyze cat-config systemd/logind.conf`로 확인한다.
- PSA 기본값은 `restricted`가 아니라 `baseline`이다. 라벨 없는 namespace는 `default`·`kube-public`·
`kube-node-lease`뿐이고 파드가 없다. 플랫폼 namespace는 모두 자기 라벨이 있어 영향이 없다.
## 넣지 않은 것: `protect-kernel-defaults`
7.3절에서 권했지만 이 묶음에서 뺐다.
- CIS Kubernetes Benchmark는 v1.7부터 이 항목(옛 4.2.6)을 뺐다.
- 켜려면 kubelet이 기대하는 sysctl을 먼저 맞춰야 한다(k3s hardening 가이드: `vm.overcommit_memory=1`,
`kernel.panic=10`, `kernel.panic_on_oops=1`, `vm.panic_on_oom=0`). `kernel.panic_on_oops=1`은 커널
oops가 한 번만 나도 10초 뒤 호스트를 재부팅한다. 이 기계는 GPU 드라이버가 있는 데스크톱이고
이중 부팅이며, 재부팅 자체가 11-11 시계 사고의 출발점이었다. `vm.overcommit_memory=1`은 호스트 전체의
메모리 할당 방식을 바꾼다.
- 얻는 것(kubelet이 커널 값을 못 바꾸게 함)보다 잃는 것이 커서 뺀다. 다시 넣으려면 sysctl 변경을 따로
소유자가 정한다.
## 적용 (소유자, 한 번)
전제:
- desktop, 이 디렉터리와 W1의 `infrastructure/host/systemd/*`가 **둘 다** 있는 platform-core `main` checkout
(W1 PR 병합 뒤). 없으면 스크립트가 `MISSING …(W1)`을 찍고 멈춘다.
- CI job과 에이전트 게이트가 없는 때. k3s 재시작 중 1~2분 API가 멈춘다. k3s는 `KillMode=process`라
컨테이너는 계속 돈다(파드가 재시작되지는 않는다).
```bash
cd /home/donghyeon/workspace/platform && git pull --ff-only
bash scripts/bootstrap/apply-k3s-restart-bundle.sh --check
sudo bash scripts/bootstrap/apply-k3s-restart-bundle.sh --execute
```
`--execute`가 하는 일:
1. 클러스터 Ready와 health 셋(Keycloak OIDC discovery, tech-log health, Gitea healthz)을 본다.
2. 바꿀 파일을 `/var/backups/platform-k3s-restart-bundle/<시각>/`에 복사하고, datastore를
`sqlite3 .backup`으로 복사한다(`token`도 같이, 모드 0600, `PRAGMA integrity_check` 출력).
3. 파일을 설치하고 `systemd-time-wait-sync`를 켠 뒤 k3s를 한 번 재시작한다.
4. 확인: kubelet configz(예약·eviction·shutdown 값), 감사 로그가 쓰이는지, `default` namespace에서
privileged 파드가 서버 dry-run으로 거절되는지, health 셋이 200인지, time-wait-sync가 enabled인지.
5. 3·4 중 하나라도 실패하면 백업한 파일을 되돌리고 새 파일을 지운 뒤 k3s를 다시 재시작한다.
6. 다 통과하면 `local-path`의 기본 표시를 끈다.
마지막 줄이 `K3S RESTART BUNDLE APPLIED`여야 한다.
## 같은 재시작 창에 이어서 할 것
PriorityClass를 파드에 붙이는 변경과 Keycloak 이미지 고정은 파드 재시작을 부르므로 이 묶음 바로 뒤에
한다(`docs/runbooks/` 재시작 창 절차). Postgres의 `priorityClassName`·Guaranteed는 W1 범위다.
## 되돌리기 (수동)
```bash
B=/var/backups/platform-k3s-restart-bundle/<시각>
sudo rm -f /var/lib/rancher/k3s/agent/etc/kubelet.conf.d/50-platform-resources.conf \
/etc/rancher/k3s/config.yaml.d/60-platform-apiserver.yaml \
/var/lib/rancher/k3s/server/manifests/local-storage.yaml.skip
sudo systemctl daemon-reload && sudo systemctl restart k3s
kubectl annotate storageclass local-path storageclass.kubernetes.io/is-default-class=true --overwrite
```
`audit.yaml`·`psa.yaml`는 참조가 없어지면 쓰이지 않으므로 남겨도 된다. W1 시간 동기화 되돌리기는
`infrastructure/host/README.md` 2단계를 따른다.
@@ -0,0 +1,12 @@
# Installed as /etc/rancher/k3s/config.yaml.d/60-platform-apiserver.yaml.
# "+" appends to kube-apiserver-arg instead of replacing it (docs.k3s.io,
# installation/configuration, multiple config files).
# Audit log: k3s hardening guide "API Server audit configuration".
# PodSecurity default: k3s hardening guide "Pod Security" (baseline here, see README).
kube-apiserver-arg+:
- "audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log"
- "audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml"
- "audit-log-maxage=30"
- "audit-log-maxbackup=10"
- "audit-log-maxsize=100"
- "admission-control-config-file=/var/lib/rancher/k3s/server/psa.yaml"
@@ -0,0 +1,29 @@
# Installed as /var/lib/rancher/k3s/agent/etc/kubelet.conf.d/50-platform-resources.conf.
# k3s starts the kubelet with --config-dir on that directory and keeps its own
# defaults in 00-k3s-defaults.conf; this file is merged after it.
# Why: docs/reviews/2026-11-11-k3s-maturity-review.md A3 and 7.2.
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
# Keep memory and CPU for the host (desktop, Docker builds, CI runner) and for
# k3s itself, so pods cannot take the whole node.
systemReserved:
cpu: "1"
memory: 2Gi
kubeReserved:
cpu: 500m
memory: 1Gi
# All three keys are listed on purpose: setting evictionHard replaces the
# defaults, and k3s's own default only had the two disk keys.
evictionHard:
memory.available: 500Mi
nodefs.available: "5%"
imagefs.available: "5%"
evictionSoft:
memory.available: 1Gi
evictionSoftGracePeriod:
memory.available: 1m
evictionMaxPodGracePeriod: 60
# Graceful node shutdown: on reboot or power-off the kubelet stops normal pods
# first (40s), then critical pods (20s), instead of leaving it to systemd.
shutdownGracePeriod: 60s
shutdownGracePeriodCriticalPods: 20s
+28
View File
@@ -0,0 +1,28 @@
# Installed as /var/lib/rancher/k3s/server/audit.yaml (mode 0600).
# Metadata level only: who did what to which object, never request or
# response bodies, so Secret values are not written to the log.
apiVersion: audit.k8s.io/v1
kind: Policy
omitStages:
- RequestReceived
rules:
# High-volume, low-value traffic.
- level: None
resources:
- group: coordination.k8s.io
resources: ["leases"]
- level: None
resources:
- group: ""
resources: ["events"]
- level: None
nonResourceURLs:
- /healthz*
- /livez*
- /readyz*
- /version
- /metrics
- level: None
users: ["system:kube-proxy"]
verbs: ["watch"]
- level: Metadata
+22
View File
@@ -0,0 +1,22 @@
# Installed as /var/lib/rancher/k3s/server/psa.yaml (mode 0600).
# Cluster-wide Pod Security default for namespaces without their own labels.
# Namespace labels still win: every platform namespace already sets its level.
apiVersion: apiserver.config.k8s.io/v1
kind: AdmissionConfiguration
plugins:
- name: PodSecurity
configuration:
apiVersion: pod-security.admission.config.k8s.io/v1
kind: PodSecurityConfiguration
defaults:
enforce: "baseline"
enforce-version: "latest"
audit: "restricted"
audit-version: "latest"
warn: "restricted"
warn-version: "latest"
exemptions:
usernames: []
runtimeClasses: []
# k3s helm-install jobs and local-path helper pods run here.
namespaces: [kube-system]
@@ -0,0 +1,9 @@
# Start k3s only after the system clock is synchronized.
# k3s writes certificates with the current time at start. If the clock is
# wrong at boot (RTC read 9 hours ahead on 2026-11-11) and NTP corrects it
# later, those certificates are "not yet valid" and every Ingress returns 502.
# time-sync.target is reached only when systemd-time-wait-sync.service is
# enabled. See infrastructure/host/README.md.
[Unit]
Wants=time-sync.target
After=time-sync.target
@@ -0,0 +1,5 @@
# The packaged unit waits forever (TimeoutStartSec=infinity). If NTP is not
# reachable at boot, k3s would never start. After 10 minutes this unit fails,
# time-sync.target is reached anyway, and k3s starts with the clock it has.
[Service]
TimeoutStartSec=10min
@@ -94,6 +94,7 @@ infrastructure/networking/traefik/
├── apply-observe.sh
├── observe-client-host.sh
├── apply-trust.sh
├── reapply-trust.sh
├── rollback-to-observe.sh
└── validate.sh
```
@@ -238,6 +239,20 @@ bash infrastructure/networking/traefik/scripts/observe-client-host.sh
`apply-trust.sh`를 실행한다. Chart, 이미지, context, API server, NodePort 경계 또는
재관측 값이 기대와 다르면 스크립트가 적용을 중단한다.
이미 trust인 상태에서 trust overlay의 다른 값(예: `resources`)만 바꿨을 때는 `apply-trust.sh`가
「observation phase must not trust forwarded headers」로 멈춘다(observe → trust 전환 전용이다). 그때는
`reapply-trust.sh`를 쓴다.
```sh
bash infrastructure/networking/traefik/scripts/reapply-trust.sh --observed-client-host 10.42.0.1 --execute
# 승인: APPLY <현재-context> RETRUST 10.42.0.1/32
```
- 적용 전에 runtime이 trust인지, 전달 헤더 신뢰가 실제로 작동하는지(Host Nginx 경유 요청의 ClientHost가
프록시 주소가 아님)를 본다. 바꿀 것을 `kubectl diff`로 보여 주고, 차이가 없으면 아무것도 하지 않는다.
- 적용 뒤 같은 검사를 다시 하고, 하나라도 실패하면 직전에 라이브였던 HelmChartConfig를 다시 적용한다.
trusted CIDR이 빠지는 일은 없다.
### 롤백
trust만 제거하고 JSON access log를 남기는 표준 롤백은 observe overlay를 적용한다.
+126
View File
@@ -0,0 +1,126 @@
#!/usr/bin/env bash
# Re-apply the trust overlay when the live Traefik is already in the trust phase,
# for example after a values change such as resources. apply-trust.sh is only for
# the observe -> trust transition and stops when the runtime already trusts the
# CIDR ("observation phase must not trust forwarded headers").
#
# bash infrastructure/networking/traefik/scripts/reapply-trust.sh \
# --observed-client-host 10.42.0.1 --execute
#
# On any failure after the apply it restores the HelmChartConfig that was live
# before, so the trusted CIDR is never dropped.
set -Eeuo pipefail
set +x
umask 077
readonly SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd -P)"
# shellcheck source=common.sh
source "${SCRIPT_DIR}/common.sh"
[[ "$#" -eq 3 && "$1" == "--observed-client-host" && "$3" == "--execute" ]] || {
sed -n '2,11p' "${BASH_SOURCE[0]}" >&2
exit 2
}
readonly REVIEWED_CLIENT_HOST="$2"
require_commands kubectl jq rg curl nc python3 awk sort date cmp find bash
reviewed_cidr="$(host_to_exact_cidr "$REVIEWED_CLIENT_HOST")"
declared_cidr="$(normalize_exact_host_cidr "$(source_trusted_proxy_cidr)")"
[[ "$declared_cidr" != "$SENTINEL_TRUSTED_PROXY_CIDR" ]] || \
fail "trust overlay still contains the non-routable sentinel CIDR"
[[ "$declared_cidr" == "$reviewed_cidr" ]] || \
fail "declared CIDR ${declared_cidr} does not match ClientHost ${REVIEWED_CLIENT_HOST}"
bash "${SCRIPT_DIR}/validate.sh"
work_dir="$(mktemp -d "${TMPDIR:-/tmp}/platform-traefik-retrust.XXXXXX")"
restore_required=false
selected_context=""
cleanup() {
local exit_code=$?
trap - EXIT
if [[ "$exit_code" -ne 0 && "$restore_required" == "true" ]]; then
printf '\nRESTORE: re-applying the HelmChartConfig that was live before.\n' >&2
set +e
kubectl --context "$selected_context" apply --filename "${work_dir}/previous.yaml"
wait_for_runtime trust "$declared_cidr"
kubectl --context "$selected_context" --namespace kube-system \
rollout status deployment/traefik --timeout=5m
wait_for_nodeport_boundary_and_health
printf 'RESTORE complete. The previous trust values are live again.\n' >&2
set -e
fi
case "$work_dir" in
/tmp/platform-traefik-retrust.*|"${TMPDIR:-/tmp}"/platform-traefik-retrust.*) rm -rf -- "$work_dir" ;;
esac
exit "$exit_code"
}
trap cleanup EXIT
trap 'exit 130' INT
trap 'exit 143' TERM
trust_render="${work_dir}/trust.yaml"
render_overlay "$TRUST_OVERLAY" "$trust_render"
assert_live_baseline
assert_runtime trust "$declared_cidr"
assert_nodeport_boundary_and_health
# In the trust phase Traefik logs the forwarded client, not the proxy. Seeing the
# proxy address (the reviewed ClientHost) would mean the trust is not working.
assert_trust_effective() {
local seen
seen="$(observe_host_nginx_client_host)"
[[ "$seen" != "$REVIEWED_CLIENT_HOST" ]] || \
fail "Traefik logged the proxy ${seen} as ClientHost; forwarded-header trust is not in effect"
printf 'Forwarded-header trust in effect (ClientHost %s, proxy %s)\n' "$seen" "$REVIEWED_CLIENT_HOST"
}
assert_trust_effective
kubectl --namespace kube-system get helmchartconfig traefik --output=json |
python3 -I -c '
import json, sys
o = json.load(sys.stdin)
md = o["metadata"]
keep = {k: md[k] for k in ("name", "namespace", "labels", "annotations") if k in md}
keep.get("annotations", {}).pop("kubectl.kubernetes.io/last-applied-configuration", None)
print(json.dumps({"apiVersion": o["apiVersion"], "kind": o["kind"], "metadata": keep, "spec": o["spec"]}))
' >"${work_dir}/previous.yaml"
printf '\nChange to apply (kubectl diff, server dry-run):\n'
set +e
KUBECTL_EXTERNAL_DIFF="diff -u -N" kubectl diff --filename "$trust_render" | grep -E '^[-+] ' | grep -v -E '^[-+] generation:'
diff_rc=${PIPESTATUS[0]}
set -e
if [[ "$diff_rc" -eq 0 ]]; then
printf 'Live HelmChartConfig already matches the trust overlay. Nothing to do.\n'
exit 0
fi
(( diff_rc == 1 )) || fail "kubectl diff failed"
selected_context="$(current_context)"
selected_api_server="$(current_api_server)"
printf '\nKubernetes context: %s\nAPI server: %s\nTrusted CIDR (unchanged): %s\n' \
"$selected_context" "$selected_api_server" "$declared_cidr"
printf 'Type APPLY %s RETRUST %s to continue: ' "$selected_context" "$declared_cidr"
read -r confirmation
[[ "$confirmation" == "APPLY ${selected_context} RETRUST ${declared_cidr}" ]] || fail "cancelled"
[[ "$(current_context)" == "$selected_context" ]] || fail "kubectl context changed after confirmation"
[[ "$(current_api_server)" == "$selected_api_server" ]] || fail "Kubernetes API server changed after confirmation"
assert_runtime trust "$declared_cidr"
restore_required=true
kubectl apply --filename "$trust_render"
wait_for_runtime trust "$declared_cidr"
kubectl --namespace kube-system rollout status deployment/traefik --timeout=5m
assert_runtime trust "$declared_cidr"
assert_live_hcc_matches_overlay "$TRUST_OVERLAY" "${work_dir}/post-apply-trust.yaml"
assert_live_baseline
wait_for_nodeport_boundary_and_health
assert_manifest_https
assert_trust_effective
restore_required=false
printf '\nTRUST PHASE RE-APPLIED\n'
printf 'Traefik web trusted CIDR: %s\n' "$declared_cidr"
printf 'Gitea health and HTTPS site-manifest checks: PASS\n'
+68
View File
@@ -0,0 +1,68 @@
#!/usr/bin/env bash
# Back up Gitea (repositories, packages, attachments, Actions data, app.ini) to
# AIStor (HDD sdb3). Root only. Run by platform-gitea-backup.timer.
#
# `gitea dump --skip-db` runs inside the live Gitea container and streams a
# tar.gz to stdout (logs go to stderr). The database is backed up separately by
# CloudNativePG (services/platform-postgres-backup). The dump contains app.ini,
# whose secrets are needed to restore, so the archive is sensitive.
set -Eeuo pipefail
umask 077
# shellcheck source=../lib/host-backup.sh
source "$(dirname -- "${BASH_SOURCE[0]}")/../lib/host-backup.sh"
readonly NAME=gitea_backup
readonly KCFG=/etc/rancher/k3s/k3s.yaml
[[ "$(id -u)" -eq 0 ]] || hb_fail "run as root"
hb_load_aistor_env
kubectl() { command /usr/local/bin/k3s kubectl --kubeconfig "$KCFG" "$@"; }
started="$(date +%s)"
previous_success="$(awk '/^platform_backup_last_success_timestamp_seconds/ {print $2}' "$TEXTFILE_DIR/$NAME.prom" 2>/dev/null || echo 0)"
on_error() {
hb_write_metrics "$NAME" <<PROM
platform_backup_last_run_success{backup="gitea"} 0
platform_backup_last_run_timestamp_seconds{backup="gitea"} $(date +%s)
platform_backup_last_success_timestamp_seconds{backup="gitea"} ${previous_success:-0}
PROM
}
trap on_error ERR
work="$(mktemp -d "$HB_WORK_ROOT/gitea.XXXXXX")"
trap 'rm -rf -- "$work"' EXIT
ts="$(date -u +%Y%m%dT%H%M%SZ)"
archive="gitea-dump-$ts.tar.gz"
kubectl -n gitea exec deploy/gitea -c gitea -- \
gitea dump --skip-db --type tar.gz --file - >"$work/$archive" 2>"$work/dump.log"
tar -tzf "$work/$archive" >"$work/list.txt"
grep -qx 'app.ini' "$work/list.txt" || hb_fail "the dump has no app.ini"
repos="$(grep -c -E '^repos/[^/]+/[^/]+\.git/HEAD$' "$work/list.txt" || true)"
(( repos > 0 )) || hb_fail "the dump has no repositories"
sha256sum "$work/$archive" | awk '{print $1}' >"$work/$archive.sha256"
size="$(stat -c %s "$work/$archive")"
# What a restore must match: restore with the same Gitea version (docs/runbooks/gitea-restore.md).
{
echo "created=$ts"
echo "gitea_image=$(kubectl -n gitea get deploy gitea -o jsonpath='{.spec.template.spec.containers[0].image}')"
echo "gitea_version=$(kubectl -n gitea exec deploy/gitea -c gitea -- gitea --version 2>/dev/null | awk '{print $3}')"
echo "repositories=$repos"
} >"$work/$archive.meta"
hb_mc "$work" cp --quiet "/work/$archive" "/work/$archive.sha256" "/work/$archive.meta" "backup/$AISTOR_BUCKET/gitea/" >/dev/null
hb_mc "$work" stat "backup/$AISTOR_BUCKET/gitea/$archive" >/dev/null
trap - ERR
hb_write_metrics "$NAME" <<PROM
# HELP platform_backup_last_success_timestamp_seconds Unix time of the last successful backup.
# TYPE platform_backup_last_success_timestamp_seconds gauge
platform_backup_last_success_timestamp_seconds{backup="gitea"} $(date +%s)
platform_backup_last_run_success{backup="gitea"} 1
platform_backup_last_run_timestamp_seconds{backup="gitea"} $(date +%s)
platform_backup_last_size_bytes{backup="gitea"} $size
platform_backup_last_duration_seconds{backup="gitea"} $(( $(date +%s) - started ))
# HELP platform_backup_gitea_repositories Repositories in the last successful Gitea dump.
# TYPE platform_backup_gitea_repositories gauge
platform_backup_gitea_repositories $repos
PROM
hb_log "gitea backup $archive ($size bytes, $repos repositories) uploaded"
+99
View File
@@ -0,0 +1,99 @@
#!/usr/bin/env bash
# Back up the k3s datastore (sqlite) and the server token to AIStor (HDD sdb3).
# Root only. Run by platform-k3s-datastore-backup.timer (infrastructure/host/backup).
#
# k3s docs (datastore/backup-restore, SQLite): copy server/db and server/token;
# the token is required to read the datastore back. `sqlite3 .backup` takes a
# consistent copy of the live database.
#
# Off-host copy (prepared, off by default): set in /etc/platform-backup/aistor.env
# OFFHOST_GPG_RECIPIENT_FILE=/etc/platform-backup/offhost-recipient.asc
# OFFHOST_RCLONE_TARGET=<rclone remote:path>
# The archive is then encrypted to that public key and copied with rclone.
set -Eeuo pipefail
umask 077
# shellcheck source=../lib/host-backup.sh
source "$(dirname -- "${BASH_SOURCE[0]}")/../lib/host-backup.sh"
readonly SERVER=/var/lib/rancher/k3s/server
readonly NAME=k3s_datastore_backup
[[ "$(id -u)" -eq 0 ]] || hb_fail "run as root"
command -v sqlite3 >/dev/null || hb_fail "sqlite3 is required"
hb_load_aistor_env
started="$(date +%s)"
on_error() {
hb_write_metrics "$NAME" <<PROM
# HELP platform_backup_last_run_success 1 when the last run of this backup succeeded.
# TYPE platform_backup_last_run_success gauge
platform_backup_last_run_success{backup="k3s-datastore"} 0
platform_backup_last_run_timestamp_seconds{backup="k3s-datastore"} $(date +%s)
platform_backup_last_success_timestamp_seconds{backup="k3s-datastore"} ${previous_success:-0}
PROM
}
previous_success="$(awk '/^platform_backup_last_success_timestamp_seconds/ {print $2}' "$TEXTFILE_DIR/$NAME.prom" 2>/dev/null || echo 0)"
trap on_error ERR
work="$(mktemp -d "$HB_WORK_ROOT/k3s.XXXXXX")"
trap 'rm -rf -- "$work"' EXIT
ts="$(date -u +%Y%m%dT%H%M%SZ)"
stage="$work/k3s-datastore-$ts"
mkdir -m 0700 "$stage"
sqlite3 "$SERVER/db/state.db" ".backup '$stage/state.db'"
check="$(sqlite3 "$stage/state.db" 'PRAGMA integrity_check;')"
[[ "$check" == "ok" ]] || hb_fail "integrity_check on the copy returned: $check"
install -m 0600 "$SERVER/token" "$stage/token"
{
echo "created=$ts"
echo "k3s_version=$(/usr/local/bin/k3s --version | head -1)"
echo "host=$(hostname)"
echo "kine_rows=$(sqlite3 "$stage/state.db" 'SELECT COUNT(*) FROM kine;')"
} >"$stage/META"
tar -C "$work" -czf "$work/k3s-datastore-$ts.tar.gz" "k3s-datastore-$ts"
sha256sum "$work/k3s-datastore-$ts.tar.gz" | awk '{print $1}' >"$work/k3s-datastore-$ts.tar.gz.sha256"
size="$(stat -c %s "$work/k3s-datastore-$ts.tar.gz")"
hb_mc "$work" cp --quiet "/work/k3s-datastore-$ts.tar.gz" "/work/k3s-datastore-$ts.tar.gz.sha256" "backup/$AISTOR_BUCKET/k3s-datastore/" >/dev/null
hb_mc "$work" stat "backup/$AISTOR_BUCKET/k3s-datastore/k3s-datastore-$ts.tar.gz" >/dev/null
# Local copy outside AIStor on the same HDD (sdb3). AIStor runs inside the
# cluster this archive restores, so after losing the cluster the AIStor copy
# cannot be read until k3s is back; this copy can. The directory sits next to
# AIStor's drive directory (data/), so AIStor does not see it as a bucket.
readonly LOCAL_DIR=/srv/k3s/aistor/host-backups/k3s-datastore
install -d -m 0700 -o root -g root "$(dirname "$LOCAL_DIR")" "$LOCAL_DIR"
install -m 0600 "$work/k3s-datastore-$ts.tar.gz" "$work/k3s-datastore-$ts.tar.gz.sha256" "$LOCAL_DIR/"
ls -1t "$LOCAL_DIR"/k3s-datastore-*.tar.gz | tail -n +8 | while read -r old; do rm -f -- "$old" "$old.sha256"; done
local_copies="$(ls -1 "$LOCAL_DIR"/k3s-datastore-*.tar.gz | wc -l)"
offhost=0
if [[ -n "${OFFHOST_GPG_RECIPIENT_FILE:-}" && -n "${OFFHOST_RCLONE_TARGET:-}" ]]; then
command -v rclone >/dev/null || hb_fail "OFFHOST_RCLONE_TARGET is set but rclone is not installed"
gpg --batch --yes --trust-model always --recipient-file "$OFFHOST_GPG_RECIPIENT_FILE" \
--output "$work/k3s-datastore-$ts.tar.gz.gpg" --encrypt "$work/k3s-datastore-$ts.tar.gz"
rclone copyto "$work/k3s-datastore-$ts.tar.gz.gpg" "$OFFHOST_RCLONE_TARGET/k3s-datastore-$ts.tar.gz.gpg"
offhost=1
fi
trap - ERR
hb_write_metrics "$NAME" <<PROM
# HELP platform_backup_last_success_timestamp_seconds Unix time of the last successful backup.
# TYPE platform_backup_last_success_timestamp_seconds gauge
platform_backup_last_success_timestamp_seconds{backup="k3s-datastore"} $(date +%s)
# HELP platform_backup_last_run_success 1 when the last run of this backup succeeded.
# TYPE platform_backup_last_run_success gauge
platform_backup_last_run_success{backup="k3s-datastore"} 1
platform_backup_last_run_timestamp_seconds{backup="k3s-datastore"} $(date +%s)
# HELP platform_backup_last_size_bytes Size of the last successful backup archive.
# TYPE platform_backup_last_size_bytes gauge
platform_backup_last_size_bytes{backup="k3s-datastore"} $size
platform_backup_last_duration_seconds{backup="k3s-datastore"} $(( $(date +%s) - started ))
# HELP platform_backup_local_copies Archives kept outside AIStor on the HDD.
# TYPE platform_backup_local_copies gauge
platform_backup_local_copies{backup="k3s-datastore"} $local_copies
# HELP platform_backup_offhost_copy 1 when the last run also wrote the encrypted off-host copy.
# TYPE platform_backup_offhost_copy gauge
platform_backup_offhost_copy{backup="k3s-datastore"} $offhost
PROM
hb_log "k3s datastore backup k3s-datastore-$ts.tar.gz ($size bytes) uploaded; off-host=$offhost"
@@ -0,0 +1,238 @@
#!/usr/bin/env bash
# Install the k3s restart bundle and restart k3s exactly once.
#
# bash scripts/bootstrap/apply-k3s-restart-bundle.sh --check # no root, changes nothing
# sudo bash scripts/bootstrap/apply-k3s-restart-bundle.sh --execute
#
# The bundle (infrastructure/host/k3s/README.md):
# W6: kubelet reservations, memory eviction and graceful shutdown (kubelet drop-in),
# API server audit log and cluster Pod Security default (k3s drop-in + two files),
# local-path StorageClass no longer the default (manifest .skip + annotation).
# W1: k3s starts after time sync (infrastructure/host/systemd/*, installed as is).
#
# --execute backs up every file it replaces plus a datastore copy, installs, restarts
# k3s, verifies, and on any failure restores the previous files and restarts again.
set -Eeuo pipefail
umask 077
readonly ROOT="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/../.." && pwd -P)"
readonly KCFG=/etc/rancher/k3s/k3s.yaml
readonly SERVER=/var/lib/rancher/k3s/server
readonly SKIP_FILE="$SERVER/manifests/local-storage.yaml.skip"
readonly AUDIT_DIR="$SERVER/logs"
readonly BACKUP_ROOT=/var/backups/platform-k3s-restart-bundle
# source | destination | mode | owner of the source file
readonly -a FILES=(
"infrastructure/host/k3s/kubelet.conf.d/50-platform-resources.conf|/var/lib/rancher/k3s/agent/etc/kubelet.conf.d/50-platform-resources.conf|0600|W6"
"infrastructure/host/k3s/config.yaml.d/60-platform-apiserver.yaml|/etc/rancher/k3s/config.yaml.d/60-platform-apiserver.yaml|0644|W6"
"infrastructure/host/k3s/server/audit.yaml|$SERVER/audit.yaml|0600|W6"
"infrastructure/host/k3s/server/psa.yaml|$SERVER/psa.yaml|0600|W6"
"infrastructure/host/systemd/k3s.service.d/10-after-time-sync.conf|/etc/systemd/system/k3s.service.d/10-after-time-sync.conf|0644|W1"
"infrastructure/host/systemd/systemd-time-wait-sync.service.d/10-timeout.conf|/etc/systemd/system/systemd-time-wait-sync.service.d/10-timeout.conf|0644|W1"
)
readonly -a HEALTH_URLS=(
https://id.learn.hyeonworks.com/realms/hyeonworks/.well-known/openid-configuration
https://hyeonworks.com/api/healthcheck
https://git.learn.hyeonworks.com/api/healthz
)
say() { printf '%s\n' "$*"; }
fail() { printf 'ERROR: %s\n' "$*" >&2; exit 1; }
mode="${1:-}"
[[ "$#" -eq 1 && ( "$mode" == "--check" || "$mode" == "--execute" ) ]] || {
sed -n '2,15p' "${BASH_SOURCE[0]}" >&2
exit 2
}
check_sources() {
local entry src dst perm owner missing=0
for entry in "${FILES[@]}"; do
IFS='|' read -r src dst perm owner <<<"$entry"
if [[ ! -f "$ROOT/$src" ]]; then
say "MISSING $src (${owner})"
missing=1
fi
done
(( missing == 0 )) || fail "run this from a platform-core checkout that has both the W6 and the W1 host files on main"
python3 -I - "$ROOT" <<'PY' || fail "bundle files do not parse"
import sys, pathlib, yaml
root = pathlib.Path(sys.argv[1]) / "infrastructure/host/k3s"
k = yaml.safe_load((root / "kubelet.conf.d/50-platform-resources.conf").read_text())
assert k["kind"] == "KubeletConfiguration", "kubelet drop-in kind"
assert set(k["evictionHard"]) == {"memory.available", "nodefs.available", "imagefs.available"}, "evictionHard keys"
assert k["shutdownGracePeriod"] == "60s" and k["shutdownGracePeriodCriticalPods"] == "20s", "shutdown periods"
a = yaml.safe_load((root / "config.yaml.d/60-platform-apiserver.yaml").read_text())
assert list(a) == ["kube-apiserver-arg+"], "apiserver drop-in must only append kube-apiserver-arg"
args = dict(x.split("=", 1) for x in a["kube-apiserver-arg+"])
assert args["audit-policy-file"] == "/var/lib/rancher/k3s/server/audit.yaml"
assert args["admission-control-config-file"] == "/var/lib/rancher/k3s/server/psa.yaml"
p = yaml.safe_load((root / "server/audit.yaml").read_text())
assert p["kind"] == "Policy" and p["rules"][-1] == {"level": "Metadata"}, "audit policy ends with Metadata"
assert all(r["level"] in ("None", "Metadata") for r in p["rules"]), "audit policy must not log bodies"
s = yaml.safe_load((root / "server/psa.yaml").read_text())
d = s["plugins"][0]["configuration"]["defaults"]
assert d["enforce"] == "baseline", "PSA default enforce"
assert s["plugins"][0]["configuration"]["exemptions"]["namespaces"] == ["kube-system"], "PSA exemptions"
print("bundle files parse and keep their invariants")
PY
}
plan() {
local entry src dst perm owner state
for entry in "${FILES[@]}"; do
IFS='|' read -r src dst perm owner <<<"$entry"
if [[ -d "$(dirname "$dst")" && ! -r "$(dirname "$dst")" ]] ||
[[ ! -e "$(dirname "$dst")" && ! -r "$(dirname "$(dirname "$dst")")" ]]; then
state="unknown (needs root to read)"
elif [[ ! -e "$dst" ]]; then
state="new"
elif cmp -s "$ROOT/$src" "$dst"; then
state="same"
else
state="replace"
fi
say " ${owner} ${state} ${dst}"
done
say " W6 touch $SKIP_FILE"
say " W6 mkdir $AUDIT_DIR (0700)"
say " W1 systemctl enable systemd-time-wait-sync.service"
say " -- systemctl daemon-reload && systemctl restart k3s (once)"
}
say "== check"
check_sources
say "== plan"
plan
[[ "$mode" == "--execute" ]] || { say "CHECK ONLY: nothing changed"; exit 0; }
[[ "$(id -u)" -eq 0 ]] || fail "--execute needs root (sudo)"
command -v sqlite3 >/dev/null || fail "sqlite3 is required for the pre-restart datastore copy"
kubectl() { command k3s kubectl --kubeconfig "$KCFG" "$@"; }
wait_ready() {
local i
for i in $(seq 1 60); do
if kubectl get --raw=/readyz >/dev/null 2>&1 &&
[[ "$(kubectl get nodes -o jsonpath='{.items[0].status.conditions[?(@.type=="Ready")].status}' 2>/dev/null)" == "True" ]]; then
return 0
fi
sleep 5
done
return 1
}
say "== preflight"
wait_ready || fail "the cluster is not Ready before the change; not touching it"
for url in "${HEALTH_URLS[@]}"; do
code="$(curl -s -o /dev/null -m 15 -w '%{http_code}' "$url" || true)"
say " before $code $url"
done
ts="$(date -u +%Y%m%dT%H%M%SZ)"
backup="$BACKUP_ROOT/$ts"
install -d -m 0700 "$backup"
time_wait_was_enabled="$(systemctl is-enabled systemd-time-wait-sync.service 2>/dev/null || true)"
say "== backup to $backup"
for entry in "${FILES[@]}"; do
IFS='|' read -r src dst perm owner <<<"$entry"
if [[ -e "$dst" ]]; then
install -D -m 0600 "$dst" "$backup/files$dst"
fi
done
sqlite3 "$SERVER/db/state.db" ".backup '$backup/state.db'"
install -m 0600 "$SERVER/token" "$backup/token"
say " datastore copy: $(sqlite3 "$backup/state.db" 'PRAGMA integrity_check;') $(du -h "$backup/state.db" | cut -f1)"
rollback() {
local entry src dst perm owner
trap - ERR
say "== ROLLBACK"
set +e
for entry in "${FILES[@]}"; do
IFS='|' read -r src dst perm owner <<<"$entry"
if [[ -e "$backup/files$dst" ]]; then
install -D -m "$perm" "$backup/files$dst" "$dst"
else
rm -f -- "$dst"
fi
done
rm -f -- "$SKIP_FILE"
[[ "$time_wait_was_enabled" == "enabled" ]] || systemctl disable systemd-time-wait-sync.service
systemctl daemon-reload
systemctl restart --no-block k3s
sleep 10
if wait_ready; then say "ROLLBACK done: previous files restored, k3s Ready"; else say "ROLLBACK: k3s still not Ready; see journalctl -u k3s"; fi
say "backup kept at $backup"
exit 1
}
trap rollback ERR
say "== install"
for entry in "${FILES[@]}"; do
IFS='|' read -r src dst perm owner <<<"$entry"
install -D -o root -g root -m "$perm" "$ROOT/$src" "$dst"
done
install -d -o root -g root -m 0700 "$AUDIT_DIR"
install -D -o root -g root -m 0600 /dev/null "$SKIP_FILE"
systemctl daemon-reload
systemctl enable systemd-time-wait-sync.service
say "== restart k3s"
# k3s.service has TimeoutStartSec=0, so a blocking restart could wait forever.
systemctl restart --no-block k3s
sleep 10
wait_ready || { say "k3s did not become Ready in 5 minutes"; false; }
say "== verify"
node="$(kubectl get nodes -o jsonpath='{.items[0].metadata.name}')"
kubectl get --raw "/api/v1/nodes/$node/proxy/configz" | python3 -I -c '
import json, sys
k = json.load(sys.stdin)["kubeletconfig"]
want = {
("systemReserved", "memory"): "2Gi", ("kubeReserved", "memory"): "1Gi",
("evictionHard", "memory.available"): "500Mi", ("evictionHard", "nodefs.available"): "5%",
("evictionHard", "imagefs.available"): "5%", ("evictionSoft", "memory.available"): "1Gi",
}
bad = [f"{a}.{b}={k.get(a, {}).get(b)}" for (a, b), v in want.items() if k.get(a, {}).get(b) != v]
if k.get("shutdownGracePeriod") != "1m0s": bad.append(f"shutdownGracePeriod={k.get(\"shutdownGracePeriod\")}")
if bad: sys.exit("kubelet config not applied: " + ", ".join(bad))
print(" kubelet: reservations, eviction and graceful shutdown applied")
'
for i in $(seq 1 12); do [[ -s "$AUDIT_DIR/audit.log" ]] && break; sleep 5; done
[[ -s "$AUDIT_DIR/audit.log" ]] || { say "audit log is empty"; false; }
say " audit log: $(wc -l <"$AUDIT_DIR/audit.log") lines"
psa_out="$(kubectl apply --dry-run=server -f - 2>&1 <<'YAML' || true
apiVersion: v1
kind: Pod
metadata:
name: psa-default-probe
namespace: default
spec:
containers:
- name: probe
image: registry.k8s.io/pause:3.10
securityContext:
privileged: true
YAML
)"
grep -q 'violates PodSecurity "baseline' <<<"$psa_out" || { say "PSA default did not reject a privileged pod: $psa_out"; false; }
say " pod security: default namespace rejects a privileged pod (baseline)"
for url in "${HEALTH_URLS[@]}"; do
ok=0
for i in $(seq 1 30); do
[[ "$(curl -s -o /dev/null -m 15 -w '%{http_code}' "$url" || true)" == "200" ]] && { ok=1; break; }
sleep 10
done
(( ok == 1 )) || { say "health check failed: $url"; false; }
say " after 200 $url"
done
say " time sync: $(systemctl is-enabled systemd-time-wait-sync.service), $(systemctl show systemd-time-wait-sync.service -p TimeoutStartUSec)"
trap - ERR
say "== local-path is no longer the default StorageClass"
kubectl annotate storageclass local-path storageclass.kubernetes.io/is-default-class=false --overwrite
kubectl get storageclass local-path -o jsonpath=' local-path default={.metadata.annotations.storageclass\.kubernetes\.io/is-default-class}{"\n"}'
say "K3S RESTART BUNDLE APPLIED (backup: $backup)"
+2 -2
View File
@@ -189,8 +189,8 @@ assert_regex_count \
assert_regex_count \
"${REPOSITORY_ROOT}/infrastructure/controllers/keycloak-operator/kustomization.yaml" \
'github\.com/keycloak/keycloak-k8s-resources/kubernetes\?ref=26\.7\.0' 1 \
"Keycloak Operator 26.7.0 source pin"
'github\.com/keycloak/keycloak-k8s-resources/kubernetes\?ref=38fd7a0c84e15d4b66b37d1c2ea728ca2f5416a4' 1 \
"Keycloak Operator 26.7.0 commit pin"
assert_regex_count \
"${render_temp_dir}/keycloak-operator.yaml" \
'^kind:[[:space:]]CustomResourceDefinition$' 4 \
@@ -0,0 +1,81 @@
#!/usr/bin/env bash
# Create the AIStor bucket, lifecycle rules, policy and user that the host backup
# timers use, and the root-only /etc/platform-backup/aistor.env.
#
# sudo bash scripts/bootstrap/create-host-backup-credentials.sh
#
# Run once, by the owner, on the desktop. Refuses to run when the env file exists;
# never rotates or deletes a bucket, policy, user or object. No key is printed.
set -Eeuo pipefail
set +x
umask 077
readonly ROOT="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/../.." && pwd -P)"
# shellcheck source=../lib/host-backup.sh
source "$ROOT/scripts/lib/host-backup.sh"
readonly BUCKET=platform-host-backups
readonly POLICY=platform-host-backups
readonly QUOTA=50GiB
readonly POLICY_FILE="$ROOT/services/platform-host-checks/bucket-policy.json"
readonly PASS_LINE=PLATFORM_HOST_BACKUP_BOOTSTRAP_PASS
kubectl() { command /usr/local/bin/k3s kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml "$@"; }
[[ "$(id -u)" -eq 0 ]] || hb_fail "run with sudo"
for c in docker jq openssl; do command -v "$c" >/dev/null || hb_fail "missing command: $c"; done
jq -e . "$POLICY_FILE" >/dev/null || hb_fail "bucket policy is not valid JSON"
[[ ! -e "$HOST_BACKUP_ENV" ]] || hb_fail "$HOST_BACKUP_ENV already exists; this script does not rotate it"
ip="$(kubectl -n object-storage get service minio -o jsonpath='{.spec.clusterIP}')"
[[ -n "$ip" ]] || hb_fail "object-storage/minio Service not found"
endpoint="http://${ip}:80"
work="$(mktemp -d)"
trap 'rm -rf -- "$work"' EXIT
kubectl -n object-storage get secret aistor-root-configuration -o jsonpath='{.data.config\.env}' | base64 -d >"$work/root.env"
grep -q '^export MINIO_ROOT_USER=' "$work/root.env" || hb_fail "aistor-root-configuration has an unexpected format"
printf 'pcbackup%s' "$(openssl rand -hex 8)" >"$work/ACCESS_KEY"
openssl rand -hex 32 | tr -d '\n' >"$work/SECRET_KEY"
cp "$POLICY_FILE" "$work/policy.json"
printf '%s\n' \
"Bucket: $BUCKET (quota $QUOTA; expire gitea/ after 56 days, k3s-datastore/ after 30 days) on $endpoint" \
"Policy: $POLICY (${POLICY_FILE#"$ROOT/"})" \
"User: a new generated access key attached to $POLICY" \
"File: $HOST_BACKUP_ENV (root, 0600)" \
'No key value will be printed.'
printf 'Type APPLY to create them: '
IFS= read -r confirmation
[[ "$confirmation" == "APPLY" ]] || hb_fail "confirmation did not match APPLY"
# Every mc call is silenced so no key reaches the terminal.
docker run --rm --network host -v "$work:/run/b:ro" --tmpfs /tmp -e MC_CONFIG_DIR=/tmp/mc \
--entrypoint /bin/sh "$MC_IMAGE" -ec '
. /run/b/root.env
ak="$(cat /run/b/ACCESS_KEY)"; sk="$(cat /run/b/SECRET_KEY)"
mc alias set root "'"$endpoint"'" "$MINIO_ROOT_USER" "$MINIO_ROOT_PASSWORD" --api S3v4 >/dev/null
mc mb --ignore-existing "root/'"$BUCKET"'" >/dev/null
mc quota set "root/'"$BUCKET"'" --size "'"$QUOTA"'" >/dev/null
mc ilm rule add --prefix gitea/ --expire-days 56 "root/'"$BUCKET"'" >/dev/null
mc ilm rule add --prefix k3s-datastore/ --expire-days 30 "root/'"$BUCKET"'" >/dev/null
mc admin policy create root "'"$POLICY"'" /run/b/policy.json >/dev/null
mc admin user add root "$ak" "$sk" >/dev/null
mc admin policy attach root "'"$POLICY"'" --user "$ak" >/dev/null
mc alias set backup "'"$endpoint"'" "$ak" "$sk" --api S3v4 >/dev/null
probe=".bootstrap-probe-$(date +%s)"
printf probe | mc pipe "backup/'"$BUCKET"'/$probe" >/dev/null
[ "$(mc cat "backup/'"$BUCKET"'/$probe")" = probe ]
mc rm "backup/'"$BUCKET"'/$probe" >/dev/null
if mc ls backup/loki >/dev/null 2>&1; then echo "the backup user can list another bucket" >&2; exit 1; fi
echo "'"$PASS_LINE"'"' | grep -qx "$PASS_LINE" || hb_fail "bootstrap did not finish; $HOST_BACKUP_ENV was not written"
install -d -m 0700 -o root -g root "$(dirname "$HOST_BACKUP_ENV")"
{
echo "AISTOR_BUCKET=$BUCKET"
echo "AISTOR_ACCESS_KEY=$(cat "$work/ACCESS_KEY")"
echo "AISTOR_SECRET_KEY=$(cat "$work/SECRET_KEY")"
echo "# Off-host encrypted copy of the k3s datastore archive (off until both are set):"
echo "# OFFHOST_GPG_RECIPIENT_FILE=/etc/platform-backup/offhost-recipient.asc"
echo "# OFFHOST_RCLONE_TARGET=remote:path"
} >"$HOST_BACKUP_ENV"
chmod 0600 "$HOST_BACKUP_ENV"
printf '%s\n' "$PASS_LINE" "Created bucket $BUCKET, policy $POLICY, one user and $HOST_BACKUP_ENV."
+170
View File
@@ -0,0 +1,170 @@
#!/usr/bin/env bash
# Create the AIStor bucket, policy and user that CloudNativePG uses to back up
# platform-postgres, and the platform-data/platform-postgres-backup-s3 Secret.
#
# Run once, by the owner, from a platform-core checkout on the desktop. The
# script refuses to run when the Secret already exists; it never rotates or
# deletes a bucket, policy, user or object. No credential value is printed.
set -Eeuo pipefail
set +x
umask 077
readonly ROOT="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/../.." && pwd -P)"
readonly EXPECTED_CONTEXT=default
readonly OS_NAMESPACE=object-storage
readonly PG_NAMESPACE=platform-data
readonly ROOT_SECRET=aistor-root-configuration
readonly TARGET_SECRET=platform-postgres-backup-s3
readonly BUCKET=platform-postgres-backups
readonly POLICY=platform-postgres-backups
readonly QUOTA=100GiB
readonly S3_ENDPOINT=http://minio.object-storage.svc.cluster.local
# Same pinned AIStor client as scripts/validate/aistor-s3-smoke.sh.
readonly MC_IMAGE="quay.io/minio/aistor/mc@sha256:c5ec777c080fd6292b7529309a0f311c58c3a92e268c9c57c0901da86881949b"
readonly POLICY_FILE="${ROOT}/services/platform-postgres-backup/bucket-policy.json"
readonly PASS_LINE=PLATFORM_POSTGRES_BACKUP_BOOTSTRAP_PASS
fail() {
printf 'FAIL: %s\n' "$*" >&2
exit 1
}
for command in kubectl jq openssl; do
command -v "$command" >/dev/null || fail "missing command: ${command}"
done
[[ -f "$POLICY_FILE" ]] || fail "missing ${POLICY_FILE}"
jq -e . "$POLICY_FILE" >/dev/null || fail "bucket policy is not valid JSON"
[[ "$(kubectl config current-context)" == "$EXPECTED_CONTEXT" ]] ||
fail "kube context must be ${EXPECTED_CONTEXT}"
if kubectl --namespace "$PG_NAMESPACE" get secret "$TARGET_SECRET" >/dev/null 2>&1; then
fail "${PG_NAMESPACE}/${TARGET_SECRET} already exists; this script does not rotate it"
fi
kubectl --namespace "$OS_NAMESPACE" get secret "$ROOT_SECRET" >/dev/null ||
fail "${OS_NAMESPACE}/${ROOT_SECRET} is missing"
work="$(mktemp -d)"
pod="pg-backup-bootstrap-$(date +%H%M%S)-$$"
handoff_secret="${pod}-keys"
policy_map="${pod}-policy"
cleanup() {
kubectl --namespace "$OS_NAMESPACE" delete pod "$pod" \
--ignore-not-found --wait=false >/dev/null 2>&1 || true
kubectl --namespace "$OS_NAMESPACE" delete secret "$handoff_secret" \
--ignore-not-found >/dev/null 2>&1 || true
kubectl --namespace "$OS_NAMESPACE" delete configmap "$policy_map" \
--ignore-not-found >/dev/null 2>&1 || true
rm -rf -- "$work"
}
trap cleanup EXIT INT TERM
printf 'pgbackup%s' "$(openssl rand -hex 8)" >"${work}/ACCESS_KEY_ID"
openssl rand -hex 32 | tr -d '\n' >"${work}/ACCESS_SECRET_KEY"
printf '%s\n' \
"Bucket: ${BUCKET} (quota ${QUOTA}) on AIStor ${S3_ENDPOINT}" \
"Policy: ${POLICY} (${POLICY_FILE#"${ROOT}/"})" \
"User: a new generated access key attached to ${POLICY}" \
"Secret: ${PG_NAMESPACE}/${TARGET_SECRET} (ACCESS_KEY_ID, ACCESS_SECRET_KEY)" \
"Temporary: one restricted client Pod, one Secret and one ConfigMap in ${OS_NAMESPACE}" \
'No credential value will be printed.'
printf 'Type APPLY to create them: '
IFS= read -r confirmation
[[ "$confirmation" == "APPLY" ]] || fail "confirmation did not match APPLY"
kubectl --namespace "$OS_NAMESPACE" create secret generic "$handoff_secret" \
--from-file=ACCESS_KEY_ID="${work}/ACCESS_KEY_ID" \
--from-file=ACCESS_SECRET_KEY="${work}/ACCESS_SECRET_KEY" >/dev/null
kubectl --namespace "$OS_NAMESPACE" create configmap "$policy_map" \
--from-file=policy.json="$POLICY_FILE" >/dev/null
# Every mc call is silenced so neither key reaches the Pod log.
bootstrap_command='set -eu
. /run/aistor-root/config.env
access_key="$(cat /run/backup/ACCESS_KEY_ID)"
secret_key="$(cat /run/backup/ACCESS_SECRET_KEY)"
mc alias set root "'"$S3_ENDPOINT"'" "$MINIO_ROOT_USER" "$MINIO_ROOT_PASSWORD" --api S3v4 >/dev/null
mc mb --ignore-existing "root/'"$BUCKET"'" >/dev/null
mc quota set "root/'"$BUCKET"'" --size "'"$QUOTA"'" >/dev/null
mc admin policy create root "'"$POLICY"'" /run/policy/policy.json >/dev/null
mc admin user add root "$access_key" "$secret_key" >/dev/null
mc admin policy attach root "'"$POLICY"'" --user "$access_key" >/dev/null
mc alias set backup "'"$S3_ENDPOINT"'" "$access_key" "$secret_key" --api S3v4 >/dev/null
probe=".bootstrap-probe-$(date +%s)"
printf "probe" | mc pipe "backup/'"$BUCKET"'/$probe" >/dev/null
[ "$(mc cat "backup/'"$BUCKET"'/$probe")" = "probe" ]
mc rm "backup/'"$BUCKET"'/$probe" >/dev/null
if mc ls backup/loki >/dev/null 2>&1; then
echo "the backup user can list another bucket" >&2
exit 1
fi
printf "%s\n" "'"$PASS_LINE"'"'
overrides="$(
jq -nc --arg command "$bootstrap_command" --arg image "$MC_IMAGE" \
--arg root "$ROOT_SECRET" --arg keys "$handoff_secret" --arg policy "$policy_map" '{
spec: {
automountServiceAccountToken: false,
securityContext: {
runAsNonRoot: true,
runAsUser: 1000,
runAsGroup: 1000,
fsGroup: 1000,
seccompProfile: {type: "RuntimeDefault"}
},
containers: [{
name: "pg-backup-bootstrap",
image: $image,
imagePullPolicy: "IfNotPresent",
command: ["/bin/sh", "-ec"],
args: [$command],
securityContext: {
allowPrivilegeEscalation: false,
capabilities: {drop: ["ALL"]},
readOnlyRootFilesystem: true
},
env: [{name: "MC_CONFIG_DIR", value: "/tmp/mc"}],
volumeMounts: [
{name: "root-config", mountPath: "/run/aistor-root", readOnly: true},
{name: "backup-keys", mountPath: "/run/backup", readOnly: true},
{name: "policy", mountPath: "/run/policy", readOnly: true},
{name: "tmp", mountPath: "/tmp"}
]
}],
volumes: [
{name: "root-config", secret: {secretName: $root, defaultMode: 256}},
{name: "backup-keys", secret: {secretName: $keys, defaultMode: 256}},
{name: "policy", configMap: {name: $policy}},
{name: "tmp", emptyDir: {}}
]
}
}'
)"
kubectl --namespace "$OS_NAMESPACE" run "$pod" \
--image="$MC_IMAGE" \
--restart=Never \
--labels='platform.hyeonworks.com/aistor-client=true,app.kubernetes.io/name=pg-backup-bootstrap' \
--overrides="$overrides" >/dev/null
phase=''
for _ in $(seq 1 90); do
phase="$(kubectl --namespace "$OS_NAMESPACE" get pod "$pod" --output=jsonpath='{.status.phase}')"
[[ "$phase" == "Succeeded" || "$phase" == "Failed" ]] && break
sleep 2
done
logs="$(kubectl --namespace "$OS_NAMESPACE" logs "$pod" 2>&1 || true)"
if [[ "$phase" != "Succeeded" || "$logs" != *"$PASS_LINE"* ]]; then
printf '%s\n' "$logs" | tail -20 >&2
fail "bootstrap Pod ended in phase '${phase}'; the platform-data Secret was not created"
fi
kubectl --namespace "$PG_NAMESPACE" create secret generic "$TARGET_SECRET" \
--from-file=ACCESS_KEY_ID="${work}/ACCESS_KEY_ID" \
--from-file=ACCESS_SECRET_KEY="${work}/ACCESS_SECRET_KEY" >/dev/null
kubectl --namespace "$PG_NAMESPACE" label secret "$TARGET_SECRET" \
app.kubernetes.io/part-of=platform app.kubernetes.io/component=platform-postgres-backup >/dev/null
printf '%s\n' "$PASS_LINE" \
"Created bucket ${BUCKET}, policy ${POLICY}, one user and ${PG_NAMESPACE}/${TARGET_SECRET}."
+78
View File
@@ -0,0 +1,78 @@
#!/usr/bin/env bash
# Install the host backup and check timers (infrastructure/host/backup).
#
# bash scripts/bootstrap/install-host-checks.sh --check # no root, changes nothing
# sudo bash scripts/bootstrap/install-host-checks.sh --execute # install, enable, first runs
#
# Scripts are copied to /usr/local/lib/platform-backup (root-owned) so the root
# timers never execute files the desktop user can edit.
set -Eeuo pipefail
umask 022
readonly ROOT="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/../.." && pwd -P)"
readonly LIB=/usr/local/lib/platform-backup
readonly UNIT_DIR=/etc/systemd/system
readonly TEXTFILE_DIR=/var/lib/node_exporter/textfile_collector
readonly WORK_ROOT=/var/lib/platform-backup/work
readonly USER_NAME=donghyeon
readonly -a SCRIPTS=(
scripts/lib/host-backup.sh:lib/host-backup.sh
scripts/backup/k3s-datastore-backup.sh:backup/k3s-datastore-backup.sh
scripts/backup/gitea-backup.sh:backup/gitea-backup.sh
scripts/validate/live-drift-check-main.sh:validate/live-drift-check-main.sh
scripts/validate/k3s-cert-expiry.sh:validate/k3s-cert-expiry.sh
services/platform-host-checks/README.md:README.md
)
readonly -a TIMERS=(platform-k3s-datastore-backup platform-gitea-backup platform-live-drift-check platform-k3s-cert-expiry)
fail() { printf 'ERROR: %s\n' "$*" >&2; exit 1; }
mode="${1:-}"
[[ "$#" -eq 1 && ( "$mode" == "--check" || "$mode" == "--execute" ) ]] || { sed -n '2,8p' "$0" >&2; exit 2; }
echo "== check"
for pair in "${SCRIPTS[@]}"; do
src="${pair%%:*}"
[[ -f "$ROOT/$src" ]] || fail "missing $src"
[[ "$src" != *.sh ]] || bash -n "$ROOT/$src" || fail "syntax error in $src"
done
for t in "${TIMERS[@]}"; do
for u in service timer; do [[ -f "$ROOT/infrastructure/host/backup/$t.$u" ]] || fail "missing $t.$u"; done
done
if command -v systemd-analyze >/dev/null; then
verify_dir="$(mktemp -d)"
trap 'rm -rf -- "$verify_dir"' EXIT
cp "$ROOT"/infrastructure/host/backup/*.service "$ROOT"/infrastructure/host/backup/*.timer "$verify_dir/"
# ExecStart paths do not exist before installation; only syntax and keys are checked.
out="$(cd "$verify_dir" && systemd-analyze verify ./*.timer ./*.service 2>&1 | grep -v -E 'is not executable|No such file|Command .* is not executable' || true)"
[[ -z "$out" ]] || { printf '%s\n' "$out" >&2; fail "systemd-analyze verify reported problems"; }
echo " units: systemd-analyze verify clean"
fi
echo "== plan"
for pair in "${SCRIPTS[@]}"; do echo " install ${pair%%:*} -> $LIB/${pair##*:}"; done
for t in "${TIMERS[@]}"; do echo " install $UNIT_DIR/$t.service, $UNIT_DIR/$t.timer; enable --now $t.timer"; done
echo " mkdir $TEXTFILE_DIR (root:$USER_NAME 0775), $WORK_ROOT (root 0700)"
echo " first runs: k3s-cert-expiry, k3s-datastore-backup, gitea-backup, live-drift-check"
[[ "$mode" == "--execute" ]] || { echo "CHECK ONLY: nothing changed"; exit 0; }
[[ "$(id -u)" -eq 0 ]] || fail "--execute needs root (sudo)"
[[ -f /etc/platform-backup/aistor.env ]] || fail "run scripts/bootstrap/create-host-backup-credentials.sh first"
echo "== install"
for pair in "${SCRIPTS[@]}"; do
install -D -o root -g root -m 0755 "$ROOT/${pair%%:*}" "$LIB/${pair##*:}"
done
install -d -o root -g "$USER_NAME" -m 0775 "$TEXTFILE_DIR"
install -d -o root -g root -m 0700 "$WORK_ROOT"
for t in "${TIMERS[@]}"; do
install -o root -g root -m 0644 "$ROOT/infrastructure/host/backup/$t.service" "$ROOT/infrastructure/host/backup/$t.timer" "$UNIT_DIR/"
done
systemctl daemon-reload
for t in "${TIMERS[@]}"; do systemctl enable --now "$t.timer"; done
echo "== first runs"
for t in platform-k3s-cert-expiry platform-k3s-datastore-backup platform-gitea-backup platform-live-drift-check; do
if systemctl start "$t.service"; then echo " $t: ok"; else echo " $t: FAILED (journalctl -u $t)"; fi
done
echo "== metrics"
grep -h -E '^platform_' "$TEXTFILE_DIR"/*.prom || echo " no metrics yet"
systemctl list-timers 'platform-*' --no-pager
echo "HOST CHECKS INSTALLED"
+55
View File
@@ -0,0 +1,55 @@
#!/usr/bin/env bash
# Shared helpers for the host backup and host check scripts. Source only.
# shellcheck shell=bash
readonly HOST_BACKUP_ENV=/etc/platform-backup/aistor.env
readonly TEXTFILE_DIR=/var/lib/node_exporter/textfile_collector
# Backup staging (root 0700, same SSD). Not /var/tmp: dockerd must see the path.
readonly HB_WORK_ROOT=/var/lib/platform-backup/work
# Same pinned AIStor client as scripts/validate/aistor-s3-smoke.sh.
readonly MC_IMAGE="quay.io/minio/aistor/mc@sha256:c5ec777c080fd6292b7529309a0f311c58c3a92e268c9c57c0901da86881949b"
hb_log() { printf '%s %s\n' "$(date -u +%FT%TZ)" "$*"; }
hb_fail() { hb_log "FAIL: $*" >&2; exit 1; }
# Write Prometheus textfile metrics atomically: hb_write_metrics NAME <<'PROM' ... PROM
hb_write_metrics() {
local name="$1" tmp
[[ -d "$TEXTFILE_DIR" ]] || { hb_log "textfile dir $TEXTFILE_DIR missing; metrics not written"; cat >/dev/null; return 0; }
tmp="$(mktemp "$TEXTFILE_DIR/.${name}.XXXXXX")"
cat >"$tmp"
chmod 0644 "$tmp"
mv -f -- "$tmp" "$TEXTFILE_DIR/${name}.prom"
}
# Load the AIStor endpoint and the backup user's keys (root-only file).
hb_load_aistor_env() {
[[ -f "$HOST_BACKUP_ENV" ]] || hb_fail "$HOST_BACKUP_ENV is missing; run scripts/bootstrap/create-host-backup-credentials.sh"
[[ "$(stat -c '%u %a' "$HOST_BACKUP_ENV")" == "0 600" ]] || hb_fail "$HOST_BACKUP_ENV must be owned by root with mode 0600"
# shellcheck disable=SC1090
. "$HOST_BACKUP_ENV"
: "${AISTOR_BUCKET:?}" "${AISTOR_ACCESS_KEY:?}" "${AISTOR_SECRET_KEY:?}"
# The host cannot resolve cluster DNS, so use the Service ClusterIP (port 80 -> 9000).
if [[ -z "${AISTOR_ENDPOINT:-}" ]]; then
local ip
ip="$(/usr/local/bin/k3s kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml -n object-storage \
get service minio -o jsonpath='{.spec.clusterIP}')"
[[ -n "$ip" ]] || hb_fail "cannot find the object-storage/minio Service"
AISTOR_ENDPOINT="http://${ip}:80"
fi
}
# Run the pinned mc client on the host network with the backup user's keys.
# The keys go through a 0600 env file, never the command line (ps would show it).
# Usage: hb_mc <dir mounted at /work> <mc args...>
hb_mc() {
local work="$1"; shift
local envfile="$work/.mc.env"
if [[ ! -f "$envfile" ]]; then
( umask 077; printf 'MC_HOST_backup=%s\nMC_CONFIG_DIR=/tmp/mc\n' \
"${AISTOR_ENDPOINT/:\/\//://${AISTOR_ACCESS_KEY}:${AISTOR_SECRET_KEY}@}" >"$envfile" )
fi
docker run --rm --network host --user "$(id -u):$(id -g)" \
--env-file "$envfile" -v "$work:/work" --tmpfs /tmp \
"$MC_IMAGE" "$@"
}
+10
View File
@@ -596,3 +596,13 @@ PLATFORM_HELM_BIN=/home/donghyeon/.local/bin/helm \
- `--textfile FILE`: `platform_live_drift_objects`, `platform_live_drift_check_success`,
`platform_live_drift_last_run_timestamp_seconds`를 Prometheus textfile 형식으로 쓴다.
- Secret 값의 차이는 보지 않는다.
## 호스트 백업·점검 시험
- `test-platform-host-checks-alerts.sh`: `services/platform-host-checks/prometheus-rules.yaml`의 promtool
단위 시험(백업 신선도, 실패, 지표 없음, drift, k3s 인증서). docker 필요, 클러스터 접근 없음.
- `k3s-cert-expiry.sh`: 6443·10250 serving 인증서 만료를 textfile 지표로 쓴다(타이머가 매일 돌림).
- `live-drift-check-main.sh`: checkout의 작업 트리가 아니라 `origin/main`으로 `live-drift-check.sh`를 돌린다.
- `k3s-datastore-restore-drill.sh`, `gitea-backup-restore-drill.sh`: 복원 리허설. 인자 없이 root로 돌리면 AIStor의
최신 백업을 쓰고, `--from-dir`·`--from-file`로 로컬 사본을 줄 수 있다.
- `test-k3s-datastore-restore-drill.sh`: 운영 데이터 없이 k3s 복원 리허설 경로를 시험한다(일회용 k3s, 시험 Secret 셋).
@@ -0,0 +1,75 @@
#!/usr/bin/env bash
# Restore drill for the Gitea backup (file level).
#
# sudo bash scripts/validate/gitea-backup-restore-drill.sh # latest dump in AIStor
# bash scripts/validate/gitea-backup-restore-drill.sh --from-file DUMP.tar.gz
#
# Extracts the dump into a throwaway directory and checks what a restore needs:
# app.ini is present, every repository passes `git fsck`, and the repository
# count matches the live Gitea. The database is restored from the CloudNativePG
# backup instead (services/platform-postgres-backup); the full restore order is
# in docs/runbooks/gitea-restore.md. Nothing is written to the cluster.
set -Eeuo pipefail
umask 077
readonly ROOT="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/../.." && pwd -P)"
# shellcheck source=../lib/host-backup.sh
source "$ROOT/scripts/lib/host-backup.sh"
readonly NAME=gitea_restore_drill
from_file=""
case "${1:-}" in
--from-file) from_file="${2:?--from-file needs a path}" ;;
"") ;;
*) sed -n '2,11p' "$0" >&2; exit 2 ;;
esac
if [[ "$(id -u)" -eq 0 ]]; then
kubectl() { command /usr/local/bin/k3s kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml "$@"; }
fi
base="$HB_WORK_ROOT"; [[ -w "$base" ]] || base="$(mktemp -d)"
work="$(mktemp -d "$base/gitea-drill.XXXXXX")"
trap 'rm -rf -- "$work"' EXIT
if [[ -z "$from_file" ]]; then
[[ "$(id -u)" -eq 0 ]] || hb_fail "run as root to read the backup credentials"
hb_load_aistor_env
latest="$(hb_mc "$work" ls "backup/$AISTOR_BUCKET/gitea/" | awk '{print $NF}' | grep -E '^gitea-dump-.*\.tar\.gz$' | sort | tail -1)"
[[ -n "$latest" ]] || hb_fail "no Gitea dump found"
hb_mc "$work" cp --quiet "backup/$AISTOR_BUCKET/gitea/$latest" "backup/$AISTOR_BUCKET/gitea/$latest.sha256" /work/ >/dev/null
[[ "$(sha256sum "$work/$latest" | awk '{print $1}')" == "$(cat "$work/$latest.sha256")" ]] || hb_fail "checksum mismatch for $latest"
from_file="$work/$latest"
hb_log "drill on $latest"
fi
mkdir "$work/restore"
tar -C "$work/restore" -xzf "$from_file"
[[ -s "$work/restore/app.ini" ]] || hb_fail "app.ini is missing from the dump"
for key in INTERNAL_TOKEN SECRET_KEY; do
grep -q "^${key}[[:space:]]*=" "$work/restore/app.ini" || hb_fail "app.ini has no ${key} (needed to restore)"
done
total=0 broken=0
while IFS= read -r -d '' repo; do
total=$((total + 1))
if ! git -c safe.directory='*' --git-dir="$repo" fsck --no-dangling --no-progress >/dev/null 2>"$work/fsck.err"; then
broken=$((broken + 1))
hb_log "fsck failed: ${repo#"$work/restore/"}: $(head -1 "$work/fsck.err")"
fi
done < <(find "$work/restore/repos" -mindepth 2 -maxdepth 2 -type d -name '*.git' -print0)
hb_log "repositories in dump: $total, fsck failures: $broken"
(( total > 0 )) || hb_fail "the dump has no repositories"
(( broken == 0 )) || hb_fail "$broken repositories failed git fsck"
live="$(kubectl -n gitea exec deploy/gitea -c gitea -- sh -c 'find /data/git/gitea-repositories -mindepth 2 -maxdepth 2 -name "*.git" -type d | wc -l' 2>/dev/null || echo "")"
if [[ -n "$live" ]]; then
hb_log "live Gitea repositories: $live"
(( total >= live - 2 )) || hb_fail "the dump has $total repositories, live Gitea has $live"
fi
hb_write_metrics "$NAME" <<PROM
# HELP platform_backup_restore_drill_last_success_timestamp_seconds Unix time of the last successful restore drill.
# TYPE platform_backup_restore_drill_last_success_timestamp_seconds gauge
platform_backup_restore_drill_last_success_timestamp_seconds{backup="gitea"} $(date +%s)
PROM
echo "GITEA RESTORE DRILL PASS"
+25
View File
@@ -0,0 +1,25 @@
#!/usr/bin/env bash
# Write the notAfter time of the k3s API server (6443) and kubelet (10250)
# serving certificates as Prometheus textfile metrics. No root needed.
# k3s renews these only when it starts and they are within 120 days of expiry
# (docs.k3s.io cli/certificate), and the kubelet exposes no certificate metric,
# so nothing else in the cluster watches them.
set -Eeuo pipefail
# shellcheck source=../lib/host-backup.sh
source "$(dirname -- "${BASH_SOURCE[0]}")/../lib/host-backup.sh"
lines=""
for target in apiserver:6443 kubelet:10250; do
name="${target%%:*}" port="${target##*:}"
end="$(echo | timeout 10 openssl s_client -connect "127.0.0.1:$port" 2>/dev/null | openssl x509 -noout -enddate 2>/dev/null | sed 's/^notAfter=//')" || true
[[ -n "$end" ]] || hb_fail "could not read the $name certificate on 127.0.0.1:$port"
lines+="platform_k3s_certificate_not_after_seconds{certificate=\"$name\"} $(date -d "$end" +%s)"$'\n'
hb_log "$name certificate notAfter $end"
done
hb_write_metrics platform_k3s_certificates <<PROM
# HELP platform_k3s_certificate_not_after_seconds notAfter of a k3s serving certificate (Unix time).
# TYPE platform_k3s_certificate_not_after_seconds gauge
${lines}# HELP platform_k3s_certificate_check_timestamp_seconds Unix time of the last certificate check.
# TYPE platform_k3s_certificate_check_timestamp_seconds gauge
platform_k3s_certificate_check_timestamp_seconds $(date +%s)
PROM
@@ -0,0 +1,99 @@
#!/usr/bin/env bash
# Restore drill for the k3s datastore backup.
#
# sudo bash scripts/validate/k3s-datastore-restore-drill.sh # latest backup in AIStor
# bash scripts/validate/k3s-datastore-restore-drill.sh --from-dir DIR [--expect-secrets N]
# DIR holds state.db and token (an extracted archive, or a test datastore)
#
# Starts a throwaway k3s server (same version as production, --network none,
# no agent) on a copy of the backup, and checks that the API answers, that the
# Secret count matches and that every Secret decrypts (the token unlocks the
# encryption config stored in the datastore). Prints no Secret value. The
# container and the copy are removed at the end.
set -Eeuo pipefail
umask 077
readonly ROOT="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/../.." && pwd -P)"
# shellcheck source=../lib/host-backup.sh
source "$ROOT/scripts/lib/host-backup.sh"
# Same version as the desktop server (k3s --version: v1.36.2+k3s1).
readonly K3S_IMAGE="docker.io/rancher/k3s:v1.36.2-k3s1@sha256:6a47cea22c4b834d4ba72c89d291696b79ebe406251f90b446e4dff03513dd87"
readonly NAME=k3s_datastore_restore_drill
from_dir="" expect_secrets=""
while (( $# > 0 )); do
case "$1" in
--from-dir) from_dir="${2:?}"; shift 2 ;;
--expect-secrets) expect_secrets="${2:?}"; shift 2 ;;
*) sed -n '2,13p' "$0" >&2; exit 2 ;;
esac
done
base="${HB_WORK_ROOT}"
[[ -w "$base" ]] || base="$(mktemp -d)"
work="$(mktemp -d "$base/k3s-drill.XXXXXX")"
container="k3s-restore-drill-$$"
cleanup() {
docker rm -f "$container" >/dev/null 2>&1 || true
# The drill container writes as root; remove its files with the same image.
docker run --rm -v "$work:/w" --entrypoint /bin/sh "$K3S_IMAGE" -c 'rm -rf /w/*' >/dev/null 2>&1 || true
rm -rf -- "$work"
}
trap cleanup EXIT
if [[ -z "$from_dir" ]]; then
[[ "$(id -u)" -eq 0 ]] || hb_fail "run as root to read the backup credentials"
hb_load_aistor_env
latest="$(hb_mc "$work" ls "backup/$AISTOR_BUCKET/k3s-datastore/" | awk '{print $NF}' | grep -E '^k3s-datastore-.*\.tar\.gz$' | sort | tail -1)"
[[ -n "$latest" ]] || hb_fail "no k3s datastore backup found"
hb_mc "$work" cp --quiet "backup/$AISTOR_BUCKET/k3s-datastore/$latest" "backup/$AISTOR_BUCKET/k3s-datastore/$latest.sha256" /work/ >/dev/null
[[ "$(sha256sum "$work/$latest" | awk '{print $1}')" == "$(cat "$work/$latest.sha256")" ]] || hb_fail "checksum mismatch for $latest"
tar -C "$work" -xzf "$work/$latest"
from_dir="$(find "$work" -mindepth 1 -maxdepth 1 -type d -name 'k3s-datastore-*' | head -1)"
hb_log "drill on $latest"
if [[ -z "$expect_secrets" ]]; then
expect_secrets="$(/usr/local/bin/k3s kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml get secrets -A --no-headers | wc -l)"
hb_log "live cluster has $expect_secrets Secrets (counts may differ by objects created since the backup)"
fi
fi
[[ -f "$from_dir/state.db" && -f "$from_dir/token" ]] || hb_fail "$from_dir must contain state.db and token"
install -d -m 0700 "$work/server/db"
cp "$from_dir/state.db" "$work/server/db/state.db"
cp "$from_dir/token" "$work/server/token"
[[ "$(sqlite3 "$work/server/db/state.db" 'PRAGMA integrity_check;')" == "ok" ]] || hb_fail "integrity_check failed"
started="$(date +%s)"
docker run -d --name "$container" --network none --tmpfs /run --tmpfs /var/run \
-v "$work/server:/var/lib/rancher/k3s/server" \
"$K3S_IMAGE" server --disable-agent --secrets-encryption \
--token-file /var/lib/rancher/k3s/server/token \
--bind-address 127.0.0.1 --advertise-address 127.0.0.1 --node-ip 127.0.0.1 \
--disable traefik,servicelb,metrics-server,local-storage,coredns >/dev/null
ready=0
for _ in $(seq 1 60); do
if docker exec "$container" kubectl get --raw /readyz >/dev/null 2>&1; then ready=1; break; fi
sleep 5
done
if (( ready == 0 )); then
docker logs --tail 20 "$container" 2>&1 | sed 's/^/ k3s: /' >&2
hb_fail "restored k3s did not become ready in 5 minutes"
fi
namespaces="$(docker exec "$container" kubectl get namespaces --no-headers | wc -l)"
secrets="$(docker exec "$container" kubectl get secrets -A --no-headers | wc -l)"
# -o json decrypts every Secret on the server side; any failure makes kubectl exit non-zero.
decrypted="$(docker exec "$container" kubectl get secrets -A -o json | python3 -I -c 'import json,sys; d=json.load(sys.stdin)["items"]; print(sum(1 for s in d if s.get("data")))')"
hb_log "restored: namespaces=$namespaces secrets=$secrets with-data=$decrypted in $(( $(date +%s) - started ))s"
if [[ -n "$expect_secrets" ]]; then
diff=$(( secrets - expect_secrets ))
(( ${diff#-} <= 5 )) || hb_fail "Secret count $secrets is far from the expected $expect_secrets"
fi
(( decrypted > 0 )) || hb_fail "no Secret carried data after restore"
hb_write_metrics "$NAME" <<PROM
# HELP platform_backup_restore_drill_last_success_timestamp_seconds Unix time of the last successful restore drill.
# TYPE platform_backup_restore_drill_last_success_timestamp_seconds gauge
platform_backup_restore_drill_last_success_timestamp_seconds{backup="k3s-datastore"} $(date +%s)
PROM
echo "K3S DATASTORE RESTORE DRILL PASS"
+6 -1
View File
@@ -3,8 +3,13 @@
Cluster platform-data platform-postgres 2026-11-30 pg_hba의 techlog 두 줄이 repo에 없다. w1이 CNPG 백업과 함께 넣는다(1c4ef9a).
DatabaseRole platform-data platform-postgres-gitea 2026-11-30 렌더가 false 기본값 필드를 명시할 뿐 값은 같다. 다음 Phase 1 적용 때 사라진다.
DatabaseRole platform-data platform-postgres-keycloak 2026-11-30 렌더가 false 기본값 필드를 명시할 뿐 값은 같다. 다음 Phase 2 적용 때 사라진다.
Deployment keycloak keycloak-operator 2026-11-30 원격 ref 26.7.0 태그 내용이 바뀌어 build-timestamp 라벨만 다르다. 재시작 창에 커밋 SHA 고정과 함께 적용한다.
Deployment keycloak keycloak-operator 2026-11-30 커밋 SHA·이미지 digest·IfNotPresent 고정(재시작). 재시작 창에 apply-keycloak.sh로 적용한다.
Service keycloak keycloak-operator 2026-11-30 위와 같은 build-timestamp 라벨.
ServiceAccount keycloak keycloak-operator 2026-11-30 위와 같은 build-timestamp 라벨.
Deployment observability observability-core-kube-pr-operator 2026-11-30 이미지 문자열이 tag@digest와 digest로 다르고 digest는 같다. 적용하면 재시작되므로 재시작 창에.
Deployment observability observability-core-kube-state-metrics 2026-11-30 위와 같은 이미지 문자열 차이.
Keycloak keycloak keycloak 2026-11-30 이미지 digest·startOptimized false·platform-critical(Keycloak 재시작). 재시작 창에 PriorityClass를 먼저 만든 뒤 적용한다.
Deployment gitea gitea 2026-11-30 priorityClassName platform-critical(Gitea 재시작). 재시작 창에 apply-gitea-oidc.sh로 적용한다.
StatefulSet observability loki 2026-11-30 priorityClassName platform-low(Loki 재시작). 재시작 창에 관측 core 적용으로.
StatefulSet observability tempo 2026-11-30 priorityClassName platform-low(Tempo 재시작). 재시작 창에 관측 core 적용으로.
PrometheusRule observability observability-core-kube-pr-kubernetes-system-kubelet 2026-11-30 Kubelet 인증서 경보 둘을 values에서 껐다(PR #6). 관측 core를 다시 적용하면 사라진다(재시작 창 2단계).
+16
View File
@@ -0,0 +1,16 @@
#!/usr/bin/env bash
# Daily drift check against origin/main (not the working tree of the checkout),
# so local edits or a stale checkout do not hide or invent drift.
# Run by platform-live-drift-check.timer as the desktop user.
set -Eeuo pipefail
readonly CHECKOUT="${PLATFORM_CHECKOUT:-/home/donghyeon/workspace/platform}"
readonly TEXTFILE=/var/lib/node_exporter/textfile_collector/platform_live_drift.prom
git -C "$CHECKOUT" fetch --quiet origin main
work="$(mktemp -d /tmp/platform-drift-main.XXXXXX)"
trap 'rm -rf -- "$work"' EXIT
git -C "$CHECKOUT" archive origin/main | tar -x -C "$work"
printf 'origin/main %s\n' "$(git -C "$CHECKOUT" rev-parse --short origin/main)"
args=(--accepted "$work/scripts/validate/live-drift-accepted.txt")
[[ -d "$(dirname "$TEXTFILE")" ]] && args+=(--textfile "$TEXTFILE")
bash "$work/scripts/validate/live-drift-check.sh" "${args[@]}"
+24 -5
View File
@@ -64,10 +64,29 @@ die() {
work="$(mktemp -d /tmp/platform-drift.XXXXXX)"; made_dirs+=("$work")
mkdir -p "$work/in" "$work/out"
# Render from a clean copy when run inside a git working tree. The observability
# renderer refuses to run when an ignored chart cache (.helm/) is left over from an
# earlier render, which happens in long-lived checkouts. The copy keeps tracked and
# untracked-but-not-ignored files, so local edits are still checked.
SOURCE_ROOT="$REPOSITORY_ROOT"
if git -C "$REPOSITORY_ROOT" rev-parse --is-inside-work-tree >/dev/null 2>&1; then
SOURCE_ROOT="$work/src"
mkdir -p "$SOURCE_ROOT"
python3 -I - "$REPOSITORY_ROOT" <<'PY' | tar -C "$REPOSITORY_ROOT" --null -T - -cf - | tar -x -C "$SOURCE_ROOT" || die "could not copy the working tree"
import subprocess, sys
root = sys.argv[1]
def ls(*args):
out = subprocess.run(["git", "-C", root, "ls-files", "-z", *args], check=True, capture_output=True).stdout
return [p for p in out.decode().split("\0") if p]
keep = set(ls("--cached", "--others", "--exclude-standard")) - set(ls("--deleted"))
sys.stdout.write("\0".join(sorted(keep)) + "\0")
PY
fi
render_root() {
local script="$1" prefix="$2" dir
dir="$(mktemp -d "/tmp/${prefix}.XXXXXX")"; made_dirs+=("$dir")
bash "$REPOSITORY_ROOT/scripts/validate/${script}" --verified-output-dir "$dir" \
bash "$SOURCE_ROOT/scripts/validate/${script}" --verified-output-dir "$dir" \
>"$work/${script}.log" 2>&1 || { tail -5 "$work/${script}.log" >&2; die "${script} failed"; }
printf '%s\n' "$dir"
}
@@ -86,10 +105,10 @@ declare -a inputs=(
for f in "$phase2"/*.yaml "$admin"/*.yaml; do inputs+=("$f"); done
# Live Traefik is the trust overlay (infrastructure/networking/traefik/README.md).
kubectl kustomize "$REPOSITORY_ROOT/infrastructure/networking/traefik/overlays/trust" >"$work/traefik-trust.yaml" || die "traefik render failed"
kubectl kustomize "$REPOSITORY_ROOT/services/tenants/techlog" >"$work/tenant-techlog.yaml" || die "tenant render failed"
kubectl kustomize "$REPOSITORY_ROOT/services/priority-classes" >"$work/priority-classes.yaml" || die "priority class render failed"
kubectl kustomize "$REPOSITORY_ROOT/infrastructure/networking/keycloak" >"$work/keycloak-network-policies.yaml" || die "keycloak policy render failed"
kubectl kustomize "$SOURCE_ROOT/infrastructure/networking/traefik/overlays/trust" >"$work/traefik-trust.yaml" || die "traefik render failed"
kubectl kustomize "$SOURCE_ROOT/services/tenants/techlog" >"$work/tenant-techlog.yaml" || die "tenant render failed"
kubectl kustomize "$SOURCE_ROOT/services/priority-classes" >"$work/priority-classes.yaml" || die "priority class render failed"
kubectl kustomize "$SOURCE_ROOT/infrastructure/networking/keycloak" >"$work/keycloak-network-policies.yaml" || die "keycloak policy render failed"
inputs+=("$work/traefik-trust.yaml" "$work/tenant-techlog.yaml" "$work/priority-classes.yaml" "$work/keycloak-network-policies.yaml")
n=0
@@ -412,9 +412,9 @@ elif mode == "alertmanager_routing":
if item.get("kind") == "AlertmanagerConfig"
and (item.get("metadata") or {}).get("name") == "platform-alertmanager"
]
expected_config = json.loads(r'''{"apiVersion":"monitoring.coreos.com/v1alpha1","kind":"AlertmanagerConfig","metadata":{"name":"platform-alertmanager","namespace":"observability","labels":{"observability.hyeonworks.com/instance":"home"}},"spec":{"route":{"receiver":"platform-slack","groupBy":["cluster","namespace","alertname","severity"],"groupWait":"30s","groupInterval":"5m","repeatInterval":"4h","routes":[{"receiver":"platform-null","matchers":[{"name":"alertname","matchType":"=","value":"InfoInhibitor"}]}]},"inhibitRules":[{"sourceMatch":[{"name":"alertname","matchType":"=","value":"InfoInhibitor"}],"targetMatch":[{"name":"severity","matchType":"=","value":"info"}],"equal":["namespace"]}],"receivers":[{"name":"platform-null"},{"name":"platform-slack","slackConfigs":[{"apiURL":{"name":"alertmanager-slack-webhook","key":"url"},"sendResolved":true,"linkNames":false,"mrkdwnIn":["text","fields"],"fallback":"{{ if eq .Status \"firing\" }}FIRING{{ else }}RESOLVED{{ end }}: {{ if .CommonLabels.severity }}{{ .CommonLabels.severity | toUpper }}{{ else }}UNKNOWN{{ end }} · {{ .CommonLabels.alertname }}","title":"{{ if eq .Status \"firing\" }}[FIRING:{{ .Alerts.Firing | len }}]{{ else }}[RESOLVED]{{ end }} {{ if .CommonLabels.severity }}{{ .CommonLabels.severity | toUpper }}{{ else }}UNKNOWN{{ end }} · {{ .CommonLabels.alertname }}","titleLink":"https://grafana.learn.hyeonworks.com/","color":"{{ if eq .Status \"resolved\" }}good{{ else if or (eq .CommonLabels.severity \"emergency\") (eq .CommonLabels.severity \"critical\") }}danger{{ else if eq .CommonLabels.severity \"warning\" }}warning{{ else }}#439FE0{{ end }}","fields":[{"title":"Status","value":"{{ .Status | toUpper }}","short":true},{"title":"Severity","value":"{{ if .CommonLabels.severity }}{{ .CommonLabels.severity | toUpper }}{{ else }}UNKNOWN{{ end }}","short":true},{"title":"Location","value":"{{ if .CommonLabels.cluster }}{{ .CommonLabels.cluster }}{{ else }}unknown-cluster{{ end }} / {{ if .CommonLabels.namespace }}{{ .CommonLabels.namespace }}{{ else }}cluster-scoped{{ end }}","short":true},{"title":"Alert count","value":"{{ len .Alerts }}","short":true}],"text":"{{ range .Alerts }}\n*Alert status:* {{ .Status | toUpper }}\n*Target:* {{ $target := .Labels.Remove $.GroupLabels.Names }}{{ if $target }}{{ range $target.SortedPairs }}{{ .Name }}={{ .Value }} {{ end }}{{ else }}unknown{{ end }}\n*Summary:* {{ with .Annotations.summary }}{{ . }}{{ else }}No summary provided{{ end }}\n*Details:* {{ with .Annotations.description }}{{ . }}{{ else }}No description provided{{ end }}\n*Started:* {{ .StartsAt.Format \"2006-01-02T15:04:05Z07:00\" }}\n{{ if eq .Status \"resolved\" }}*Ended:* {{ .EndsAt.Format \"2006-01-02T15:04:05Z07:00\" }}{{ end }}\n{{ with .Annotations.runbook_url }}*Runbook:* <{{ . }}|대응 절차 열기>{{ end }}\n\n{{ end }}\n<https://grafana.learn.hyeonworks.com/|Grafana> · <https://grafana.learn.hyeonworks.com/explore|Explore>","footer":"hyeonworks observability · Alertmanager"}]}]}}''')
expected_config = json.loads(r'''{"apiVersion":"monitoring.coreos.com/v1alpha1","kind":"AlertmanagerConfig","metadata":{"name":"platform-alertmanager","namespace":"observability","labels":{"observability.hyeonworks.com/instance":"home"}},"spec":{"route":{"receiver":"platform-slack","groupBy":["cluster","namespace","alertname","severity"],"groupWait":"30s","groupInterval":"5m","repeatInterval":"4h","routes":[{"receiver":"platform-null","matchers":[{"name":"alertname","matchType":"=","value":"InfoInhibitor"}]},{"receiver":"platform-heartbeat","matchers":[{"name":"alertname","matchType":"=","value":"PlatformHeartbeat"}],"groupWait":"0s","groupInterval":"1m","repeatInterval":"5m"}]},"inhibitRules":[{"sourceMatch":[{"name":"alertname","matchType":"=","value":"InfoInhibitor"}],"targetMatch":[{"name":"severity","matchType":"=","value":"info"}],"equal":["namespace"]}],"receivers":[{"name":"platform-null"},{"name":"platform-heartbeat","webhookConfigs":[{"urlSecret":{"name":"alertmanager-heartbeat","key":"url"},"sendResolved":false,"maxAlerts":1}]},{"name":"platform-slack","slackConfigs":[{"apiURL":{"name":"alertmanager-slack-webhook","key":"url"},"sendResolved":true,"linkNames":false,"mrkdwnIn":["text","fields"],"fallback":"{{ if eq .Status \"firing\" }}FIRING{{ else }}RESOLVED{{ end }}: {{ if .CommonLabels.severity }}{{ .CommonLabels.severity | toUpper }}{{ else }}UNKNOWN{{ end }} · {{ .CommonLabels.alertname }}","title":"{{ if eq .Status \"firing\" }}[FIRING:{{ .Alerts.Firing | len }}]{{ else }}[RESOLVED]{{ end }} {{ if .CommonLabels.severity }}{{ .CommonLabels.severity | toUpper }}{{ else }}UNKNOWN{{ end }} · {{ .CommonLabels.alertname }}","titleLink":"https://grafana.learn.hyeonworks.com/","color":"{{ if eq .Status \"resolved\" }}good{{ else if or (eq .CommonLabels.severity \"emergency\") (eq .CommonLabels.severity \"critical\") }}danger{{ else if eq .CommonLabels.severity \"warning\" }}warning{{ else }}#439FE0{{ end }}","fields":[{"title":"Status","value":"{{ .Status | toUpper }}","short":true},{"title":"Severity","value":"{{ if .CommonLabels.severity }}{{ .CommonLabels.severity | toUpper }}{{ else }}UNKNOWN{{ end }}","short":true},{"title":"Location","value":"{{ if .CommonLabels.cluster }}{{ .CommonLabels.cluster }}{{ else }}unknown-cluster{{ end }} / {{ if .CommonLabels.namespace }}{{ .CommonLabels.namespace }}{{ else }}cluster-scoped{{ end }}","short":true},{"title":"Alert count","value":"{{ len .Alerts }}","short":true}],"text":"{{ range .Alerts }}\n*Alert status:* {{ .Status | toUpper }}\n*Target:* {{ $target := .Labels.Remove $.GroupLabels.Names }}{{ if $target }}{{ range $target.SortedPairs }}{{ .Name }}={{ .Value }} {{ end }}{{ else }}unknown{{ end }}\n*Summary:* {{ with .Annotations.summary }}{{ . }}{{ else }}No summary provided{{ end }}\n*Details:* {{ with .Annotations.description }}{{ . }}{{ else }}No description provided{{ end }}\n*Started:* {{ .StartsAt.Format \"2006-01-02T15:04:05Z07:00\" }}\n{{ if eq .Status \"resolved\" }}*Ended:* {{ .EndsAt.Format \"2006-01-02T15:04:05Z07:00\" }}{{ end }}\n{{ with .Annotations.runbook_url }}*Runbook:* <{{ . }}|대응 절차 열기>{{ end }}\n\n{{ end }}\n<https://grafana.learn.hyeonworks.com/|Grafana> · <https://grafana.learn.hyeonworks.com/explore|Explore>","footer":"hyeonworks observability · Alertmanager"}]}]}}''')
if configs != [expected_config]:
reject("platform AlertmanagerConfig is not the exact global Slack contract")
reject("platform AlertmanagerConfig is not the exact global Slack and heartbeat contract")
policies = [
item
+4
View File
@@ -402,6 +402,10 @@ fi
fail "the Keycloak role must authenticate only to the Keycloak database"
[[ "$(rg --count -- '^[[:space:]]*-[[:space:]]host all keycloak all reject[[:space:]]*$' "${render_dir}/platform-postgres.yaml" || true)" == "1" ]] || \
fail "the Keycloak role must be rejected from every other database"
[[ "$(rg --count -- '^[[:space:]]*-[[:space:]]host techlog techlog all scram-sha-256[[:space:]]*$' "${render_dir}/platform-postgres.yaml" || true)" == "1" ]] || \
fail "the tech-log role must authenticate only to the tech-log database"
[[ "$(rg --count -- '^[[:space:]]*-[[:space:]]host all techlog all reject[[:space:]]*$' "${render_dir}/platform-postgres.yaml" || true)" == "1" ]] || \
fail "the tech-log role must be rejected from every other database"
rg --quiet '^kind: CustomResourceDefinition$' "${render_dir}/cnpg-operator.yaml" || \
+5 -1
View File
@@ -266,7 +266,8 @@ aistor_objectstore_kustomization="${REPOSITORY_ROOT}/services/minio-aistor/base/
aistor_operator_values="${REPOSITORY_ROOT}/infrastructure/controllers/aistor-operator/values/home.yaml"
aistor_objectstore_values="${REPOSITORY_ROOT}/services/minio-aistor/base/deployment/values/home.yaml"
assert_regex_count "$keycloak_operator_kustomization" '^[[:space:]]*-[[:space:]]github\.com/keycloak/keycloak-k8s-resources/kubernetes\?ref=26\.7\.0[[:space:]]*$' 1 "Keycloak Operator remote ref must be 26.7.0"
assert_regex_count "$keycloak_operator_kustomization" '^[[:space:]]*-[[:space:]]github\.com/keycloak/keycloak-k8s-resources/kubernetes\?ref=38fd7a0c84e15d4b66b37d1c2ea728ca2f5416a4[[:space:]]*$' 1 "Keycloak Operator remote ref must be the 26.7.0 commit"
assert_regex_count "$keycloak_operator_kustomization" '^[[:space:]]*digest:[[:space:]]sha256:257ee324da2c701b7fc1ed1b2ce0e03979d1bf149952383bdcf6be904ae5b4f3[[:space:]]*$' 1 "Keycloak Operator image must be pinned by digest"
assert_regex_count "$keycloak_operator_kustomization" 'github\.com/keycloak/keycloak-k8s-resources/kubernetes' 1 "exactly one Keycloak Operator remote resource is allowed"
assert_regex_count "$keycloak_operator_kustomization" '\?ref=' 1 "exactly one pinned remote ref is allowed in the Keycloak Operator root"
@@ -342,6 +343,9 @@ assert_regex_count "${render_dir}/keycloak.yaml" '^kind: Ingress$' 1 "Keycloak I
assert_regex_count "${render_dir}/minio-aistor.yaml" '^kind: ObjectStore$' 1 "AIStor ObjectStore count"
assert_regex_count "${render_dir}/keycloak.yaml" '^[[:space:]]*instances:[[:space:]]1[[:space:]]*$' 1 "Keycloak instance count field"
assert_regex_count "${render_dir}/keycloak.yaml" '^[[:space:]]*image:[[:space:]]quay\.io/keycloak/keycloak@sha256:[0-9a-f]{64}[[:space:]]*$' 1 "Keycloak server image must be pinned by digest"
assert_regex_count "${render_dir}/keycloak.yaml" '^[[:space:]]*startOptimized:[[:space:]]false[[:space:]]*$' 1 "the stock Keycloak image must not start optimized"
assert_regex_count "${render_dir}/keycloak.yaml" '^[[:space:]]*priorityClassName:[[:space:]]platform-critical[[:space:]]*$' 1 "Keycloak priority class"
assert_regex_count "${render_dir}/keycloak.yaml" '^[[:space:]]*hostname:[[:space:]]https://id\.learn\.hyeonworks\.com[[:space:]]*$' 1 "Keycloak external hostname"
assert_regex_count "${render_dir}/keycloak.yaml" '^[[:space:]]*strict:[[:space:]]true[[:space:]]*$' 1 "Keycloak strict hostname mode"
assert_regex_count "${render_dir}/keycloak.yaml" '^[[:space:]]*httpEnabled:[[:space:]]true[[:space:]]*$' 1 "Keycloak internal HTTP mode"
@@ -2,9 +2,13 @@
set -o pipefail
readonly PLATFORM_SENSITIVE_SOURCE_EXACT_PLATFORM_ROOT=/home/donghyeon/workspace/platform
readonly PLATFORM_SENSITIVE_SOURCE_EXACT_DOCS_ROOT=/home/donghyeon/workspace/docs/platform
readonly PLATFORM_SENSITIVE_SOURCE_EXACT_SCANNER="$PLATFORM_SENSITIVE_SOURCE_EXACT_PLATFORM_ROOT/scripts/validate/scan-platform-sensitive-source.sh"
# The platform root is the checkout this scanner lives in, so a temporary copy
# or a CI runner scans itself. The central documentation root stays on the
# desktop host unless --docs-root or --platform-only says otherwise.
PLATFORM_SENSITIVE_SOURCE_SELF_ROOT="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/../.." && pwd -P)"
readonly PLATFORM_SENSITIVE_SOURCE_SELF_ROOT
readonly PLATFORM_SENSITIVE_SOURCE_SELF="$PLATFORM_SENSITIVE_SOURCE_SELF_ROOT/scripts/validate/scan-platform-sensitive-source.sh"
readonly PLATFORM_SENSITIVE_SOURCE_DEFAULT_DOCS_ROOT=/home/donghyeon/workspace/docs/platform
_sensitive_source_fail() {
printf 'ERROR: %s\n' "$*" >&2
@@ -14,17 +18,22 @@ _sensitive_source_fail() {
_sensitive_source_usage() {
cat <<'USAGE'
Usage:
bash scripts/validate/scan-platform-sensitive-source.sh
bash scripts/validate/scan-platform-sensitive-source.sh [--docs-root DIR | --platform-only]
Scans the complete platform source and central platform documentation trees.
Only .git and .helm directory contents, plus the exact platform-root
.superpowers/sdd scratch subtree, are excluded. Diagnostics contain file names
and finding classes only; matched values and lines are never printed.
Scans the platform checkout that contains this script and the central platform
documentation tree (default /home/donghyeon/workspace/docs/platform, or
--docs-root DIR). --platform-only skips the documentation tree, for a
temporary copy or a CI runner that has none. Only .git and .helm directory
contents, plus the platform-root .superpowers/sdd scratch subtree, are
excluded. Diagnostics contain file names and finding classes only; matched
values and lines are never printed.
The production run invokes all five renderer entrypoints in private temporary
contexts. Four output-producing renderers publish private handoffs. The access
renderer uses its output-free Grafana verifier because it does not require
historical metric inventories; its exact two-line success contract is checked.
contexts. In a git checkout they run from a private copy of the tracked and
not-ignored files, so an ignored chart cache cannot stop them. Four
output-producing renderers publish private handoffs. The access renderer uses
its output-free Grafana verifier because it does not require historical metric
inventories; its exact two-line success contract is checked.
It still traverses the pinned core renderer, whose separately published handoff
contains the only allowlisted credential-free Secret and is decoded here.
USAGE
@@ -43,7 +52,7 @@ _sensitive_source_report_path() {
local path=$1 platform_root=$2 docs_root=$3 relative
if [[ "$path" == "$platform_root"/* ]]; then
relative="platform/${path#"$platform_root"/}"
elif [[ "$path" == "$docs_root"/* ]]; then
elif [[ -n "$docs_root" && "$path" == "$docs_root"/* ]]; then
relative="docs/${path#"$docs_root"/}"
else
relative="$(basename -- "$path")"
@@ -82,13 +91,19 @@ structured_source = suffix in (".yaml", ".yml", ".json")
python_source = suffix == ".py"
shell_source = suffix in (".sh", ".bash", ".zsh") or not suffix
prose_source = suffix in (".md", ".markdown", ".txt", ".rst")
embedded_python_source = path in (
"/home/donghyeon/workspace/platform/scripts/validate/observability-smoke.sh",
"/home/donghyeon/workspace/platform/scripts/validate/test-scan-platform-sensitive-source.sh",
"/home/donghyeon/workspace/platform/scripts/validate/test-configure-keycloak-grafana-oidc.sh",
"/home/donghyeon/workspace/platform/scripts/bootstrap/create-observability-secrets.sh",
"/home/donghyeon/workspace/platform/scripts/bootstrap/configure-keycloak-grafana-oidc.sh",
)
# Shell files with embedded Python, relative to the checkout that holds this
# scanner (scanner_source is <root>/scripts/validate/<scanner>).
scanner_root = Path(scanner_source).parents[2]
embedded_python_source = path in {
str(scanner_root / relative)
for relative in (
"scripts/validate/observability-smoke.sh",
"scripts/validate/test-scan-platform-sensitive-source.sh",
"scripts/validate/test-configure-keycloak-grafana-oidc.sh",
"scripts/bootstrap/create-observability-secrets.sh",
"scripts/bootstrap/configure-keycloak-grafana-oidc.sh",
)
}
try:
raw = open(path, "rb").read()
@@ -717,6 +732,19 @@ def permitted_reference_container(key, value):
)
def permitted_s3_credentials_reference(path, key, value):
# CloudNativePG barmanObjectStore.s3Credentials: each entry is a
# SecretKeySelector (name and key only), never the value itself.
return (
len(path) >= 1
and path[-1] == "s3Credentials"
and key in ("accessKeyId", "secretAccessKey")
and isinstance(value, dict)
and set(value) == {"name", "key"}
and all(isinstance(value[field], str) and value[field] for field in value)
)
def permitted_webhook_feature(path, key, value):
return (
tuple(str(part) for part in path) == ("operators", "object-store")
@@ -752,6 +780,8 @@ def walk(value, path=()):
pass
elif permitted_reference_container(key, child):
pass
elif permitted_s3_credentials_reference(path, key, child):
pass
elif not isinstance(child, str) or unsafe_scalar(child):
return True
if walk(child, path + (key,)):
@@ -1382,9 +1412,13 @@ scan_platform_sensitive_roots() (
return 1
}
done
[[ "$platform_root" == /* && "$docs_root" == /* && "$platform_root" != "$docs_root" ]] ||
# An empty docs root means a platform-only scan.
local -a scan_roots=("$platform_root")
[[ -z "$docs_root" ]] || scan_roots+=("$docs_root")
[[ "$platform_root" == /* ]] && { [[ -z "$docs_root" ]] ||
[[ "$docs_root" == /* && "$platform_root" != "$docs_root" ]]; } ||
_sensitive_source_fail 'scan roots must be distinct absolute paths' || return
for argument in "$platform_root" "$docs_root"; do
for argument in "${scan_roots[@]}"; do
[[ -d "$argument" && ! -L "$argument" && "$(cd -- "$argument" && pwd -P)" == "$argument" ]] ||
_sensitive_source_fail "scan root is not a physical directory: $argument" || return
done
@@ -1397,7 +1431,7 @@ scan_platform_sensitive_roots() (
printf 'SCANNER_ERROR: file inventory hardening failed\n' >&2
return 1
}
if ! find "$platform_root" "$docs_root" \
if ! find "${scan_roots[@]}" \
\( -type d -path "$platform_root/.superpowers/sdd" -prune \) -o \
\( -type d \( -name .git -o -name .helm \) -prune \) -o \
\( -type f -o -type l \) -print0 >"$list_file"; then
@@ -1432,7 +1466,7 @@ scan_platform_sensitive_roots() (
fi
finding="$(_sensitive_source_python_scan \
"$file" "$test_file" "$PLATFORM_SENSITIVE_SOURCE_EXACT_SCANNER")" || {
"$file" "$test_file" "$PLATFORM_SENSITIVE_SOURCE_SELF")" || {
printf 'SCANNER_ERROR: content scan failed for FILE=%s\n' "$report" >&2
return 1
}
@@ -1505,7 +1539,7 @@ scan_platform_sensitive_roots() (
scan_platform_renderer_secrets() (
set -o pipefail
local platform_root=$1 renderer label output_prefix publishes log file results line rc entry artifact
local helm_bin list_file=''
local helm_bin list_file='' render_root source_copy=''
local failed=false
local -a renderers=(
'phase1|render-phase1.sh|platform-phase1-apply|yes'
@@ -1546,6 +1580,11 @@ scan_platform_renderer_secrets() (
rm -f -- "$list_file"
;;
esac
case "$source_copy" in
/tmp/platform-sensitive-source-render-src.??????)
rm -rf -- "$source_copy"
;;
esac
}
trap cleanup_renderer_scan EXIT HUP INT TERM
@@ -1570,9 +1609,56 @@ scan_platform_renderer_secrets() (
return 1
}
# In a git checkout, render from a private copy of the tracked and not-ignored
# files (minus files deleted from the working tree). Ignored state such as a
# leftover .helm chart cache would otherwise make the renderers refuse to run.
render_root=$platform_root
if [[ -e "$platform_root/.git" ]]; then
for label in git tar; do
command -v "$label" >/dev/null 2>&1 || {
printf 'SCANNER_ERROR: renderer command unavailable: %s\n' "$label" >&2
return 1
}
done
[[ "$(git -C "$platform_root" rev-parse --show-toplevel 2>/dev/null)" == "$platform_root" ]] || {
printf 'SCANNER_ERROR: renderer root is not the top of its git checkout\n' >&2
return 1
}
source_copy="$(mktemp -d /tmp/platform-sensitive-source-render-src.XXXXXX)" || {
printf 'SCANNER_ERROR: renderer source copy allocation failed\n' >&2
return 1
}
chmod 0700 "$source_copy" || return 1
if ! python3 -I - "$platform_root" <<'PY' |
import subprocess
import sys
root = sys.argv[1]
def listed(*options):
result = subprocess.run(
["git", "-C", root, "ls-files", "-z", *options],
check=True,
stdout=subprocess.PIPE,
)
return [entry for entry in result.stdout.split(b"\0") if entry]
keep = set(listed("--cached", "--others", "--exclude-standard")) - set(listed("--deleted"))
for entry in sorted(keep):
sys.stdout.buffer.write(entry + b"\0")
PY
tar -C "$platform_root" --null -T - -cf - | tar -x -C "$source_copy"; then
printf 'SCANNER_ERROR: renderer source copy failed\n' >&2
return 1
fi
render_root=$source_copy
fi
for renderer in "${renderers[@]}"; do
IFS='|' read -r label renderer output_prefix publishes <<<"$renderer"
renderer="$platform_root/scripts/validate/$renderer"
renderer="$render_root/scripts/validate/$renderer"
[[ -f "$renderer" && ! -L "$renderer" ]] || {
printf 'SCANNER_ERROR: required renderer is unsafe: %s\n' "$label" >&2
return 1
@@ -1706,14 +1792,23 @@ scan_platform_renderer_secrets() (
)
scan_platform_sensitive_source_main() {
(( $# == 0 )) || {
local docs_root=$PLATFORM_SENSITIVE_SOURCE_DEFAULT_DOCS_ROOT
if (( $# == 1 )) && [[ "$1" == --platform-only ]]; then
docs_root=''
elif (( $# == 2 )) && [[ "$1" == --docs-root && "$2" == /* ]]; then
docs_root=$2
elif (( $# != 0 )); then
_sensitive_source_usage >&2
return 2
}
fi
if [[ -n "$docs_root" && ! -d "$docs_root" ]]; then
printf 'ERROR: documentation root %s does not exist; pass --docs-root DIR or --platform-only\n' \
"$docs_root" >&2
return 2
fi
scan_platform_sensitive_roots \
"$PLATFORM_SENSITIVE_SOURCE_EXACT_PLATFORM_ROOT" \
"$PLATFORM_SENSITIVE_SOURCE_EXACT_DOCS_ROOT" >/dev/null || return
scan_platform_renderer_secrets "$PLATFORM_SENSITIVE_SOURCE_EXACT_PLATFORM_ROOT" || return
"$PLATFORM_SENSITIVE_SOURCE_SELF_ROOT" "$docs_root" >/dev/null || return
scan_platform_renderer_secrets "$PLATFORM_SENSITIVE_SOURCE_SELF_ROOT" || return
printf 'PLATFORM SENSITIVE SOURCE SCAN PASS\n'
}
@@ -0,0 +1,41 @@
#!/usr/bin/env bash
# Test k3s-datastore-restore-drill.sh without production data: start a fresh,
# network-less k3s with secrets encryption, add three test Secrets, stop it,
# and run the drill on its state.db and token. Needs docker; no root.
set -Eeuo pipefail
umask 077
readonly ROOT="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/../.." && pwd -P)"
readonly K3S_IMAGE="$(sed -n 's/^readonly K3S_IMAGE="\(.*\)"$/\1/p' "$ROOT/scripts/validate/k3s-datastore-restore-drill.sh")"
[[ -n "$K3S_IMAGE" ]] || { echo "cannot read K3S_IMAGE from the drill script" >&2; exit 1; }
work="$(mktemp -d)"
seed="w6-k3s-drill-seed-$$"
cleanup() {
docker rm -f "$seed" >/dev/null 2>&1 || true
docker run --rm -v "$work:/w" --entrypoint /bin/sh "$K3S_IMAGE" -c 'rm -rf /w/*' >/dev/null 2>&1 || true
rm -rf -- "$work"
}
trap cleanup EXIT
mkdir -p "$work/server" "$work/copy"
docker run -d --name "$seed" --network none --tmpfs /run --tmpfs /var/run \
-v "$work/server:/var/lib/rancher/k3s/server" "$K3S_IMAGE" server --disable-agent --secrets-encryption \
--bind-address 127.0.0.1 --advertise-address 127.0.0.1 --node-ip 127.0.0.1 \
--disable traefik,servicelb,metrics-server,local-storage,coredns >/dev/null
for _ in $(seq 1 60); do docker exec "$seed" kubectl get --raw /readyz >/dev/null 2>&1 && break; sleep 5; done
for i in 1 2 3; do docker exec "$seed" kubectl create secret generic "drill-test-$i" --from-literal=value="test-$i" >/dev/null; done
expected="$(docker exec "$seed" kubectl get secrets -A --no-headers | wc -l)"
docker stop -t 30 "$seed" >/dev/null
docker run --rm -v "$work:/w" --entrypoint /bin/sh "$K3S_IMAGE" -c \
"cp /w/server/db/state.db* /w/server/token /w/copy/ && chown $(id -u):$(id -g) /w/copy/*"
# Fold the WAL into state.db, as `sqlite3 .backup` does in the real backup.
sqlite3 "$work/copy/state.db" 'PRAGMA wal_checkpoint(TRUNCATE);' >/dev/null
rm -f "$work/copy/state.db-wal" "$work/copy/state.db-shm"
# The test Secrets must be stored encrypted (k8s:enc:...), so the drill really
# proves that the token unlocks the encryption key.
encrypted="$(sqlite3 "$work/copy/state.db" "SELECT COUNT(*) FROM kine WHERE name LIKE '/registry/secrets/default/drill-test-%' AND CAST(value AS TEXT) LIKE 'k8s:enc:%';")"
echo "seed: $expected Secrets, encrypted test Secret rows=$encrypted"
(( encrypted >= 3 )) || { echo "test Secrets are not encrypted at rest in the seed datastore" >&2; exit 1; }
bash "$ROOT/scripts/validate/k3s-datastore-restore-drill.sh" --from-dir "$work/copy" --expect-secrets "$expected"
echo "K3S DATASTORE RESTORE DRILL TEST PASS"
@@ -87,7 +87,20 @@ if alert != {
"value": "InfoInhibitor",
}
],
}
},
{
"receiver": "platform-heartbeat",
"matchers": [
{
"name": "alertname",
"matchType": "=",
"value": "PlatformHeartbeat",
}
],
"groupWait": "0s",
"groupInterval": "1m",
"repeatInterval": "5m",
},
],
},
"inhibitRules": [
@@ -111,6 +124,16 @@ if alert != {
],
"receivers": [
{"name": "platform-null"},
{
"name": "platform-heartbeat",
"webhookConfigs": [
{
"urlSecret": {"name": "alertmanager-heartbeat", "key": "url"},
"sendResolved": False,
"maxAlerts": 1,
}
],
},
{
"name": "platform-slack",
"slackConfigs": [
@@ -156,7 +179,7 @@ if alert != {
],
},
}:
fail("platform-alertmanager must match the exact global Slack contract")
fail("platform-alertmanager must match the exact global Slack and heartbeat contract")
if policy != {
"apiVersion": "networking.k8s.io/v1",
+263
View File
@@ -0,0 +1,263 @@
#!/usr/bin/env bash
# Unit tests for the alerts that guard the desktop host: core-rules.yaml and
# services/platform-postgres-backup/prometheus-rule.yaml.
#
# Root disk: the kubelet starts deleting unused images at 85% use and evicts
# pods at 90% (nodefs.available < 10%). The alerts sit in front of those two
# points: warning at 80%, critical at 85%, and a forecast that fires when the
# 6h trend reaches the eviction point within 6 hours.
# PostgreSQL backups: no successful base backup for 26 hours, or WAL waiting
# to be archived. Monitoring: ingestion stopped, and the always-firing
# heartbeat that an external dead man's switch watches.
#
# Needs docker, jq and kubectl (client only, no cluster access).
set -euo pipefail
readonly ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
readonly RULE_FILE="${ROOT_DIR}/services/observability/rules/platform/core-rules.yaml"
readonly BACKUP_RULE_FILE="${ROOT_DIR}/services/platform-postgres-backup/prometheus-rule.yaml"
# Same pinned image as test-platform-observability-rules.sh.
readonly PROMETHEUS_IMAGE="quay.io/prometheus/prometheus:v3.13.2-distroless@sha256:ce95cfa77eff5aad28bd7a65aff19868cf78d9e17e4c254da7dfe22ade78318b"
readonly RUNBOOK_URL="https://git.learn.hyeonworks.com/donghyeon.kang/project-infra/src/branch/main/docs/runbooks/2026-07-31-observability-access-cutover.md"
readonly BACKUP_RUNBOOK_URL="https://git.learn.hyeonworks.com/donghyeon.kang/platform-core/src/branch/main/services/platform-postgres-backup/README.md"
temp_dir=''
cleanup() {
if [[ -n "$temp_dir" && -d "$temp_dir" ]]; then
rm -rf -- "$temp_dir"
fi
}
trap cleanup EXIT
fail() {
printf 'FAIL: %s\n' "$*" >&2
exit 1
}
for command in docker jq kubectl; do
command -v "$command" >/dev/null || fail "missing command: ${command}"
done
temp_dir="$(mktemp -d)"
chmod 0755 "$temp_dir"
kubectl create --dry-run=client --validate=false -f "$RULE_FILE" -o json |
jq '.spec' >"${temp_dir}/core.rules.json"
kubectl create --dry-run=client --validate=false -f "$BACKUP_RULE_FILE" -o json |
jq '.spec' >"${temp_dir}/backup.rules.json"
# Every core alert keeps the runbook the other platform rules use.
jq -e --arg url "$RUNBOOK_URL" '
all(.groups[].rules[] | select(has("alert")); .annotations.runbook_url == $url)
' "${temp_dir}/core.rules.json" >/dev/null || fail "core alert without the shared runbook_url"
jq -e --arg url "$BACKUP_RUNBOOK_URL" '
all(.groups[].rules[] | select(has("alert")); .annotations.runbook_url == $url)
' "${temp_dir}/backup.rules.json" >/dev/null || fail "backup alert without the backup README runbook_url"
# node-exporter labels as scraped by serviceMonitor/observability-agent/node-exporter.
readonly SERIES_LABELS='job="prometheus-node-exporter",instance="10.42.0.95:9100",device="/dev/sdc1",fstype="ext4",mountpoint="/"'
readonly PG_LABELS='job="platform-postgres",namespace="platform-data",pod="platform-postgres-1",instance="10.42.0.130:9187"'
readonly ALERT_LABELS=' job: prometheus-node-exporter
instance: 10.42.0.95:9100
device: /dev/sdc1
fstype: ext4
mountpoint: /'
usage_case() {
# name, used percent, eval minute, expected alerts (YAML list body or "[]")
local name=$1 used=$2 eval_minute=$3 expected=$4
local avail=$((100 - used))
cat <<EOF
- name: ${name}
interval: 1m
input_series:
- series: 'node_filesystem_avail_bytes{${SERIES_LABELS}}'
values: '${avail}+0x40'
- series: 'node_filesystem_size_bytes{${SERIES_LABELS}}'
values: '100+0x40'
alert_rule_test:
- eval_time: ${eval_minute}m
alertname: PlatformRootFilesystemUsage
exp_alerts:${expected}
EOF
}
usage_alert() {
local severity=$1 threshold=$2 summary=$3 description=$4
cat <<EOF
- exp_labels:
${ALERT_LABELS}
severity: ${severity}
threshold: "${threshold}"
exp_annotations:
summary: ${summary}
description: ${description}
runbook_url: ${RUNBOOK_URL}
EOF
}
readonly WARNING_ALERT="$(usage_alert warning 80 \
'루트 파일시스템 사용률이 80%를 초과함' \
'노드 루트 파일시스템 사용률이 80%를 10분간 초과했습니다. 85%에서 kubelet이 이미지를 지우기 시작합니다.')"
readonly CRITICAL_ALERT="$(usage_alert critical 85 \
'루트 파일시스템 사용률이 85%를 초과함' \
'노드 루트 파일시스템 사용률이 85%를 5분간 초과했습니다. kubelet이 안 쓰는 이미지를 지우는 중이고 90%에서 파드를 내보냅니다.')"
forecast_case() {
# name, starting avail, avail change per minute, eval minute, expected alerts
local name=$1 start=$2 step=$3 eval_minute=$4 expected=$5
cat <<EOF
- name: ${name}
interval: 1m
input_series:
- series: 'node_filesystem_avail_bytes{${SERIES_LABELS}}'
values: '${start}${step}x60'
- series: 'node_filesystem_size_bytes{${SERIES_LABELS}}'
values: '1000+0x60'
alert_rule_test:
- eval_time: ${eval_minute}m
alertname: PlatformRootFilesystemFillingFast
exp_alerts:${expected}
EOF
}
readonly FORECAST_ALERT="
- exp_labels:
${ALERT_LABELS}
severity: critical
exp_annotations:
summary: 루트 파일시스템이 6시간 안에 축출 문턱에 닿을 것으로 예측됨
description: 최근 6시간 추세대로면 루트 파일시스템 여유가 6시간 안에 10% 아래로 내려갑니다. 10% 아래에서 kubelet이 파드를 내보냅니다.
runbook_url: ${RUNBOOK_URL}"
{
printf '%s\n' \
'rule_files:' \
' - /work/core.rules.json' \
' - /work/backup.rules.json' \
'evaluation_interval: 1m' \
'tests:'
usage_case 'warning stays quiet at 79 percent' 79 11 ' []'
usage_case 'warning fires at 81 percent after 10 minutes' 81 11 "$WARNING_ALERT"
usage_case 'warning is still pending at 81 percent after 9 minutes' 81 9 ' []'
usage_case 'critical fires at 86 percent after 5 minutes' 86 6 "$CRITICAL_ALERT"
usage_case 'warning and critical both fire at 86 percent after 10 minutes' 86 11 "${WARNING_ALERT}${CRITICAL_ALERT}"
# 1000 bytes, 400 free, losing 1 per minute: the 6h trend reaches the
# 100-byte eviction point in 300 minutes.
forecast_case 'forecast fires for a fast fill after 15 minutes' 400 '-1' 16 "$FORECAST_ALERT"
forecast_case 'forecast is still pending before 15 minutes' 400 '-1' 14 ' []'
# Losing 0.1 per minute takes far longer than 6 hours to reach 100.
forecast_case 'forecast stays quiet for a slow fill' 400 '-0.1' 30 ' []'
forecast_case 'forecast stays quiet for flat usage' 400 '+0' 30 ' []'
# PostgreSQL backups. Timestamps are seconds since the test epoch.
cat <<EOF
# The test clock starts at 0, so "27.8 hours ago" is a negative timestamp.
# In production a cluster without any backup reports 0 and fires the same way.
- name: backup alert fires when the last backup is older than 26 hours
interval: 1m
input_series:
- series: 'cnpg_collector_last_available_backup_timestamp{${PG_LABELS}}'
values: '-100000+0x40'
alert_rule_test:
- eval_time: 31m
alertname: PlatformPostgresBackupMissing
exp_alerts:
- exp_labels:
job: platform-postgres
namespace: platform-data
severity: critical
exp_annotations:
summary: platform-postgres에 26시간 안에 끝난 백업이 없음
description: platform-postgres의 마지막 성공한 base backup이 26시간보다 오래됐거나 없습니다. 매일 03:00 KST 백업(ScheduledBackup platform-postgres-daily)을 확인하세요.
runbook_url: ${BACKUP_RUNBOOK_URL}
- name: backup alert stays quiet after a fresh backup
interval: 1m
input_series:
- series: 'cnpg_collector_last_available_backup_timestamp{${PG_LABELS}}'
values: '60+0x40'
alert_rule_test:
- eval_time: 31m
alertname: PlatformPostgresBackupMissing
exp_alerts: []
- name: WAL backlog fires after 15 minutes
interval: 1m
input_series:
- series: 'cnpg_collector_pg_wal_archive_status{${PG_LABELS},value="ready"}'
values: '11+0x30'
- series: 'cnpg_collector_pg_wal_archive_status{${PG_LABELS},value="done"}'
values: '500+0x30'
alert_rule_test:
- eval_time: 16m
alertname: PlatformPostgresWALArchiveBacklog
exp_alerts:
- exp_labels:
job: platform-postgres
namespace: platform-data
severity: critical
exp_annotations:
summary: platform-postgres의 WAL이 보관소로 올라가지 않음
description: 보관을 기다리는 WAL 파일이 10개를 넘은 채 15분이 지났습니다. 보관이 막히면 WAL이 PostgreSQL 볼륨에 쌓여 디스크가 찹니다.
runbook_url: ${BACKUP_RUNBOOK_URL}
- name: WAL backlog stays quiet at 10 ready files
interval: 1m
input_series:
- series: 'cnpg_collector_pg_wal_archive_status{${PG_LABELS},value="ready"}'
values: '10+0x30'
alert_rule_test:
- eval_time: 20m
alertname: PlatformPostgresWALArchiveBacklog
exp_alerts: []
- name: ingestion alert fires when node-exporter samples stop
interval: 1m
input_series:
- series: 'up{job="prometheus-node-exporter",instance="10.42.0.95:9100"}'
values: '1x5 _x20'
alert_rule_test:
- eval_time: 20m
alertname: PlatformMetricsIngestionStopped
exp_alerts:
- exp_labels:
job: prometheus-node-exporter
severity: critical
exp_annotations:
summary: Prometheus에 새 지표가 저장되지 않음
description: 5분 넘게 node-exporter의 up 샘플이 저장되지 않았습니다. Prometheus가 샘플을 버리고 있으면(예 시계가 되돌아감) 다른 지표 경보도 울리지 않습니다.
runbook_url: ${RUNBOOK_URL}
- name: ingestion alert stays quiet while samples arrive
interval: 1m
input_series:
- series: 'up{job="prometheus-node-exporter",instance="10.42.0.95:9100"}'
values: '1x30'
alert_rule_test:
- eval_time: 20m
alertname: PlatformMetricsIngestionStopped
exp_alerts: []
- name: heartbeat always fires
interval: 1m
alert_rule_test:
- eval_time: 1m
alertname: PlatformHeartbeat
exp_alerts:
- exp_labels:
severity: none
exp_annotations:
summary: 경보 경로 heartbeat
description: 항상 울린다. Alertmanager가 이 경보를 외부 dead man's switch로 5분마다 보내고, 끊기면 외부 서비스가 알린다.
runbook_url: ${RUNBOOK_URL}
EOF
} >"${temp_dir}/disk.test.yaml"
chmod 0444 "${temp_dir}"/*
docker run --rm \
--network none \
--volume "${temp_dir}:/work:ro" \
--entrypoint /bin/promtool \
"$PROMETHEUS_IMAGE" check rules /work/core.rules.json /work/backup.rules.json
docker run --rm \
--network none \
--volume "${temp_dir}:/work:ro" \
--entrypoint /bin/promtool \
"$PROMETHEUS_IMAGE" test rules /work/disk.test.yaml
printf 'PASS: root filesystem, PostgreSQL backup, ingestion and heartbeat alerts\n'
@@ -0,0 +1,151 @@
#!/usr/bin/env bash
# promtool unit tests for services/platform-host-checks/prometheus-rules.yaml.
# Needs docker and python3 (PyYAML). No cluster access.
set -Eeuo pipefail
readonly ROOT="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/../.." && pwd -P)"
# Same pinned image as test-platform-observability-rules.sh.
readonly PROMETHEUS_IMAGE="quay.io/prometheus/prometheus:v3.13.2-distroless@sha256:ce95cfa77eff5aad28bd7a65aff19868cf78d9e17e4c254da7dfe22ade78318b"
work="$(mktemp -d)"
trap 'rm -rf -- "$work"' EXIT
python3 -I - "$ROOT/services/platform-host-checks/prometheus-rules.yaml" "$work/rules.yaml" <<'PY'
import sys, yaml
rule = yaml.safe_load(open(sys.argv[1]))
yaml.safe_dump({"groups": rule["spec"]["groups"]}, open(sys.argv[2], "w"), sort_keys=False)
PY
# promtool evaluates from Unix time 0, so "now" in each test is the eval_time.
cat >"$work/tests.yaml" <<'EOF'
rule_files:
- rules.yaml
evaluation_interval: 1m
tests:
# k3s datastore: last success at t=0; stale after 2 days, alert after 1h more.
- interval: 1m
input_series:
- series: 'platform_backup_last_success_timestamp_seconds{backup="k3s-datastore"}'
values: '0x5500'
- series: 'platform_backup_last_run_timestamp_seconds{backup="k3s-datastore"}'
values: '0x5500'
- series: 'platform_backup_last_run_timestamp_seconds{backup="gitea"}'
values: '0x5500'
- series: 'platform_backup_last_success_timestamp_seconds{backup="gitea"}'
values: '0x5500'
alert_rule_test:
- eval_time: 24h
alertname: PlatformK3sDatastoreBackupTooOld
exp_alerts: []
- eval_time: 50h
alertname: PlatformK3sDatastoreBackupTooOld
exp_alerts:
- exp_labels: {severity: warning}
exp_annotations:
summary: k3s datastore has not been backed up for more than 2 days
description: Check `journalctl -u platform-k3s-datastore-backup` on desktop. Daily timer, 03:30 KST.
- eval_time: 90h
alertname: PlatformGiteaBackupTooOld
exp_alerts: []
- eval_time: 90h
alertname: PlatformHostBackupMetricsAbsent
exp_alerts: []
- interval: 1m
input_series:
- series: 'platform_backup_last_success_timestamp_seconds{backup="gitea"}'
values: '0x11700'
alert_rule_test:
- eval_time: 194h
alertname: PlatformGiteaBackupTooOld
exp_alerts:
- exp_labels: {severity: warning}
exp_annotations:
summary: Gitea has not been backed up for more than 8 days
description: Check `journalctl -u platform-gitea-backup` on desktop. Weekly timer, Sunday 04:00 KST.
# A failed run is reported with its backup label.
- interval: 1m
input_series:
- series: 'platform_backup_last_run_success{backup="gitea",instance="desktop"}'
values: '0x30'
- series: 'platform_backup_last_run_success{backup="k3s-datastore",instance="desktop"}'
values: '1x30'
alert_rule_test:
- eval_time: 15m
alertname: PlatformHostBackupFailed
exp_alerts:
- exp_labels: {severity: warning, backup: gitea}
exp_annotations:
summary: "The last gitea backup run failed"
description: "Check `journalctl -u platform-gitea-backup` on desktop."
# Metrics never written (timers or textfile collector missing).
- interval: 1h
input_series:
- series: 'up{job="prometheus-node-exporter"}'
values: '1x30'
alert_rule_test:
- eval_time: 25h
alertname: PlatformHostBackupMetricsAbsent
exp_alerts:
- exp_labels: {severity: warning, backup: k3s-datastore}
exp_annotations:
summary: Host backup metrics are missing
description: The backup timers or the node-exporter textfile collector (/var/lib/node_exporter/textfile_collector) are not running.
- exp_labels: {severity: warning, backup: gitea}
exp_annotations:
summary: Host backup metrics are missing
description: The backup timers or the node-exporter textfile collector (/var/lib/node_exporter/textfile_collector) are not running.
# Drift for 7h fires; drift 0 does not.
- interval: 1m
input_series:
- series: 'platform_live_drift_objects'
values: '3x500'
- series: 'platform_live_drift_last_run_timestamp_seconds'
values: '0x500'
- series: 'platform_live_drift_check_success'
values: '1x500'
alert_rule_test:
- eval_time: 5h
alertname: PlatformLiveDriftDetected
exp_alerts: []
- eval_time: 7h
alertname: PlatformLiveDriftDetected
exp_alerts:
- exp_labels: {severity: warning}
exp_annotations:
summary: "3 live objects differ from platform-core origin/main"
description: Run `scripts/validate/live-drift-check.sh --accepted scripts/validate/live-drift-accepted.txt` on desktop to list them.
- eval_time: 7h
alertname: PlatformLiveDriftCheckStale
exp_alerts: []
# Certificates: 50 days left -> warning only; 10 days left -> warning and critical.
- interval: 1m
input_series:
- series: 'platform_k3s_certificate_not_after_seconds{certificate="apiserver"}'
values: '4320000x200'
- series: 'platform_k3s_certificate_not_after_seconds{certificate="kubelet"}'
values: '864000x200'
- series: 'platform_k3s_certificate_check_timestamp_seconds'
values: '0x200'
alert_rule_test:
- eval_time: 2h
alertname: PlatformK3sServingCertificateExpiry
exp_alerts:
- exp_labels: {severity: warning, certificate: apiserver}
exp_annotations:
summary: "k3s apiserver serving certificate expires in less than 100 days"
description: k3s renews certificates only when it starts within 120 days of expiry. Plan a k3s restart window (docs/runbooks).
- exp_labels: {severity: warning, certificate: kubelet}
exp_annotations:
summary: "k3s kubelet serving certificate expires in less than 100 days"
description: k3s renews certificates only when it starts within 120 days of expiry. Plan a k3s restart window (docs/runbooks).
- exp_labels: {severity: critical, certificate: kubelet}
exp_annotations:
summary: "k3s kubelet serving certificate expires in less than 14 days"
description: Restart k3s (it renews certificates within 120 days of expiry) or run `k3s certificate rotate`.
- eval_time: 2h
alertname: PlatformK3sCertificateCheckStale
exp_alerts: []
EOF
docker run --rm --user "$(id -u):$(id -g)" -v "$work:/work:ro" -w /work --entrypoint /bin/promtool "$PROMETHEUS_IMAGE" check rules rules.yaml
docker run --rm --user "$(id -u):$(id -g)" -v "$work:/work:ro" -w /work --entrypoint /bin/promtool "$PROMETHEUS_IMAGE" test rules tests.yaml
echo "PLATFORM HOST CHECK ALERTS PASS"
@@ -197,7 +197,7 @@ write_file "$alertmanager_fixture" \
' alertmanagerConfiguration:' \
' name: platform-alertmanager'
alerting_fixture="$fixture_dir/alerting.yaml"
write_file "$alerting_fixture" '{"apiVersion":"monitoring.coreos.com/v1alpha1","kind":"AlertmanagerConfig","metadata":{"name":"platform-alertmanager","namespace":"observability","labels":{"observability.hyeonworks.com/instance":"home"}},"spec":{"route":{"receiver":"platform-slack","groupBy":["cluster","namespace","alertname","severity"],"groupWait":"30s","groupInterval":"5m","repeatInterval":"4h","routes":[{"receiver":"platform-null","matchers":[{"name":"alertname","matchType":"=","value":"InfoInhibitor"}]}]},"inhibitRules":[{"sourceMatch":[{"name":"alertname","matchType":"=","value":"InfoInhibitor"}],"targetMatch":[{"name":"severity","matchType":"=","value":"info"}],"equal":["namespace"]}],"receivers":[{"name":"platform-null"},{"name":"platform-slack","slackConfigs":[{"apiURL":{"name":"alertmanager-slack-webhook","key":"url"},"sendResolved":true,"linkNames":false,"mrkdwnIn":["text","fields"],"fallback":"{{ if eq .Status \"firing\" }}FIRING{{ else }}RESOLVED{{ end }}: {{ if .CommonLabels.severity }}{{ .CommonLabels.severity | toUpper }}{{ else }}UNKNOWN{{ end }} · {{ .CommonLabels.alertname }}","title":"{{ if eq .Status \"firing\" }}[FIRING:{{ .Alerts.Firing | len }}]{{ else }}[RESOLVED]{{ end }} {{ if .CommonLabels.severity }}{{ .CommonLabels.severity | toUpper }}{{ else }}UNKNOWN{{ end }} · {{ .CommonLabels.alertname }}","titleLink":"https://grafana.learn.hyeonworks.com/","color":"{{ if eq .Status \"resolved\" }}good{{ else if or (eq .CommonLabels.severity \"emergency\") (eq .CommonLabels.severity \"critical\") }}danger{{ else if eq .CommonLabels.severity \"warning\" }}warning{{ else }}#439FE0{{ end }}","fields":[{"title":"Status","value":"{{ .Status | toUpper }}","short":true},{"title":"Severity","value":"{{ if .CommonLabels.severity }}{{ .CommonLabels.severity | toUpper }}{{ else }}UNKNOWN{{ end }}","short":true},{"title":"Location","value":"{{ if .CommonLabels.cluster }}{{ .CommonLabels.cluster }}{{ else }}unknown-cluster{{ end }} / {{ if .CommonLabels.namespace }}{{ .CommonLabels.namespace }}{{ else }}cluster-scoped{{ end }}","short":true},{"title":"Alert count","value":"{{ len .Alerts }}","short":true}],"text":"{{ range .Alerts }}\n*Alert status:* {{ .Status | toUpper }}\n*Target:* {{ $target := .Labels.Remove $.GroupLabels.Names }}{{ if $target }}{{ range $target.SortedPairs }}{{ .Name }}={{ .Value }} {{ end }}{{ else }}unknown{{ end }}\n*Summary:* {{ with .Annotations.summary }}{{ . }}{{ else }}No summary provided{{ end }}\n*Details:* {{ with .Annotations.description }}{{ . }}{{ else }}No description provided{{ end }}\n*Started:* {{ .StartsAt.Format \"2006-01-02T15:04:05Z07:00\" }}\n{{ if eq .Status \"resolved\" }}*Ended:* {{ .EndsAt.Format \"2006-01-02T15:04:05Z07:00\" }}{{ end }}\n{{ with .Annotations.runbook_url }}*Runbook:* <{{ . }}|대응 절차 열기>{{ end }}\n\n{{ end }}\n<https://grafana.learn.hyeonworks.com/|Grafana> · <https://grafana.learn.hyeonworks.com/explore|Explore>","footer":"hyeonworks observability · Alertmanager"}]}]}}' \
write_file "$alerting_fixture" '{"apiVersion":"monitoring.coreos.com/v1alpha1","kind":"AlertmanagerConfig","metadata":{"name":"platform-alertmanager","namespace":"observability","labels":{"observability.hyeonworks.com/instance":"home"}},"spec":{"route":{"receiver":"platform-slack","groupBy":["cluster","namespace","alertname","severity"],"groupWait":"30s","groupInterval":"5m","repeatInterval":"4h","routes":[{"receiver":"platform-null","matchers":[{"name":"alertname","matchType":"=","value":"InfoInhibitor"}]},{"receiver":"platform-heartbeat","matchers":[{"name":"alertname","matchType":"=","value":"PlatformHeartbeat"}],"groupWait":"0s","groupInterval":"1m","repeatInterval":"5m"}]},"inhibitRules":[{"sourceMatch":[{"name":"alertname","matchType":"=","value":"InfoInhibitor"}],"targetMatch":[{"name":"severity","matchType":"=","value":"info"}],"equal":["namespace"]}],"receivers":[{"name":"platform-null"},{"name":"platform-heartbeat","webhookConfigs":[{"urlSecret":{"name":"alertmanager-heartbeat","key":"url"},"sendResolved":false,"maxAlerts":1}]},{"name":"platform-slack","slackConfigs":[{"apiURL":{"name":"alertmanager-slack-webhook","key":"url"},"sendResolved":true,"linkNames":false,"mrkdwnIn":["text","fields"],"fallback":"{{ if eq .Status \"firing\" }}FIRING{{ else }}RESOLVED{{ end }}: {{ if .CommonLabels.severity }}{{ .CommonLabels.severity | toUpper }}{{ else }}UNKNOWN{{ end }} · {{ .CommonLabels.alertname }}","title":"{{ if eq .Status \"firing\" }}[FIRING:{{ .Alerts.Firing | len }}]{{ else }}[RESOLVED]{{ end }} {{ if .CommonLabels.severity }}{{ .CommonLabels.severity | toUpper }}{{ else }}UNKNOWN{{ end }} · {{ .CommonLabels.alertname }}","titleLink":"https://grafana.learn.hyeonworks.com/","color":"{{ if eq .Status \"resolved\" }}good{{ else if or (eq .CommonLabels.severity \"emergency\") (eq .CommonLabels.severity \"critical\") }}danger{{ else if eq .CommonLabels.severity \"warning\" }}warning{{ else }}#439FE0{{ end }}","fields":[{"title":"Status","value":"{{ .Status | toUpper }}","short":true},{"title":"Severity","value":"{{ if .CommonLabels.severity }}{{ .CommonLabels.severity | toUpper }}{{ else }}UNKNOWN{{ end }}","short":true},{"title":"Location","value":"{{ if .CommonLabels.cluster }}{{ .CommonLabels.cluster }}{{ else }}unknown-cluster{{ end }} / {{ if .CommonLabels.namespace }}{{ .CommonLabels.namespace }}{{ else }}cluster-scoped{{ end }}","short":true},{"title":"Alert count","value":"{{ len .Alerts }}","short":true}],"text":"{{ range .Alerts }}\n*Alert status:* {{ .Status | toUpper }}\n*Target:* {{ $target := .Labels.Remove $.GroupLabels.Names }}{{ if $target }}{{ range $target.SortedPairs }}{{ .Name }}={{ .Value }} {{ end }}{{ else }}unknown{{ end }}\n*Summary:* {{ with .Annotations.summary }}{{ . }}{{ else }}No summary provided{{ end }}\n*Details:* {{ with .Annotations.description }}{{ . }}{{ else }}No description provided{{ end }}\n*Started:* {{ .StartsAt.Format \"2006-01-02T15:04:05Z07:00\" }}\n{{ if eq .Status \"resolved\" }}*Ended:* {{ .EndsAt.Format \"2006-01-02T15:04:05Z07:00\" }}{{ end }}\n{{ with .Annotations.runbook_url }}*Runbook:* <{{ . }}|대응 절차 열기>{{ end }}\n\n{{ end }}\n<https://grafana.learn.hyeonworks.com/|Grafana> · <https://grafana.learn.hyeonworks.com/explore|Explore>","footer":"hyeonworks observability · Alertmanager"}]}]}}' \
'---' \
'apiVersion: networking.k8s.io/v1' \
'kind: NetworkPolicy' \
@@ -378,6 +378,44 @@ envValueFrom:
YAML
expect_reject 'a secretKeyRef container with a literal child fails closed' \
'platform/bootstrap/reference-container.yaml'
rm -f -- "$WORK/platform/bootstrap/reference-container.yaml"
cat >"$WORK/platform/bootstrap/s3-credentials.yaml" <<'YAML'
spec:
backup:
barmanObjectStore:
s3Credentials:
accessKeyId:
name: platform-postgres-backup-s3
key: ACCESS_KEY_ID
secretAccessKey:
name: platform-postgres-backup-s3
key: ACCESS_SECRET_KEY
YAML
expect_clean 'a CloudNativePG s3Credentials Secret reference remains a reference'
cat >"$WORK/platform/bootstrap/s3-credentials.yaml" <<'YAML'
spec:
s3Credentials:
secretAccessKey:
name: platform-postgres-backup-s3
key: ACCESS_SECRET_KEY
value: should-never-be-printed-s3-reference
YAML
expect_reject 'an s3Credentials reference with an extra literal child fails closed' \
'platform/bootstrap/s3-credentials.yaml'
cat >"$WORK/platform/bootstrap/s3-credentials.yaml" <<'YAML'
spec:
storage:
secretAccessKey:
name: platform-postgres-backup-s3
key: ACCESS_SECRET_KEY
YAML
expect_reject 'a name and key container outside s3Credentials still fails closed' \
'platform/bootstrap/s3-credentials.yaml'
printf 'spec:\n s3Credentials:\n secretAccess%sKey: Actual%s\n' '' 'ProductionSecretValue9283' \
>"$WORK/platform/bootstrap/s3-credentials.yaml"
expect_reject 'a literal s3Credentials secret access key is rejected' \
'platform/bootstrap/s3-credentials.yaml'
new_fixture
cat >"$WORK/platform/bootstrap/webhook-feature.yaml" <<'YAML'
@@ -967,6 +1005,16 @@ label="$(basename -- "$0")"
handoff=no
[[ -z "$out" ]] || handoff=yes
printf '%s\n' "$label${component:+:$component}:handoff=$handoff" >>"$PLATFORM_SENSITIVE_SOURCE_RENDER_LOG"
root="$(cd -- "$(dirname -- "$0")/../.." && pwd -P)"
if [[ "${PLATFORM_SENSITIVE_SOURCE_FAKE_REJECT_CACHE:-0}" != 0 ]] && \
[[ -n "$(find "$root" -name .helm -type d -print -quit)" ]]; then
printf 'REJECT: exact chart cache already exists\n' >&2
exit 23
fi
if [[ -n "${PLATFORM_SENSITIVE_SOURCE_FAKE_REQUIRE_INPUT:-}" && \
! -f "$root/$PLATFORM_SENSITIVE_SOURCE_FAKE_REQUIRE_INPUT" ]]; then
exit 42
fi
emit_phase1_log() {
local publish=$1 name
@@ -1438,6 +1486,41 @@ for injected_renderer in \
fail "$injected_renderer unexpected stdout leaked content"
pass "$injected_renderer stdout is restricted to its bounded grammar"
done
# A git checkout renders from a private copy of its tracked and not-ignored files,
# so an ignored chart cache left in the checkout cannot stop the renderers (the
# real renderers refuse a pre-existing exact cache with RC=23).
command -v git >/dev/null 2>&1 || fail 'git is required for the clean render copy test'
printf '.helm/\n' >"$WORK/platform/.gitignore"
git -C "$WORK/platform" init -q
git -C "$WORK/platform" add -A
git -C "$WORK/platform" -c user.name=fixture -c user.email=fixture@example.invalid \
commit -q -m fixture
mkdir -p "$WORK/platform/services/observability/.helm/charts/stale-1.0.0"
printf 'untracked\n' >"$WORK/platform/services/observability/untracked-input.yaml"
: >"$WORK/renderer.log"
renderer_rc=0
renderer_output="$(PLATFORM_SENSITIVE_SOURCE_FAKE_REJECT_CACHE=1 \
PLATFORM_SENSITIVE_SOURCE_FAKE_REQUIRE_INPUT=services/observability/untracked-input.yaml \
scan_platform_renderer_secrets "$WORK/platform" 2>&1)" || renderer_rc=$?
(( renderer_rc == 0 )) || {
printf '%s\n' "$renderer_output" >&2
fail 'an ignored chart cache in a git checkout blocked the renderers'
}
[[ -d "$WORK/platform/services/observability/.helm/charts/stale-1.0.0" ]] || \
fail 'the renderer scan changed the ignored cache in the scanned checkout'
pass 'a git checkout renders from a clean copy, so an ignored chart cache cannot block it'
pass 'the clean render copy keeps untracked files that git does not ignore'
rm -f -- "$WORK/platform/services/observability/untracked-input.yaml"
rm -f -- "$WORK/platform/scripts/validate/render-phase2.sh"
renderer_rc=0
renderer_output="$(scan_platform_renderer_secrets "$WORK/platform" 2>&1)" || renderer_rc=$?
(( renderer_rc != 0 )) || fail 'a renderer deleted from the working tree was still run'
[[ "$renderer_output" == *'required renderer is unsafe: phase2'* ]] || {
printf '%s\n' "$renderer_output" >&2
fail 'a renderer deleted from the working tree did not fail at the renderer check'
}
pass 'the clean render copy leaves out files deleted from the working tree'
unset PLATFORM_SENSITIVE_SOURCE_RENDER_LOG
new_fixture
@@ -1468,6 +1551,39 @@ output="$(run_scanner 2>&1)" || scanner_error_rc=$?
fail 'scanner error did not fail closed'
}
pass 'rg errors fail closed'
PLATFORM_SENSITIVE_SOURCE_TEST_PATH=''
[[ "$PLATFORM_SENSITIVE_SOURCE_SELF_ROOT" == "$ROOT" ]] || \
fail 'the scanner does not scan the checkout it lives in'
pass 'the scanner scans the checkout it lives in, not a fixed path'
main_rc=0
main_output="$(scan_platform_sensitive_source_main --unknown 2>&1)" || main_rc=$?
(( main_rc == 2 )) || fail 'an unknown entrypoint argument was accepted'
main_rc=0
main_output="$(scan_platform_sensitive_source_main --docs-root relative/docs 2>&1)" || main_rc=$?
(( main_rc == 2 )) || fail 'a relative docs root was accepted'
new_fixture
main_rc=0
main_output="$(scan_platform_sensitive_source_main --docs-root "$WORK/missing-docs" 2>&1)" || main_rc=$?
(( main_rc == 2 )) || fail 'a missing docs root was accepted'
[[ "$main_output" == *'--platform-only'* ]] || \
fail 'a missing docs root did not name --platform-only'
pass 'the entrypoint rejects unknown, relative and missing docs roots before scanning'
printf 'ordinary input\n' >"$WORK/platform/bootstrap/ordinary.md"
platform_only_output="$(scan_platform_sensitive_roots "$WORK/platform" '' 2>&1)" || {
printf '%s\n' "$platform_only_output" >&2
fail 'a platform-only root scan failed'
}
[[ "$platform_only_output" == *'PLATFORM SENSITIVE SOURCE SCAN PASS'* ]] || \
fail 'a platform-only root scan did not emit the PASS marker'
printf 'Slack: https://hooks.%s/services/T1/B2/C3\n' 'slack.com' \
>"$WORK/platform/bootstrap/live.env"
platform_only_rc=0
platform_only_output="$(scan_platform_sensitive_roots "$WORK/platform" '' 2>&1)" || platform_only_rc=$?
(( platform_only_rc != 0 )) || fail 'a platform-only root scan accepted a Slack webhook'
[[ "$platform_only_output" == *'platform/bootstrap/live.env'* ]] || \
fail 'a platform-only root scan did not report the filename'
pass 'a platform-only root scan still scans the platform root'
cleanup_before="$(
find /tmp -maxdepth 1 -type d -name 'platform-sensitive-source-test.??????' \
@@ -2,6 +2,9 @@ namespace: gitea
replicaCount: 1
# services/priority-classes: evicted after everything that is not critical.
priorityClassName: platform-critical
image:
registry: docker.gitea.com
repository: gitea
+9
View File
@@ -9,6 +9,15 @@ metadata:
spec:
instances: 1
# The 26.7.0 server image running on 2026-11-12, by digest. With spec.image set
# the operator assumes a pre-built (optimized) image; this is the stock image,
# so startOptimized stays false and Keycloak builds at start as before.
image: quay.io/keycloak/keycloak@sha256:0f198be292568439d700cdbfb893e69a6009bb43a94a06a945b1d3d506c76b13
startOptimized: false
scheduling:
priorityClassName: platform-critical
db:
vendor: postgres
host: platform-postgres-rw.platform-data.svc.cluster.local
+43
View File
@@ -0,0 +1,43 @@
# Alertmanager 경로
`platform-alertmanager.yaml`은 kube-prometheus-stack이 전역
`alertmanagerConfiguration`으로 읽는 AlertmanagerConfig 하나다. 경로는 셋이다.
| 경보 | receiver | 보내는 곳 |
| --- | --- | --- |
| `InfoInhibitor` | `platform-null` | 버린다 |
| `PlatformHeartbeat` | `platform-heartbeat` | 외부 heartbeat 서비스(dead man's switch), 5분마다 |
| 나머지 | `platform-slack` | Slack, 4시간마다 다시 |
## heartbeat (dead man's switch)
`PlatformHeartbeat`(`rules/platform/core-rules.yaml`)는 항상 울린다. Alertmanager가 이것을
5분마다 외부 서비스의 ping URL로 보낸다. ping이 끊기면 그 서비스가 소유자에게 알린다.
- 막는 것: Prometheus·Alertmanager가 죽었을 때, desktop이나 인터넷 연결이 끊겼을 때.
이 경우는 클러스터 안의 어떤 규칙도 알리지 못한다.
- 막지 못하는 것: Prometheus가 돌면서 샘플만 버리는 경우(2026-11-11 시계 사고). 규칙은 돌고
heartbeat도 계속 나간다. 그 경우는 `PlatformMetricsIngestionStopped`가 Slack으로 알린다.
### 소유자가 할 일 (적용 전에 한 번)
1. 외부 heartbeat 서비스에 check를 하나 만든다. 예: healthchecks.io(무료 요금제, 오픈소스라
나중에 직접 띄울 수도 있다). 주기 5분, 유예 10분. 알림은 Slack이 아닌 곳(메일·휴대폰)도
하나 건다. Slack 경로가 죽었을 때도 알아야 하기 때문이다.
2. ping URL을 Secret으로 만든다. URL 자체가 자격 증명이라 화면에 남기지 않는다.
```bash
read -rs HEARTBEAT_URL # ping URL 붙여 넣기, 화면에 안 보임
printf '%s' "$HEARTBEAT_URL" | kubectl -n observability create secret generic alertmanager-heartbeat \
--from-file=url=/dev/stdin
unset HEARTBEAT_URL
```
3. 그 뒤에 rules·alerts를 다시 적용한다(`scripts/bootstrap/apply-observability-access.sh
--execute --rules-alerts`). **Secret보다 먼저 적용하지 않는다.** operator가 없는 Secret을
가리키는 AlertmanagerConfig로는 설정을 만들지 못한다.
4. 확인: 외부 서비스의 check가 5분 안에 「up」이 된다. Alertmanager 지표
`alertmanager_notifications_total{integration="webhook"}`가 5분마다 는다.
Alertmanager egress는 공개 IPv4 TCP/443만 허용한다(`alertmanager-public-https-egress.yaml`).
ping URL은 HTTPS여야 한다.
@@ -22,6 +22,16 @@ spec:
- name: alertname
matchType: "="
value: InfoInhibitor
# PlatformHeartbeat always fires. Sending it every 5 minutes keeps the
# external dead man's switch quiet; when it stops, that service alerts.
- receiver: platform-heartbeat
matchers:
- name: alertname
matchType: "="
value: PlatformHeartbeat
groupWait: 0s
groupInterval: 1m
repeatInterval: 5m
inhibitRules:
- sourceMatch:
- name: alertname
@@ -35,6 +45,15 @@ spec:
- namespace
receivers:
- name: platform-null
- name: platform-heartbeat
webhookConfigs:
# Ping URL of the external heartbeat check. Create the Secret before
# applying this file; see services/observability/alerting/README.md.
- urlSecret:
name: alertmanager-heartbeat
key: url
sendResolved: false
maxAlerts: 1
- name: platform-slack
slackConfigs:
- apiURL:
@@ -34,6 +34,11 @@ kubeProxy:
defaultRules:
disabled:
Watchdog: true
# k3s kubelet exposes no kubelet_certificate_manager_* metrics, so these two
# can never fire. Serving certificates are watched by platform-host-checks
# (PlatformK3sServingCertificateExpiry) instead. Review 2026-11-11 A11.
KubeletClientCertificateExpiration: true
KubeletServerCertificateExpiration: true
rules:
etcd: false
kubeControllerManager: false
@@ -80,6 +80,8 @@ loki:
max_compaction_parallelism: 1
singleBinary:
# services/priority-classes: logs can be lost first under node pressure.
priorityClassName: platform-low
replicas: 1
sidecar: false
podLabels:
@@ -32,7 +32,11 @@ resources:
cpu: 100m
memory: 128Mi
limits:
cpu: 250m
# Average use is about 2m, but each scrape runs the collectors in
# parallel. At 250m (25ms per 100ms CFS period) 26% of periods were
# throttled on 2026-11-11. One core removes the throttling and still caps
# a runaway collector.
cpu: "1"
memory: 256Mi
serviceAccount:
@@ -62,6 +66,14 @@ hostNetwork: false
hostPID: false
hostIPC: false
# Host timers (infrastructure/host/backup, owned by the k3s maturity work)
# write *.prom files here: last successful Gitea dump and k3s datastore
# backup, live drift count, API server and kubelet certificate expiry. The
# host root is already mounted read-only at /host/root. A missing directory
# only produces an error metric.
extraArgs:
- --collector.textfile.directory=/host/root/var/lib/node_exporter/textfile_collector
hostRootFsMount:
enabled: true
mountPropagation: None
@@ -41,14 +41,14 @@ spec:
node_filesystem_avail_bytes{job="prometheus-node-exporter",mountpoint="/",fstype!~"rootfs|tmpfs|ramfs|squashfs|overlay|nsfs|tracefs|debugfs|securityfs|proc|sysfs|cgroup2?"}
/
node_filesystem_size_bytes{job="prometheus-node-exporter",mountpoint="/",fstype!~"rootfs|tmpfs|ramfs|squashfs|overlay|nsfs|tracefs|debugfs|securityfs|proc|sysfs|cgroup2?"}
) > 85
for: 15m
) > 80
for: 10m
labels:
severity: warning
threshold: "85"
threshold: "80"
annotations:
summary: 루트 파일시스템 사용률이 85%를 초과함
description: 노드 루트 파일시스템 사용률이 경고 임계치인 85%를 15분간 초과했습니다.
summary: 루트 파일시스템 사용률이 80%를 초과함
description: 노드 루트 파일시스템 사용률이 80%를 10분간 초과했습니다. 85%에서 kubelet이 이미지를 지우기 시작합니다.
runbook_url: https://git.learn.hyeonworks.com/donghyeon.kang/project-infra/src/branch/main/docs/runbooks/2026-07-31-observability-access-cutover.md
- alert: PlatformRootFilesystemUsage
expr: >-
@@ -56,14 +56,32 @@ spec:
node_filesystem_avail_bytes{job="prometheus-node-exporter",mountpoint="/",fstype!~"rootfs|tmpfs|ramfs|squashfs|overlay|nsfs|tracefs|debugfs|securityfs|proc|sysfs|cgroup2?"}
/
node_filesystem_size_bytes{job="prometheus-node-exporter",mountpoint="/",fstype!~"rootfs|tmpfs|ramfs|squashfs|overlay|nsfs|tracefs|debugfs|securityfs|proc|sysfs|cgroup2?"}
) > 90
) > 85
for: 5m
labels:
severity: critical
threshold: "90"
threshold: "85"
annotations:
summary: 루트 파일시스템 사용률이 90%를 초과함
description: 노드 루트 파일시스템 사용률이 심각 임계치인 90%를 5분간 초과했습니다.
summary: 루트 파일시스템 사용률이 85%를 초과함
description: 노드 루트 파일시스템 사용률이 85%를 5분간 초과했습니다. kubelet이 안 쓰는 이미지를 지우는 중이고 90%에서 파드를 내보냅니다.
runbook_url: https://git.learn.hyeonworks.com/donghyeon.kang/project-infra/src/branch/main/docs/runbooks/2026-07-31-observability-access-cutover.md
# kubelet evicts pods when nodefs.available drops below 10%. Fire when
# the 6h trend says that happens within the next 6 hours, so a fast
# fill (17G in 9h on 2026-11-07) is caught before the 80% warning.
- alert: PlatformRootFilesystemFillingFast
expr: >-
predict_linear(
node_filesystem_avail_bytes{job="prometheus-node-exporter",mountpoint="/",fstype!~"rootfs|tmpfs|ramfs|squashfs|overlay|nsfs|tracefs|debugfs|securityfs|proc|sysfs|cgroup2?"}[6h],
6 * 3600
)
<
0.10 * node_filesystem_size_bytes{job="prometheus-node-exporter",mountpoint="/",fstype!~"rootfs|tmpfs|ramfs|squashfs|overlay|nsfs|tracefs|debugfs|securityfs|proc|sysfs|cgroup2?"}
for: 15m
labels:
severity: critical
annotations:
summary: 루트 파일시스템이 6시간 안에 축출 문턱에 닿을 것으로 예측됨
description: 최근 6시간 추세대로면 루트 파일시스템 여유가 6시간 안에 10% 아래로 내려갑니다. 10% 아래에서 kubelet이 파드를 내보냅니다.
runbook_url: https://git.learn.hyeonworks.com/donghyeon.kang/project-infra/src/branch/main/docs/runbooks/2026-07-31-observability-access-cutover.md
- alert: PlatformRootFilesystemInodes
expr: >-
@@ -95,6 +113,32 @@ spec:
summary: 루트 파일시스템 inode 사용률이 90%를 초과함
description: 노드 루트 파일시스템 inode 사용률이 심각 임계치인 90%를 15분간 초과했습니다.
runbook_url: https://git.learn.hyeonworks.com/donghyeon.kang/project-infra/src/branch/main/docs/runbooks/2026-07-31-observability-access-cutover.md
- name: platform.monitoring
rules:
# When the TSDB rejects every sample (2026-11-11: the clock went back
# after samples were stored 9 hours ahead) rules still run, but on no
# data. absent() is one of the few expressions that still fires then.
- alert: PlatformMetricsIngestionStopped
expr: absent(up{job="prometheus-node-exporter"})
for: 5m
labels:
severity: critical
annotations:
summary: Prometheus에 새 지표가 저장되지 않음
description: 5분 넘게 node-exporter의 up 샘플이 저장되지 않았습니다. Prometheus가 샘플을 버리고 있으면(예 시계가 되돌아감) 다른 지표 경보도 울리지 않습니다.
runbook_url: https://git.learn.hyeonworks.com/donghyeon.kang/project-infra/src/branch/main/docs/runbooks/2026-07-31-observability-access-cutover.md
# Dead man's switch. Alertmanager forwards this to an external
# heartbeat service; when the heartbeat stops, that service alerts.
# It covers what no rule here can: Prometheus, Alertmanager or the
# whole node being down.
- alert: PlatformHeartbeat
expr: vector(1)
labels:
severity: none
annotations:
summary: 경보 경로 heartbeat
description: 항상 울린다. Alertmanager가 이 경보를 외부 dead man's switch로 5분마다 보내고, 끊기면 외부 서비스가 알린다.
runbook_url: https://git.learn.hyeonworks.com/donghyeon.kang/project-infra/src/branch/main/docs/runbooks/2026-07-31-observability-access-cutover.md
- name: platform.workloads
rules:
- alert: PlatformPodRestartBurst
@@ -4,6 +4,9 @@ global:
replicas: 1
# services/priority-classes: traces can be lost first under node pressure.
priorityClassName: platform-low
config: |
stream_over_http_enabled: false
memberlist:
+86
View File
@@ -0,0 +1,86 @@
# 호스트 백업과 점검
desktop 호스트에서 systemd 타이머로 도는 백업과 점검이다. 결과는 node-exporter textfile 수집기
(`/var/lib/node_exporter/textfile_collector/*.prom`)로 Prometheus에 올라가고, 이 폴더의 PrometheusRule이
경보를 낸다. 근거와 결정은 `docs/reviews/2026-11-11-k3s-maturity-review.md` A1·A11·A13·A14, 8절 결정 1·6이다.
## 무엇을 어디에 얼마나
| 대상 | 방법 | 주기 | 보관 | 위치 |
| --- | --- | --- | --- | --- |
| k3s datastore + `server/token` | `sqlite3 .backup` → tar.gz + sha256 | 매일 03:30 KST | AIStor 30일 + 로컬 최근 7개 | AIStor `platform-host-backups/k3s-datastore/`와 `/srv/k3s/aistor/host-backups/k3s-datastore/`(root 0700). 둘 다 HDD sdb3 |
| Gitea (저장소·패키지·첨부·Actions·`app.ini`) | 라이브 파드에서 `gitea dump --skip-db` | 일요일 04:00 KST | 56일(8주) | AIStor `platform-host-backups/gitea/` |
| platform-postgres (Gitea·Keycloak·tech-log DB) | CloudNativePG base backup + WAL | 매일 03:00 KST | 30일 | AIStor `platform-postgres-backups/` (W1, `services/platform-postgres-backup`) |
| AIStor 데이터(Loki·Tempo) | **백업하지 않는다** | — | — | 8절 결정 6: 잃어도 되는 관측 데이터. 디스크를 잃으면 로그·추적 기록만 사라진다 |
| Prometheus·Grafana·Alertmanager PV | 백업하지 않는다 | — | — | 설정은 이 저장소가 정본이고, 지표는 다시 쌓인다 |
- Gitea DB는 dump에 없다(`--skip-db`). Gitea를 복원하려면 CNPG 복원이 먼저다(`docs/runbooks/gitea-restore.md`).
- k3s 백업에는 `token`이 들어 있다. token과 datastore가 같이 있으면 클러스터의 모든 Secret을 풀 수 있다.
Gitea dump의 `app.ini`에도 비밀 값이 있다. 그래서 둘 다 AIStor의 전용 사용자만 읽고 쓴다.
- AIStor는 같은 desktop의 HDD다. SSD가 죽으면 남지만 desktop 자체를 잃으면 같이 잃는다.
- k3s 백업을 AIStor 밖에도 두는 이유: AIStor는 바로 그 k3s 위에서 돈다. 클러스터를 통째로 잃으면 AIStor를
다시 띄우기 전에는 AIStor 안의 사본을 꺼낼 수 없다. 로컬 사본은 sdb3를 mount하기만 하면 읽힌다
(`docs/runbooks/k3s-datastore-restore.md`). 폴더는 AIStor 드라이브 폴더(`data/`) 옆이라 bucket으로 보이지 않는다.
## off-host 사본 (준비만, 기본 꺼짐)
위치는 소유자가 정한다(8절 결정 1). `/etc/platform-backup/aistor.env`에 두 줄을 넣으면 k3s datastore
백업이 그 공개 키로 암호화한 사본을 `rclone`으로 보낸다.
```bash
OFFHOST_GPG_RECIPIENT_FILE=/etc/platform-backup/offhost-recipient.asc # 공개 키. 개인 키는 desktop에 두지 않는다
OFFHOST_RCLONE_TARGET=<rclone remote>:<path> # rclone 설치·설정 필요
```
지표 `platform_backup_offhost_copy{backup="k3s-datastore"}`가 1이 된다. Gitea dump의 off-host 사본은 아직 없다.
## 점검
| 타이머 | 사용자 | 하는 일 | 지표 |
| --- | --- | --- | --- |
| `platform-k3s-cert-expiry` | donghyeon | 6443·10250 serving 인증서 notAfter | `platform_k3s_certificate_not_after_seconds` |
| `platform-live-drift-check` | donghyeon | origin/main 렌더 → `kubectl diff`(서버 dry-run) | `platform_live_drift_objects` |
## 경보 (`prometheus-rules.yaml`)
| 경보 | 조건 |
| --- | --- |
| `PlatformK3sDatastoreBackupTooOld` | 마지막 성공이 2일보다 오래됨 |
| `PlatformGiteaBackupTooOld` | 8일보다 오래됨 |
| `PlatformHostBackupFailed` | 마지막 실행 실패 |
| `PlatformHostBackupMetricsAbsent` | 지표가 하루 동안 없음(타이머나 textfile 수집기가 안 돎) |
| `PlatformLiveDriftDetected` | drift가 6시간 넘게 남음 |
| `PlatformLiveDriftCheckStale` | drift 검사가 2일 넘게 완료되지 않음 |
| `PlatformK3sServingCertificateExpiry` | 100일 미만 warning, 14일 미만 critical |
| `PlatformK3sCertificateCheckStale` | 인증서 점검이 2일 넘게 안 돎 |
`scripts/validate/test-platform-host-checks-alerts.sh`가 promtool 단위 시험으로 각 경보가 울려야 할 때 울리고
아닐 때 안 울리는지 본다. 같은 PR에서 kube-prometheus-stack의 `KubeletClientCertificateExpiration`·
`KubeletServerCertificateExpiration`을 껐다. k3s kubelet은 그 지표를 내지 않아 울릴 수 없었다.
## 적용 (소유자, 순서대로)
전제: node-exporter textfile 수집기(W1, `services/observability/node-exporter/values/home.yaml`의
`--collector.textfile.directory=/host/root/var/lib/node_exporter/textfile_collector`)가 들어간 main checkout.
```bash
cd /home/donghyeon/workspace/platform && git pull --ff-only
sudo bash scripts/bootstrap/create-host-backup-credentials.sh # bucket·정책·사용자·/etc/platform-backup/aistor.env (APPLY 입력)
bash scripts/bootstrap/install-host-checks.sh --check
sudo bash scripts/bootstrap/install-host-checks.sh --execute # 타이머 설치·켜기·첫 실행
kubectl apply -k services/platform-host-checks # 경보 규칙
sudo bash scripts/validate/k3s-datastore-restore-drill.sh # 첫 백업으로 복원 리허설
sudo bash scripts/validate/gitea-backup-restore-drill.sh
```
`install-host-checks.sh --execute` 마지막에 `platform_backup_last_run_success{backup="k3s-datastore"} 1`과
`{backup="gitea"} 1`이 보여야 한다. 리허설 둘은 각각 `K3S DATASTORE RESTORE DRILL PASS`, `GITEA RESTORE DRILL PASS`로 끝나야 한다.
## 복원 리허설
- k3s: 백업을 내려받아 sha256을 맞춘 뒤, 운영과 같은 버전의 k3s를 **네트워크 없는** 일회용 컨테이너로
띄워 API가 뜨는지, Secret 수가 라이브와 비슷한지(±5), 모든 Secret이 복호화되는지 본다.
`scripts/validate/test-k3s-datastore-restore-drill.sh`는 운영 데이터 없이 같은 경로를 시험한다(새 k3s에
시험 Secret 셋 → 암호화돼 저장된 것을 확인 → 복원).
- Gitea: dump를 풀어 `app.ini`(INTERNAL_TOKEN·SECRET_KEY), 모든 저장소의 `git fsck`, 라이브와 저장소 수를 본다.
- 분기마다 한 번 돌린다. 성공하면 `platform_backup_restore_drill_last_success_timestamp_seconds`가 갱신된다.
@@ -0,0 +1,15 @@
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": ["s3:GetBucketLocation", "s3:ListBucket"],
"Resource": ["arn:aws:s3:::platform-host-backups"]
},
{
"Effect": "Allow",
"Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject"],
"Resource": ["arn:aws:s3:::platform-host-backups/*"]
}
]
}
@@ -0,0 +1,5 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- prometheus-rules.yaml
@@ -0,0 +1,88 @@
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: platform-host-checks
namespace: observability
labels:
observability.hyeonworks.com/instance: home
app.kubernetes.io/part-of: platform
spec:
groups:
- name: platform-host-backups
rules:
- alert: PlatformK3sDatastoreBackupTooOld
expr: time() - max(platform_backup_last_success_timestamp_seconds{backup="k3s-datastore"}) > 2 * 86400
for: 1h
labels:
severity: warning
annotations:
summary: k3s datastore has not been backed up for more than 2 days
description: Check `journalctl -u platform-k3s-datastore-backup` on desktop. Daily timer, 03:30 KST.
- alert: PlatformGiteaBackupTooOld
expr: time() - max(platform_backup_last_success_timestamp_seconds{backup="gitea"}) > 8 * 86400
for: 1h
labels:
severity: warning
annotations:
summary: Gitea has not been backed up for more than 8 days
description: Check `journalctl -u platform-gitea-backup` on desktop. Weekly timer, Sunday 04:00 KST.
- alert: PlatformHostBackupFailed
expr: max by (backup) (platform_backup_last_run_success) == 0
for: 10m
labels:
severity: warning
annotations:
summary: "The last {{ $labels.backup }} backup run failed"
description: "Check `journalctl -u platform-{{ $labels.backup }}-backup` on desktop."
- alert: PlatformHostBackupMetricsAbsent
expr: absent(platform_backup_last_run_timestamp_seconds{backup="k3s-datastore"}) or absent(platform_backup_last_run_timestamp_seconds{backup="gitea"})
for: 1d
labels:
severity: warning
annotations:
summary: Host backup metrics are missing
description: The backup timers or the node-exporter textfile collector (/var/lib/node_exporter/textfile_collector) are not running.
- name: platform-host-drift
rules:
- alert: PlatformLiveDriftDetected
expr: max(platform_live_drift_objects) > 0
for: 6h
labels:
severity: warning
annotations:
summary: "{{ $value }} live objects differ from platform-core origin/main"
description: Run `scripts/validate/live-drift-check.sh --accepted scripts/validate/live-drift-accepted.txt` on desktop to list them.
- alert: PlatformLiveDriftCheckStale
expr: time() - max(platform_live_drift_last_run_timestamp_seconds) > 2 * 86400 or max(platform_live_drift_check_success) == 0
for: 1h
labels:
severity: warning
annotations:
summary: The daily live drift check has not completed
description: Check `journalctl -u platform-live-drift-check` on desktop.
- name: platform-k3s-certificates
rules:
- alert: PlatformK3sServingCertificateExpiry
expr: (min by (certificate) (platform_k3s_certificate_not_after_seconds) - time()) < 100 * 86400
for: 1h
labels:
severity: warning
annotations:
summary: "k3s {{ $labels.certificate }} serving certificate expires in less than 100 days"
description: k3s renews certificates only when it starts within 120 days of expiry. Plan a k3s restart window (docs/runbooks).
- alert: PlatformK3sServingCertificateExpiry
expr: (min by (certificate) (platform_k3s_certificate_not_after_seconds) - time()) < 14 * 86400
for: 10m
labels:
severity: critical
annotations:
summary: "k3s {{ $labels.certificate }} serving certificate expires in less than 14 days"
description: Restart k3s (it renews certificates within 120 days of expiry) or run `k3s certificate rotate`.
- alert: PlatformK3sCertificateCheckStale
expr: time() - max(platform_k3s_certificate_check_timestamp_seconds) > 2 * 86400
for: 1h
labels:
severity: warning
annotations:
summary: The daily k3s certificate expiry check has not run
description: Check `journalctl -u platform-k3s-cert-expiry` on desktop.
+176
View File
@@ -0,0 +1,176 @@
# 플랫폼 PostgreSQL 백업
`platform-data/platform-postgres` 하나에 Gitea, Keycloak, tech-log의 DB가 같이 있다
(2026-11-11: gitea 608MB, keycloak 14MB, techlog 14MB). 2026-11-11까지 백업 설정도,
ScheduledBackup도 없었다. 그 PV는 루트 SSD 하나에만 있었다.
이 폴더는 매일 base backup과 WAL 연속 보관을 AIStor(HDD `sdb3`)에 두고, 30일 안의
아무 시점으로나 되돌릴 수 있게 한다.
## 무엇이 어디로 가나
```text
platform-postgres-1 (SSD)
├─ WAL 파일이 찰 때마다, 그리고 5분마다 ─▶ s3://platform-postgres-backups/platform-postgres/wals/
└─ 매일 03:00 KST base backup ─────────▶ s3://platform-postgres-backups/platform-postgres/base/
AIStor ObjectStore minio-aistor (HDD sdb3, aistor-local-xfs-retain)
```
| 파일 | 하는 일 |
| --- | --- |
| `cluster-backup-patch.yaml` | live Cluster에 `spec.backup`(보관소, 30일 보존, gzip)을 더한다 |
| `scheduled-backup.yaml` | 매일 18:00 UTC(03:00 KST) base backup. 만들자마자 한 번 |
| `object-storage-counterpart.yaml` | `object-storage` 쪽 ingress: platform-data의 PostgreSQL 파드 → AIStor 9000 |
| `bucket-policy.json` | 백업 사용자가 자기 bucket만 읽고 쓰는 정책 |
| `prometheus-rule.yaml` | 백업 경보 둘(아래). 관측 core 렌더러가 백업 규칙을 core 범위 밖으로 두므로 여기 둔다 |
| `restore-drill/cluster.yaml` | 복원 리허설용 임시 Cluster |
| `../../scripts/bootstrap/create-postgres-backup-credentials.sh` | bucket·정책·사용자·Secret 생성 |
## 고른 방법
- **내장 `barmanObjectStore`.** CloudNativePG 1.26부터 deprecated이고 Barman Cloud Plugin이
권장이다. 하지만 plugin은 cert-manager가 필요한데 이 클러스터에는 없다(2026-11-11 CRD 없음).
백업이 하나도 없는 지금은 새 operator 두 개(cert-manager, plugin)를 먼저 세우는 것보다 지금
operator(1.30.0)가 바로 지원하는 쪽이 빠르고 위험이 작다. bucket 구조가 같아 plugin으로
옮겨도 백업을 그대로 쓴다.
- **기한:** operator 경고는 「1.31.0에서 완전히 제거」라고 한다(문서는 1.32.0). 그래서
**CNPG를 1.31로 올리기 전에** Barman Cloud Plugin으로 옮긴다.
- **백업은 HDD에만 둔다(소유자 지시 2026-11-12: 「운영 db 백업은 SSD 말고 HDD쪽에」).** 보관소는
AIStor다. AIStor의 데이터 볼륨은 하나뿐이고, PV `aistor-data-local-pv` → `/srv/k3s/aistor` →
`/dev/sdb3`(xfs, WDC WD20EZAZ 2TB HDD, `lsblk` `ROTA=1`)이다. 루트 SSD(`/dev/sdc1`, Crucial MX500)에는
백업이 가지 않는다. barman-cloud는 WAL과 base backup을 SSD에 따로 모으지 않고 곧바로 올린다.
- 버린 대안: HDD 보관소 `/mnt/hdd-archive`(`/dev/sdb2`, NTFS). POSIX 권한이 없어 PostgreSQL 파일을
그대로 두기 어렵고, barman-cloud가 쓰는 S3 API도 없다. 거기에 `pg_dump`를 두면 매일 한 시점만
남아 30일 안의 아무 시점으로 되돌릴 수 없다. AIStor는 같은 HDD 위에서 bucket별 사용자·정책과
quota를 준다.
- SSD를 잃어도 백업은 남는다. desktop 자체를 잃으면 백업도 같이 잃는다. 다른 기계나 외부로 복사하는
일은 남은 과제다.
- 복원 리허설의 임시 Cluster는 local-path(루트 SSD) 5Gi를 잠깐 쓰고 바로 지운다. 백업 보관이 아니라
복원 결과를 확인하는 자리다. HDD에는 동적으로 볼륨을 만드는 StorageClass가 없고, `sdb3`는 통째로
AIStor가 쓴다.
- **보존 30일, quota 100GiB.** 압축한 base backup이 하루 수백 MB, WAL은 5분마다 한 파일
(`archive_timeout` 기본값)이라 30일에 수십 GB 안쪽이다.
- **볼륨 스냅숏은 쓰지 않는다.** `ssd-local-postgres-retain`은 정적 local PV라 CSI 스냅숏이 없다.
## 적용 (소유자, 순서대로)
전제: 이 PR이 병합된 platform-core checkout, desktop의 kube context `default`.
**백업 설정은 `kubectl patch`로만 더한다.** `services/platform-postgres` 렌더를 다시 적용하면
`priorityClassName`과 Guaranteed resources도 같이 들어가 PostgreSQL이 재시작된다. 그것은
k3s 재시작 창(`scripts/bootstrap/apply-k3s-restart-bundle.sh`)에서 한다. 2026-11-11까지는
live `pg_hba`의 `techlog` 두 줄이 원본에 없어서 다시 적용하면 tech-log 백엔드가 끊겼다.
이 브랜치에서 원본에 넣었으므로(`fix(platform-postgres): keep the tech-log pg_hba rules`)
지금 렌더의 `kubectl diff`에는 pg_hba 차이가 없다.
1. bucket, 정책, 사용자, Secret을 만든다. 화면에 자격 증명이 나오지 않는다.
```bash
bash scripts/bootstrap/create-postgres-backup-credentials.sh
```
마지막 줄에 `PLATFORM_POSTGRES_BACKUP_BOOTSTRAP_PASS`가 나와야 한다. 새 사용자가 자기
bucket에 쓰고 읽고 지울 수 있고, `loki` bucket은 못 보는 것까지 확인한다.
2. AIStor 쪽 ingress를 연다. 그 전에는 PostgreSQL이 9000에 닿지 못한다.
```bash
kubectl apply -f services/platform-postgres-backup/object-storage-counterpart.yaml
```
3. live Cluster에 백업 설정을 더한다. PostgreSQL은 재시작하지 않는다.
```bash
kubectl -n platform-data patch cluster platform-postgres --type merge \
--patch-file services/platform-postgres-backup/cluster-backup-patch.yaml --dry-run=server -o name
kubectl -n platform-data patch cluster platform-postgres --type merge \
--patch-file services/platform-postgres-backup/cluster-backup-patch.yaml
kubectl -n platform-data get cluster platform-postgres -o jsonpath='{.spec.postgresql.pg_hba}{"\n"}'
```
마지막 줄에 `techlog` 두 줄이 있어야 한다. 몇 분 뒤 WAL이 올라가는지 본다.
```bash
kubectl -n platform-data get cluster platform-postgres \
-o jsonpath='{.status.conditions[?(@.type=="ContinuousArchiving")].status}{" "}{.status.firstRecoverabilityPoint}{"\n"}'
```
4. 매일 백업을 건다. 만들자마자 첫 base backup이 돈다.
```bash
kubectl apply -k services/platform-postgres-backup # ScheduledBackup, AIStor ingress, 경보
kubectl -n platform-data get backup -w
```
`PHASE`가 `completed`가 되면 된다. `failed`면
`kubectl -n platform-data describe backup <이름>`의 마지막 줄을 본다.
5. 아래 「복원 리허설」을 한 번 한다.
경보(`prometheus-rule.yaml`, 4단계에서 같이 들어간다. platform-data는 Prometheus가 규칙을 읽는 namespace다):
- `PlatformPostgresBackupMissing`: 26시간 안에 끝난 base backup이 없다. 위 4단계 전까지는 계속
울린다(백업이 정말 없으니까).
- `PlatformPostgresWALArchiveBacklog`: 보관을 기다리는 WAL이 10개를 넘은 채 15분.
보관이 막히면 WAL이 SSD 볼륨에 쌓여 PostgreSQL이 멈춘다. 2·3단계의 순서가 바뀌었거나
Secret이 틀리면 이것이 울린다.
### 되돌리기
```bash
kubectl delete -k services/platform-postgres-backup # ScheduledBackup, ingress, 경보
kubectl -n platform-data patch cluster platform-postgres --type json -p '[{"op":"remove","path":"/spec/backup"}]'
```
bucket, 사용자, Secret, 이미 올린 백업은 지우지 않는다. 지울 때는 소유자가 `mc`로 직접 한다.
## 복원 리허설
백업이 실제로 복원되는지는 복원해 봐야 안다. 첫 base backup이 끝난 뒤 한 번, 그 뒤로는 CNPG나
PostgreSQL을 올릴 때마다 한다. 운영 Cluster는 건드리지 않는다. 임시 Cluster는 bucket을 읽기만 한다
(`backup` 칸이 없다).
1. 임시 Cluster를 만든다. local-path 5Gi(루트 SSD)를 잠깐 쓴다.
```bash
kubectl apply -f services/platform-postgres-backup/restore-drill/cluster.yaml
kubectl -n platform-data wait cluster/platform-postgres-restore-drill \
--for=condition=Ready --timeout=20m
```
2. 두 쪽의 표마다 행 수를 비교한다. 운영은 그 사이에도 쓰이므로 활동이 많은 표(Gitea의
`action`, `notice` 등)는 조금 다를 수 있다. 표가 없거나 크게 다르면 실패다.
```bash
count_rows() { # $1 = cluster, $2 = database
kubectl -n platform-data exec "$1-1" -c postgres -- psql -At -d "$2" -c \
"select string_agg(format('select %L, count(*) from %I.%I', schemaname||'.'||relname, schemaname, relname), ' union all ')
from pg_stat_user_tables" |
kubectl -n platform-data exec -i "$1-1" -c postgres -- psql -At -d "$2" | sort
}
for db in gitea keycloak techlog; do
diff <(count_rows platform-postgres "$db") <(count_rows platform-postgres-restore-drill "$db") \
&& echo "$db: same" || echo "$db: differs (see above)"
done
```
3. 복원된 시점을 기록한다.
```bash
kubectl -n platform-data exec platform-postgres-restore-drill-1 -c postgres -- \
psql -At -c "select now(), pg_last_xact_replay_timestamp()"
```
4. 임시 Cluster를 지운다. PVC도 같이 지워진다.
```bash
kubectl -n platform-data delete cluster platform-postgres-restore-drill
```
결과(날짜, 복원된 시점, 표 비교, 걸린 시간)는 아래 「기록」에 남긴다.
## 기록
- 2026-11-11: 설계와 원본만. bucket·Secret·적용·리허설은 하지 않았다(소유자 단계).
`kubectl apply --dry-run=server`로 ScheduledBackup, ingress, 리허설 Cluster가 admission을
통과했고, `kubectl patch --dry-run=server`로 patch 뒤에도 `techlog` pg_hba 두 줄이 남는 것을 봤다.
@@ -0,0 +1,29 @@
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:GetBucketLocation",
"s3:ListBucket",
"s3:ListBucketMultipartUploads"
],
"Resource": [
"arn:aws:s3:::platform-postgres-backups"
]
},
{
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:PutObject",
"s3:DeleteObject",
"s3:AbortMultipartUpload",
"s3:ListMultipartUploadParts"
],
"Resource": [
"arn:aws:s3:::platform-postgres-backups/*"
]
}
]
}
@@ -0,0 +1,26 @@
# Merge patch for the live Cluster. Apply it with kubectl patch, not by
# re-applying services/platform-postgres: the live Cluster carries pg_hba lines
# for techlog that the Phase 1 source does not have (README.md, "적용").
spec:
backup:
# Base backups and WAL older than the newest point that still lets a
# restore reach 30 days back are deleted by the operator.
retentionPolicy: "30d"
barmanObjectStore:
# AIStor ObjectStore on the HDD (sdb3, aistor-local-xfs-retain), not the
# root SSD that holds the database itself.
destinationPath: s3://platform-postgres-backups/
endpointURL: http://minio.object-storage.svc.cluster.local
s3Credentials:
accessKeyId:
name: platform-postgres-backup-s3
key: ACCESS_KEY_ID
secretAccessKey:
name: platform-postgres-backup-s3
key: ACCESS_SECRET_KEY
wal:
compression: gzip
maxParallel: 2
data:
compression: gzip
jobs: 2
@@ -0,0 +1,17 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
# Applied after platform-postgres (Phase 1) and AIStor (Phase 2) exist and
# after scripts/bootstrap/create-postgres-backup-credentials.sh has created
# the bucket and the platform-data/platform-postgres-backup-s3 Secret.
# cluster-backup-patch.yaml and restore-drill/ are applied by hand; see README.md.
resources:
- object-storage-counterpart.yaml
- scheduled-backup.yaml
- prometheus-rule.yaml
labels:
- pairs:
app.kubernetes.io/part-of: platform
app.kubernetes.io/managed-by: kustomize
includeSelectors: false
@@ -0,0 +1,30 @@
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: object-storage-allow-platform-postgres-backup
namespace: object-storage
labels:
app.kubernetes.io/component: platform-postgres-backup
spec:
podSelector:
matchLabels:
aistor.min.io/objectStore: minio-aistor
policyTypes:
- Ingress
ingress:
# barman-cloud in the PostgreSQL pods writes base backups and WAL; the
# restore drill cluster reads them.
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: platform-data
podSelector:
matchExpressions:
- key: cnpg.io/cluster
operator: In
values:
- platform-postgres
- platform-postgres-restore-drill
ports:
- protocol: TCP
port: 9000
@@ -0,0 +1,46 @@
# Backup alerts for platform-postgres. They live with the backup and not in
# the observability core, whose renderer keeps backup rules out of its scope
# (scripts/validate/render-observability-core.sh, no_unscoped_monitors_rules).
# Prometheus loads PrometheusRules from namespaces labelled
# observability.hyeonworks.com/enabled=true, which platform-data is.
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: platform-postgres-backup
namespace: platform-data
labels:
app.kubernetes.io/name: platform-postgres
app.kubernetes.io/component: backup
observability.hyeonworks.com/instance: home
spec:
groups:
- name: platform.backup
rules:
# cnpg_collector_last_available_backup_timestamp is 0 until the first
# base backup, so a cluster without backups fires as well.
- alert: PlatformPostgresBackupMissing
expr: >-
time() - max by (job, namespace) (
cnpg_collector_last_available_backup_timestamp{job="platform-postgres",namespace="platform-data"}
) > 26 * 3600
for: 30m
labels:
severity: critical
annotations:
summary: platform-postgres에 26시간 안에 끝난 백업이 없음
description: platform-postgres의 마지막 성공한 base backup이 26시간보다 오래됐거나 없습니다. 매일 03:00 KST 백업(ScheduledBackup platform-postgres-daily)을 확인하세요.
runbook_url: https://git.learn.hyeonworks.com/donghyeon.kang/platform-core/src/branch/main/services/platform-postgres-backup/README.md
# A failing archive_command keeps WAL in pg_wal, which then fills the
# PostgreSQL volume.
- alert: PlatformPostgresWALArchiveBacklog
expr: >-
max by (job, namespace) (
cnpg_collector_pg_wal_archive_status{job="platform-postgres",namespace="platform-data",value="ready"}
) > 10
for: 15m
labels:
severity: critical
annotations:
summary: platform-postgres의 WAL이 보관소로 올라가지 않음
description: 보관을 기다리는 WAL 파일이 10개를 넘은 채 15분이 지났습니다. 보관이 막히면 WAL이 PostgreSQL 볼륨에 쌓여 디스크가 찹니다.
runbook_url: https://git.learn.hyeonworks.com/donghyeon.kang/platform-core/src/branch/main/services/platform-postgres-backup/README.md
@@ -0,0 +1,49 @@
# Restore drill: a throwaway Cluster bootstrapped from the platform-postgres
# backups. It has no backup section, so it never writes to the bucket.
# Apply, compare, delete; see ../README.md ("복원 리허설").
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
name: platform-postgres-restore-drill
namespace: platform-data
labels:
app.kubernetes.io/name: platform-postgres-restore-drill
app.kubernetes.io/component: restore-drill
spec:
instances: 1
imageName: ghcr.io/cloudnative-pg/postgresql:17.9-standard-trixie
storage:
# Dynamic local-path volume on the root SSD, deleted with the Cluster.
storageClass: local-path
size: 5Gi
bootstrap:
recovery:
source: platform-postgres
externalClusters:
- name: platform-postgres
barmanObjectStore:
destinationPath: s3://platform-postgres-backups/
endpointURL: http://minio.object-storage.svc.cluster.local
serverName: platform-postgres
s3Credentials:
accessKeyId:
name: platform-postgres-backup-s3
key: ACCESS_KEY_ID
secretAccessKey:
name: platform-postgres-backup-s3
key: ACCESS_SECRET_KEY
wal:
maxParallel: 2
managed:
services:
disabledDefaultServices:
- ro
- r
affinity:
enablePodAntiAffinity: false
resources:
requests:
cpu: 100m
memory: 256Mi
limits:
memory: 1Gi
@@ -0,0 +1,17 @@
apiVersion: postgresql.cnpg.io/v1
kind: ScheduledBackup
metadata:
name: platform-postgres-daily
namespace: platform-data
labels:
app.kubernetes.io/name: platform-postgres
app.kubernetes.io/component: backup
spec:
# Seconds field first. The operator runs in UTC: 18:00 UTC is 03:00 KST.
schedule: "0 0 18 * * *"
# Take the first base backup as soon as this object is created.
immediate: true
backupOwnerReference: self
method: barmanObjectStore
cluster:
name: platform-postgres
+10 -3
View File
@@ -21,9 +21,16 @@ platform-postgres-rw.platform-data.svc.cluster.local:5432
업그레이드한다.
Cluster의 사용자 정의 `pg_hba` 규칙은 `gitea`가 `gitea` Database에만,
`keycloak`이 `keycloak` Database에만 인증하도록 허용한 뒤 각 Role의 다른 모든
Database 접근을 거부한다. 이 규칙들은 CloudNativePG의 기본 fallback 인증 규칙보다
앞에 배치된다.
`keycloak`이 `keycloak` Database에만, `techlog`가 `techlog` Database에만 인증하도록
허용한 뒤 각 Role의 다른 모든 Database 접근을 거부한다. 이 규칙들은
CloudNativePG의 기본 fallback 인증 규칙보다 앞에 배치된다.
`techlog` 규칙 쌍은 운영 tech-log(hyeonworks.com)가 쓰는 것으로, 클러스터에는
이 저장소 밖에서 먼저 들어갔다(2026-11-11 점검에서 확인). 이 규칙 쌍이 없는 렌더를
적용하면 운영 backend가 DB에 접속하지 못하므로 공통 Cluster에 둔다. `techlog`
`DatabaseRole`·`Database`, `platform-data/techlog-db-credentials`와 tech-log용
NetworkPolicy는 아직 어느 저장소에도 없다. 둘 곳은 소유자가 정한다
(`docs/reviews/2026-11-11-k3s-maturity-review.md` 4절).
## Phase 1 Secret 계약
+9 -2
View File
@@ -24,6 +24,8 @@ spec:
- host all gitea all reject
- host keycloak keycloak all scram-sha-256
- host all keycloak all reject
- host techlog techlog all scram-sha-256
- host all techlog all reject
storage:
size: 20Gi
@@ -37,10 +39,15 @@ spec:
affinity:
enablePodAntiAffinity: false
# The DB is the last thing that should go when the node runs short.
# platform-critical comes from services/priority-classes. requests equal
# limits so the pod is Guaranteed QoS and its oom_score_adj drops from 985
# to -997 (k3s maturity review 2026-11-11, 7.2 items 2 and 3).
priorityClassName: platform-critical
resources:
requests:
cpu: 250m
memory: 512Mi
cpu: "2"
memory: 4Gi
limits:
cpu: "2"
memory: 4Gi
+4 -3
View File
@@ -6,13 +6,14 @@
| 이름 | 값 | 선점 | 붙일 곳 |
| --- | --- | --- | --- |
| `platform-critical` | 1000000 | 낮은 것을 선점 | platform-postgres, Gitea, Keycloak, tech-log |
| `platform-low` | -1000 | 선점 안 함 | Grafana, pgAdmin, Loki, Tempo |
| `platform-critical` | 1000000 | 낮은 것을 선점 | Gitea(`services/gitea` values), Keycloak(CR `spec.scheduling`). platform-postgres는 W1, tech-log는 tech-log `deploy/` |
| `platform-low` | -1000 | 선점 안 함 | Loki(`singleBinary`), Tempo. Grafana·pgAdmin은 chart에 최상위 키가 없어 아직 안 붙였다 |
- 아무것도 안 붙인 파드는 기본값 0이다. 시스템 클래스(`system-*`, 20억)는 그대로 가장 높다.
- `globalDefault`는 둘 다 끈다. 새 파드가 모르게 우선순위를 얻지 않게 하려는 것이다.
- 클래스를 만드는 것은 아무 파드도 바꾸지 않는다. **붙이는 것은 파드 재시작을 부르므로**
k3s 재시작 창에 한다(`infrastructure/host/k3s/README.md`).
k3s 재시작 창에 한다(`docs/runbooks/k3s-restart-window.md`).
- **클래스를 먼저 만든다.** 없는 클래스를 가리키는 파드는 생성이 거절돼 Deployment가 새 파드를 못 만든다.
```sh
kubectl apply -k services/priority-classes