A Gitea dump must be restored with the Gitea version that wrote it, and
a restore checks the repository count against the dump. Neither was
recorded outside the archive, so the restore runbook had nothing to
compare with.
Upload a small .meta file with each dump: creation time, the
Deployment image, the gitea --version output and the repository count.
docs/runbooks/gitea-restore.md reads it.
Risk: none for the backup itself; the file holds no secret.
Verification: bash -n; kubectl -n gitea exec deploy/gitea -c gitea --
gitea --version prints "gitea version 1.27.0 built with ..." on the
desktop, so the third field is the version.
AIStor runs inside the k3s cluster that the datastore archive restores.
After losing the whole cluster the AIStor copy cannot be read until k3s
is running again, which is what the archive is for.
Also keep the seven newest archives in
/srv/k3s/aistor/host-backups/k3s-datastore (root 0700) on the same HDD
filesystem (sdb3). The directory is next to AIStor's drive directory
data/, so AIStor does not list it as a bucket, and it is readable by
mounting sdb3 alone. A metric reports how many local copies exist.
Rejected: the NTFS archive partition (sdb2). It has no Unix
permissions and Windows on the same machine can read it; the archive
holds the server token.
Risk: the files live inside the AIStor PersistentVolume root, which is
mounted into the MinIO pod (uid 1000); mode 0700 root keeps the pod
from reading them.
Verification: bash -n; /srv/k3s/aistor contains only data/ (2026-11-12).
A backup nobody has restored is not known to work. The 2026-08 k3s
recovery bundle never got its isolated restore drill.
k3s-datastore-restore-drill.sh downloads the latest archive, checks its
sha256 and integrity, starts the same k3s version in a network-less
throwaway container on the copy, and checks the API comes up, the
Secret count is close to live and every Secret decrypts with the
token. gitea-backup-restore-drill.sh extracts the latest dump and
checks app.ini (INTERNAL_TOKEN, SECRET_KEY), git fsck on every
repository and the repository count against live Gitea. Both accept a
local copy and write a drill timestamp metric.
Risk: read-only for the cluster. The k3s drill runs the restored
control plane with --network none, so its controllers reach nothing.
Verification (desktop):
- test-k3s-datastore-restore-drill.sh: a fresh k3s with three test
Secrets (stored encrypted, k8s:enc:) restores with all Secrets
decrypted in 7s; before folding the WAL the test saw 2 of 3 rows,
which is why the real backup uses sqlite3 .backup.
- gitea-backup-restore-drill.sh on a live repository dump: 29
repositories, 0 fsck failures, 29 live; the dump was deleted after.
Backups, the daily drift check and the k3s serving certificates have no
alert. k3s renews certificates only when it starts within 120 days of
expiry, and the two kube-prometheus-stack kubelet certificate alerts
can never fire because the k3s kubelet exports no
kubelet_certificate_manager_* metric.
Add a PrometheusRule (platform-host-checks) for backup age, failed
runs, missing metrics, drift left for 6h, a stale drift check, and
serving certificate expiry (warning under 100 days, critical under 14).
Add the two remaining host timers (certificate expiry via openssl on
6443 and 10250; the drift check against origin/main) and an installer
that copies the scripts to a root-owned path, creates the textfile and
work directories and enables all four timers. Disable the two dead
kubelet certificate rules in kube-prometheus-stack.
Risk: the metrics need the node-exporter textfile collector, a one-line
change in W1's node-exporter values that was asked of W1. Until then
PlatformHostBackupMetricsAbsent fires after a day, which is the point.
Verification (desktop):
- test-platform-host-checks-alerts.sh: promtool check and unit tests
pass; with the k3s threshold changed from 2 to 5 days they fail.
- render-observability-core.sh passes; the render has no Kubelet*
certificate alerts and still has both KubeClientCertificateExpiration.
- k3s-cert-expiry.sh as the desktop user reads both certificates.
- install-host-checks.sh --check: systemd-analyze verify is clean.
Nothing backs up the k3s datastore or Gitea's repositories, packages and
app.ini. W1 covers platform-postgres only. One SSD failure loses the
source, CI artefacts, published libraries and every cluster Secret.
Add two root systemd timers that write to a new AIStor bucket on the
HDD (platform-host-backups, same AIStor as W1's postgres backups):
- k3s datastore, daily: sqlite3 .backup of the live state.db plus the
server token (k3s backup/restore docs: the token is needed to read
it back), integrity_check, tar.gz, sha256, upload. Off-host copy is
prepared behind two variables (gpg recipient file, rclone target) and
stays off until the owner chooses a place.
- Gitea, weekly: gitea dump --skip-db streamed from the live container,
listing checked for app.ini and repositories, sha256, upload. The DB
is in the CNPG backup.
Each run writes textfile metrics for the alerts in the next commit.
create-host-backup-credentials.sh makes the bucket (50GiB quota,
lifecycle 56 days for gitea/, 30 days for k3s-datastore/), a
bucket-scoped policy and user, and a root-only env file; no key is
printed and mc gets keys only through a 0600 env file.
Rejected: in-cluster CronJobs. The datastore is a root-owned host file
and the dump needs the Gitea DB connection; a hostPath root pod or a
second Gitea-labelled pod is a larger privilege than a root timer.
Risk: not installed here (owner sudo steps). The k3s archive holds the
token; together with the datastore it decrypts every Secret. It lives
in a bucket only the backup user can read.
Verification (desktop): a dump streamed with --file - is a valid
tar.gz with app.ini; repositories appear as repos/<owner>/<name>.git
(29, matching the live count). The scripts parse with bash -n.
The 2026-11-11 review found a pg_hba rule that existed only in the
cluster by rendering every root by hand and running kubectl diff.
Nothing does that regularly, so drift is found by accident.
live-drift-check.sh renders all live roots (Phase 1 and 2, admin
services, observability core, the Traefik trust overlay, the tenant,
PriorityClass and keycloak policy roots), strips Secrets and runs
kubectl diff, which is a server-side dry run. It prints one line per
drifted object, honours a dated accepted-drift list and can write
Prometheus textfile metrics for a later alert.
Risk: read-only. It consumes the render scripts' verified-output-dir
handoff and deletes those directories on exit; the README notes this
second consumer. Secret values are not compared.
Verification (desktop, before this series is applied): 17 drifted
objects, matching the review plus the new roots; exit 1; the textfile
holds platform_live_drift_objects 17.
The tech-log tenant needs objects inside platform namespaces: its
database and role, ingress to Postgres and AIStor, blackbox probes,
alert rules, a dashboard and two observability NetworkPolicies. They
were only ever applied with kubectl and no repository holds them, so a
deleted object cannot be recreated and a platform re-apply can drop
tech-log access without anyone noticing.
Declare them in services/tenants/techlog, copied from each object's
last-applied configuration. The dashboard follows the repository's
dashboard pattern (JSON file + configMapGenerator, same name, same
labels). The hyeonworks-prod Namespace moves here as the tenant
boundary and gets PSA enforce=baseline, warn/audit=restricted.
Rejected: enforce=restricted now. A server dry-run shows all three
tech-log pods would violate it; baseline has zero violations.
Risk: the render matches the live objects exactly except the added PSA
labels. The pg_hba rules and the Secret stay out of this root (Cluster
owner and Git policy).
Verification (desktop):
- render-tenants-and-policies.sh passes.
- kubectl diff of the render shows only the Namespace PSA labels.
- kubectl label --dry-run=server enforce=baseline: no warnings.