feat(backup): back up the k3s datastore and Gitea to AIStor

Nothing backs up the k3s datastore or Gitea's repositories, packages and
app.ini. W1 covers platform-postgres only. One SSD failure loses the
source, CI artefacts, published libraries and every cluster Secret.

Add two root systemd timers that write to a new AIStor bucket on the
HDD (platform-host-backups, same AIStor as W1's postgres backups):
- k3s datastore, daily: sqlite3 .backup of the live state.db plus the
  server token (k3s backup/restore docs: the token is needed to read
  it back), integrity_check, tar.gz, sha256, upload. Off-host copy is
  prepared behind two variables (gpg recipient file, rclone target) and
  stays off until the owner chooses a place.
- Gitea, weekly: gitea dump --skip-db streamed from the live container,
  listing checked for app.ini and repositories, sha256, upload. The DB
  is in the CNPG backup.
Each run writes textfile metrics for the alerts in the next commit.
create-host-backup-credentials.sh makes the bucket (50GiB quota,
lifecycle 56 days for gitea/, 30 days for k3s-datastore/), a
bucket-scoped policy and user, and a root-only env file; no key is
printed and mc gets keys only through a 0600 env file.

Rejected: in-cluster CronJobs. The datastore is a root-owned host file
and the dump needs the Gitea DB connection; a hostPath root pod or a
second Gitea-labelled pod is a larger privilege than a root timer.

Risk: not installed here (owner sudo steps). The k3s archive holds the
token; together with the datastore it decrypts every Secret. It lives
in a bucket only the backup user can read.

Verification (desktop): a dump streamed with --file - is a valid
tar.gz with app.ini; repositories appear as repos/<owner>/<name>.git
(29, matching the live count). The scripts parse with bash -n.
This commit is contained in:
DongHyeonka
2026-11-12 01:13:02 +09:00
parent b484110b6b
commit 77daa39a41
9 changed files with 341 additions and 0 deletions
+55
View File
@@ -0,0 +1,55 @@
#!/usr/bin/env bash
# Shared helpers for the host backup and host check scripts. Source only.
# shellcheck shell=bash
readonly HOST_BACKUP_ENV=/etc/platform-backup/aistor.env
readonly TEXTFILE_DIR=/var/lib/node_exporter/textfile_collector
# Backup staging (root 0700, same SSD). Not /var/tmp: dockerd must see the path.
readonly HB_WORK_ROOT=/var/lib/platform-backup/work
# Same pinned AIStor client as scripts/validate/aistor-s3-smoke.sh.
readonly MC_IMAGE="quay.io/minio/aistor/mc@sha256:c5ec777c080fd6292b7529309a0f311c58c3a92e268c9c57c0901da86881949b"
hb_log() { printf '%s %s\n' "$(date -u +%FT%TZ)" "$*"; }
hb_fail() { hb_log "FAIL: $*" >&2; exit 1; }
# Write Prometheus textfile metrics atomically: hb_write_metrics NAME <<'PROM' ... PROM
hb_write_metrics() {
local name="$1" tmp
[[ -d "$TEXTFILE_DIR" ]] || { hb_log "textfile dir $TEXTFILE_DIR missing; metrics not written"; cat >/dev/null; return 0; }
tmp="$(mktemp "$TEXTFILE_DIR/.${name}.XXXXXX")"
cat >"$tmp"
chmod 0644 "$tmp"
mv -f -- "$tmp" "$TEXTFILE_DIR/${name}.prom"
}
# Load the AIStor endpoint and the backup user's keys (root-only file).
hb_load_aistor_env() {
[[ -f "$HOST_BACKUP_ENV" ]] || hb_fail "$HOST_BACKUP_ENV is missing; run scripts/bootstrap/create-host-backup-credentials.sh"
[[ "$(stat -c '%u %a' "$HOST_BACKUP_ENV")" == "0 600" ]] || hb_fail "$HOST_BACKUP_ENV must be owned by root with mode 0600"
# shellcheck disable=SC1090
. "$HOST_BACKUP_ENV"
: "${AISTOR_BUCKET:?}" "${AISTOR_ACCESS_KEY:?}" "${AISTOR_SECRET_KEY:?}"
# The host cannot resolve cluster DNS, so use the Service ClusterIP (port 80 -> 9000).
if [[ -z "${AISTOR_ENDPOINT:-}" ]]; then
local ip
ip="$(/usr/local/bin/k3s kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml -n object-storage \
get service minio -o jsonpath='{.spec.clusterIP}')"
[[ -n "$ip" ]] || hb_fail "cannot find the object-storage/minio Service"
AISTOR_ENDPOINT="http://${ip}:80"
fi
}
# Run the pinned mc client on the host network with the backup user's keys.
# The keys go through a 0600 env file, never the command line (ps would show it).
# Usage: hb_mc <dir mounted at /work> <mc args...>
hb_mc() {
local work="$1"; shift
local envfile="$work/.mc.env"
if [[ ! -f "$envfile" ]]; then
( umask 077; printf 'MC_HOST_backup=%s\nMC_CONFIG_DIR=/tmp/mc\n' \
"${AISTOR_ENDPOINT/:\/\//://${AISTOR_ACCESS_KEY}:${AISTOR_SECRET_KEY}@}" >"$envfile" )
fi
docker run --rm --network host --user "$(id -u):$(id -g)" \
--env-file "$envfile" -v "$work:/work" --tmpfs /tmp \
"$MC_IMAGE" "$@"
}