feat(backup): back up the k3s datastore and Gitea to AIStor
Nothing backs up the k3s datastore or Gitea's repositories, packages and app.ini. W1 covers platform-postgres only. One SSD failure loses the source, CI artefacts, published libraries and every cluster Secret. Add two root systemd timers that write to a new AIStor bucket on the HDD (platform-host-backups, same AIStor as W1's postgres backups): - k3s datastore, daily: sqlite3 .backup of the live state.db plus the server token (k3s backup/restore docs: the token is needed to read it back), integrity_check, tar.gz, sha256, upload. Off-host copy is prepared behind two variables (gpg recipient file, rclone target) and stays off until the owner chooses a place. - Gitea, weekly: gitea dump --skip-db streamed from the live container, listing checked for app.ini and repositories, sha256, upload. The DB is in the CNPG backup. Each run writes textfile metrics for the alerts in the next commit. create-host-backup-credentials.sh makes the bucket (50GiB quota, lifecycle 56 days for gitea/, 30 days for k3s-datastore/), a bucket-scoped policy and user, and a root-only env file; no key is printed and mc gets keys only through a 0600 env file. Rejected: in-cluster CronJobs. The datastore is a root-owned host file and the dump needs the Gitea DB connection; a hostPath root pod or a second Gitea-labelled pod is a larger privilege than a root timer. Risk: not installed here (owner sudo steps). The k3s archive holds the token; together with the datastore it decrypts every Secret. It lives in a bucket only the backup user can read. Verification (desktop): a dump streamed with --file - is a valid tar.gz with app.ini; repositories appear as repos/<owner>/<name>.git (29, matching the live count). The scripts parse with bash -n.
This commit is contained in:
@@ -0,0 +1,55 @@
|
||||
#!/usr/bin/env bash
|
||||
# Shared helpers for the host backup and host check scripts. Source only.
|
||||
# shellcheck shell=bash
|
||||
|
||||
readonly HOST_BACKUP_ENV=/etc/platform-backup/aistor.env
|
||||
readonly TEXTFILE_DIR=/var/lib/node_exporter/textfile_collector
|
||||
# Backup staging (root 0700, same SSD). Not /var/tmp: dockerd must see the path.
|
||||
readonly HB_WORK_ROOT=/var/lib/platform-backup/work
|
||||
# Same pinned AIStor client as scripts/validate/aistor-s3-smoke.sh.
|
||||
readonly MC_IMAGE="quay.io/minio/aistor/mc@sha256:c5ec777c080fd6292b7529309a0f311c58c3a92e268c9c57c0901da86881949b"
|
||||
|
||||
hb_log() { printf '%s %s\n' "$(date -u +%FT%TZ)" "$*"; }
|
||||
hb_fail() { hb_log "FAIL: $*" >&2; exit 1; }
|
||||
|
||||
# Write Prometheus textfile metrics atomically: hb_write_metrics NAME <<'PROM' ... PROM
|
||||
hb_write_metrics() {
|
||||
local name="$1" tmp
|
||||
[[ -d "$TEXTFILE_DIR" ]] || { hb_log "textfile dir $TEXTFILE_DIR missing; metrics not written"; cat >/dev/null; return 0; }
|
||||
tmp="$(mktemp "$TEXTFILE_DIR/.${name}.XXXXXX")"
|
||||
cat >"$tmp"
|
||||
chmod 0644 "$tmp"
|
||||
mv -f -- "$tmp" "$TEXTFILE_DIR/${name}.prom"
|
||||
}
|
||||
|
||||
# Load the AIStor endpoint and the backup user's keys (root-only file).
|
||||
hb_load_aistor_env() {
|
||||
[[ -f "$HOST_BACKUP_ENV" ]] || hb_fail "$HOST_BACKUP_ENV is missing; run scripts/bootstrap/create-host-backup-credentials.sh"
|
||||
[[ "$(stat -c '%u %a' "$HOST_BACKUP_ENV")" == "0 600" ]] || hb_fail "$HOST_BACKUP_ENV must be owned by root with mode 0600"
|
||||
# shellcheck disable=SC1090
|
||||
. "$HOST_BACKUP_ENV"
|
||||
: "${AISTOR_BUCKET:?}" "${AISTOR_ACCESS_KEY:?}" "${AISTOR_SECRET_KEY:?}"
|
||||
# The host cannot resolve cluster DNS, so use the Service ClusterIP (port 80 -> 9000).
|
||||
if [[ -z "${AISTOR_ENDPOINT:-}" ]]; then
|
||||
local ip
|
||||
ip="$(/usr/local/bin/k3s kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml -n object-storage \
|
||||
get service minio -o jsonpath='{.spec.clusterIP}')"
|
||||
[[ -n "$ip" ]] || hb_fail "cannot find the object-storage/minio Service"
|
||||
AISTOR_ENDPOINT="http://${ip}:80"
|
||||
fi
|
||||
}
|
||||
|
||||
# Run the pinned mc client on the host network with the backup user's keys.
|
||||
# The keys go through a 0600 env file, never the command line (ps would show it).
|
||||
# Usage: hb_mc <dir mounted at /work> <mc args...>
|
||||
hb_mc() {
|
||||
local work="$1"; shift
|
||||
local envfile="$work/.mc.env"
|
||||
if [[ ! -f "$envfile" ]]; then
|
||||
( umask 077; printf 'MC_HOST_backup=%s\nMC_CONFIG_DIR=/tmp/mc\n' \
|
||||
"${AISTOR_ENDPOINT/:\/\//://${AISTOR_ACCESS_KEY}:${AISTOR_SECRET_KEY}@}" >"$envfile" )
|
||||
fi
|
||||
docker run --rm --network host --user "$(id -u):$(id -g)" \
|
||||
--env-file "$envfile" -v "$work:/work" --tmpfs /tmp \
|
||||
"$MC_IMAGE" "$@"
|
||||
}
|
||||
Reference in New Issue
Block a user