Nothing backs up the k3s datastore or Gitea's repositories, packages and app.ini. W1 covers platform-postgres only. One SSD failure loses the source, CI artefacts, published libraries and every cluster Secret. Add two root systemd timers that write to a new AIStor bucket on the HDD (platform-host-backups, same AIStor as W1's postgres backups): - k3s datastore, daily: sqlite3 .backup of the live state.db plus the server token (k3s backup/restore docs: the token is needed to read it back), integrity_check, tar.gz, sha256, upload. Off-host copy is prepared behind two variables (gpg recipient file, rclone target) and stays off until the owner chooses a place. - Gitea, weekly: gitea dump --skip-db streamed from the live container, listing checked for app.ini and repositories, sha256, upload. The DB is in the CNPG backup. Each run writes textfile metrics for the alerts in the next commit. create-host-backup-credentials.sh makes the bucket (50GiB quota, lifecycle 56 days for gitea/, 30 days for k3s-datastore/), a bucket-scoped policy and user, and a root-only env file; no key is printed and mc gets keys only through a 0600 env file. Rejected: in-cluster CronJobs. The datastore is a root-owned host file and the dump needs the Gitea DB connection; a hostPath root pod or a second Gitea-labelled pod is a larger privilege than a root timer. Risk: not installed here (owner sudo steps). The k3s archive holds the token; together with the datastore it decrypts every Secret. It lives in a bucket only the backup user can read. Verification (desktop): a dump streamed with --file - is a valid tar.gz with app.ini; repositories appear as repos/<owner>/<name>.git (29, matching the live count). The scripts parse with bash -n.
82 lines
4.1 KiB
Bash
82 lines
4.1 KiB
Bash
#!/usr/bin/env bash
|
|
# Create the AIStor bucket, lifecycle rules, policy and user that the host backup
|
|
# timers use, and the root-only /etc/platform-backup/aistor.env.
|
|
#
|
|
# sudo bash scripts/bootstrap/create-host-backup-credentials.sh
|
|
#
|
|
# Run once, by the owner, on the desktop. Refuses to run when the env file exists;
|
|
# never rotates or deletes a bucket, policy, user or object. No key is printed.
|
|
set -Eeuo pipefail
|
|
set +x
|
|
umask 077
|
|
|
|
readonly ROOT="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/../.." && pwd -P)"
|
|
# shellcheck source=../lib/host-backup.sh
|
|
source "$ROOT/scripts/lib/host-backup.sh"
|
|
readonly BUCKET=platform-host-backups
|
|
readonly POLICY=platform-host-backups
|
|
readonly QUOTA=50GiB
|
|
readonly POLICY_FILE="$ROOT/services/platform-host-checks/bucket-policy.json"
|
|
readonly PASS_LINE=PLATFORM_HOST_BACKUP_BOOTSTRAP_PASS
|
|
kubectl() { command /usr/local/bin/k3s kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml "$@"; }
|
|
|
|
[[ "$(id -u)" -eq 0 ]] || hb_fail "run with sudo"
|
|
for c in docker jq openssl; do command -v "$c" >/dev/null || hb_fail "missing command: $c"; done
|
|
jq -e . "$POLICY_FILE" >/dev/null || hb_fail "bucket policy is not valid JSON"
|
|
[[ ! -e "$HOST_BACKUP_ENV" ]] || hb_fail "$HOST_BACKUP_ENV already exists; this script does not rotate it"
|
|
|
|
ip="$(kubectl -n object-storage get service minio -o jsonpath='{.spec.clusterIP}')"
|
|
[[ -n "$ip" ]] || hb_fail "object-storage/minio Service not found"
|
|
endpoint="http://${ip}:80"
|
|
|
|
work="$(mktemp -d)"
|
|
trap 'rm -rf -- "$work"' EXIT
|
|
kubectl -n object-storage get secret aistor-root-configuration -o jsonpath='{.data.config\.env}' | base64 -d >"$work/root.env"
|
|
grep -q '^export MINIO_ROOT_USER=' "$work/root.env" || hb_fail "aistor-root-configuration has an unexpected format"
|
|
printf 'pcbackup%s' "$(openssl rand -hex 8)" >"$work/ACCESS_KEY"
|
|
openssl rand -hex 32 | tr -d '\n' >"$work/SECRET_KEY"
|
|
cp "$POLICY_FILE" "$work/policy.json"
|
|
|
|
printf '%s\n' \
|
|
"Bucket: $BUCKET (quota $QUOTA; expire gitea/ after 56 days, k3s-datastore/ after 30 days) on $endpoint" \
|
|
"Policy: $POLICY (${POLICY_FILE#"$ROOT/"})" \
|
|
"User: a new generated access key attached to $POLICY" \
|
|
"File: $HOST_BACKUP_ENV (root, 0600)" \
|
|
'No key value will be printed.'
|
|
printf 'Type APPLY to create them: '
|
|
IFS= read -r confirmation
|
|
[[ "$confirmation" == "APPLY" ]] || hb_fail "confirmation did not match APPLY"
|
|
|
|
# Every mc call is silenced so no key reaches the terminal.
|
|
docker run --rm --network host -v "$work:/run/b:ro" --tmpfs /tmp -e MC_CONFIG_DIR=/tmp/mc \
|
|
--entrypoint /bin/sh "$MC_IMAGE" -ec '
|
|
. /run/b/root.env
|
|
ak="$(cat /run/b/ACCESS_KEY)"; sk="$(cat /run/b/SECRET_KEY)"
|
|
mc alias set root "'"$endpoint"'" "$MINIO_ROOT_USER" "$MINIO_ROOT_PASSWORD" --api S3v4 >/dev/null
|
|
mc mb --ignore-existing "root/'"$BUCKET"'" >/dev/null
|
|
mc quota set "root/'"$BUCKET"'" --size "'"$QUOTA"'" >/dev/null
|
|
mc ilm rule add --prefix gitea/ --expire-days 56 "root/'"$BUCKET"'" >/dev/null
|
|
mc ilm rule add --prefix k3s-datastore/ --expire-days 30 "root/'"$BUCKET"'" >/dev/null
|
|
mc admin policy create root "'"$POLICY"'" /run/b/policy.json >/dev/null
|
|
mc admin user add root "$ak" "$sk" >/dev/null
|
|
mc admin policy attach root "'"$POLICY"'" --user "$ak" >/dev/null
|
|
mc alias set backup "'"$endpoint"'" "$ak" "$sk" --api S3v4 >/dev/null
|
|
probe=".bootstrap-probe-$(date +%s)"
|
|
printf probe | mc pipe "backup/'"$BUCKET"'/$probe" >/dev/null
|
|
[ "$(mc cat "backup/'"$BUCKET"'/$probe")" = probe ]
|
|
mc rm "backup/'"$BUCKET"'/$probe" >/dev/null
|
|
if mc ls backup/loki >/dev/null 2>&1; then echo "the backup user can list another bucket" >&2; exit 1; fi
|
|
echo "'"$PASS_LINE"'"' | grep -qx "$PASS_LINE" || hb_fail "bootstrap did not finish; $HOST_BACKUP_ENV was not written"
|
|
|
|
install -d -m 0700 -o root -g root "$(dirname "$HOST_BACKUP_ENV")"
|
|
{
|
|
echo "AISTOR_BUCKET=$BUCKET"
|
|
echo "AISTOR_ACCESS_KEY=$(cat "$work/ACCESS_KEY")"
|
|
echo "AISTOR_SECRET_KEY=$(cat "$work/SECRET_KEY")"
|
|
echo "# Off-host encrypted copy of the k3s datastore archive (off until both are set):"
|
|
echo "# OFFHOST_GPG_RECIPIENT_FILE=/etc/platform-backup/offhost-recipient.asc"
|
|
echo "# OFFHOST_RCLONE_TARGET=remote:path"
|
|
} >"$HOST_BACKUP_ENV"
|
|
chmod 0600 "$HOST_BACKUP_ENV"
|
|
printf '%s\n' "$PASS_LINE" "Created bucket $BUCKET, policy $POLICY, one user and $HOST_BACKUP_ENV."
|