project-gitops/docs/runbooks/sealed-secrets-recovery.md

Sealed Secrets backup and recovery

Existing SealedSecrets are decryptable only with a controller private key. Back up every key after first install and after rotation.

Backup

Use an encrypted operator workstation:

umask 077
kubectl -n kube-system get secret \
  -l sealedsecrets.bitnami.com/sealed-secrets-key \
  -o json > .local/sealed-secrets-keys.json
chmod 0600 .local/sealed-secrets-keys.json

Encrypt the file with the organization's recovery mechanism, store at least two independently controlled copies, and delete the plaintext working copy. Record cluster, date and checksum without recording private key data.

Restore

Restore keys before application SealedSecrets sync:

kubectl -n kube-system apply -f .local/sealed-secrets-keys.json
kubectl -n kube-system rollout restart deployment/sealed-secrets-controller
kubectl -n kube-system rollout status deployment/sealed-secrets-controller

Verify both auth-dev/ghcr-regcred and api-dev/ghcr-regcred are created.

If no key backup exists, old ciphertext cannot be recovered. Create a new controller key and reseal every Secret from the original credential source.

Rotation

The chart requests periodic key renewal. Old keys must remain until all ciphertext has been resealed and verified. GHCR credential rotation requires:

1. issue an organization-owned read-only package credential;
2. create namespace-scoped SealedSecrets with the current controller cert;
3. merge and verify image pulls;
4. revoke the previous credential;
5. update the encrypted key backup.