Backup and restore

PVC Backups

PVC-backed backups are useful for labs, isolated clusters, and short-retention local copies. They are not a disaster-recovery substitute for off-cluster storage.

When to use PVC backups

Use PVC backups when:

  • You are testing backup and restore behavior locally.
  • Your cluster has a durable shared storage system.
  • You need a short-lived staging copy before exporting elsewhere.

Avoid PVC backups when:

  • Cluster loss must not lose backups.
  • The StorageClass is node-local.
  • The PVC cannot be mounted by backup, restore, and cleanup Jobs.

Operator-managed PVC

backup:
  profiles:
    - name: pvc-local
      storage:
        type: PVC
        pvc:
          provisioned:
            storageClassName: fast-ssd
            size: 500Gi
          subPath: orders
      retention:
        keepLast: 7
  schedules:
    - name: nightly
      profileName: pvc-local
      schedule: "0 6 * * *"
      timeZone: Etc/UTC
      concurrencyPolicy: Forbid

pvc.provisioned makes the operator create and own the claim mysql-<group>-backup-<profile> with that class and size; without storageClassName the cluster's default StorageClass is used. Set exactly one of provisioned and claimName.

provisioned is applied only when the claim is created, so admission refuses changing it, adding it, or removing it for as long as the profile stays in spec.backup.profiles. That includes switching between provisioned and claimName, and switching the profile's storage to S3 and back. A size is compared by value, so rewriting 500Gi as 512000Mi is not a change. To grow the volume, expand the claim itself if its StorageClass allows it; the operator never changes an existing claim.

If a profile is removed and later added back under the same name with another class or a larger size, the claim the earlier profile created no longer matches. Each backup checks the claim against provisioned before its Job starts:

  • storageClassName unset matches any class on the claim (the cluster fills in its default class).
  • A set storageClassName must equal the claim's class.
  • The size is compared with the claim's request (spec.resources.requests.storage), not with the provisioned capacity. A larger request, such as a claim expanded out of band, is accepted with a Normal BackupStorageExpansionAccepted Event.
  • Another class or a smaller request fails the backup with Ready=False, reason BackupStorageMismatch, and a Warning Event of the same name. No backup Job writes to the claim. The check covers backup Jobs only: when spec.backup.pitr.profileName names this profile, the PITR archiver in the MySQL pods keeps writing binlogs to the claim.

To fix a mismatch, restore the profile's values, or replace the claim:

  1. If PITR archives to this profile, disable PITR (spec.backup.pitr.enabled: false) or switch spec.backup.pitr.profileName to another profile, and wait until every MySQL pod of the group has rolled, so no pod mounts the claim.
  2. Keep or copy what the claim holds.
  3. Delete the claim. The operator provisions a new one from provisioned.

mysql-<group>-backup-<profile> belongs to the group: deleting the group deletes it, and every backup on it, even a backup with deletionPolicy: Retain. To keep such backups, see Deleting a group without losing backups.

subPath is a directory inside the PVC; backups land under <subPath>/<backup-name>/. The API server rejects a subPath containing a .. segment, and backup and restore Jobs refuse one stored before that validation existed.

User-managed PVC

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: orders-backup-store
  namespace: orders
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: fast-ssd
  resources:
    requests:
      storage: 500Gi
---
apiVersion: shipstream.io/v1alpha1
kind: MysqlFailoverGroup
metadata:
  name: orders
  namespace: orders
spec:
  backup:
    profiles:
      - name: pvc-local
        storage:
          type: PVC
          pvc:
            claimName: orders-backup-store
            subPath: orders

On-demand backup

apiVersion: shipstream.io/v1alpha1
kind: MysqlBackup
metadata:
  name: orders-pvc-manual-20260427
  namespace: orders
  annotations:
    bloodraven.shipstream.io/triggered-by: manual
spec:
  failoverGroupRef:
    name: orders
  profileName: pvc-local
kubectl apply -f orders-pvc-backup.yaml
kubectl get mysqlbackup orders-pvc-manual-20260427 -n orders -o wide

Restore from PVC

apiVersion: shipstream.io/v1alpha1
kind: MysqlFailoverGroup
metadata:
  name: orders-restore
  namespace: orders
spec:
  initFromBackup:
    source:
      pvc:
        claimName: orders-backup-store
        subPath: orders/orders-pvc-manual-20260427

Include normal sites, credentials, and dns fields in the recovery failover group.

Sizing and retention

  • Size the PVC for peak dump size, compression ratio uncertainty, and retention overlap.
  • Keep at least two successful backups: retention.keepLast of 2 or more.
  • Watch PVC usage; a full PVC usually causes backup Jobs to fail after doing expensive work.

Backups that never succeeded

A backup Job writes its backup stamp (<backup>.bloodraven-backup.json beside the dump directory) before the dump. When a backup that never succeeded (Failed, or deleted while Running) is deleted, the finalizer stops its Job and runs artifact cleanup at the planned location: it deletes the stamp and any partial dump, but only when the stamp names that backup's UID. The cleanup is best effort; if it cannot run or fails, the finalizer is released with an ArtifactCleanupSkipped Warning naming what is left. The same happens when there is no stamp at the location but the directory still exists (the cleanup Job exits 0 with BLOODRAVEN_CLEANUP_SKIPPED): nothing is deleted, and an encrypted backup's dump-key snapshot is kept, because those files may be its artifacts. A backup re-created with the same name replaces an unfinished stamp of an earlier backup of that name when nothing remains under the location, and otherwise fails with backup stamp conflict naming the first remaining file.

More generally, a backup never dumps into a location that already holds files without its stamp, since abandoned cleanup deletes everything under a location whose stamp names its backup. It fails with backup stamp conflict naming the first file; delete the files, or use another backup name or subPath.

Common failures

FailureCheck
PVC fullkubectl describe pvc orders-backup-store -n orders, Job logs
Wrong access modeWhether the backup Job can mount the claim on its scheduled node
Node-local storageRestore Job scheduled to a different node cannot see data
Reclaim policyDeleting a claim might delete the only backup copy
Deleted MysqlBackup stuck TerminatingThe ArtifactCleanup condition on the CR. Cleanup mounts the claim recorded in status.artifactStorage and runs from the operator image; see Artifact cleanup
Restore fails with Read-only file system on load-progressThe restore Job was built by an operator that predates disabled load-progress tracking; upgrade it. The restore mounts the claim read-only by design