S3 Backups

Use S3 or an S3-compatible object store for production backups that must survive cluster or site loss.

Prerequisites

• A bucket, prefix, and region or endpoint URL. With an endpoint URL and no region in the profile or the credentials Secret, the operator's own S3 clients (artifact cleanup, encrypted upload and download, PITR archiving) sign requests for us-east-1, which S3-compatible stores accept. Artifact cleanup also signs for us-east-1 on plain AWS when no region is configured anywhere, as mysqlsh does when it writes the dump.
• Credentials that can put, get, list, and delete objects under the backup prefix (Backup credential permissions).
• spec.credentials.backupSecretName or operator credentials with backup privileges.
• Network egress from backup Jobs to the S3 endpoint.
• The operator image pullable in the group's namespace: deleting a MysqlBackup runs bloodraven artifact-cleanup from it, which needs s3:ListBucket and s3:DeleteObject on the prefix.

Credentials Secret

kubectl create secret generic orders-s3-backup -n orders \
  --from-literal=AWS_ACCESS_KEY_ID='replace-with-access-key' \
  --from-literal=AWS_SECRET_ACCESS_KEY='replace-with-secret-key' \
  --from-literal=AWS_REGION='us-east-1'

Profile and schedule

apiVersion: shipstream.io/v1alpha1
kind: MysqlFailoverGroup
metadata:
  name: orders
  namespace: orders
spec:
  backup:
    image: container-registry.oracle.com/mysql/community-server:9.7
    maxLagSecondsForSource: 300
    profiles:
      - name: s3-daily
        storage:
          type: S3
          s3:
            bucket: orders-mysql-backups
            prefix: prod/orders
            region: us-east-1
            credentialsSecretName: orders-s3-backup
        dump:
          threads: 4
          compression: zstd
        retention:
          keepLast: 14
          keepWithinDays: 35
        verification:
          enabled: true
          schedule: "30 8 * * *"
    schedules:
      - name: nightly
        profileName: s3-daily
        schedule: "0 6 * * *"
        timeZone: Etc/UTC
        concurrencyPolicy: Forbid

The snippet shows only spec.backup; keep your existing sites, credentials, tls, and dns fields in the same failover group.

The credentialsSecretName must contain non-empty AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. Backup, restore, and cleanup Jobs fail when either key is missing; they do not fall back to IRSA, Workload Identity, or instance-profile credentials. The operator checks the Secret on every reconcile and reports a gap through the S3CredentialsReady condition and an S3CredentialsIncomplete warning event, before the next scheduled Job fails.

On-demand backup

apiVersion: shipstream.io/v1alpha1
kind: MysqlBackup
metadata:
  name: orders-manual-20260427
  namespace: orders
  annotations:
    bloodraven.shipstream.io/triggered-by: manual
spec:
  failoverGroupRef:
    name: orders
  profileName: s3-daily

Apply and check status:

kubectl apply -f orders-manual-backup.yaml
kubectl get mysqlbackup orders-manual-20260427 -n orders -o wide
kubectl describe mysqlbackup orders-manual-20260427 -n orders

List artifacts

aws s3 ls s3://orders-mysql-backups/prod/orders/ --recursive

For MinIO or another compatible store:

AWS_ACCESS_KEY_ID=minio AWS_SECRET_ACCESS_KEY=minio123 \
aws --endpoint-url https://minio.example.com \
  s3 ls s3://orders-mysql-backups/prod/orders/ --recursive

S3-compatible endpoint

storage:
  type: S3
  s3:
    bucket: orders-mysql-backups
    prefix: lab/orders
    region: us-east-1
    endpointURL: https://minio.minio.svc.cluster.local:9000
    credentialsSecretName: orders-s3-backup

endpointURL must be an absolute http:// or https:// URL with no credentials, query string, or fragment. In-cluster and private addresses are allowed. A MysqlStandbyCluster source is read by the operator pod itself, so whoever can create one can make the operator send S3 requests, signed with that object's credentials, to any address the operator can reach.

Backup credential permissions

The profile's credentials Secret is used by the backup Job, by artifact cleanup when a MysqlBackup is deleted, and by restores and verifications that read the dump. On <prefix>/* they need:

A role with put and list but no get fails every backup with backup stamp: read …: AccessDenied.

The prefix is normalized: leading and trailing / are ignored, so /prod/orders and prod/orders name the same dumps, stamps and archive. It is at most 512 characters, and empty segments (a//b) and . or .. segments are refused at admission, because the dump writer and the archive readers would otherwise resolve it to different keys. A MysqlBackup name may not end in .bloodraven-backup.json.

A backup never dumps into a location that already holds objects without its stamp: the stamp is what proves whose objects a location holds, and cleanup of an abandoned backup deletes everything under it. The Job then fails with backup stamp conflict naming the first object; delete the objects, or use another backup name or prefix.

Backups that never succeeded

A backup Job writes its stamp before the dump. When a backup that never succeeded (Failed, or deleted while Running) is deleted, the finalizer stops its Job and runs artifact cleanup in abandoned mode at the planned location: it deletes the stamp and whatever part of a dump is there, but only when the stamp names that backup's UID. An encrypted backup's stamp is encrypted, so the cleanup Job mounts the passphrase Secret. This cleanup is best effort: if it cannot run or fails, the finalizer is released with an ArtifactCleanupSkipped Warning naming what is left. The same happens when there is no stamp at the location but objects remain under it (the cleanup Job exits 0 with BLOODRAVEN_CLEANUP_SKIPPED): nothing is deleted, and an encrypted backup's dump-key snapshot is kept, because those objects may be its artifacts. A backup re-created with the same name replaces an unfinished stamp of an earlier backup of that name when no dump object remains under the location, and otherwise fails with backup stamp conflict naming the first remaining object.

PITR archive requirements

When the profile is spec.backup.pitr.profileName, every site's sidecar writes the binlog archive under <prefix>/binlogs/. Its credentials need:

• s3:ListBucket on the bucket. The sidecar lists the prefix to classify it before creating the archive descriptor; without it the store reports archive store init: … naming s3:ListBucket and PITRArchiveReady=False / ArchiveStoreInitFailed.
• s3:PutObject, s3:GetObject and s3:DeleteObject on the prefix, for binlogs, manifests, the descriptor, the conditional-write probe and retention deletes.

The store must enforce conditional writes. The archive descriptor (<prefix>/binlogs/archive.json) is created with If-None-Match: * so two sites can never both create it. Before creating it, the sidecar probes the store: it creates <prefix>/binlogs/.bloodraven-cas-probe-<16 hex>, creates the same key again, and deletes it. Only a 412 Precondition Failed on the second create certifies the store. A store that accepts both creates, or rejects the header (501, NotImplemented, or a 400 naming it), refuses PITRArchiveStoreUnsupported and archives nothing. A 409 ConditionalRequestConflict that persists after retries is inconclusive: the sidecar reports ArchiveStoreInitFailed and probes again on the next cycle. The probe runs only while the descriptor is absent.

• AWS S3, MinIO and RustFS 1.0 or later enforce conditional writes.
• Active-active multi-zone or multi-site buckets (where each zone accepts writes and replicates asynchronously) are unsupported for archive writing: the precondition is checked per zone, so it cannot prevent two creates. Replicating the archive one way to a read-only copy is fine; see Restoring from a replicated archive.

S3 prefixes are normalized: leading, trailing and repeated / are dropped, so /prod/orders and prod/orders are the same archive. Backup stamps (<location>.bloodraven-backup.json) also use a conditional create, but fall back to a plain write plus a read-back compare, so backups themselves work on stores without conditional writes.

Restore from S3

apiVersion: shipstream.io/v1alpha1
kind: MysqlFailoverGroup
metadata:
  name: orders-restore
  namespace: orders
spec:
  initFromBackup:
    source:
      s3:
        bucket: orders-mysql-backups
        prefix: prod/orders/orders-manual-20260427
        region: us-east-1
        credentialsSecretName: orders-s3-backup

Include the normal sites, credentials, and dns fields in the recovery failover group.

Common failures

Next steps

• Backup Verification
• Backup Encryption
• Backup And Restore Reference