Multi-cluster disaster recovery

This page is the operational runbook for recovering a MysqlFailoverGroup into a separate Kubernetes cluster when the source cluster is lost. It covers every step from topology to DNS cutover using only the existing MysqlFailoverGroup CRD surface (available today on main). No new CRDs or controllers are required to execute this runbook.

WISHLIST #7 introduces MysqlStandbyCluster as a passive observability layer that surfaces source-archive freshness via BucketReadable / SourceConfigKnown conditions, summarised by Ready. Phase 2 adds a readiness gate (Restorable); Phase 3 adds one-command activation. See What MysqlStandbyCluster adds at the bottom of this page for details on what ships in that first phase and what remains in later phases.

Topology overview

Cross-cluster DR in Bloodraven v1 uses the shared object store (S3 or compatible) as the only channel between the source and DR clusters. There is no cross-cluster replication link, no operator-to-operator RPC, and no federation.

Key points:

• The source cluster writes: full dumps (operator-driven backup Jobs) and sealed binlogs (sidecar archiver on the active primary, gated on !@@read_only). The upload switches to a new primary within one archiver scan cycle after an intra-cluster failover.
• The DR cluster only reads during bootstrap. It does not run a binlog archiver of its own — there is nothing to archive from a cluster that does not yet exist.
• DisasterRecovery site roles (SiteRoleDisasterRecovery in api/v1alpha1/types.go, Multi-site topology) are a separate concept: they are passive replicas inside the same cluster as the source MFG and are never auto-promoted. Cross-cluster DR is distinct from this.

Bucket and IAM layout

Bucket structure

Bloodraven writes to the bucket prefix configured in spec.backup.profiles[].storage.s3.prefix. The layout under that prefix:

<prefix>/
  <mysqlbackup-name>/          # one directory per successful full dump
    @.json                     # dump sentinel: GTIDs, size, completion time
    <shard-files>.sql.gz       # mysqlsh dumpInstance artifacts
  <mysqlbackup-name>.bloodraven-backup.json  # backup stamp (identity, server clock)
  binlogs/                     # PITR archive (when spec.backup.pitr.enabled=true)
    archive.json               # write-once archive descriptor (plaintext)
    manifest-<site>.json       # per-site sealed-binlog manifest (version 2)
    <site>/
      <binlog-file>.enc        # sealed binlogs (possibly encrypted)
  dr-cursors/                  # Phase 2: reserved, not written in this release

The DR cluster only needs read access to <prefix>/ to execute this runbook. Write access to dr-cursors/ is reserved for the MysqlStandbyCluster verification/readiness feature (Phase 2 of WISHLIST #7).

Minimum IAM policy — AWS

Grant the DR cluster's service account or IAM role the following policy. Replace shipstream-backups and orders/west with your actual bucket name and prefix.

DR cluster read-only policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DRReadOnly",
      "Effect": "Allow",
      "Action": ["s3:ListBucket", "s3:GetObject"],
      "Resource": [
        "arn:aws:s3:::shipstream-backups",
        "arn:aws:s3:::shipstream-backups/orders/west/*"
      ]
    }
  ]
}

Scope the ListBucket resource to the bucket name (not a prefix — S3 requires bucket-level ListBucket), and the GetObject resource to the prefix subtree. The DR restore Job uses the credentialsSecretName you reference in spec.initFromBackup.source.s3; that Secret must contain AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY for credentials that the above policy covers. Backup, restore, and cleanup Jobs fail if either key is missing or empty; they do not fall back to IRSA, Workload Identity, or instance-profile credentials.

GCS / S3-compatible (MinIO, Ceph)

Bloodraven uses the AWS SDK v2 with a configurable endpoint. Set spec.initFromBackup.source.s3.endpointURL (or spec.backup.profiles[].storage.s3.endpointURL on the source) to your endpoint. Minimum equivalent permissions:

Encryption passphrase distribution

When the source profile uses spec.backup.profiles[].encryption, every dump artifact and archived binlog is wrapped in the Bloodraven BRV1 format (AES-256-GCM envelope). The DR restore needs the keys they were written with, and those are not necessarily the bytes in the profile's Secret today: the source group records what it used in its own key snapshots, and keeps using them after a refused or in-progress rotation. Copy the snapshots, never the live profile Secret.

There are up to two keys:

• Dump key. Each backup records the snapshot it was encrypted with in status.artifactStorage.passphraseSecretKeyRef (a mysqlbackup-<name>-dump-key-… Secret).
• Archive key (PITR only). The source group's Bound binding for the archive's destination names it in encryption.keySecretName (a mysql-<group>-pitr-key-… Secret).

They can differ, so copy each one separately. The operator does not distribute keys across clusters. Before triggering a recovery:

Step 1. On the source cluster, read the two snapshot names: the selected backup's dump key, and the archive key of the destination you replay (copy the destination exactly as status.pitrArchiveBindings[] shows it):

DUMP_KEY=$(kubectl -n orders get mysqlbackup <backup> \
  -o jsonpath='{.status.artifactStorage.passphraseSecretKeyRef.name}')
ARCHIVE_KEY=$(kubectl -n orders get mfg orders -o json | jq -r --arg d '<destination>' \
  '.status.pitrArchiveBindings[] | select(.destination == $d and .state == "Bound") | .encryption.keySecretName')

An empty ARCHIVE_KEY means the destination has no Bound binding; do not substitute the profile's Secret. Resolve the binding first, or use the original passphrase from your own key backup.

Step 2. Copy each snapshot's passphrase through a temporary file on a trusted workstation into the DR namespace:

for pair in "orders-dr-dump-key:$DUMP_KEY" "orders-dr-archive-key:$ARCHIVE_KEY"; do
  PASSFILE=$(mktemp)
  kubectl --context=<source-context> -n orders get secret "${pair#*:}" \
    -o jsonpath='{.data.passphrase}' | base64 -d > "$PASSFILE"
  kubectl --context=<dr-context> -n orders create secret generic "${pair%%:*}" \
    --from-file=passphrase="$PASSFILE"
  shred -u "$PASSFILE"
done

Step 3. Configure each reader with its own key:

• the dump: spec.initFromBackup.decryption.passphraseSecretKeyRef.name: orders-dr-dump-key;
• the archive: the DR group's archive profile (the pointInTime.archiveProfileName profile on the source prefix) with encryption.passphraseSecretKeyRef.name: orders-dr-archive-key.

See Backup encryption for the field semantics and Archive binding and keys for how the restore checks the archive key.

If the archive key is wrong, the DR group's read binding is refused KeyMismatch before any restore Job is created, and the restore waits in Pending with a PITRArchiveBindingNotReady message: fix the Secret's bytes and the binding is checked again. A wrong dump key fails the restore Job's decrypt-download init container, because the envelope does not authenticate under it. The profile's live Secret is never a fallback for either key.

Source fencing checklist

Before you trigger a recovery into the DR cluster, confirm the source is genuinely down. Promoting the DR copy while the source is still accepting writes produces two writable clusters. Bloodraven v1 does not automatically detect or resolve cross-cluster split-brain — divergent GTIDs accumulate on the source side and can only be audited post-hoc (see Durability and RPO).

Use at least two of these three signals before proceeding:

Signal 1 — Source operator /active-site endpoint returns 5xx:

curl -f 'http://<source-operator-aux-svc>:8082/active-site?namespace=orders&group=orders'
# Healthy: {"activeSite":"iad"}
# Down:    curl: (22) The requested URL returned error: 503

The operator's auxiliary HTTP server (cmd/bloodraven/main.go:361) serves /active-site on port 8082 and requires namespace plus group query parameters. It returns 503 when it cannot reach any writable site.

Signal 2 — Source cluster API server is unreachable:

kubectl --context=<source-context> get nodes --request-timeout=10s
# Error from server: timeout waiting for server response

Signal 3 — Source MySQL endpoint is TCP-unreachable from a third vantage point:

# From a network location that is NOT one of the two clusters
nc -zv <source-mysql-lb-ip> 3306
# Connection refused / timeout

If any signal is ambiguous — for example, the API server is reachable but appears degraded — wait for a clearer signal rather than promoting. The cost of delaying a few minutes is far lower than the cost of a split-brain incident.

Recovery procedure

This walkthrough assumes:

• The source MysqlFailoverGroup is named orders, in namespace orders, and had PITR enabled under a profile named nightly-s3 stored at s3://shipstream-backups/orders/west.
• The DR cluster has network access to the same S3 bucket.
• The passphrase Secret has been mirrored (see above).
• The DR cluster has a fresh namespace orders with Bloodraven installed (same operator version as the source; see Install production).

Step 1 — Identify the recovery point

Locate the most recent successful dump in the source bucket:

aws s3 ls s3://shipstream-backups/orders/west/ --recursive \
  | grep '@\.json' | sort -k1,2 | tail -5

Each directory under orders/west/ that contains @.json is a completed dump. The @.json holds the completion timestamp, GTID set, and dump size. Pick the most recent @.json whose directory you want to recover from. Note its prefix — for example, orders/west/orders-nightly-20260520.

To target a specific point in time (e.g., 14:32 UTC before a bad migration), note the desired stopTime. Binlogs replayed after the dump will be bounded by the newest sealed binlog archived before the event; the unarchived tail on the lost primary's PVC is not recoverable (see PITR and the RPO window).

Step 2 — Apply the DR MysqlFailoverGroup

Create a MysqlFailoverGroup on the DR cluster configured for the DR environment's nodes, IPs, and DNS. Set spec.initFromBackup to pull from the source bucket.

Source group configuration (reference — source cluster):

apiVersion: shipstream.io/v1alpha1
kind: MysqlFailoverGroup
metadata:
  name: orders
  namespace: orders
spec:
  sites:
    - name: iad
      serverID: 101
      role: PrimaryCandidate
      zone: us-east-1a
      taintNodeSelector:
        bloodraven.shipstream.io/failover-group.orders: "true"
        bloodraven.shipstream.io/site.orders: iad
      loadBalancerIP: 10.0.10.11
      storage: { storageClassName: gp3, size: 500Gi }
    - name: iad-2
      serverID: 102
      role: PrimaryCandidate
      zone: us-east-1b
      taintNodeSelector:
        bloodraven.shipstream.io/failover-group.orders: "true"
        bloodraven.shipstream.io/site.orders: iad-2
      loadBalancerIP: 10.0.10.12
      storage: { storageClassName: gp3, size: 500Gi }
  credentials:
    operatorSecretName: mysql-operator-creds
  dns:
    hostname: orders-east.example.com
    ttlSeconds: 60
  backup:
    profiles:
      - name: nightly-s3
        storage:
          type: S3
          s3:
            bucket: shipstream-backups
            prefix: orders/west
            region: us-west-2
            credentialsSecretName: s3-backup-creds
        encryption:
          passphraseSecretKeyRef:
            name: orders-backup-passphrase
    pitr:
      enabled: true
      profileName: nightly-s3

DR cluster recovery manifest:

apiVersion: shipstream.io/v1alpha1
kind: MysqlFailoverGroup
metadata:
  name: orders
  namespace: orders
spec:
  # DR-local node topology — different nodes, IPs, and zones from source
  sites:
    - name: east-1
      serverID: 101
      role: PrimaryCandidate
      zone: us-east-1a
      taintNodeSelector:
        bloodraven.shipstream.io/failover-group.orders: "true"
        bloodraven.shipstream.io/site.orders: east-1
      loadBalancerIP: 10.1.20.11           # DR cluster's LB IP
      storage: { storageClassName: gp3, size: 500Gi }
    - name: east-2
      serverID: 102
      role: PrimaryCandidate
      zone: us-east-1b
      taintNodeSelector:
        bloodraven.shipstream.io/failover-group.orders: "true"
        bloodraven.shipstream.io/site.orders: east-2
      loadBalancerIP: 10.1.20.12
      storage: { storageClassName: gp3, size: 500Gi }
  credentials:
    operatorSecretName: mysql-operator-creds   # must exist in DR namespace
  dns:
    hostname: orders-east.example.com
    ttlSeconds: 60
  backup:
    profiles:
      - name: west-archive                           # source archive, read-only
        storage:
          type: S3
          s3:
            bucket: shipstream-backups
            prefix: orders/west
            region: us-west-2
            credentialsSecretName: s3-dr-readonly-creds
        encryption:
          passphraseSecretKeyRef:
            name: orders-backup-passphrase
      - name: east-nightly                           # this group's own archive
        storage:
          type: S3
          s3:
            bucket: shipstream-backups
            prefix: orders/east
            region: us-west-2
            credentialsSecretName: s3-backup-creds
        encryption:
          passphraseSecretKeyRef:
            name: orders-backup-passphrase
    pitr:
      enabled: true
      profileName: east-nightly
  initFromBackup:
    source:
      s3:
        bucket: shipstream-backups
        prefix: orders/west/orders-nightly-20260520  # exact dump directory
        region: us-west-2
        credentialsSecretName: s3-dr-readonly-creds      # DR read-only creds
    decryption:
      passphraseSecretKeyRef:
        name: orders-backup-passphrase               # mirrored from source
    pointInTime:                                     # omit if recovering to dump GTID
      stopTime: "2026-05-20T14:32:00Z"
      archiveProfileName: west-archive               # required: the source's archive
      # allowPartial: true                           # see below; off by default

The restore reads the source archive through west-archive, including every site manifest under it whatever the source's site names were, while the DR group archives into its own orders/east prefix. archiveProfileName is required, and it may not resolve to the DR group's own active PITR destination: pointing spec.backup.pitr at orders/west too fails the restore with RestoreBuildFailed / PITRArchiveDestinationShared before any Job is created. Its sidecars would refuse the source's archive anyway (PITRArchiveForeign), because the descriptor names the source group.

The example leaves allowPartial off. A direct dump is as well bounded as a backupRef one: dump-meta reads its GTID set from @.json. With allowPartial: true, a target later than anything the source managed to archive (the usual case when the source region failed) restores to the archive head instead of refusing PITRTargetNotCovered, and reports outcome: Partial with a Warning PITRReplayTruncated. Enable it only when "as late as the archive reaches" is an acceptable recovery point, and then compare newestReplayedCommitTime with your recovery objective before routing traffic. Without it, a refused target tells you to pick an earlier one.

Restoring from a replicated archive

If the source bucket is copied to the DR region (S3 replication, rclone, or a storage-level mirror), point the read-only profile at the copy, for example bucket: shipstream-backups-dr, and name it in archiveProfileName. Readers accept the copy: its archive.json still names the source group and its archive ID, and the restore checks the encryption, the passphrase's keyCheck, and that every manifest names that archive.

• Copies are read-only. A writer accepts a descriptor only when it names its own group's UID and an archive ID derived for its own destination. The DR group refuses the copy as PITRArchiveForeign. The source group, if it were ever pointed at the copy, refuses it as PITRArchiveCopy. Neither writes into it, so replication cannot fight a second writer.
• Replication lag shows as PITRArchiveDescriptorMissing. Replication does not preserve write order, so a copy can hold version 2 manifests before archive.json arrives. Restores and the standby reconciler refuse it with PITRArchiveDescriptorMissing until it does. Wait for replication, or copy archive.json from the source.
• Objects can arrive after the manifest that lists them. For the same reason a copied manifest can list a binlog object that has not been replicated yet. A restore then sees a missing or unreadable object and can refuse PITRArchiveObjectInvalid, PITRArchiveGap (naming an unreadable row), or PITRArchiveNamespaceReuse ("cannot be read and verified"). Their documented remedies (quarantine, a new prefix, a full backup) assume permanent damage. On a copy, first check replication is complete: compare the named object with the source archive, or wait for replication to catch up (S3 replication metrics, rclone check) and retry. Only an object missing at the source too is damage.
• The destination differs, the archive ID does not. The archive ID is derived from the source's own destination, so it stays the same in every copy. status.discovered.binlogArchive.identity.destination on a MysqlStandbyCluster shows where the copy was read.

A direct S3 dump works for PITR when it carries its backup stamp (<prefix>.bloodraven-backup.json, copied with the dump): dump-meta reads the backup GTID set from the dump's own @.json and checks the stamp, and refuses a dump without one (PITRBackupIdentityUnknown). A backupRef additionally checks the stamp against the MysqlBackup and its source group. The outcome is in status.restore.provenance.pointInTime; compare its outcome, newestReplayedCommitTime and gtidExecuted with your recovery objective.

Apply to the DR cluster:

kubectl --context=<dr-context> apply -f dr-orders.yaml

Step 3 — Monitor the restore

kubectl --context=<dr-context> -n orders get mysqlfailovergroup orders -w

Watch status.restore.phase. The sequence:

Full status:

kubectl --context=<dr-context> -n orders describe mysqlfailovergroup orders

The initFromBackup path is one-shot and runs before the failover group is considered ready (api/v1alpha1/backup_types.go:552-557). Once status.restore.phase == Succeeded, bootstrap (clone, replication, fencing) completes normally.

Step 4 — Verify the DR group is healthy

kubectl --context=<dr-context> -n orders get mysqlfailovergroup orders -o yaml \
  | grep -A5 'conditions:'

Wait for Ready=True. Check that at least one site is Writable and replication is up on the peer:

kubectl --context=<dr-context> -n orders get mysqlfailovergroup orders \
  -o jsonpath='{.status.activeSite}'

Step 5 — DNS cutover

Bloodraven writes a DNSEndpoint CR per site using external-dns/endpoint-controller.io/v1 (api/v1alpha1/types.go:371-384, internal/platform/dns.go:23-31). The operator controls the per-cluster DNS record for spec.dns.hostname in whatever cluster it runs in. It does not control the global application-facing record.

Choose one of these cutover patterns:

Option A — Weighted CNAME with health checks (recommended for production)

Maintain a weighted CNAME (Route53, Google Cloud DNS, Akamai GTM) or a GSLB record that points at both clusters. On source loss:

1. Set source weight to 0 (or let health checks drop it automatically if the source LB IP is unreachable).
2. Set DR weight to 100.
3. TTL propagation time depends on your DNS TTL. Pre-reducing TTL to ≤ 60 s before an expected maintenance window shortens this.

Option B — Manual flip (simplest)

# Point the application CNAME at the DR cluster's DNS name
aws route53 change-resource-record-sets --hosted-zone-id Z123 --change-batch '{
  "Changes": [{
    "Action": "UPSERT",
    "ResourceRecordSet": {
      "Name": "db.example.com",
      "Type": "CNAME",
      "TTL": 60,
      "ResourceRecords": [{"Value": "orders-east.example.com"}]
    }
  }]
}'

Option C — GSLB / Akamai GTM

Mark the source origin as offline in the GTM policy. The GSLB health monitor will have already removed it from rotation if the TCP probe to MySQL is failing.

After DNS propagation, confirm applications are connecting to the DR cluster (check status.activeSite and replication lag on the DR group).

Failback narrative

When the source cluster returns, the original path is reversible. This is a manual, prose-only procedure in v1; Phase 4 of WISHLIST #7 will document and partially automate it.

The high-level steps:

1. Confirm the DR cluster is steady-state primary. New writes are landing on the DR cluster; intra-cluster replication (if multi-site) is healthy.
2. Wipe the source cluster's MySQL state. Delete the old MysqlFailoverGroup CR. The operator cascades to Deployments, Services, and DNSEndpoints; PVCs are not cascaded — delete them explicitly:

   kubectl --context=<source-context> -n orders delete mysqlfailovergroup orders
   kubectl --context=<source-context> -n orders delete pvc \
     -l bloodraven.shipstream.io/failover-group=orders
   

3. Ensure the DR group's backup prefix does not collide with the original source prefix. A directional prefix convention avoids ambiguity — for example, use orders/east/ for the DR group's dumps and orders/west/ for the original source. Update the DR group's spec.backup.profiles[].storage.s3.prefix accordingly before the DR group takes its first backup.
4. Stand up a fresh MysqlFailoverGroup on the source cluster with spec.initFromBackup pointing at the DR cluster's new prefix (the backup that the now-primary DR cluster took). This is the same pattern as the original recovery, but in reverse. Wait for status.restore.phase=Succeeded.
5. Cut DNS back if desired. This step is optional and human-driven. Bloodraven v1 does not auto-rebalance traffic across clusters.

Phase 4 (WISHLIST #7) introduces MysqlStandbyCluster symmetric failback: a second standby CR on the returning source cluster pointing at the DR cluster's new prefix, with the same continuous verification and one-command activation path that Phase 3 provides for the forward direction.

What MysqlStandbyCluster (Phase 1) adds today

WISHLIST #7 Phase 1 introduces MysqlStandbyCluster — a new CRD that lives on the DR cluster and continuously monitors the source bucket.

What Phase 1 ships:

• A new MysqlStandbyCluster CR (shipstream.io/v1alpha1, short name msc) that names the relationship between a source MFG's backup archive and this DR cluster.
• A passive reconciler (MysqlStandbyClusterReconciler) that runs a discovery loop on a configurable cadence (default 5 minutes):
  • Lists objects under spec.source.s3.prefix to find the most recent successful full dump and per-site binlog manifests.
  • Populates status.discovered.latestDump (name, location, completionTime, gtidExecuted, sizeBytes), status.discovered.binlogArchive and status.discovered.lastScanTime.
  • Stamps BucketReadable=True/False based on whether the bucket scan succeeded.
  • Stamps SourceConfigKnown=True/False based on whether at least one dump and one binlog manifest were found.
  • Stamps Ready=True (reason SourceDiscovered) when both are True; otherwise Ready=False with the reason and message of the first of the two that is not (BucketReadable first).
  • Reads the archive descriptor and every version 2 manifest, and publishes the identity and the window together as status.discovered.binlogArchive: identity (archiveID, destination, source{namespace, name, uid}, encryptionMode, algorithm), manifestCount, fileCount, sizeBytes, oldestEventTime and newestEventTime. A refused archive sets SourceConfigKnown=False with the token minus PITR as the reason (ArchiveLegacy, ArchiveDescriptorMissing, ArchiveIdentityMismatch, ArchiveEncryptionMismatch, ArchiveKeyMismatch, ArchiveVersionUnsupported, ArchiveUnreadable), emits a Warning PITRArchiveRefused when the reason changes, and still publishes latestDump, with no binlogArchive. With source.decryption set the standby expects a Passphrase archive and checks the passphrase against the descriptor's keyCheck; without it, a None archive. The passphrase file is trimmed of leading and trailing \n, \r, space and tab only.
• No mysqld is started, no dump is loaded, and no MysqlFailoverGroup is materialized. Phase 1 is purely observational.

What Phase 1 does NOT ship:

• Continuous restore verification (Restorable condition) — Phase 2.
• One-command activation (dr-activate kubectl plugin + a future activate-request state machine) — Phase 3.
• Failback tooling — Phase 4.

The spec fields and status that drive Phase 2 verification and Phase 3 activation are not part of the API. They will be added (backward-compatibly) when that code ships, rather than shipping inert fields that enforce nothing today. That includes the shape of the group an activation would create: keep the MysqlFailoverGroup manifest you would apply in the DR cluster (with spec.initFromBackup pointed at this prefix) in Git next to the standby.

An on-call engineer recovering from a cluster loss still executes this runbook manually. Phase 1's value is operational visibility: you can see at a glance whether the source bucket is readable and whether a dump + binlog window exist, without having to run aws s3 ls yourself.

Sample MysqlStandbyCluster CR:

apiVersion: shipstream.io/v1alpha1
kind: MysqlStandbyCluster
metadata:
  name: orders-east-from-west
  namespace: orders
spec:
  transport: ObjectStore           # the only transport, and the default
  source:
    # The S3 destination of the source group's backup profile.
    s3:
      bucket: shipstream-backups
      prefix: orders/west          # required
      region: us-west-2
      credentialsSecretName: s3-dr-readonly-creds
    decryption:
      passphraseSecretKeyRef:
        name: orders-backup-passphrase
  freshness:
    discoveryInterval: 5m          # default 5m, minimum 30s

The standby identifies its source by the prefix alone; the archive descriptor it verifies there names the source group (namespace, name and UID). Use metadata labels if you want a display name for the source cluster. See examples/standby-cluster.yaml for the annotated sample.

Check status:

kubectl -n orders get mysqlstandbycluster orders-east-from-west -o yaml

Look for status.discovered.latestDump, status.discovered.binlogArchive (the verified archive identity and its window), and the Ready, BucketReadable and SourceConfigKnown conditions. kubectl get msc shows the latest dump's name, the newest archived event time and Ready.

Planned follow-up phases (tracking issue: WISHLIST #7):

• Phase 2 — Continuous DR readiness: the reconciler schedules periodic MysqlBackupVerification runs against the discovered dump + binlog window, publishing a Restorable condition and a bloodraven_dr_restorable_timestamp_seconds gauge. The source operator gains awareness of dr-cursors/ sentinel objects to prevent premature binlog pruning. Prerequisite: WISHLIST #43 (PITR E2E scenarios).
• Phase 3 — One-command activation: kubectl bloodraven dr-activate
  • a future confirm-token gate materializes a writable MysqlFailoverGroup from the discovered dump. The activation state machine is phase-stamped and durable across operator restarts. The spec field that carries the confirm token will be added when this lands.
• Phase 4 — Failback tooling: symmetric MysqlStandbyCluster on the returning source cluster + extended runbook.

Until Phase 3 ships, activation always requires the manual spec.initFromBackup steps described in this runbook.