Backup and restore

Bloodraven drives scheduled and on-demand MySQL backups using the mysqlsh util.dumpInstance() / util.loadDump() utilities against a configurable S3 or PVC destination. Restores are bootstrap-only in v1: populate a brand-new MysqlFailoverGroup from a previous dump before cross-site replication starts.

Backup decision table

Restore workflow checklist

• Confirm whether this is a new recovery group or a destructive in-place restore.
• Confirm source artifact, namespace, bucket/PVC path, and decryption Secret.
• Disable production traffic to the recovery DNS name until validation completes.
• Verify MySQL starts, expected schemas exist, and application smoke tests pass.
• Only then move traffic or update application configuration.

Recovery time and sizing considerations

Restore time depends on dump size, compression, object-store or PVC throughput, loadOptions.threads, MySQL startup time, and any PITR replay window. Size staging volumes and the node ephemeral storage used by verification for at least the compressed artifact plus restore working space; production teams should measure this with a full-size backup before go-live.

Concepts

• Backup profile — a named, reusable configuration living on MysqlFailoverGroup.spec.backup.profiles[]. Selects a storage target (S3 or PVC), dump options, and retention.
• Backup schedule — a cron expression on MysqlFailoverGroup.spec.backup.schedules[] that references a profile by name. Each schedule becomes a Kubernetes CronJob owned by the operator, named mysql-<group>-backup-<schedule> (a name over 52 characters, the longest CronJob name Kubernetes accepts, keeps its first 43 characters plus - and an 8-character hash).
• MysqlBackup CRD — one CR per backup run. Created ad-hoc via kubectl create or by the schedule CronJob. Tracks phase, start / completion times, dump location, size, GTID, and binlog coordinates.
• Restore — spec.initFromBackup on the failover group. Gates initial bootstrap on a one-shot util.loadDump() into a dynamically- selected target site.
• Verification — a parallel MysqlBackupVerification CRD and spec.backup.profiles[].verification block that periodically restores the latest Succeeded backup into a throwaway MySQL instance to prove it's actually loadable. See Backup verification.
• Encryption at rest — opt-in, per-profile spec.backup.profiles[].encryption block that turns on client-side envelope encryption (AES-256-GCM) for every dump artifact and archived PITR binlog. The passphrase lives in a Kubernetes Secret the operator controls, so keys and storage credentials can be kept on separate blast radii. See Backup encryption.

The backup image

The default image is pinned to container-registry.oracle.com/mysql/community-server:9.7. This bundles the mysqlsh binary; the community-shell repository that appears in some docs does not exist in the Oracle registry — a common stumbling block. Production deployments should always pin this explicitly via spec.backup.image and avoid floating tags like :9 or :latest, since mysqlsh dump/load compatibility across versions is not guaranteed.

Backup source selection

The reconciler prefers the replica site as the dump source so the primary is not loaded by long-running backups. It falls back to the primary when the replica is unreachable, not replicating, or lagging beyond spec.backup.maxLagSecondsForSource (default 300). Override per-backup with spec.sourceSite on a MysqlBackup CR.

If no healthy source is available before a Job is created, the backup stays Pending with Ready=False and reason WaitingForHealthySource, and retries for up to 30 minutes; a normal failover window does not consume the one-shot backup as a terminal failure. The reconciler emits a WaitingForHealthySource warning Event once on entering the wait. If no healthy source appears within that window the CR fails terminally with reason NoHealthySource so retention pruning can reclaim it. The other pending reasons (BackupJobConflict, PITRArchiveRolloutPending, DumpKeySnapshotConflict, PassphraseSecretMissing) have the same 30-minute bound; past it the backup fails with reason PendingTimedOut, and the message names the wait (pending for <duration> (<pending reason>): <message>). A pinned spec.sourceSite that cannot be used fails at once with NoHealthySource. Once the Job is created, its selected source is pinned. Later reconciles observe that Job without re-running source selection against a topology that may be changing.

When ActiveSite changes while a backup is in flight, the backup reconciler emits an InFlightFailover warning event on the CR. This is a soft signal: the artifact is still a valid point-in-time snapshot of the original source, but operators should know a failover happened mid-dump so they can correlate it with any replication-gap alerts.

Security context defaults

Every backup, restore, and cleanup Job pod runs with a hardened pod- and container-level SecurityContext matching the Restricted Pod Security Standard:

Because readOnlyRootFilesystem=true, the backup Job pod also mounts two ephemeral emptyDir volumes: mysqlsh-home at /home/mysqlsh (mysqlsh needs a writable home for ~/.mysqlsh) and tmp at /tmp. HOME is set to /home/mysqlsh so mysqlsh discovers them cleanly.

Users can override individual fields via spec.backup.podSecurityContext and spec.backup.containerSecurityContext. Overrides are merged on top of the defaults — any unset field stays at the default, and any set field wins. This lets you tighten further (e.g. set a specific seLinuxOptions) without reintroducing capabilities or root access.

Credential layout

MySQL credentials are never injected into the backup pod via envFrom. Instead the reconciler derives a Secret per backup run (with MYSQL_USER / MYSQL_PASSWORD keys) and mounts it as files at /run/bloodraven/mysql-creds/ with mode 0400. The embedded Python dump script reads the files via the BLOODRAVEN_MYSQL_CREDS_DIR env var. This keeps plaintext passwords out of /proc/$PID/environ on the pod and makes accidental env-var leaks (e.g. in panic stack traces) much less dangerous.

Backup and restore scripts read mounted MySQL usernames and passwords as UTF-8 without trimming whitespace or changing line endings. A trailing newline in a Secret is part of the password. Readable files take precedence over the legacy MYSQL_USER / MYSQL_PASSWORD environment variables, including an empty password file. Only missing or unreadable files use the environment fallback; an empty username still fails validation.

The derived Secret reads username/password directly from spec.credentials.backupSecretName (or falls back to operatorSecretName if no dedicated backup secret is configured).

For S3 profiles the same pattern applies to AWS credentials: the profile's credentialsSecretName is mounted at /run/bloodraven/aws-creds/ (mode 0400), pointed at by BLOODRAVEN_AWS_CREDS_DIR, and the dump script assembles a standard ~/.aws/credentials file on the fly so the AWS SDK inside mysqlsh picks it up natively.

AWS configuration files retain whitespace normalization; the exact-content rule applies to MySQL credentials.

S3 backup profile

apiVersion: shipstream.io/v1alpha1
kind: MysqlFailoverGroup
metadata:
  name: orders
  namespace: orders
spec:
  image: mysql:9.7
  credentials:
    operatorSecretName: mysql-operator-creds
    backupSecretName: mysql-backup-creds
  # ... sites, dns, sidecar omitted ...
  backup:
    image: container-registry.oracle.com/mysql/community-server:9.7
    maxLagSecondsForSource: 120
    retry:
      maxAttempts: 3
      initialBackoff: 1m
      maxBackoff: 30m
    profiles:
      - name: nightly-s3
        retention:
          keepLast: 14
          keepWithinDays: 30
        storage:
          type: S3
          s3:
            bucket: shipstream-backups
            prefix: orders
            region: us-east-1
            credentialsSecretName: s3-backup-creds
        dump:
          threads: 8
          bytesPerChunk: "128M"
          compression: zstd
          consistent: true
    schedules:
      - name: nightly
        profileName: nightly-s3
        schedule: "0 2 * * *"
        timeZone: "America/Los_Angeles"

The referenced s3-backup-creds Secret must contain AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and optionally AWS_SESSION_TOKEN / AWS_REGION. The keys are each mounted as a separate file under /run/bloodraven/aws-creds/ and the dump script stitches them into a standard credentials file at startup.

TimeZone

schedules[].timeZone is an IANA zone name (default Etc/UTC) that is passed to the CronJob spec's .timeZone field. This matters because kube-controller-manager's local timezone is environment- dependent and unreliable — two clusters running the same manifest can fire the same cron at different wall-clock times. Pinning the TZ per-schedule makes backup scheduling reproducible no matter where the control plane runs.

MinIO or other S3-compatible stores

Set storage.s3.endpointURL to the target endpoint (e.g. https://minio.internal:9000). The reconciler maps this onto mysqlsh's s3EndpointOverride option.

PVC-backed profile

backup:
  profiles:
    - name: daily-local
      retention:
        keepLast: 7
      storage:
        type: PVC
        pvc:
          provisioned:
            storageClassName: fast
            size: 50Gi
      dump:
        compression: zstd

With pvc.provisioned the operator provisions a PVC named mysql-<fg>-backup-<profile>; with pvc.claimName the user-managed claim is used. Exactly one of them is set, and provisioned cannot change (see PVC backups). Either claim is mounted read-write into the backup Job at /backups.

Retention

After each backup finishes, the operator applies the profile's retention to that profile's MysqlBackup objects. Deleting a MysqlBackup removes its artifacts (see Artifact cleanup), so retention is what bounds storage use. A backup with spec.deletionPolicy: Retain is pruned the same way, but its artifact stays in storage (see Keeping an artifact).

A successful backup is kept when any rule keeps it, so it is deleted only when it is outside keepLast and older than keepWithinDays. Because keepLast is at least 1, retention never deletes the last good backup, even when every attempt after a long outage has failed. Failed backups are counted separately and never push a successful one out. Omit retention, or any of its fields, to get the defaults; an omitted keepFailed means 10, while an explicit keepFailed: 0 deletes failed backups two minutes after the operator records their verdict (or after an operator restart, whichever is later), not after the Job ended, so a backup the operator observes late is not deleted in the same pass. The delay lets kubectl bloodraven backup --wait, or anyone else watching the backup, read its verdict before it goes. One failed backup is always kept, whatever keepFailed says: the one that is still the latest of a schedule in spec.backup.schedules. It stays until a newer backup of that schedule exists (its retry or the next scheduled run), because spec.backup.retry creates the retry from it and status.backupSchedules reports it as the schedule's last result. Once you remove or rename the schedule, its last backup is pruned like any other.

Retention always prunes. There is no setting that keeps every successful backup; set keepLast (and keepWithinDays) to the history you need.

The PITR binlog archive follows backup retention, not the reverse: archived binlogs are kept back to the oldest successful MysqlBackup of the group, of any profile, that retention has not deleted or that a verification still reads. Pruning backups is what lets the archive release older binlogs; see Binlog retention.

Moving from the released retention fields

Earlier releases had a retention: N integer and a retentionPolicy object. Both are gone. The API server refuses retention: 7. retentionPolicy is an unknown field: kubectl apply refuses it, but a client that does not validate fields gets it dropped with a warning, and the defaults then apply. Translate them like this:

See Backup verification for verification.retention.

A successful backup that a MysqlBackupVerification is reading is never pruned while the verification runs. It stays past the keepLast and keepWithinDays windows, with a RetentionDeferred Event, and is pruned on the next pass after the run stops. See Backup verification.

Artifact cleanup

Cleanup deletes the destination recorded in status.artifactStorage, which the operator writes when it creates the backup Job. If that status write was lost, the operator recovers the destination from the backup's own producing Job (its main container's destination environment) while that owned Job still exists, never from today's profile.

If an already-Succeeded record has no recoverable destination, repair records DestinationUnavailable once and continues retention without permanent reconcile errors. On deletion, ArtifactCleanupSkipped names the producing Job and the finalizer is released. An unlocated artifact may remain in storage and needs separate inspection; deletion does not guess a cleanup path.

When a MysqlBackup CR with spec.deletionPolicy: Delete (the default) is deleted (manually or by retention) the reconciler runs a short-lived cleanup Job via the bloodraven.shipstream.io/mysqlbackup finalizer. The Job runs bloodraven artifact-cleanup from the operator image. It mounts only the storage: the profile's S3 credentialsSecretName, or the backup claim. It never connects to MySQL, so the group's MySQL credentials are not needed.

Keeping an artifact

Set spec.deletionPolicy: Retain on a MysqlBackup to keep its artifact when the CR goes. Retain only stops the cleanup Job. It does not keep the storage: an artifact on a claim the group provisioned (mysql-<group>-backup-<profile>, from pvc.provisioned) is deleted with that claim when the group is deleted, whatever the policy says. Before deleting a group whose dumps must outlive it, copy those artifacts to S3 or to a claim you created, check the copy, copy the data of an encrypted backup's dump-key snapshot Secret (named in status.artifactStorage.passphraseSecretKeyRef) into a Secret you own (deleting a Delete backup deletes its snapshot), and delete the group's backups while their storage still exists (see Deleting a group without losing backups). Do not remove the claim's owner reference by hand: the operator sets it again. The policy is read when the deletion is processed, so it can be set on an existing backup up to then. Set it before you delete the backup: once a Delete deletion has started its cleanup (status.cleanupStartTime is recorded just before the first cleanup Job is created, and stays set if that Job is later removed; it is taken back only if the Job cannot be created), a later switch to Retain is not applied, because the artifact may already be gone or partly deleted. A switch that lands before the cleanup start is recorded wins: the operator records the start only against the spec revision that still said Delete. The cleanup runs to its end, and its ArtifactCleanupSucceeded, ArtifactCleanupNothingFound or ArtifactCleanupFailed Event adds "deletionPolicy was changed to Retain after the cleanup started, so it was not applied". Deleting a Retain backup runs no cleanup Job: the artifact, its backup stamp and, for an encrypted backup, its dump-key snapshot Secret (mysqlbackup-<name>-dump-key-<uid>, the only copy of the key that decrypts it) stay where they are, and a Normal ArtifactRetained Event names the location and the Secret. A backup that never succeeded is released once its producing Job's pods have stopped, as in the abandoned cleanup, and no Retain backup is released while a cleanup pod labelled with its name is still pending or running. Like a Delete backup, a Retain backup that a MysqlBackupVerification still reads keeps its finalizer until that verification's pods have stopped, so the PITR binlog cutoff keeps counting it while a replay may still need its binlogs. Nothing tracks the kept objects afterwards; delete them by hand when they are no longer needed. A backup re-created with the same name fails at its backup stamp step while the retained dump is still at that location.

Where cleanup deletes

Cleanup deletes from where the backup was written, never from wherever the profile points today. When the backup Job is created, the operator records in status.artifactStorage the exact artifact location the Job writes (prefix or subdirectory included), plus its bucket, endpoint and region (S3) or claim (PVC). A later edit of the profile's bucket, endpoint, prefix, subPath or claim does not redirect the delete. Once the location is recorded, the API server rejects any change to status.artifactStorage and its removal, including removing the whole status. Backups that succeeded under an older operator get it filled in from their producing Job on their next reconcile, not only at deletion. Credentials are still read from the profile's current credentialsSecretName, so rotating keys needs no change on old backups.

Restores and verifications of a backupRef read the same record: the bucket, endpoint, region and claim come from status.artifactStorage, never from the live profile, and only the S3 credentials (credentialsSecretName) come from the profile. A PVC backup therefore needs no profile to be restored or verified.

Backups taken before status.artifactStorage existed recover it from their producing Job, which is kept while the CR exists. A successful backup with no recorded destination that cannot be recovered is released with a Warning ArtifactCleanupSkipped ("no recorded artifact destination"); its artifact, if any, stays in storage and needs separate inspection.

status.artifactStorage.location is the only artifact location. Before any cleanup Job is created, the operator checks it, so a backup cannot delete another namespace's backup of the same name in a shared bucket or claim:

• the last path segment must be the backup name;
• an s3:// location must be in the recorded bucket;
• a PVC location must be under /backups;
• while the producing Job still exists, the recorded destination (storage type, location, bucket, endpoint, region, claim and passphrase key) must match the one that Job writes.

What gets deleted:

• S3 — every object under <location>/. The trailing slash means x/backup-1 never matches x/backup-10. Deletes run 16 at a time with DeleteObject, one listing page at a time, so a prefix with millions of objects never has to fit in memory. Then the prefix is listed again; anything left is a transient failure. On a versioned bucket, this leaves delete markers and keeps noncurrent versions; expire them with a lifecycle rule.
• PVC — the <location> directory tree. Paths are resolved inside the mount at the moment of deletion, so a directory replaced by a symlink cannot redirect the delete outside the claim; such a path is a refusal (exit 64), not a transient error. The mount root itself is never deleted.

The container's termination message carries the verdict: BLOODRAVEN_CLEANUP_COMPLETE objects=<n> on success, or BLOODRAVEN_CLEANUP_FAILED: <reason> on failure. Exit code 64 means a refusal or a configuration error; exit code 75 means a transient storage error. Any other exit, including a crash, is treated as transient. Cleanup of an abandoned backup has a third verdict, exit 0 with BLOODRAVEN_CLEANUP_SKIPPED: <reason>: no backup stamp at the location names this backup, but objects remain there, so nothing was deleted.

The cleanup Job is named mysqlcleanup-<backup>. Older operators named it mysqlbackup-<backup>-cleanup, which is also the backup Job name of a backup called <backup>-cleanup. A Job with the old name is only treated as this backup's cleanup Job when the backup owns it.

Failures and retries

A failed cleanup is a state, not a dead end. The ArtifactCleanup condition on the Terminating MysqlBackup names the cause, and a fresh cleanup Job runs automatically:

Retries never stop, so fixing credentials inside the same Secret, an IAM policy or a bucket ACL clears the failure without touching the backup. A replacement Job starts only after the failed Job and its pods are gone. Cleanup Jobs left behind by older operator versions (which ran cleanup.py and always failed on S3, because MySQL Shell has no util.rmdump) are replaced the same way after upgrade.

A backup that never succeeded, or whose failover group or profile is gone, is released in abandoned mode: the operator first deletes its producing backup Job and waits until none of its pods is pending or running, so neither the best-effort cleanup of its stamp and partial dump nor a new backup of the same name races a dump that is still writing. A status update lost after the Job was created is recovered from the Job itself, running or failed. An encrypted backup's stamp is read with the producing Job's passphrase Secret (the backup's dump-key snapshot), or, once the Job is gone, the one recorded in status.artifactStorage.passphraseSecretKeyRef. A recorded Secret must exist: if it is missing, the cleanup blocks with PassphraseSecretMissing naming it. The profile's live Secret is used only for a backup that recorded no key at all. A foreign stamp, or no stamp over an empty location, deletes nothing and completes with BLOODRAVEN_CLEANUP_COMPLETE objects=0 (the dump-key snapshot is deleted). When there is no stamp but objects remain at the location, the cleanup Job exits 0 with BLOODRAVEN_CLEANUP_SKIPPED: no backup stamp at <stamp key> names this backup, but <n> objects remain under <prefix>; nothing was deleted (on a PVC: ... but <path> still exists; nothing was deleted) and logs WARN artifact cleanup skipped. The finalizer is then released without deleting the dump-key snapshot, and a Warning ArtifactCleanupSkipped names the location, the stamp key and the kept snapshot, because the objects may be this backup's encrypted artifacts: delete them by hand, and the snapshot once none remain. If a producer pod has not stopped after 10 minutes, the condition turns ProducerStopStuck and a Warning ArtifactCleanupStalledProducerStop names it; the deletion stays held, because the pod may still write. Once its node is confirmed down, run kubectl delete pod <pod> -n <ns> --force --grace-period=0; never force-delete a pod on a live node.

For DestinationMismatch, inspect storage, delete the artifact by hand if it still exists, and then remove the bloodraven.shipstream.io/mysqlbackup finalizer. Do not remove the finalizer while the artifact is still in storage, or the artifact will never be cleaned up.

Events:

• ArtifactCleanupStarted — cleanup Job created.
• ArtifactCleanupSucceeded — artifact removed; the message gives the object count.
• ArtifactCleanupNothingFound (Warning) — cleanup found nothing at the recorded destination. The finalizer is released, because the artifact may already have been removed by hand. Otherwise, check that the backup really was written there.
• ArtifactCleanupFailed (Warning) — a cleanup Job failed. Emitted once per Job, with the cause and the retry delay.
• ArtifactCleanupRetry — a failed cleanup Job is being replaced.
• ArtifactCleanupDeferred — a verification is still reading the artifact; cleanup waits for its run to stop.
• RetentionDeferred — retention kept a backup past its window because a verification is reading it.
• ArtifactCleanupBlocked (Warning) — cleanup cannot start; the message carries the ArtifactCleanup reason.
• ArtifactCleanupStalledProducerStop (Warning) — an abandoned backup's producer pod has not stopped after 10 minutes; see above.
• ArtifactCleanupSkipped — the referenced failover group or profile is gone, or the producing Job cannot supply a missing artifact destination, so cleanup cannot run. It also fires when the best-effort cleanup of a backup that never succeeded (its stamp and any partial dump) cannot run or fails, or skips because objects remain at the location with no stamp naming the backup (BLOODRAVEN_CLEANUP_SKIPPED); the message names what is left. An encrypted backup that recorded neither its producing Job nor its artifact destination, or whose abandoned cleanup was skipped, keeps its dump-key snapshot, which the message names: delete it once you have confirmed no encrypted artifact of the backup remains.

bloodraven_backup_artifact_cleanup_failures_total{group,profile,reason} counts failed cleanup Jobs, once per Job.

Retries for scheduled backups

spec.backup.retry configures operator-level retries for scheduled CRs that land in Failed. This is independent of the Job-level backoffLimit: Job backoff retries the container inside a single CR, whereas this retries the whole CR and produces a fresh MysqlBackup object with its own Job and attempt counter.

backup:
  retry:
    maxAttempts: 3         # total attempts including the original
    initialBackoff: 1m     # default 1m; 1s to 1h
    maxBackoff: 30m        # default 30m; 1s to 24h, at least initialBackoff

Backoff is exponential: the Nth retry waits initialBackoff * 2^(N-1) from the failed CR's completion time, capped at maxBackoff. Retention never deletes the failed CR a retry is waiting on, even with keepFailed: 0: a schedule's latest backup is kept until a newer one of that schedule exists. spec.backup.retry is ignored for ad-hoc CRs — those are operator-initiated and do not participate in automatic retries.

Each scheduled run has its own retry chain. Retained retries from earlier runs do not consume or suppress the current run's attempts. New retry names use backup-retry-<predecessor UID hash>-r<N>, so repeated reconciles, operator restarts, and a stale list cache converge on the same successor. The hash is derived from the failed backup's UID, so reusing a backup name for a different object starts a distinct chain.

Retry creation records bloodraven.shipstream.io/backup-retry-of as a label containing the failed predecessor's UID. Annotations bloodraven.shipstream.io/backup-retry-root and bloodraven.shipstream.io/backup-retry-root-uid identify the original run; bloodraven.shipstream.io/backup-attempt records the launch attempt. These values are persisted with the Create request. The backup reconciler restores status.attempt from launch metadata if the initial status patch was lost, and retry limits use the greater of the recorded status and launch attempt. An existing successor must match its expected lineage. A conflicting object at the deterministic name emits a BackupRetryConflict warning and skips only that retry; other schedules' retries and status rollups continue. status.backupSchedules[].retryConflict names the blocking object. The warning and conflict log are emitted when that marker changes, including after a resolved conflict recurs. Unchanged conflicts do not repeat warnings across reconciles or operator restarts. The marker clears when the conflict is no longer observed; removing a schedule removes its rollup entry. If Create sees an existing object before it is visible to the cache, the operator requests a five-second wake-up without blocking other schedules.

Older records without lineage metadata remain supported through their status attempt, but their higher counter alone cannot suppress another run. An unlinked legacy retry becomes the recorded root when its next successor is created; its existing attempt counter still applies.

One-off backups

apiVersion: shipstream.io/v1alpha1
kind: MysqlBackup
metadata:
  name: orders-preupgrade
  namespace: orders
spec:
  failoverGroupRef:
    name: orders
  profileName: nightly-s3

Inspect with kubectl get mysqlbackups. The Phase and Ready columns (and, with -o wide, Location and Size) reflect the current state. The Ready condition is False from creation; its reason says why a backup is waiting, that it is Running, or why it failed.

Restore at bootstrap

To recover a lost failover group from a previous backup, create a new MysqlFailoverGroup that references an existing MysqlBackup CR in the same namespace:

apiVersion: shipstream.io/v1alpha1
kind: MysqlFailoverGroup
metadata:
  name: orders
  namespace: orders
spec:
  image: mysql:9.7
  credentials:
    operatorSecretName: mysql-operator-creds
  # ... sites etc. ...
  backup:
    profiles:
      - name: nightly-s3
        storage:
          type: S3
          s3:
            bucket: shipstream-backups
            prefix: orders
            credentialsSecretName: s3-backup-creds
  initFromBackup:
    source:
      backupRef:
        name: orders-preupgrade

Dynamic target-site resolution

The restore reconciler does not hard-code spec.sites[0] as the restore target. Instead it resolves dynamically via restoreTargetSite, and creates the load Job only once the target is writable:

1. If status.activeSite is set and that site is observed Writable, use it.
2. If status.activeSite is set but that site has no observed state yet or is observed ReadOnly, wait in Pending. A new group's seed is published as the active site just before the poll that observes it writable.
3. If status.activeSite is set but that site is observed in any other state (Unreachable, Unknown), refuse: the reconciler emits a RestoreTargetUnavailable warning event and parks the restore in Pending. This prevents accidentally overwriting a recovering site with a stale dump.
4. Otherwise (true fresh deploy) wait in Pending for the default seed site (the first splitBrainPolicy.sitePriorities primary candidate, else the first primary candidate in spec.sites).

Every MySQL server of a new group boots fenced (read_only=ON, and the sidecar adds super_read_only until the operator names the site active), so the load cannot start before the operator lifts the seed's fence. While initFromBackup has not succeeded, the topology manager's fresh-deploy bootstrap only seeds: once every site is reachable and empty, it fences the other sites, makes the seed writable and publishes it as the active site, but clones nothing. When the restore succeeds, the empty sites are cloned from the loaded seed. A Job that already exists is observed to completion whatever its target's state.

Applications wait for the restore

The seed is writable and published as status.activeSite while the Job loads into it, but it is not the application's database until the restore succeeds: application writes would mix with the load, and a point-in-time restore's proofs would fail. Until status.restore.phase is Succeeded (while it is unobserved, Pending, Running, or Failed) the operator keeps applications off the group:

• No DNS record. The spec.dns.hostname record is not written. It is published on the first poll after the restore succeeds.
• No application Service endpoints. No pod is labelled bloodraven.shipstream.io/role=primary or bloodraven.shipstream.io/healthy=true, so the -primary and -replicas Services have no endpoints, and every per-site Service (including a LoadBalancer one) also selects on bloodraven.shipstream.io/healthy=true and has none either.
• No clones. The other sites stay empty, and no site other than the seed is promoted: an emergency failover would otherwise make an empty server the primary the group serves once the restore succeeds.
• Ready=False, reason InitFromBackupInProgress, or InitFromBackupFailed after a failure.

The restore Job is unaffected: it connects through the seed's internal Service (mysql-<group>-<site>-internal), which selects on the site alone. Dragonfly is not gated: its -active Service follows the active site as usual, and holds no MySQL data.

If the seed's mysqld or sidecar restarts during the restore it comes back fenced (read_only, plus the sidecar's super_read_only), and with part of the load in it the fresh-deploy seeding no longer applies. The operator lifts that fence on the restore's seed, also after an operator restart: it takes the seed from status.restore.targetSite once the Job exists (else status.activeSite), and lifts the fence as long as every other site is read-only with no user schemas and no replication history.

initFromBackup is fixed at creation

spec.initFromBackup is the group's bootstrap record. Admission refuses adding it to a group that was created without it, removing it, and any change to it:

spec.initFromBackup is fixed at creation: it cannot be added, changed or removed. To retry a bootstrap restore that never succeeded, delete the group and create it again with the corrected block (see Backup and restore > A failed bootstrap restore)

Adding it to a serving group would load a dump into its primary with none of the fencing restoreInPlace has, and removing it and adding it back would run the restore again. To restore into a running group, use spec.restoreInPlace. The comparison is by value: the operator's own writes, which re-encode a stopTime offset in UTC or drop an explicit allowPartial: false, are not changes.

A failed bootstrap restore

A failed load can leave partial data on the seed. The group then stays as it was during the restore: no DNS record, no application Service endpoints, nothing cloned, Ready=False with reason InitFromBackupFailed. The RestoreFailed Event and the Ready message give the recovery.

When the cause is outside spec.initFromBackup (a missing Secret, the backup's profile, the source PVC's contents, the restore account), fix it and retry in place:

1. Read the restore Job's log for the cause and fix it.
2. If the load had started (BLOODRAVEN_LOAD_START in the Job log), empty the target: delete the group's MySQL pods and PVCs together. A claim stays Terminating while a pod still uses it, so deleting the PVCs alone does not empty anything.

   kubectl delete pvc,pod -l app.kubernetes.io/instance=<group>,app.kubernetes.io/name=mysql -n <namespace>
   

   
   Wait until the operator has re-created every claim and it is Bound, and every MySQL pod is Running:

   kubectl get pvc,pod -l app.kubernetes.io/instance=<group>,app.kubernetes.io/name=mysql -n <namespace>
   

3. Delete the restore Job. The operator seeds again if needed and rebuilds the Job on the next reconcile. Deleting the Job before the claims are replaced rebuilds it against the same partial load.

When the cause is in spec.initFromBackup itself (a wrong source, a missing decryption block, a stopTime the archive cannot reach), the block cannot be edited. Delete the group and create it again with the corrected block. Deleting a MysqlFailoverGroup to recreate it is allowed only for a group that holds no data: this case, a group whose bootstrap restore never succeeded, and a group that passes every check in Does the group hold data?. Check that first:

1. status.restore.phase has never been Succeeded: the group never served. Check kubectl get mfg <group> -o jsonpath='{.status.restore.phase}' and the group's Events. If the restore ever succeeded, the group holds application data: do not delete it, use restoreInPlace instead.
2. No application is connected. The group publishes no DNS record and no Service endpoints until the restore succeeds, so a connection means something reaches the pods directly: find it first.
3. The backup is intact: the MysqlBackup named in source.backupRef is Succeeded, or the S3 or PVC source still holds the dump.
4. Save the manifest and correct spec.initFromBackup in it. Delete the group as described in Deleting a group without losing backups: its PVCs, Services and the partial load are deleted with it (they are owned by the group), and so are its own backups and the backup claims it provisioned. The source backup of spec.initFromBackup is affected only if its spec.failoverGroupRef names this group.
5. Apply the corrected manifest under the same name once the group, its PVCs and its backups are gone.

The cost is the partial load: the new group's PVCs are provisioned and the restore runs again from the start.

Rolled-out Deployment gate

Before spawning the load Job, the reconciler waits until the target site's Deployment is fully rolled out:

• generation <= 1 → accept as long as ReadyReplicas >= 1 (fresh deploy; ObservedGeneration may briefly lag).
• Otherwise, require ObservedGeneration >= Generation, UpdatedReplicas >= 1, and ReadyReplicas >= 1.

This guards against firing a load against a Deployment mid-rolling- update, which would race the ready MySQL container against the terminating one.

Direct S3 / PVC sources

initFromBackup.source also accepts a direct S3 URL or a read-only PVC, which is useful when the source backup CR has been garbage- collected:

initFromBackup:
  source:
    s3:
      bucket: shipstream-backups
      prefix: orders/orders-preupgrade
      credentialsSecretName: s3-backup-creds

Restores are one-shot. If the Job fails, inspect the logs, delete the Job, and the reconciler will rebuild it on the next pass (see A failed bootstrap restore). Restore Jobs mount local dump sources (PVCs, decrypted staging) read-only and disable mysqlsh load-progress tracking, so no restore state is written to backup storage. A rebuilt Job therefore loads the whole dump again rather than resuming. If the failed Job had already started loading (BLOODRAVEN_LOAD_START in its log), empty the target before deleting the Job: delete the new group's MySQL pods and PVCs together and wait for the re-created claims to be Bound and the pods Running (see A failed bootstrap restore), or run ./playground/reset-mysql.sh in the playground. Otherwise the reload stops on the objects the first attempt created. Restoring into a non-empty data directory is not supported — util.loadDump() fails fast with a clear error.

Per-schema bootstrap (tenant migration pattern)

loadOptions.includeSchemas and loadOptions.excludeSchemas are forwarded to util.loadDump() and let you carve a subset of a full-instance dump into a brand-new failover group. Both are sets of schema names (no duplicates, at most 256 names of at most 64 characters), as are a profile's dump.includeSchemas and dump.excludeSchemas. This is useful for migrating a single tenant off a shared cluster onto its own dedicated one:

spec:
  initFromBackup:
    source:
      backupRef:
        name: shared-nightly
    loadOptions:
      includeSchemas: ["tenant_42"]

Operationally, the migration pattern is: put the tenant into application-level maintenance mode, take a backup of the shared cluster, deploy a new MysqlFailoverGroup with includeSchemas pointing at the tenant's schema, then flip the app's connection string to the new group. System users are not part of a util.dumpInstance dump, so pair the restore with whatever user-provisioning process the new group uses.

In-place restore (spec.restoreInPlace)

initFromBackup is one-shot and runs before the failover group is considered ready. For rolling an existing, live cluster back to a previous dump without teardown or rename, use spec.restoreInPlace. In-place restores always restore to the backup itself: see Point-in-time recovery is not supported in place. It is re-triggerable and operates directly against the active primary.

Two modes are supported, selected by loadOptions.includeSchemas. The API server rejects a restoreInPlace with more than one includeSchemas entry; use a full-instance restore instead.

Full-instance in-place restore

When includeSchemas is empty, the restore wipes every user schema on the primary and reloads from the dump. Choreography:

1. Preflight — validates that status.activeSite is writable and the confirm timestamp is newer than the last consumed one. It also checks that the operator can stop every peer's replication receiver (see Peer replication during a full-instance restore), and waits while an ordered update, a planned failover or a clone is running (see the peer-replication Preflight step below for which planned failovers it waits for). When the restore is accepted, the operator records every other site in status.restoreReclones (see Peer reclones).
2. Fencing — the operator strips the primary role label on the active pod(s), so the -primary Service sheds endpoints and takes no new connections, and sets offline_mode=ON on the primary (see Application sessions during a full-instance restore). It then stops the replication receiver of every peer. The restore Job is created only after no client session from before the fence remains and every peer's receiver is verified stopped. The topology manager is frozen: no promotion, auto-clone, or recovery actions fire during the restore.
3. Restoring — a Job runs util.loadDump() after a STOP REPLICA; RESET REPLICA ALL; DROP DATABASE <each-user-schema> preflight. The load uses loadOptions.skipBinlog as given. When a loadOptions block is present the CRD defaults it to true (an unlogged load). When loadOptions is omitted the operator passes nothing and mysqlsh's own default applies: the load is binlogged and replicates to the peers. Either way every peer is recloned afterwards.
4. Resuming — the primary role label is restored, the topology manager is unfrozen, the peer reclone queue is released, and the restore reports Succeeded. The peers are recloned from the restored primary after that, in the background.

Application sessions during a full-instance restore

Removing the primary label does not disconnect established connections (kube-proxy keeps existing flows), and clients of the per-site Service or of spec.dns.hostname never go through the -primary Service. A full-instance restore therefore also fences the database itself:

1. Preflight checks that the operator account holds the privileges the fence and the restore Job need on the target: CONNECTION_ADMIN, SYSTEM_VARIABLES_ADMIN, PROCESS and REPLICATION_SLAVE_ADMIN (see The restore account). Without them the restore is rejected (RestoreInPlaceRejected naming the missing privileges, nothing fenced, confirm not consumed). There is no fallback to a label-only fence. While the rejected spec stays applied the operator re-checks about once a minute, so the restore starts on its own once the grants are in place.
2. The operator records the fence in status.restoreOfflineFence, reads and records the current offline_mode, and only then runs SET GLOBAL offline_mode = ON. The setting is never persisted.
3. MySQL disconnects every session without CONNECTION_ADMIN/SUPER and refuses new ones with error 3032 (ER_SERVER_OFFLINE_MODE). The operator also kills every remaining client session except its own account's and replication dump threads. It waits until no session from before the fence is left (RestoreOfflineFenceApplied), and only then creates the restore Job. Sessions of SYSTEM_USER accounts cannot be killed by an operator account without SYSTEM_USER; they are left connected and named in a RestoreOfflineFenceSessionsKept Warning instead of holding the fence forever.
4. While the restore runs the topology manager re-asserts offline_mode=ON on every poll, so a restarted mysqld (which comes back with it OFF) does not readmit applications. The restore Job itself refuses to drop anything unless offline_mode is ON (BLOODRAVEN_OFFLINE_FENCE_MISSING).
5. When the restore ends (Succeeded, Failed or cancelled), the operator puts the recorded value back, ON included, and checks that it took (RestoreOfflineFenceReleased). Only then does it clear status.restoreOfflineFence; it retries until that works. Peer reclones start after the release. Each run owns its own fence (status.restoreOfflineFence.runID), so a cancelled restore that is applied again with the same confirm fences the primary afresh.

What the fence does and does not cover:

• Accounts with CONNECTION_ADMIN or SUPER are not fenced. MySQL exempts them from offline_mode. Sessions from such accounts are disconnected once, when the fence goes up, but can reconnect and write during the restore. Do not give application accounts these privileges.
• offline_mode does not stop replication; the operator does. With the default spec.credentials setup the peers replicate as the operator account, which offline_mode exempts. A dedicated replication user (MYSQL_REPLICATION_USER) without CONNECTION_ADMIN is disconnected, but its receiver keeps retrying and reconnects once the fence is released. Either way the restore's binlogged DROP DATABASE statements would reach the peers, so the operator stops each peer's replication receiver before the restore Job exists (next section).
• Monitoring without CONNECTION_ADMIN (for example mysqld_exporter on the monitor account) loses the primary for the window. Expect gaps and connection errors in those metrics.
• If the operator cannot restore offline_mode when the restore ends (for example the primary is unreachable from the operator), it keeps retrying and emits RestoreOfflineFenceReleaseFailed. If applications must come back before that succeeds, run SET GLOBAL offline_mode = OFF on the primary yourself once status.restoreInPlace is terminal. The operator's own retry is then a no-op that clears the record.
• Upgrading and downgrading. Operators before this fence never checked these privileges. After the upgrade, full-instance restores with a narrowed operator account are rejected until the grants below are in place. That includes a restore an older operator accepted and left in Fencing: it is checked before offline_mode is touched. Per-schema restores and installs whose operator account Bloodraven created are not affected. Do not downgrade the operator while a full-instance restore is in Fencing, Restoring or Resuming: an older operator does not know status.restoreOfflineFence and never releases it. If that happened, run SET GLOBAL offline_mode = OFF on the primary once the restore is terminal (or leave it ON if status.restoreOfflineFence.priorOfflineMode is true).

Peer replication during a full-instance restore

Before the restore Job can drop anything, the operator stops the replication receiver of every site in status.restoreReclones.pending with STOP REPLICA IO_THREAD. Only the receiver stops: the applier keeps applying the relay log the peer already received, which holds only transactions committed before the fence.

1. Preflight checks every peer before anything is fenced. A peer whose operator account lacks REPLICATION_SLAVE_ADMIN (or SUPER) rejects the restore without consuming confirm, like the primary's privilege check, and it is re-checked about once a minute; so does a peer whose MySQL checker cannot stop a receiver. An unreachable peer makes Preflight wait (waiting for peer <site> to be reachable, no application impact yet). Preflight also waits for a running ordered update, planned failover or clone, any of which could start a peer's replication; a planned failover already in flight keeps running to completion meanwhile. For a planned failover it waits only for a run that owns the sites, is Publishing, or is Admitting with a provisional claim; a Deferred run, or an Admitting run without a claim, owns nothing and does not block the restore (its own admission is refused while the restore is in flight, so the two would only wait for each other). When the deployment admission reservation is refused (a planned-failover claim or automation hold, an ordered update, or a topology change not yet persisted), Preflight waits and retries every 15 seconds instead of failing the reconcile, with status.restoreInPlace.message set to waiting for a concurrent operation to finish (<reason>), or, while an automation hold stands, waiting for the automation hold of planned failover <runID> to be acknowledged (bloodraven.shipstream.io/acknowledge-planned-failover-hold=<runID>). The hold's acknowledgement is processed while the restore waits. A new restore also does not start until the previous restore's reclone queue is settled (see below), which can take up to ten minutes while that restore's Job pods are still running (RestoreInPlaceDeferredPreviousQueue; status.restoreInPlace keeps showing the previous run, and the rest of the group's reconciliation carries on). A clone whose admission is under way when the restore freezes the topology or queues its recipient is refused before it starts (BootstrapRefusedRestoreHold).
2. In Fencing, after offline_mode is on and the old sessions are gone, the operator records the primary's gtid_executed in status.restoreReclones.fencedGtidExecuted. A peer whose receiver is running gets up to 30 seconds to receive that set before its receiver is stopped. A receiver that is Connecting (a dedicated replication user after offline_mode) or already stopped is stopped at once. A peer stopped before it had everything is named, with the missing GTIDs, in a RestorePeerReplicationStoppedBehind Warning: it keeps its pre-restore data without those transactions.
3. The receiver must then report neither running nor connecting (RestorePeerReplicationStopped). If that cannot be verified for every peer within two minutes (unreachable peer, refused STOP), the run ends Failed with RestoreInPlacePeerStopFailed before any Job exists. Nothing was restored; the confirm is consumed (as for a Job that could not be built) so offline_mode is not re-applied in a loop. Bring the peer back, then bump confirm (a site cannot be removed from spec.sites; to drop one, see Replacing a site). A site added to spec.sites after the restore was accepted cancels the run (RestoreInPlaceCanceled, confirm not consumed); it re-runs with the current peers.
4. While a site is queued, nothing in the operator starts its replication: source convergence reports it Pending/RestorePeerHold, old-primary recovery and ordered updates leave it alone, and no clone runs into it while the queue is Armed or Held. Starts that were already running when the restore began finish before the receivers are stopped. While the queue is Armed past Fencing, or Held, the topology manager also re-stops a receiver someone started by hand. A Draining site is restarted only by its own reclone.

Expected signals while sites are queued: Degraded=True with reason RestoreReclonePending, Ready=False ("await their post-restore reclone"), the deployment API reports RestoreReclonePending, bloodraven_replication_running{thread="io"}=0 on the queued sites, and bloodraven_restore_reclone_pending{site,phase}=1. Exclude queued sites from replication alerts with that gauge (see Monitoring).

Per-schema restores are not affected: replication carries their DROP and load to the peers.

Upgrading and downgrading. An operator upgraded while a restore is already in Restoring re-stops the peers' receivers on its first poll, but DROP statements may already have reached them. Do not downgrade the operator while status.restoreReclones is Armed or Held: an older operator restarts the held receivers, which then pull the restore's DROP statements. Every restore guard the topology manager enforces (the replication hold and re-stop, the clone block, the promotion exclusion, the freeze and the offline_mode fence) comes from status read through the API server: a manager that starts (an operator restart, or a configuration change) reads it before its first poll, and later changes seen only in the operator's cache are applied directly only when they add a guard; releasing one waits for a confirming read. If a starting manager cannot read the status, it keeps every guard of the manager it replaces, or does not start until the read succeeds. Standby clusters outside spec.sites that replicate from this primary are not held.

Peer reclones after a full-instance restore

A full-instance restore replaces the primary's data, so every other site (PrimaryCandidate sites, DisasterRecovery followers and ReadOnly readers) still holds the old data until it is recloned. The operator owns that work through status.restoreReclones:

• The queue is fixed before anything is destroyed. When the restore is accepted, status.restoreReclones.pending lists every peer: PrimaryCandidate sites first (in splitBrainPolicy.sitePriorities order, then declared order), then DisasterRecovery, then ReadOnly sites. The queue is Armed while the restore runs and Draining once it has succeeded.
• Queued sites are fenced off. Until a site leaves the queue it is never promoted automatically (emergency failover, split-brain resolution, the primary re-assert, the ordered-update handoff) and it is labelled bloodraven.shipstream.io/healthy=false, which removes it from the -replica Service and from a ReadOnly site's own Service. The per-site Service of a PrimaryCandidate or DisasterRecovery site does not select on that label, so clients that connect to it directly still read the old data. If the primary fails while only queued sites remain, the group does not fail over: the Degraded condition reads NoPrimary with a pending post-restore reclone message and a PromotionRefusedReclonePending Warning event is emitted.
• One reclone at a time. The operator writes bloodraven.shipstream.io/reclone-site=<site>:confirm=<group>:uid=<uid> (plus :gtidPrefix=<source UUID> when the site has recorded divergence) for the first queued site, waits for that clone, then moves on. It never overwrites a reclone annotation someone else set; it waits for that request to finish first. No new reclone is requested while a planned failover is in flight.
• A newer full-instance restore takes over cleanly. Before it replaces a queue that is still draining, the operator waits for any clone that is running to finish, and withdraws the queue's request that has not started yet (RestoreRecloneWithdrawn). No request from the old queue survives into the new restore.
• Nothing is cloned to a queued site while its queue is Armed or Held. A reclone annotation for such a site, the queue's own left over from before or an administrator's, is refused (RecloneRejected) and removed, also after an operator restart. To clone a Held site, set the queue back to Draining (see below).
• A site leaves the queue only after its clone succeeds. A failed clone, or a request whose clone has not started within ten minutes, keeps the site queued (status.restoreReclones.failures). The other queued sites go first, and the failed site is retried a minute later, so one broken peer does not hold up the rest. The queue survives operator restarts.
• Failed restores do not clone from, or replicate, the broken primary. When the restore ends Failed (or is cancelled), the queue is settled before the offline_mode fence is released:
  • If the run never created its Job, or every pod of its Job has terminated and the primary's gtid_executed still equals fencedGtidExecuted (for example the Job failed before its DROP), the binlog holds nothing replication could carry. The queue is discarded (RestoreReclonesDiscarded) and the peers resume replicating and become eligible for promotion again. While a pod of the Job (even one whose Job object is gone) has not terminated, the queue stays Armed and the offline_mode fence stays up (and is re-asserted if mysqld restarts), for at most ten minutes after the run ended; an unreadable primary is retried for two minutes from the first failed read. After that the queue is Held. The fence is also kept until the discard of the queue is written.
  • Otherwise (the primary changed, or it could not be compared) the queue is put Held (RestoreReclonesHeld) with every peer. Their receivers stay stopped, so they keep the data they had before the restore and may be the only intact copies. They stay out of promotion and the reader Services, and nothing is cloned or replicated into them until you decide (see the next bullet).
  • A queue still Armed from an earlier restore whose settlement was lost (for example the operator restarted right after that restore failed) when the next restore starts is put Held: whether the earlier restore changed the primary is unknown.
• An earlier restore's queue is kept, but never cloned from a failed restore's primary. If the failed restore had replaced an earlier restore's queue that was still draining, the earlier queue's sites still hold data from before that earlier restore, so they stay queued. What happens next depends on whether the failed restore touched the primary:
  • It ended before it started creating its restore Job (cancelled, rejected, or the Job could not be built). The primary is unchanged, so the earlier queue drains again (RestoreReclonesRestored).
  • It started creating its Job (status.restoreInPlace.jobCreating, recorded before the Job is created), whether the Job then failed, vanished, or its name was never recorded, and the primary changed. The earlier queue is put back as Held (RestoreReclonesHeld), together with the failed restore's other peers.
• Deciding about a Held queue. The primary may hold dropped or half-loaded schemas; the held sites keep their pre-restore data. While the queue is held, scheduled backups skip the held sites (they do not replicate) and fall back to the primary, so suspend backup schedules whose retention could age out the last good backup. Then choose one:
  • Restore the held data into the primary. Take a backup from a held site with spec.sourceSite: <held-site> on a MysqlBackup, then run a new full-instance restore from it. It reclones every peer once it succeeds.
  • Keep the primary's data. Once it is verified good, set the phase to Draining; the held sites are recloned from it: kubectl patch mfg <group> --subresource=status --type=merge -p '{"status":{"restoreReclones":{"phase":"Draining"}}}'.
  • Release the hold. Removing status.restoreReclones resumes the held sites' replication from the current primary, which replicates whatever the failed restore did to it. Only do this when that is what you want.
  
  A planned failover to a held site is refused (TargetUnhealthy: it is not replicating), so there is no in-place way to promote a held site while the primary is up.
• Explicit promotion. A queued site's receiver is stopped, so it cannot catch up, and kubectl bloodraven promote <group> <queued-site> is refused (TargetUnhealthy). If a queued site becomes the primary some other way, the rest of the queue is abandoned (RestoreReclonesAbandoned). If the primary is down and only queued sites remain, either bring the primary back (the queue resumes) or accept the stale copy by clearing the queue: kubectl patch mfg <group> --subresource=status --type=json -p '[{"op":"remove","path":"/status/restoreReclones"}]'. Automatic failover then proceeds.

Watch progress with kubectl get mfg <group> -o jsonpath='{.status.restoreReclones}' and the RestoreRecloneRequested, RestoreRecloneSucceeded, RestoreRecloneFailed and RestoreReclonesDrained events. A requested reclone counts as stalled only after this operator process has watched it sit unstarted for ten minutes, so an operator restart does not fail a clone that was running.

spec:
  restoreInPlace:
    confirm: "2026-04-17T14:32:00Z"
    source:
      backupRef:
        name: orders-nightly-20260416

Per-schema in-place restore

When includeSchemas contains exactly one entry, the restore drops and reloads only that schema. The primary Service stays up — other tenants keep writing — and replication carries the DROP + load through the primary's binlog to the peer. No reclone is scheduled.

spec:
  restoreInPlace:
    confirm: "2026-04-17T14:32:00Z"
    source:
      backupRef:
        name: shared-nightly
    loadOptions:
      includeSchemas: ["tenant_42"]

The caller is responsible for putting the affected tenant into application-level maintenance mode. The operator forces skipBinlog=false on the load so the propagation to the peer works.

Point-in-time recovery is not supported in place

restoreInPlace has no pointInTime field. PITR replays archived binlogs with their original GTIDs, and the live primary already executed (or replicated) every one of those transactions, so MySQL would skip them all and the restore would stop at the backup.

To recover to a point in time, restore into a new failover group with initFromBackup.pointInTime, then move the data or the traffic across. Operator versions before v1.7.0 accepted the field: earlier ones reported success without replaying anything (if you used it, check the restored data), and later ones refused it.

Support is planned through a staging instance: the restore Job would load the dump and replay the binlogs into a temporary mysqld inside the Job, then dump that result and only then fence and load it into the live primary.

The confirm timestamp

spec.restoreInPlace.confirm must be an RFC 3339 timestamp (e.g. 2026-04-17T14:32:00Z). It is the anti-fat-finger gate: the operator refuses to run the restore unless confirm parses and is strictly greater than status.restoreInPlace.confirmTokenUsed. On every new run, bump confirm to a newer timestamp. A common idiom in automation is to set confirm: $(date -u +%FT%TZ) at the moment the user authorizes the restore.

This gives programmatic callers a simple "just send now()" pattern while also protecting against replay: an older manifest applied accidentally does not re-trigger a restore. Admission refuses a confirm that is not an RFC 3339 timestamp. On an object stored before that rule, a confirm that does not parse on a finished restore starts nothing and leaves its terminal status in place, with a Warning RestoreInPlaceConfirmIgnored.

Changing confirm to a newer value before the restore Job is created re-authorizes the run in progress: the Job is built from the current spec and runs under the new confirm, which is recorded in status.restoreInPlace.runConfirm before the Job is created (RestoreInPlaceConfirmRebound). No second restore follows.

Changing confirm to a value that does not advance past the run's own before the Job is created cancels the run, like removing the field: Failed, RestoreInPlaceCanceled, nothing restored, confirmTokenUsed unchanged. That covers the GitOps way of cancelling, reverting the commit to the previous, already executed manifest: its Job would otherwise be built from the reverted spec and restore the older backup again.

Changing confirm while a restore Job is already running does not interrupt it. The running Job may be mid-DROP or mid-load against the live primary, so the operator lets it finish and records its outcome against the confirm it was accepted under (visible as status.restoreInPlace.confirmTokenUsed); a newer confirm then starts its own run once the first reaches a terminal phase, and one that does not advance past it starts nothing. Until then the CR says so:

status.restoreInPlace.message: in-flight restore Job mysql-orders-pdx-inplace-restore
  is running under confirm=2026-04-17T14:32:00Z; confirm=2026-04-17T15:05:00Z
  starts a new run once it finishes

Removing or editing spec.restoreInPlace mid-run

Once a restore is accepted, status.restoreInPlace drives it, not the spec. Removing the field (by hand, kubectl replace, or a GitOps apply or prune) does not cancel destructive work:

The terminal status is never cleared, so confirmTokenUsed keeps blocking replays: re-applying an old manifest whose confirm is not newer does nothing. confirmTokenUsed holds the last confirm that actually ran (or may have, for RestoreInPlaceJobLost). A newer request that is rejected or cancelled does not replace it.

Changing loadOptions.includeSchemas after the restore was accepted does not change what the running restore fences. If the change lands before the restore Job is created, the run ends Failed (RestoreInPlaceRejected) without restoring anything; bump confirm to restore with the new scope. If the restore Job has already been created, the Job runs with the scope it was built with.

If a restore Job the run created disappears before the run sees it finish, the run ends Failed with RestoreInPlaceJobLost, because the outcome is unknown. Check the target's data before relying on it. The Job may have run, so its confirm counts as used: confirmTokenUsed records it (from status.restoreInPlace.runConfirm, the confirm of the run in flight), and re-applying the removed manifest does not start the restore again. The operator never builds a second Job for the same run.

Before it creates the Job, the operator records status.restoreInPlace.jobCreating together with that confirm, and it creates the Job only once that write has succeeded. A Job whose name never reached the status (an operator crash or a lost status write right after the Job was created) therefore still counts as created: if it disappears, the run ends RestoreInPlaceJobLost, not RestoreInPlaceCanceled. If the API server refuses the Job outright (validation or admission), the mark is withdrawn and the run still counts as never started. Any other create error keeps it, because the Job may exist.

Observing the state machine

kubectl get mysqlfailovergroup orders -o jsonpath='{.status.restoreInPlace}' | jq .

Phase sequence: Preflight → Fencing → Restoring → Resuming → Succeeded (or Failed at any point). Failed is terminal; inspect the Job logs, fix the underlying issue, then bump confirm to re-arm.

Required MySQL privileges

Consistent dumps (dump.consistent: true, the default) need BACKUP_ADMIN on the backup user in addition to standard replication and dump grants:

GRANT RELOAD, LOCK TABLES, PROCESS, REPLICATION CLIENT,
      SELECT, SHOW VIEW, EVENT, TRIGGER, BACKUP_ADMIN
      ON *.* TO 'backup'@'%';

Without BACKUP_ADMIN, set dump.consistent: false on the profile.

The restore account

Restore Jobs (bootstrap and in-place) run as the operator account (spec.credentials.operatorSecretName), not the backup account. A restore drops and creates schemas, sets global variables, and must keep working under the full-instance offline_mode fence. The backup role (credentials.backupSecretName) has none of those privileges. With the default spec.credentials setup the operator account holds ALL PRIVILEGES ON *.*. A custom operator account needs at least:

GRANT CONNECTION_ADMIN, SYSTEM_VARIABLES_ADMIN, PROCESS,
      REPLICATION_SLAVE_ADMIN ON *.* TO 'operator'@'%';

plus DDL and DML on the schemas being restored. CONNECTION_ADMIN keeps the account working under offline_mode, SYSTEM_VARIABLES_ADMIN sets it, PROCESS lets the eviction see other accounts' sessions, and REPLICATION_SLAVE_ADMIN lets the restore Job reset the primary's replication. A full-instance restore is rejected in Preflight without them. SUPER stands in for all but PROCESS. SYSTEM_USER is optional: without it the operator cannot kill sessions of SYSTEM_USER accounts during the fence.

local_infile and the restore account

Loading a dump uses LOAD DATA LOCAL INFILE, which the server rejects unless @@GLOBAL.local_infile is ON. MySQL 8 ships it OFF, so the restore Job turns it on for the duration of the load and puts the prior value back afterwards — the hardened posture outside the load window is unchanged. That requires SYSTEM_VARIABLES_ADMIN on the restore (operator) account:

GRANT SYSTEM_VARIABLES_ADMIN ON *.* TO 'operator'@'%';

This is checked before anything destructive runs. If local_infile is OFF and the operator cannot enable it, the Job exits with BLOODRAVEN_LOCAL_INFILE_UNAVAILABLE and no schema is dropped — the load could not have succeeded, so the restore fails with the data still intact rather than dropping the target and then failing to reload it. (A server already running local_infile=ON needs no extra grant.)

Point-in-time recovery (PITR)

Bloodraven can continuously archive MySQL binary logs so restores can target any second after a retained full dump, not just the dump's own capture point. For the interaction between PITR and emergency-failover RPO — specifically, when PITR does and does not narrow the data-loss window — see Durability and RPO → PITR and the RPO window.

How it works

Binlog archival runs inside the existing sidecar on every MySQL pod. The archiver goroutine:

1. Uses inotify on mysql-bin.index in the data PVC to detect rotations. A poll-tick every archivePollInterval (default 60s) is a belt-and-suspenders safety net in case an inotify event is missed.
2. Gates on @@read_only — only the primary uploads, and only sealed files. After a failover the new primary takes over within one scan cycle. A site keeps one binlog across role changes, so its files can hold both its own and replicated transactions; the replay planner orders them by their recorded predecessor sets. Before the first upload of each writable episode the sidecar appends a promotion record to its manifest.
3. For each newly sealed binlog file, extracts timestamp + GTID metadata, uploads the file to the referenced backup profile's storage, and appends an entry to a per-site manifest. Nothing is uploaded until the archive's identity has been created or verified.
4. Maintains its site's archive independently of upload eligibility. Read-only sites and sites whose MySQL role cannot be queried still run retention. Archive writes and cleanup remain serialized in the same sidecar goroutine.

Storage layout under the profile's prefix:

<profile-prefix>/binlogs/
├── archive.json              # write-once descriptor (plaintext)
├── manifest-<site-a>.json
├── manifest-<site-b>.json    # post-failover (or different primary)
├── <site-a>/mysql-bin.000042.<id>
├── <site-a>/mysql-bin.000043.<id>
└── <site-b>/mysql-bin.000001.<id>

Archive identity

Every archive has a write-once descriptor at <profile-prefix>/binlogs/archive.json. It is plaintext, also on an encrypted profile, so a reader can identify the archive before it has a key:

{"kind":"BloodravenPITRArchive","version":1,"archiveID":"ar1-<26 base32 chars>",
 "source":{"namespace":"prod","name":"lion","uid":"<group metadata.uid>"},
 "encryption":{"mode":"Passphrase","algorithm":"AES-256-GCM","keyCheck":"<43 chars>"},
 "createdAt":"2026-11-05T09:30:00.000000Z","createdBy":"iad"}

• archiveID is derived from the source group's metadata.uid and the normalized destination. A group deleted and re-created under the same name has a new UID, so it gets a new archive ID. The destination is s3://<bucket>/<prefix>/binlogs?endpoint=<endpoint> or pvc://<namespace>/<claim>/<subPath>/binlogs. Leading, trailing and repeated / in an S3 prefix are dropped, so /prod and prod name the same archive. Endpoint scheme and host are lowercased, the default port is dropped, and an empty endpoint means AWS. The operator publishes the result as status.pitr.archive.destination.
• encryption records the mode (None or Passphrase) and algorithm. For Passphrase the keyCheck lets a sidecar or Job prove it holds the archive's passphrase without decrypting anything; see Backup encryption.
• Manifests are version 2. Every manifest-<site>.json carries the descriptor's archiveId. Readers refuse version 1 manifests (PITRArchiveLegacy), newer versions (PITRArchiveVersionUnsupported) and a manifest that names another archive.

The first writer creates the descriptor, and only on an empty prefix. On a prefix that already holds version 2 manifests but no descriptor it refuses PITRArchiveDescriptorMissing (restore the descriptor, or wait for a DR replica to catch up); anything else there refuses PITRArchiveLegacy. The create is atomic (If-None-Match: * on S3, link(2) on a PVC), and a writer first proves the store enforces it: it creates .bloodraven-cas-probe-<16 hex> under the prefix twice, and only a precondition failure on the second create certifies the store. A store that accepts both creates, or rejects the header, refuses PITRArchiveStoreUnsupported. See S3 backups.

A writer (a site's sidecar, or pitr-quarantine) accepts a descriptor only when it names its own group's UID, an archive ID derived for its own destination, and its profile's encryption mode, algorithm and key:

A refusing site uploads nothing, lastError starts with the token, and status.pitr.archive turns state: Refused with the reason. Each writer re-reads the descriptor before every manifest save and retention delete, and stops (WARN archive descriptor changed) if it no longer verifies. Readers (pitr-download, the standby reconciler) accept any archive ID, but still verify the encryption, the keyCheck, and that every manifest names the descriptor's archive.

Promotion records and history integrity

Promotion records. The sidecar samples the server's role in its own goroutine every PEER_CHECK_INTERVAL (default 5 s), independent of archive I/O, so a busy upload never delays it. Each probe reads @@read_only, @@server_uuid, @@server_id and @@gtid_executed. A new writable episode starts at the first writable observation after the sidecar starts, after every read-only or failed probe, and also while writable when the server becomes a new incarnation: server_uuid or server_id changed, or gtid_executed no longer contains the previous observation's set (a reclone, a reset, a re-seed, a replaced datadir). For each episode the sidecar appends a promotion record {serverUuid, serverId, gtidExecuted, observedAt, probeInterval, readOnlyGtidExecuted} to its own manifest and reads the save back (INFO promotion recorded):

• observedAt is when the probe returned, on the sidecar's clock. A probe that ran longer than 1 s cannot open a record; uploads wait for one that does (lastError: promotion record: no role probe of this writable episode returned within 1s …).
• probeInterval bounds how long before observedAt the change the record reports happened: the time since the previous probe that still showed the earlier state, and at least PEER_CHECK_INTERVAL. For the first observation after a sidecar start that finds the server changed, it reaches back to the site's latest record.
• readOnlyGtidExecuted is the gtid_executed of the read-only observation of the same server directly before the episode, when there was one. It is recorded for diagnosis; the planner does not use it.

It uploads nothing in a writable episode until that episode's record is confirmed; a failed save shows as lastError: promotion record: …. A scan whose episode the sampler ended mid-scan does not write the manifest; its files wait for the next episode's record.

Incarnation-reset records. In any role, the sidecar also appends a record of kind Reset ({kind: Reset, serverUuid, serverId, gtidExecuted, observedAt, probeInterval}, WARN server incarnation reset recorded) when an observation shows another server_uuid or server_id than the previous one, or a gtid_executed that no longer contains it (a reclone, a reset, a re-seed). Before its first record or upload after a start, it compares its first observation with the site's latest record the same way (WARN server incarnation changed while the sidecar was not running); such a reset record's probeInterval reaches back to that latest record. A change seen at the first observation of a writable episode is carried by that episode's promotion record instead. Uploads wait until a reset record is saved.

Promotion and reset records share one list; the newest 256 are kept. Promotion records tell the replay planner which history each promotion started from (anchors for replica-copy witnesses, and loss evidence in latest mode); every record of a site also tells it when that site's server may have started a new episode or incarnation (see Witnesses archived only on a replica).

History integrity. On every scan, before and whatever the state of its archive session, every site checks the settings an archive needs to certify history:

• no replication filters (performance_schema.replication_applier_filters and replication_applier_global_filters are empty);
• no binlog filters (Binlog_Do_DB and Binlog_Ignore_DB are empty);
• binlog_format=ROW and binlog_row_image=FULL;
• replica_preserve_commit_order=ON (the operator owns this setting).

On the first violation the sidecar persists historyIntegrity{violation, detail, at} in its own manifest (WARN archive history unsafe), stops uploading, and reports ArchiveUnsafeHistory. violation is one of ReplicationFilter, BinlogFilter, BinlogFormat, BinlogRowImage or CommitOrder. The marker is permanent for that prefix: removing the setting does not clear it. Fix the setting, point spec.backup.pitr at a new prefix and take a full backup. A violation found while the archive is not yet verified (a replica before the descriptor exists, or a refused session) is reported at once and persisted at the first verified session; until then it is kept in a file in the sidecar's pod-lifetime state directory, so a sidecar container restart keeps it, but a pod restart before then loses a violation the server no longer shows. When the check itself cannot run (a query error or a missing privilege), uploads stop with lastError: history integrity: … and no marker is set.

A marker removes every restore target that needs that site's bytes, including targets before the violation, because nothing proves which of the site's archived files the setting affected. A new prefix restores only targets after it. While PITR is enabled, the operator therefore refuses the offending options in spec.mysqlConf instead of rendering them: they are left out of my.cnf (MySQL's safe default applies), reported by the condition PITRMysqlConfAccepted=False/PITRUnsafeSettingRefused, and a Warning PITRMysqlConfRefused. A marker then means an out-of-band change (SET GLOBAL, a hand-edited my.cnf). See Configuration.

Backup stamp

Every backup Job starts with a backup-stamp init container (operator image). It reads the dumped server's clock (UTC_TIMESTAMP(6)) and server_id over the same Service the dump uses, refuses a server_id other than the one the operator pinned for the source site (its spec.sites[].serverID), and writes <dump location>.bloodraven-backup.json, a sibling of the dump directory:

• backup and source ({namespace, name, uid} of the MysqlBackup and the failover group), pitrArchiveID, site and serverID;
• dumpFilters{includeSchemas, excludeSchemas} from the profile;
• serverClock{start, end} and createdAt.

The stamp is BRV1-encrypted with the dump passphrase when the profile encrypts. After a successful dump the Job adds serverClock.end: plaintext Jobs from dump.py, through the binary the init container installed in /run/bloodraven/tools; encrypted Jobs from backup-encrypt-upload after the upload. A failure to write or finish the stamp fails the backup, and its termination message starts backup stamp: . The mysqlsh containers run with TZ=UTC. Deleting or pruning the backup deletes the stamp with the dump, unless the backup's spec.deletionPolicy is Retain, which keeps both (and then a backup re-created with the same name fails at this step while the retained dump is there).

The operator pins the source group's UID and its active PITR archive ID on the Job (annotations bloodraven.shipstream.io/failover-group-uid and bloodraven.shipstream.io/pitr-archive-id) and copies them to MysqlBackup.status.failoverGroupUID and status.pitrArchiveID, recovering them from the Job if the first status write is lost. pitrArchiveID is empty when PITR was off. The stamp is what ties a dump to the archive it can be replayed from.

Each new manifest row includes an id: a 64-character lowercase content digest. Plaintext stores use SHA-256; encrypted stores use HMAC-SHA256 keyed by the archive passphrase with a binlog-specific domain, so object names do not expose a plaintext SHA-256. Files are identified by (name, id), so rebuilding or recloning a former primary can restart binlog numbering without overwriting or skipping retained history. Identical file bytes are recognized after sidecar restart. The sidecar caches local identities by file identity, size, and modification time; unchanged sealed files do not need to be rehashed on each poll.

Legacy rows keep their basename-only keys. Verification streams and persists contentId; historical updates are batched into one manifest write per scan. A memory cache avoids repeated local hashing while files remain unchanged. There is no persistent inode cache: restarts reread history gradually.

New rotations and changed files receive priority even when names were used before. One newest-first historical cursor hashes local files in 64 MiB/two-second yield slices and can pause mid-file. A remote legacy read that exceeds a slice continues draining in one bounded background worker; it never parks an HTTP response until the next scan. Transport failures resume S3 ciphertext at the consumed offset with Range and If-Match, preserving the decrypter and plaintext hash. Changed ETags or ignored ranges fail loudly. Only the scan goroutine publishes completed identities. These are scheduling budgets, not an I/O rate cap; foreground upload bandwidth still limits catch-up. This bounds historical work between rotations instead of making new logs wait for the entire archive.

Uploads pin an opened inode and verify both the parse/hash pass and the plaintext upload stream. Path replacement cannot substitute new bytes under old metadata. Encryption, verification and retention inspection stream through bounded buffers; no full binlog copies or writable container $TMPDIR are required. Downloads use only a final staging .part and atomic rename.

Per-site manifests remain separate, and replay considers every manifest under the prefix; there is no per-restore site list. The replay planner in pitr-download reads them all and orders the archived transactions by history, not by site or file name; see Restore to a target timestamp.

A missing legacy object is replaced only when a recorded content identity matches surviving bytes. Unidentified losses and divergent tails need explicit, reversible quarantine against a canonical full backup. The divergence audit records divergent rows (archived GTIDs a server that established its history does not hold); they stay in the manifest and feed the planner as loss evidence. Neither a divergent flag nor a quarantine is an acknowledgement: a restore still refuses until such transactions inside its window are in excludeGtids. Reuse guards require overlapping GTIDs from the same server UUID, different verified content identities, and different original commit or actual transaction-query identities, and a verified reuse refuses the whole archive. See Repair PITR history safely.

Upgrading to archive identity

v1.7.0 is a breaking change for existing archives. There is no in-place upgrade from v1.6.0: a v1.7.0 group is a new group, bootstrapped from a v1.6.0 backup (see Upgrade policy). What that group meets:

• Existing archives are refused. A prefix that holds version 1 manifests and no archive.json refuses PITRArchiveLegacy for writing and for reading. Sidecars keep running but upload nothing, and the group shows PITRArchiveReady=False / ArchiveLegacy. Give the new group's PITR profile a prefix that no earlier group has used and take a full backup; the new prefix gets a descriptor on the first upload. Keep the old prefix, its passphrase and the old cluster if you may still need to restore from it; this build cannot read it.
• S3 prefixes are normalized. /prod and prod are now the same destination. If you relied on the leading / to keep two archives apart, move one to a new prefix and take a full backup.
• Sidecar permissions. The sidecar needs s3:ListBucket, s3:PutObject and s3:DeleteObject, and the store must enforce conditional writes; see S3 backups.
• Restores. initFromBackup.pointInTime.archiveProfileName is required, and pointInTime.sites and allowUnverifiedCrossSite are removed (as is sites on verifications). Remove them from your manifests; the v1.7.0 CRDs no longer define them.
• replica-preserve-commit-order is operator-owned and forced ON; a spec.mysqlConf override is dropped.

Do not run an older sidecar against a version 2 archive. An older writer rewrites its manifest as version 1 without archiveId, and every current reader then refuses the whole archive as PITRArchiveLegacy.

Enabling PITR

Set spec.backup.pitr on the failover group and point it at an existing backup profile:

spec:
  backup:
    profiles:
    - name: primary
      storage:
        type: S3
        s3:
          bucket: lion-backups
          prefix: prod
          region: us-east-1
          credentialsSecretName: aws-creds
      retention:
        keepLast: 14
    pitr:
      enabled: true
      profileName: primary
      maxBinlogSize: "100M"     # MySQL max_binlog_size; controls
                                # rotation cadence and therefore RPO

When enabled, the operator:

• Injects max_binlog_size=<value> into the generated my.cnf (default 100M). Smaller values shorten the RPO gap (unarchived tail on a crashed primary) at the cost of more objects.
• Mounts the MySQL data PVC read-only into the sidecar container so the archiver can read sealed binlog files.
• Mounts the profile's storage credentials (for S3) or the backup PVC (for PVC) into the sidecar.
• Passes the archive identity to the sidecar (BLOODRAVEN_FAILOVER_GROUP_UID, BLOODRAVEN_PITR_ARCHIVE_ID, BLOODRAVEN_PITR_DESTINATION, BLOODRAVEN_PITR_ENCRYPTION_MODE, BLOODRAVEN_PITR_KEY_REVISION). A sidecar whose identity disagrees with its store settings refuses to start. For an encrypted profile the passphrase file is the archive's key snapshot, not your Secret.
• Rolls the MySQL pods to pick up the new config, once. The spec hash includes the resolved PITR render (bloodraven.shipstream.io/pitr-render on each site Deployment: enabled, destination, archive ID, storage fields, encryption mode, algorithm, key snapshot, region, endpoint, credentials Secret, poll interval) and maxBinlogSize. The profile name, the encryption selector, pvc.provisioned.size and pvc.provisioned.storageClassName are not part of it, so switching to an alias profile with identical storage and key does not roll pods. The ordered update also reports drift when a managed site's pitr-render annotation differs from the resolved render although its spec hash matches, so a hand-edited or lost annotation is repaired.
• Leaves the Deployments of a site that is no longer in spec.sites alone: they are never re-rendered. One that is scaled to zero with no pod is ignored; while one may still run, a backup of the active profile that sees different renders says so ("Deployments <names> belong to sites no longer in spec.sites and are never re-rendered: scale them to zero or delete them"). Admission refuses removing a site from spec.sites, so such a Deployment is a leftover from a write that skipped validation: scale it to zero or delete it.
• While the render is deferred (pre-binding Deployments), does not rewrite an existing site's ConfigMap either: its my.cnf keeps the pre-upgrade PITR settings, and only sites being created get the candidate's ConfigMap.

The archive's encryption identity is governed by the group's archive encryption binding: once it is Bound, a change to the profile's encryption or its Secret's bytes is refused on the PITRArchiveBindingReady condition and does not reach the pods. Rotation means a new prefix; see Backup encryption > Rotation.

Binlog retention

Archived binlogs are pruned only when every retained successful MysqlBackup for the group covers every transaction in that file, regardless of backup profile. The backup's pre-dump status.gtidExecuted is a conservative lower bound; completion time does not establish a safe replay boundary. Checking every backup protects dumps taken from lagging replicas and backups from other profiles used as PITR restore sources. Raw S3/PVC dumps and other sources without a retained MysqlBackup CR are outside this retention guarantee; preserve their replay windows separately.

New retirements pause while any non-deleting group backup is Pending, Running, or has an unknown phase. An overlapping dump can still finish with an older snapshot and need files that a newer successful backup already covers. Failed backups are terminal and do not hold retention. Persisted pending deletes remain authorized and can still finish.

The mechanism is pull-driven: the sidecar archiver polls the operator at /pitr-cutoff?namespace=X&group=Y&archive=Z&coverage=gtid-v1 (default once per hour), where archive is the sidecar's BLOODRAVEN_PITR_ARCHIVE_ID. The operator answers only when it equals the archive ID derived for the group and the destination of one of its archive bindings (404 otherwise, 400 when missing), and echoes it in the reply. Sidecars still on an older render send profile= and get 400 until they roll; retention defers meanwhile and nothing is deleted. A proof includes coverage: "gtid-v1", retainedGTIDs from every successful, non-deleting group backup and the minimum known StartTime as cutoffTime. Backup labels are not required. The operator validates the archive against the group's bindings through its informer cache before scanning backups, coalesces concurrent lookups, and allows one uncached lookup per caller IP per 30 seconds across that caller's connections. Junk query keys allocate no admission state and do not list backups. A caller cannot reserve a group/profile slot against other source IPs. The caller map and safe-deferral cache are each capped at 1,024 entries; a full caller map admits a new source by evicting the oldest entry. Saturation never denies a new caller merely because the map is full.

Deferrals are cached for up to 30 seconds; positive delete authority and lookup errors are never cached. A newly observed lagging or in-flight backup must participate in the next fresh proof. A limited caller receives HTTP 429 with Retry-After: 30 and no cutoff; the sidecar treats it as an error, logs at Warn and retries after 30 seconds. A disconnected caller cannot cancel another caller's shared lookup. Polling does not generate live API-server LISTs per request (AUDIT H2). Call the auxiliary Service directly and preserve pod source IPs; a proxy/NAT that merges callers into one source shares their limit. Forwarding headers are deliberately ignored.

Missing or malformed GTIDs, unknown start times, no retained backups, or in-flight backups yield no cutoff and a retentionDeferredReason. Legacy requests without the coverage capability also receive no cutoff. A new sidecar rejects time-only replies from older operators. Retention pauses safely during version skew; upgrade the operator before sidecars to stop older clients using time-only deletion.

Manifest rows record gtidCoverage with complete, gtids, and contentId fields. A proof's content ID must match the row's verified identity (keyed for encrypted storage). Coverage is complete only when the entire sealed file parses with valid CRC32 checksums and supported GTID events. Tagged and untagged sources remain distinct. Fully parsed files containing only format/previous-GTID/rotate/stop events have complete, empty coverage and can retire without transactions to replay. ZSTD transaction payloads are supported when they immediately follow a valid GTID event; malformed, orphaned, or nested GTID payloads remain unknown. Truncated files, anonymous transactions, disabled checksums, and unsupported events are retained. Previous GTIDs alone are not transaction proof.

Older rows, including complete proofs without a matching proof content ID, are inspected from streamed archived plaintext before retirement. One inspection computes coverage and identity together; both ID-aware keys and persisted legacy IDs must match the bytes. Each batch reads at most eight objects and stops starting another read after 64 MiB or two seconds. One large object may exceed the byte/time budget. The batch persists coverage and legacy contentId together; restarts skip verified work and retirement does not download those objects again. Eligible remaining backfill retries after 30 seconds, interleaved with normal upload scans, and reports normal progress at Info. A failed identity waits one normal retention interval (one hour by default) before inspection is retried; it does not force repeated 30-second downloads or block other files. Failure state is process-local, follows the row's name/ID/content ID, and is removed when its identity changes or the row leaves the manifest. Restarting the sidecar clears it. Inspection warnings identify actual failed attempts; repair the original object/permissions and allow the cooldown to expire. Unreadable or mismatched objects stay live, and unsupported files receive an explicit incomplete bound proof. No inspection scratch files are needed. The operator needs no archive credentials; sidecars perform reads/deletes.

GTID containment is also subject to an age guard: the file's last event must precede the cutoff, future cutoffs are ignored, and files less than 24 hours old are kept. The endpoint remains unauthenticated; isolate it with NetworkPolicy. retention deferred logs noRetainedBackups and backupInFlight at Info; metadata/protocol/file coverage problems remain warnings. Do not fabricate GTIDs or delete backups to force cleanup.

Retirement and physical deletion use two manifest writes. The first atomically moves eligible rows from files to an additive pruned list with deleted: false. Restores read only files, so an expired object is no longer advertised before deletion starts. After successful object deletes, a second write records deleted: true for each success. Failed deletes remain pending; a crash or failed acknowledgement write causes the next sweep to retry the idempotent delete, including when the object is already absent.

Each sweep retries persisted pending deletes before requesting a new operator proof. Already approved cleanup can therefore continue while the operator is unavailable or cannot authorize new retirements. An object still referenced by files is protected until a fresh retirement removes that reference. Attempts remain bounded by the retention interval.

A retirement record keeps name, the original archive id, key, previousGtids, firstEventTime, and deleted. A legacy record retains its empty archive ID and adds contentId, the verified content digest of its archived plaintext (keyed on encrypted stores), captured before deletion. New records use the archive ID as their content identity. Delete targets are reconstructed from the trusted prefix/site and validated name/ID; stored key values never select the object. Acknowledged records retain this identity bookkeeping. They are excluded from restore selection and active manifest counts/bytes.

Upload scans and backlog calculations consult all retirement records, including pending deletes. A matching filename and full content ID is already retired, so the sidecar does not upload it or re-add its live row, including after restart or while the operator is unavailable. Legacy records match their contentId; names and header timestamps alone cannot suppress a new file. Reusing a retired filename with different bytes remains eligible for archival.

Remote retention is independent of local MySQL expiry. The operator defaults local binlog expiry to 14 days, which can be overridden in MySQL configuration. A file can remain locally for replica catch-up after its remote object is retired; it stays retired rather than being uploaded again. The sidecar does not issue local PURGE BINARY LOGS commands.

Every running site's sidecar prunes its own manifest, including former primaries that are now replicas. Maintenance also runs when there are no sealed local files. Active coverage counts are refreshed after cleanup and exclude retirement records. An acknowledged record is compacted only when a readable local index no longer lists its name and the local file is absent. Missing/unreadable indexes or uncertain filesystem state keep the record. Pending deletes are never compacted. A site with no running sidecar still needs a separate archive-lifecycle plan.

Restore to a target timestamp

Add a pointInTime block to spec.initFromBackup:

spec:
  initFromBackup:
    source:
      backupRef:
        name: nightly-2026-04-14
    pointInTime:
      stopTime: "2026-04-15T09:30:00Z"
      archiveProfileName: source-archive # required: the archive to read
      # allowPartial: true               # accept the archive head if nothing is archived at or after the target
      # excludeGtids: "<uuid>:42"         # transactions never replayed

stopTime (T) is a whole second in RFC 3339, with Z or a ±HH:MM offset. The API server rejects a missing zone, a space instead of T, a lowercase t or z, a fractional second and an offset outside -23:59..+23:59. Commit times are still compared to the microsecond, so T cuts between transactions committed in the same second: those committed before T are replayed, the first at or after it is the witness.

What the target means. The restore finds the witness: the first archived transaction, outside the backup and excludeGtids, whose original commit time is at or after T. The result is exactly the history that transaction was committed on top of, cut just before it. Every replayed transaction committed before T, and no archived transaction committed before T is silently left out: either it is on that history, or it is in the backup, or you excluded it, or the restore refuses. When two candidates share the earliest microsecond, the one that precedes the other wins; candidates on different histories at the same microsecond refuse PITRStopAmbiguous ("choose a target ≠ <τ>").

The witness must be archived. If nothing at or after T is archived yet (the active binlog is archived only after it rotates), the restore refuses PITRTargetNotCovered: wait for the archive, run FLUSH BINARY LOGS on the source, or set allowPartial (see Partial outcomes).

The target must be after the dump. A dump cannot be un-applied, so T must be at or after the second the dump finished (mysqlsh's @.done.json end, rounded up), and every transaction of the backup must have committed before T. The operator does not check this at reconcile: status.completionTime includes the upload and would refuse valid targets. dump-meta and pitr-download check T against the dump itself and the dumped server's clock and refuse PITRBackupAfterTarget; see The backup frontier.

Archive profile

pointInTime.archiveProfileName is required. It names the spec.backup.profiles entry whose archive the restore reads, usually a profile pointing at the source group's prefix. The API server rejects a pointInTime without it: pointInTime.archiveProfileName is required: name the backup profile whose PITR archive the restore reads. The name is a DNS label of at most 63 characters.

The profile may not resolve to the restored group's own active PITR destination (spec.backup.pitr.profileName). Destinations are compared after normalization, so another profile with the same bucket, prefix and endpoint spelled differently is refused too. The restore fails before any Job is created with a RestoreBuildFailed Event and PITRArchiveDestinationShared. Keep spec.backup.pitr on a profile with its own prefix, so the restored group never writes into the archive it restores from. Even if it tried, the source's descriptor names another group, so its sidecars would refuse it as PITRArchiveForeign.

Archive binding and keys

The archive is read through a read binding of the restored group (see Archive encryption bindings). Before the restore Job is created, the operator runs one pitr-archive-check Job against the archive with the archive profile's key. When it verifies, the binding is Bound and the restore Job's pitr-download mounts its key snapshot (never the profile's Secret) and carries the label bloodraven.shipstream.io/pitr-key-secret. Until then:

• the restore waits (status.restore.phase: Pending, message starting PITRArchiveBindingNotReady) while the binding is being proven, or is refused for a reason you can fix with a Secret, profile or RBAC edit: KeyMismatch, ArchiveEncryptionMismatch, BindingLimitReached, or a Bound binding whose snapshot is lost. Fix the Secret, and the binding is retried. A check that cannot list the prefix keeps the binding Pending/Authenticating with an ArchiveListFailed: ... message (grant s3:ListBucket or make the volume readable) and is retried every minute;
• the restore fails with PITRArchiveBindingNotReady when the archive can never verify: ArchiveNotFound, ArchiveDescriptorMissing, ArchiveLegacy, ArchiveVersionUnsupported.

The key comes from the archive profile of the restored group: a new group cannot discover the source group's key snapshots, whatever retention the source keeps. The archive profile's encryption.passphraseSecretKeyRef must therefore hold the passphrase the archive was written with. After a rotation in the source group, or once its original Secret is deleted, either keep that Secret, or name the source group's retained key snapshot (the source's Bound binding's encryption.keySecretName, key passphrase, in the restored group's namespace) as the archive profile's passphraseSecretKeyRef. Without it the binding is refused KeyMismatch (wrong bytes) or the profile cannot resolve its Secret, and the restore stays Pending.

A restore that failed before its Job was created keeps its read binding (and any refusal on it) while it is retried; no check Job loop starts.

The archive profile is read-only while the restore has not succeeded: it cannot run backups (ProfileReadOnly). When the archive belongs to another group (a restore into a new group from the source group's archive), the profile stays read-only after the restore succeeds, because the binding keeps the source group's UID until you release it (see Read-only profiles). It also stays read-only after a successful restore while its destination has no binding at all, for instance a plaintext read binding lost with the status. Give the restored group a profile on its own prefix for its own backups and archive.

The dump is decrypted with spec.initFromBackup.decryption when set, else with the backup's own dump-key snapshot recorded in status.artifactStorage.passphraseSecretKeyRef (the Secret must exist), else the restore fails PITRBackupDumpKeyMissing. The profile's live Secret is never used for a dump.

Replay considers every manifest-<site>.json under the prefix, whatever the site names in the restored group; there is no per-restore site list. The restore needs list permission on the prefix (S3 s3:ListBucket, which covers ListObjectsV2); a listing failure reports PITRArchiveListFailed with a remedy for its cause: an S3 403, grant s3:ListBucket; a PVC permission error, make the archive volume readable by the Job's user; anything else is transient and retried.

The restore Job

A PITR restore Job runs four containers in order:

1. dump-meta (operator image) reads the dump's @.json, @.done.json and backup stamp, decrypting them with the dump key when the dump is encrypted (there is no plaintext fallback), and writes /pitr-dump-meta/dump.json: the backup GTID set B, the dump's end, the dumped server's clock and the stamp. It refuses a dump with no stamp (PITRBackupIdentityUnknown, take a new backup), a stamp that does not match the backupRef (PITRBackupIdentityMismatch), an inconsistent dump (PITRBackupInconsistent), a dump taken with schema filters (PITRBackupFiltered), and a dump that never finished (PITRBackupIncomplete: no @.done.json, or no serverClock.end). The dump's end is the latest instant it may have finished: mysqlsh truncates @.done.json's end to the second, so a whole-second end E counts as E + 0.999999 s and a target inside the dump's last second refuses PITRBackupAfterTarget. An encrypted dump read without a decryption passphrase refuses PITRBackupIdentityUnknown ("… is encrypted …, but the restore has no decryption passphrase …"), and so do a plaintext dump read with one ("… is not encrypted, but the restore decrypts this dump …") and a wrong passphrase ("… does not decrypt with the configured passphrase …"). A stamp larger than 1 MiB refuses PITRBackupIdentityUnknown, an @.json or @.done.json larger than 16 MiB PITRBackupIncomplete ("larger than its bound"). Transient store errors are retried in the container (6 attempts over about 31 s); when they persist it fails without a token and the Job's next attempt retries.
2. pitr-download opens the archive as a reader (descriptor, encryption, keyCheck, every manifest version 2 of that archive, no reset epoch), expecting the archive the stamp names. It plans the replay, downloads and checks only the files the plan and its proofs need, stages the segment files in /pitr-binlogs, and writes coverage.json (version 2) beside them, also on refusal. The archive-open reads (descriptor, manifest listing, each manifest) are retried like row reads and end without a token when a transport failure persists. A row whose object is gone and which its site's live manifest no longer lists (retention retired it after the Job read the manifests) ends planning as a transient error ("the archive changed during planning (row retired); retry"), never as damage or quarantine advice. Every downloaded row is authenticated by its digest before its bytes count: bytes that do not match the row's content ID refuse PITRArchiveObjectInvalid (digest), whatever the index pass made of them, and so does a manifest row whose size disagrees with its authenticated bytes and a manifest key naming a site outside ^[a-z0-9-]+$. The index pass also refuses PITRArchiveObjectInvalid for a file that ends inside a transaction or a transaction without its terminator, and PITRReplayUnsupported for an event after a terminator (an INCIDENT, for example). Staging keeps the node's filesystem above max(10 %, 256 MiB) free ("… would leave less than its floor …", PITRPlanTooLarge).
3. decrypt-download (encrypted dumps only) decrypts the dump into a staging directory.
4. mysqlsh runs restore.py: it re-checks coverage.json and the staged files' digests, loads the dump, re-proves the plan with the server's own GTID functions, replays the segments, and proves the result. Before the load it refuses PITRCoverageInvalid for a plan whose postStop intersects the replayed history (R6: postStop holds …) and for a segment whose transaction count differs from its GTID count.

The target's own transactions. The restore target is a fresh server, but not an empty one: image initialization and the first-boot users write transactions under its own server_uuid before the load. restore.py reads the target's gtid_executed and server_uuid before the load and keeps those transactions: the baseline check (R2) and the final proof (R6) expect them exactly, they are logged as BLOODRAVEN_PITR_TARGET_LOCAL gtids=<set>, and they are part of the certified executed set. A target that holds another server's transactions, or whose server_uuid the plan names, refuses PITRCoverageInvalid before the load; empty it (delete its pod and PVC together, as in A failed bootstrap restore) and retry. Writes on the target during the load or the replay still fail the proofs, so while spec.initFromBackup has not succeeded the operator defers its own credential reconciliation, and MysqlDatabases on the group wait in Pending with reason RestoreInProgress; both run once the restore has succeeded. R2 and R6 each run on a fresh session; when it cannot be opened, or the replayed gtid_executed cannot be read, the restore fails PITRReplayFailed with the message BLOODRAVEN_PITR_FAILED: cannot open a session to prove the loaded baseline (R2)|the replay (R6): … or BLOODRAVEN_PITR_FAILED: cannot read the replayed gtid_executed to prove the replay (R6): …, not PITRReplayMismatch.

The backup frontier

Before it looks for the witness, pitr-download finds the newest archived commit time of any transaction in B, reading the archived bytes, not manifest hints. It reads the archived rows that may hold members of B newest first and stops at a row whose manifest hint says it holds nothing newer than what it has read. A hint is unverified until its row is read, so the newest commit time is proven only when every row holding members of B was read; when the search stopped on a hint (the usual case for a backup spanning several files) it is a lower bound, and the server-clock check below applies:

The last two checks use the dumped server's own clock, which the stamp records when the dump finishes. They assume the servers' clocks agree to within 1 s (NTP). A backup whose newest commit time is proven from the bytes needs no such assumption. In practice a target at least 1 s after the dump finished is never refused by the frontier. coverage.json's backupNewestCommitTime may therefore be a lower bound. The frontier runs first, so a backup that is too new never reports PITRTargetNotCovered.

Target history and excludeGtids

Call the witness's history Λ, the backup B and excludeGtids X. Then:

• B must be on Λ. A backup holding a transaction that is not on the target history refuses PITRBackupNotOnLineage. X may not intersect B (PITRExcludeIntersectsBackup).
• Every transaction committed between the backup's newest transaction and T must be accounted for. Let D be the archived transactions (active or quarantined rows) committed after the backup's newest transaction and before T that are not in Λ, B or X. They were lost at a failover, written on a replica (errant), or belong to another branch. If D is not empty, the restore refuses PITRArchiveDivergentHistory:

  PITRArchiveDivergentHistory: N archived transactions committed between the
  backup's newest transaction (<t>) and <T> are not in the history of <w>
  (site <s>, committed <tw>): <D> (sites …, <min>…<max>). They were lost at a
  failover, written on a replica (errant), or belong to another branch. To
  restore <w>'s history, add them to excludeGtids, or choose a target at or
  before <min>, or use a backup taken after <max>.
  

  
  The message prints D up to 1 KiB; the full set is logged as BLOODRAVEN_PITR_DIVERGENT_SET gtids= and written to coverage.json. Copy it into excludeGtids and retry. A successful restore that excluded D reports pointInTime.excludedDivergentTransactionCount and a Warning PITRDivergentHistoryExcluded.

Transactions committed before the backup's newest transaction are outside D: every restore from that backup omits them anyway, so they say nothing about which history T selects.

Examples, with sites a (UUID A) and b (UUID Bu):

• Lossy failover, backup taken before it. a archived A:96-100, then died; b was promoted having applied only A:1-95, and wrote Bu:1-50. With a backup from before the failover:
  • T before A:96 committed: covered, on a's history.
  • T inside the lost tail (between A:96 and A:100): covered; the result is a's state at T, including the transactions later lost.
  • T after b started writing: PITRArchiveDivergentHistory with D = A:96-100. With excludeGtids: "A:96-100" the restore is b's history at T.
• Lossy failover, backup taken after it. A backup taken on b after the promotion has its newest transaction after the lost tail, so A:96-100 are outside D: the same targets are covered with no exclusion.
• Errant transaction. Bu:e was written directly on replica b. A target whose witness is archived only through b refuses PITRStopAmbiguous ("…predecessors include Bu:e … exclude it"); with excludeGtids: "Bu:e" it restores a's history. Once b was promoted and continued from Bu:e, later targets include it as history and need no exclusion. A backup taken on b after the errant holds it, so excluding it refuses PITRExcludeIntersectsBackup: use a backup taken on the other site or before the errant.
• Split brain. Both sites wrote after a fork at A:100. A target in the overlap refuses PITRArchiveDivergentHistory; the message offers both remedies. excludeGtids set to a's post-fork GTIDs keeps b's branch, and b's post-fork GTIDs keeps a's.

Verifications take the same excludeGtids; see Backup verification.

On-history exclusions

Excluding a transaction that is on the target history, for example an accidental DROP TABLE or a bad UPDATE, is allowed, and is the usual reason to set excludeGtids. The restore skips that GTID. It is not certified, though: a later transaction on the same rows is replayed from its full row image, which can carry the excluded transaction's values. The restore reports such GTIDs (excluded transactions on Λ that precede a replayed one) as pointInTime.onHistoryExclusions and onHistoryExclusionCount, the verdict reports onHistoryExcluded=<n>, and the operator emits a Warning PITROnHistoryTransactionsExcluded:

excluded <set> precedes replayed transactions on the target history; their
GTIDs were skipped but later row images may carry their effects; the result
is not certified for them

Excluded transactions that are not on the target history (lost, errant or another branch) are not on-history exclusions, and the result is certified without them. Check the affected rows after a restore with on-history exclusions.

Segments and continuity

The plan is a list of segments: a run of consecutive transactions in one archived file, replayed with mysqlbinlog --include-gtids=<segment> <file>. Each site keeps one binlog across role changes, so a single file can hold origin transactions, replicated ones, and both sides of a failback. The planner picks the plan with the fewest segments, and every segment starts at a position whose predecessor set, minus X, is exactly the backup plus every earlier segment: each replayed transaction runs on the state it was originally committed on. All segments go through one mysql --binary-mode session. restore.py re-proves every segment boundary with the server's GTID_SUBSET and GTID_SUBTRACT before replaying, and after the replay proves @@global.gtid_executed equals Λ minus X and holds nothing that was committed after the witness (PITRReplayMismatch otherwise). If mysqlbinlog fails, mysql is killed and the restore fails PITRReplayFailed with BLOODRAVEN_PITR_FAILED: mysqlbinlog exit=<rc> segment=<k>. If mysql fails, the restore fails PITRReplayFailed with BLOODRAVEN_PITR_FAILED: mysql exit=<rc> segment=<k> (…): <k> is the segment being streamed when the failure was detected, and because mysql reads ahead the failing statement may be from an earlier segment. A pipe or read failure appends its error, and a mysql still running then is killed (exit=-9).

When no plan exists, the restore refuses:

Witnesses archived only on a replica. When the witness w exists only as a replicated copy (its origin's own binlog was never archived, for example a short era with no rotation before a failback), its history is accepted only when every predecessor is either from w's origin or is known to precede w on the origin server. Two kinds of anchor prove that, with different rules:

• A promotion record of the origin server that does not hold w, observed more than δ (5 s) before w committed (observedAt + δ < commit time of w). The record is on the sidecar's clock and w's commit time on the server's, so the margin is needed.
• An archived origin file of that server: its transactions are on the same server's binlog as w, so the latest one that w's predecessors include anchors without the δ margin, however close before w it committed.

Either anchor is set aside, and the restore refuses PITRStopAmbiguous naming the unexplained GTID and why the anchor does not count, when:

• a later record of the anchor's site (a promotion record or an incarnation-reset record, of any server) was observed after the anchor (for an origin-file anchor, after its transaction's commit time minus δ) and no later than w's commit time plus that record's probeInterval plus δ. Such a record reports a new writable episode or a new server incarnation (reclone, reset, replaced datadir) that happened within probeInterval before it, possibly before w, so w may belong to a later episode whose history need not hold the anchor. A record without probeInterval counts as unbounded; or
• another server's promotion record, on any site, was observed between the anchor and w (±δ) and does not hold w: the origin lost its role and regained it in between, and its new era's record is missing or not provably before w.

The message names the record that set the anchor aside, for example the promotion records b#0 are not anchors, because a later record of site b, b#1 (an incarnation-reset record observed at …, reporting a change that happened up to 5s before it), is not provably after …. Neither a binlog file's first event time nor a clone's bridge row (a row with baseGtids) dates an incarnation; only the sidecar's records do. A replica built by CLONE at bootstrap therefore anchors normally once its promotion record is older than δ. A sidecar restart in the middle of an era writes a new promotion record of the same incarnation, so targets whose witness committed within (its observedAt − probeInterval − δ, its observedAt + δ] refuse this way, with the safe target after it. A promotion record is written within one role probe (PEER_CHECK_INTERVAL, 5 s by default) of the promotion, so a target whose witness committed between a new primary's first commit and its record's observedAt plus δ can refuse this way; the message prints the window and the safe targets (at or before the promotion, or more than δ after observedAt).

Clock skew

Commit times come from each origin server's clock. One margin, δ, fixed at 5 s in this release (it is not configurable), absorbs skew in the reissue check, the latest-mode loss check and the quarantine waiver. D itself uses no margin. The planner refuses PITRStopAmbiguous when skew makes the target ambiguous: a replayed transaction committed at or after T, or a member of D committed on top of the witness ("choose a target ≤ … or > …"). Branches whose transactions all carry times before the backup's newest transaction are treated as before the backup. When archived bytes of the backup are missing, the frontier falls back to the dumped server's clock and assumes skew below 1 s; see The backup frontier.

Latest mode

A verification with mode: Latest replays to the archive head. The head is the unique newest archived state that covers every other one, ignoring excludeGtids; two heads that each hold transactions the other lacks refuse PITRArchiveSplitBrain, listing each head's exclusive set. The active, unsealed binlog is not part of the archive.

The planner then checks for loss with the promotion records. A transaction g of the head, of any source, is lost at a promotion record R when g committed more than δ before R was observed (commit time of g < observedAt − δ), R does not contain g, g is not in excludeGtids, and g has an archived copy on a site other than R's own (R's own site's rows are its later history). Commits after a record, or within δ of it, are never counted as lost by it. The run then refuses PITRArchiveDivergentHistory ("the newest archived state includes <set>, committed before <site> was promoted at <observedAt> without them; they were lost at that failover"). The divergence audit's divergent rows are also loss evidence, and loss and audit findings are reported together. The message prints up to 1 KiB; the complete set is in coverage.json (divergent, lost) and logged as BLOODRAVEN_PITR_DIVERGENT_SET gtids=, for latest-mode loss refusals too. Add it to excludeGtids to verify what the cluster kept. A flagged loss clears only when excludeGtids holds every flagged transaction (D ⊆ X): a record is never set aside, and excluding a promotion's own later writes does not resolve it. Nothing archived beyond the backup refuses PITRNoArchivedReplay.

Partial outcomes

With allowPartial: true, a timestamp restore whose archive holds nothing at or after T replays to the archive head instead, with the latest-mode rules above. It succeeds with status.restore.provenance.pointInTime.outcome: Partial (no status.restore.reason) and a Warning PITRReplayTruncated naming the requested target and newestReplayedCommitTime. A partial replay must apply at least one transaction (PITRNoArchivedReplay otherwise). allowPartial relaxes nothing else: when a witness exists, a gap or divergence still refuses. Verifications have no partial override.

Restore refusals

The failing step writes its reason to its termination message, token first; the restore records it in status.restore.reason and message, and a verification in its Verified condition. The containers run in order, so the first applicable refusal wins, in this order:

0. Before any Job is created (RestoreBuildFailed): a MysqlBackup whose status records no source failover group UID has no stamp and refuses PITRBackupIdentityUnknown at once, and so does a direct S3 or PVC source at the root of its store (no prefix or subPath; a leading / in an S3 prefix is ignored). PITRTargetInvalid refuses an unparsable or fractional target, an excludeGtids entry that does not start with a server UUID, an excludeGtids whose canonical set exceeds 64 KiB, and a verification against a group with spec.backup.pitr disabled. A verification records these tokens as its Verified reason.
1. The dump (dump-meta, which runs first): PITRBackupIdentityUnknown (no readable stamp), PITRBackupIdentityMismatch (the stamp names another backup or source group), PITRBackupIncomplete (no @.json), PITRBackupInconsistent (no gtidExecuted in @.json), PITRBackupIdentityMismatch (the backup's recorded gtidExecuted is not in @.json), PITRBackupInconsistent (gtidExecutedInconsistent), PITRBackupFiltered, then PITRBackupIncomplete (no @.done.json, or a stamp without serverClock.end).
2. Archive open (pitr-download): PITRArchiveListFailed, PITRArchiveNotFound, PITRArchiveVersionUnsupported, PITRArchiveLegacy, PITRArchiveDescriptorMissing, PITRArchiveIdentityMismatch, PITRArchiveEncryptionMismatch, PITRArchiveKeyMismatch; then PITRArchiveHistoryReset.
3. PITRExcludeIntersectsBackup.
4. The frontier: PITRArchiveNamespaceReuse, then PITRBackupAfterTarget.
5. The witness: PITRTargetNotCovered, PITRStopAmbiguous, PITRArchiveUnsafeHistory.
6. Lineage: PITRBackupNotOnLineage, PITRStopAmbiguous, PITRArchiveDivergentHistory.
7. Identity and order across copies: PITRArchiveNamespaceReuse, PITRArchiveOrderInversion.
8. The search: PITRArchiveUnsafeHistory, PITRArchiveGap, PITRArchiveOrderConflict, PITRPlanTooLarge.
9. PITRReplayUnsupported (XA in the replay).

PITRArchiveObjectInvalid (a digest, hint or parse mismatch, or an active row whose object cannot be read) refuses as soon as it is found. restore.py adds PITRCoverageInvalid (coverage.json or the staged files do not match the request, or a plan check fails), PITRArchiveIdentityMismatch (the plan's archive was not created by the dump's source group, or is not the archive the dump's stamp names), PITRExcludeIntersectsBackup, PITRNoArchivedReplay, PITRReplayUnsupported (no mysqlbinlog), PITRReplayMismatch and PITRReplayFailed (served for the script's BLOODRAVEN_PITR_FAILED: ... message, which status keeps). The archive identity tokens are explained in Archive identity and under Archive profile:

How the operator reads the outcome

A PITR Job is certified only by the verdict restore.py writes to the main container's termination message as its last action:

BLOODRAVEN_PITR_VERDICT v=2 mode=timestamp stop=2026-04-15T09%3A30%3A00.000000Z outcome=TargetCovered archive=ar1-… encryptionMode=Passphrase sourceUID=… sourceNamespace=prod sourceName=orders segments=2 files=2 transactions=1234 lastFile=iad%2Fmysql-bin.000042 newest=2026-04-15T09%3A29%3A59.981204Z divergent=0 onHistoryExcluded=0 executedSha256=… sites=iad%2Cpdx
BLOODRAVEN_PITR_RESULT v=1 executed=<gtid_executed>

The layout is in the log schema.

The operator accepts it only from the main container of a Succeeded pod the Job owns, terminated with exit code 0, and only when its mode, target, outcome, archive encryption and source group match what was rendered. It then writes status.restore.provenance.pointInTime (PointInTimeResult): outcome, archive identity, segment, file and transaction counts, the last replayed file, the newest replayed commit time, the excluded-divergent and on-history counts, and gtidExecuted with its SHA-256 (gtidExecuted is left empty when longer than 2048 characters). The verdict is built to fit the main container's share of the kubelet's termination messages (12 KiB split among the pod's containers, at most 4 KiB each; the operator renders it as BLOODRAVEN_TERMINATION_MESSAGE_LIMIT, default 3072): the optional on-history line is dropped first, then the executed set gives way to its digest, and a verdict that still does not fit is refused (verify.sh: PITRCoverageInvalid: restore.py's verdict is N bytes, over …) rather than published cut. A missing, truncated or mismatched verdict, or a pod that is already gone, fails the restore with PITRCoverageInvalid. Pod logs supply only best-effort GTID coordinates and never change the outcome. A failed restore reports the earliest failing container's message, is recorded once, and is not re-read.

Load options and schemas

The replayed transactions keep their original GTIDs. For a bootstrap restore the operator loads the dump with updateGtidSet: "replace" and skipBinlog: true, which sets the new primary's gtid_executed to the dump's @.json gtidExecuted (B); restore.py checks that before replaying. The operator adds updateGtidSet itself; it is not a loadOptions field. A logged load would also write the whole dump to the new primary's binlog under new local GTIDs, so initFromBackup.loadOptions.skipBinlog: false cannot be combined with pointInTime: the restore fails with a RestoreBuildFailed event before any Job is created. The peer site gets the data by cloning the restored primary once the restore succeeds.

Replay covers every schema, so the dump and the load must too. loadOptions.includeSchemas or excludeSchemas with pointInTime refuses PITRBackupFiltered at reconcile, and so does a dump made with schema filters, in dump-meta.

Upgrading and downgrading

PITR restore and verification Jobs rendered by an older operator write no version 2 verdict and fail PITRCoverageInvalid. Dumps taken before v1.7.0 have no stamp and refuse PITRBackupIdentityUnknown: take a new full backup after the archive has its descriptor. An older release cannot read version 2 archives or stamps; keep the old cluster with the old prefix if you still need it.

Observability

• Archiver status: GET /archiver/status on the sidecar (mysql-<fg>-<site>:8080) returns {enabled, storeReady, primary, lastScanAt, filesArchived, lastError, storageType, manifestPrefix, site, uploadFailures, consecutiveFailedScans, ...} — handy for kubectl exec poking or a Prometheus scraper. enabled:false means PITR is not configured on that pod. enabled:true with storeReady:false means archiving is configured but stopped because the archive store cannot be built (unreadable passphrase or S3 credentials file, invalid endpoint, empty PVC mount path, PVC mount path that is not a mount point or not writable); lastError reads archive store init: <cause> and uploadFailures (and bloodraven_archiver_upload_failures) rises by one per failed attempt. The archive identity adds archive{destination, archiveID, encryptionMode, state, reason, keyRevision} (state is AwaitingDescriptor, Verified or Refused), promotionRecordedAt, historyIntegrity (the persisted marker, or absent), and divergentRows / quarantinedRows counted in the site's own manifest.
• Archive store initialization failures do not stop the sidecar. The sidecar keeps serving /health (readiness, which checks MySQL only), the startup safety net and self-fencing, so the pod stays Ready and in every Service. It retries building the store every 5 s, doubling to at most 5 minutes (with jitter), logging failed to init archive store per attempt and archive store initialized after retries once it succeeds; fixing the Secret projection, credentials or endpoint is enough, no restart needed. A missing archive volume needs a pod spec change, which restarts the pod and its MySQL; on the primary site, run a planned failover away from it first. Sidecar images before v1.6.0 exited instead and crash-looped the container, which removed the pod from every Service.
• The PVC archive store checks its volume. At store construction the PVC mount path (/run/bloodraven/pitr-archive) must be a mount point listed in the container's /proc/self/mountinfo, not a directory on the container filesystem, and the sidecar must be able to create a file under the archive prefix. Otherwise the store fails with pvc archive store: <path> is not a mount point (it does not exist) (the usual result of a missing volume, since the image has no such directory), ... is not a mount point (it is inside <mount>, fstype <type>) or ... is not writable: <cause>, and the group shows PITRArchiveReady=False / ArchiveStoreInitFailed. The operator always renders the pitr-archive volume, so a missing mount means something removed it from the pod (a mutating webhook, a policy engine or a manual edit). A declared volume whose claim cannot be bound or mounted still leaves the pod Pending (SidecarUnreachable). The check proves that a volume is mounted, not that it is a durable PVC (an emptyDir at that path passes), and it runs when the store is built, not on every write, so PITRArchiveReady=True still does not prove PVC backing; verify the pod's volumes and claim. The same mount check runs in every Job step that opens a PVC store, which fails at start when its PVC is not mounted at the configured path: the readers dump-meta, pitr-download and decrypt-download, and the writers backup-stamp (the first step of every PVC backup, encrypted or not), encrypt-upload and pitr-quarantine. The writers also probe write access, so a PVC backup whose volume the Job's user cannot write fails in backup-stamp before mysqlsh starts. The <cause> of is not writable names the fix: permission denied is ownership or fsGroup (the sidecar runs as UID 999 unless overridden); read-only file system is the claim's access mode, or a filesystem the kernel remounted read-only after an I/O error (check the node's dmesg); no space left on device means the volume has no free inodes (expand or prune it). The probe creates an empty file and writes no data, so a volume that is only out of data blocks still starts the archiver, which reports it as ArchiverError while retention keeps deleting retired objects to free space.
• PITRArchiveReady condition: the operator turns its 30 s polls of every site into one condition on the group (kubectl describe mfg <group>), with a Warning PITRArchiveNotReady Event when it goes False ("PITRArchiveReady is False (): ") and a Normal PITRArchiveRecovered Event on recovery ("PITRArchiveReady is True again: every site reports …"). Reasons: ArchiveStoreInitFailed, ArchiverError (2 consecutive failed archiver scans, consecutiveFailedScans ≥ 2; one failed scan never sets it), ArchiverDisabled, SidecarUnreachable (3 failed polls over at least 60 s, or 5 minutes for a sidecar that has not answered since the operator started, which is also how a crash-looping older sidecar shows up), SidecarOutdated (an enabled archiver reports no archive-identity block: spec.sidecar.image pins a sidecar older than v1.7.0, so upgrade or clear it, or, right after an upgrade to v1.7.0, the site Deployments' update is deferred and the old pods keep their sidecars, so switch to a new prefix (PITR archive identity); it ranks after ArchiverError and before ArchiverDisabled, and is Unknown/AwaitingRollout while that site's ordered update is pending, at most 30 minutes), and the archive identity refusals ArchiveForeign, ArchiveCopy, ArchiveLegacy, ArchiveDescriptorMissing, ArchiveEncryptionMismatch, ArchiveKeyMismatch, ArchiveVersionUnsupported, ArchiveStoreUnsupported and ArchiveUnsafeHistory, which rank after ArchiveStoreInitFailed and before ArchiverError when sites disagree. ArchiveUnsafeHistory comes only from a site's marker, with an edge-triggered Warning PITRArchiveUnsafeHistoryDetected; unsafe keys in spec.mysqlConf are refused by the separate condition PITRMysqlConfAccepted instead (see Operator-owned MySQL settings). Right after PITR is enabled, sites still waiting for their ordered update report Unknown/AwaitingRollout instead of ArchiverDisabled, for at most 30 minutes and only when the PITR spec names a profile that renders an archiver. See the condition reference. The condition reports whether archiving is running now, not how much history is restorable. status.pitr oldest/newest archived times bound retention and freshness, not restorability: only a restore or a timestamp verification proves a target is covered.
• status.pitr.archive summarizes the identity the sidecars observed at the active destination: destination, archiveID, encryptionMode, state (Verified only when every site verified one archive ID; Refused when any site refuses, with reason), keyRevision and observedTime. Only reports for the active destination count, so a pod still on an older render does not affect it.
• Job logs: dump-meta logs dump metadata (location, backup and source UIDs, archive ID, site, dump end) and, on refusal, read dump metadata with the token; pitr-download logs pitr plan (witness, ground truth, segments, files, sites, replayed, divergent and on-history counts) and, on refusal, download site binlogs with the token, and prints BLOODRAVEN_PITR_DIVERGENT_SET gtids= with the divergent set; the restore container prints BLOODRAVEN_PITR_SEGMENT per segment, BLOODRAVEN_PITR_DIVERGENT count= and BLOODRAVEN_PITR_DIVERGENT_SET gtids=, BLOODRAVEN_PITR_RESULT and the verdict. See the log schema.
• Events: Warning PITRDivergentHistoryExcluded when the replay excluded transactions outside the target history, Warning PITROnHistoryTransactionsExcluded when an excluded transaction precedes a replayed one, and Warning PITRReplayTruncated, on the restored group or the verification.

Known limitations

• The planner refuses whenever the archive cannot prove the target history: ambiguous targets, missing or corrupt objects, copies that disagree, and orders no server produced. Preserve the evidence and follow PITR history repair: recover exact bytes, or quarantine an abandoned row and take a full backup after it. Verified GTID reuse requires a new prefix and a full backup.
• The planner sees only archived bytes. An order inversion between two sites whose bytes were never archived or were pruned is not detected; errant transactions are unsupported history. An empty transaction injected to repair an errant is detected when the backup was taken elsewhere, but a non-empty injected copy is not.
• Every limitation of point-in-time recovery is listed in Known limitations.
• Errant or lost transactions committed inside a restore window block that restore until they are in excludeGtids; a backup taken after them needs no exclusion. See Target history and excludeGtids.
• A recloned old primary keeps its server_uuid. After a lossy failover it can be promoted again and reissue the lost GTID numbers with different transactions, and every restore from the archive then refuses PITRArchiveHistoryReset. The operator warns with PITRArchiveReissueRiskDetected (also for a promotion after a NoPrimary interval on a group that has failed over before); start a new prefix and take a full backup at once. See PITR history repair.
• A writer whose binlog is not yet archived is invisible to planning. A demoted primary's active binlog is uploaded only if that site is writable again when it seals, and a reclone wipes unuploaded files, so a failback with a reclone can leave a tail that is never archived; restores that need it refuse PITRArchiveGap ("never archived"). Before a planned failover away from a primary, run FLUSH BINARY LOGS on it and wait until the sealed file is in its manifest; see Known limitations.
• Older writers may already have discarded rows, clone baseGtids, coverage, retirement or recovery provenance. New readers cannot recreate absent bytes or unknown historical issuance. Preserve original manifests, keys, compatible images and backup/generation pairing; follow the upgrade restrictions above or start a new prefix and full backup.
• Legacy verification streams archived objects, incurring storage reads and S3 egress. Comparison results and failure cooldowns are process-local; restarts can repeat I/O and defer retention. Batch byte/time budgets are soft scheduling bounds, not guarantees on individual I/O latency. Heavily reverse-ordered GTID histories still have quadratic interval-index insertion cost. Unresolved issuance can defer all new retirements for the site; failures remain in status and log at WARN with a repair/quarantine remedy. Monitor archive capacity while resolving them.

Metrics

The operator emits six Prometheus metrics for backups. All are labelled by {group, profile} unless noted.

Terminal metrics are emitted exactly once per terminal reconcile, via a semantic-equality check on the computed next status plus a stable completion timestamp derived from the Job's terminal condition. This means re-reconciling an already-terminal CR does not produce duplicate counter increments or repeat completion timestamps. Deferred size observations can refresh bloodraven_backup_last_size_bytes when the repaired backup is at least as recent as the latest observed success. Older repairs do not replace a newer backup's size.

Completion metadata can be repaired afterward without repeating terminal counters, completion Events, or the recorded completion time. A successful Job must have a known artifact destination before it becomes Succeeded; missing optional log observations do not delay success once that destination has been recovered from the Job.

Typical alerts:

• Stale backup: time() - bloodraven_backup_last_success_timestamp_seconds > 86400
• Repeated failures: increase(bloodraven_backup_runs_total{result="failure"}[24h]) > 3
• Runaway duration: histogram_quantile(0.95, rate(bloodraven_backup_duration_seconds_bucket[1h])) > 3600
• Artifacts not being deleted (deleted backups stuck Terminating): increase(bloodraven_backup_artifact_cleanup_failures_total[6h]) > 0. The counter resets when the operator restarts, so use increase().

Monitoring

• kubectl get mysqlbackups -A — phase and Ready (-o wide adds location and size).
• kubectl describe mysqlbackup <name> — the MetadataComplete condition and completion-metadata warnings.
• kubectl describe mysqlfailovergroup <name> — rollup under status.backupSchedules[] (including lastSuccessfulBackupName, lastRetryAttempt, nextRetryTime) and status.lastBackupTime.
• kubectl logs job/mysqlbackup-<name> — live dump progress, plus the final BLOODRAVEN_DUMP_COMPLETE sentinel line with the structured location, sizeBytes, gtidExecuted, binlogFile, binlogPos fields. New sentinels include v=2 and percent-encode token values (%20 for a space, %25 for a percent sign); underscores remain literal. The operator decodes these values into status.
• Prometheus metrics above.
• Events on the MysqlBackup CR: BackupStarted, BackupSucceeded, BackupFailed, BackupMetadataIncomplete, InFlightFailover, ArtifactCleanup*.
• Events on the MysqlFailoverGroup CR: BackupRetryScheduled, RestoreTargetUnavailable.

The sentinel's location is only cross-checked against the recorded destination (MetadataComplete); restore, verification and cleanup all use status.artifactStorage.location. Do not edit that location by hand.

The artifact destination is recorded in status.artifactStorage when the operator creates the backup Job, before the Job runs. If that status write is lost, the destination is recovered from the backup's own producing Job: its main container's destination environment, expanded the way Kubernetes expands it ($(NAME) uses earlier known values and $$ escapes a dollar sign). A value that depends on a runtime source the operator cannot read (a Secret or field reference) is not a known path, so such a Job yields no destination. The completion log never supplies the destination, and today's profile is never used to reconstruct one. Restore, verification and cleanup preserve literal dollars in the recorded path; status.artifactStorage.location is the artifact identity.

A backup certifies only the destination its Job really wrote. The operator compares the whole recorded destination (storage type, location, bucket, endpoint, region, claim and passphrase key) with the one the producing Job writes, at three points:

• When the backup Job is gone mid-run and a replacement would be built from the current profile: if the profile's destination changed since launch, the replacement is refused and the backup fails with Ready=False/BuildJobFailed ("... would not write to the destination recorded at launch ...; create a new backup").
• When the Job succeeds: a Job that wrote somewhere other than the recorded destination fails the backup with Ready=False/DestinationMismatch and a BackupFailed Event, because the recorded destination holds no artifact.
• Before cleanup: a mismatch blocks it with the ArtifactCleanup reason DestinationMismatch (see Failures and retries).

A Job whose destination environment cannot be read is compared by location only.

Completion logs supply only the optional observations: size, GTID and binlog position. If the logs are temporarily unavailable, the operator keeps the successful phase and retries log collection every 15 seconds for up to ten minutes after completion. Retries survive operator restarts. Logs that were read but carry no usable sentinel end the collection.

MetadataComplete=True means the completion sentinel was collected and its location agrees with the recorded destination. False with SentinelUnavailable or SentinelMissing means optional observations may be absent; the recorded status.artifactStorage.location remains usable. LocationMismatch means the sentinel named another location; the recorded destination is kept and the sentinel's is ignored. DestinationUnavailable means neither the status nor the producing Job supplied a destination: on first completion it sets Ready=False and retries with an error. An already-Succeeded record instead records the condition and ends that repair attempt without an error, allowing retention and deletion to proceed. Successful metadata repair preserves an already-True Ready condition; recovering a location only for cleanup does not mark a Running or Failed backup ready.