Backup verification

An untested backup is a schrödinger backup — it is both good and bad until you try to restore it, which you usually do during the incident that made you reach for it. Bloodraven's verification feature closes that loop: it periodically restores the most recent successful backup into a throwaway MySQL instance to prove the backup is actually loadable, and exports a freshness gauge so alerting can fire on stale or broken verifications.

The same CRD supports dump-load verification, optional point-in-time recovery (PITR) binlog replay, and configurable scalar sanity queries. You can start a run from its schedule, a raw CR, or the kubectl bloodraven verify-backup command.

Concepts

• MysqlBackupVerification CRD — one CR per verification run. Created ad-hoc via kubectl create or by the scheduled CronJob that the operator renders from a profile's verification block. Tracks phase, start/completion times, resolved backup reference, and the names of the ephemeral Pod and Job used for the run.
• Verification schedule — spec.backup.profiles[].verification on the failover group. When enabled: true, the operator materializes a CronJob whose pod runs bloodraven trigger-verification to create a MysqlBackupVerification CR.
• Ephemeral instance — each run provisions temporary volumes and a dedicated Job pod. The pod boots mysqld on its temporary datadir with --bind-address=127.0.0.1, then delegates the actual util.loadDump() to the shared restore.py script against that local instance. No external Service is created; the verification instance is never reachable from outside the pod network namespace.

What counts as "verified"

The default verification contract is the backup dump loads into an empty mysqld without error. That covers the failure modes that cause most real disasters — corrupt object store, truncated dump, missing credential rotation, storage class regression, mysqlsh version incompatibility. When you also set spec.pointInTime or spec.sanityCheck the contract extends to "archived binlogs replay on top of the dump" and "a caller-supplied SELECT returns a result that matches an expectation," respectively.

The feature deliberately does not cover:

• Logical equivalence with the live primary (a different tool).
• Application-level rehearsal of writes / traffic cutover (wishlist #7, #11).

Encrypted profiles are supported. The verification Job uses the same bloodraven decrypt-download init container as restore, with the backup's own dump-key snapshot recorded in status.artifactStorage.passphraseSecretKeyRef (never the profile's live Secret, so a later rotation does not affect it), and loads only the decrypted artifact. A missing recorded Secret fails the verification at launch, before any Job is created, with PITRBackupDumpKeyMissing ("backup <b> is encrypted and the Secret <s> it recorded as its dump key is gone; restore that Secret"). An encrypted backup taken before v1.7.0 recorded no dump key and always fails that way, since a verification has no decryption override: restore it with decryption into a scratch group to check it, and verify a backup taken after the upgrade. See Backup encryption: Verification.

Logical dumps also retain ENCRYPTION='Y' table DDL from servers using encryption at rest. Verification starts MySQL with a new writable component_keyring_file keyring in the temporary data volume. Restored tables receive new keys; the verification Job does not mount or reuse the source server's escrowed keyring. The launcher copies mysqld and writes its component configuration into the writable runtime volume so verification still works with a read-only container filesystem. The keyring lives on the same temporary volume as the restored data, so verification is not a confidentiality control for node storage.

Encrypted-table restore needs component_keyring_file.so in the MySQL image (spec.backup.image). If the component is missing or cannot be started, the Job logs a WARNING: verification keyring unavailable: <reason> line and starts the stock mysqld instead, so dumps without encrypted tables keep verifying. A dump that does contain ENCRYPTION='Y' tables then fails at restore with MySQL's own error.

Scheduling a verification

Attach a verification block to any BackupProfile:

apiVersion: shipstream.io/v1alpha1
kind: MysqlFailoverGroup
metadata:
  name: orders
spec:
  backup:
    image: container-registry.oracle.com/mysql/community-server:9.7
    profiles:
      - name: nightly
        storage:
          type: S3
          s3:
            bucket: orders-backups
            prefix: nightly
            region: us-east-2
            credentialsSecretName: orders-backups-creds
        retention:
          keepLast: 14
        verification:
          enabled: true
          schedule: "0 5 * * *"            # after the 02:00 backup finishes
          concurrencyPolicy: Forbid
          ttlSecondsAfterFinished: 0       # delete each Succeeded run's Pod at once
          retention:
            keepLast: 30                   # keep last 30 Succeeded runs (default)
            keepFailed: 10                 # keep last 10 Failed runs (default)

verification.retention applies to every MysqlBackupVerification of the profile, scheduled or created by hand. Succeeded and Failed runs are counted separately, so a run of failures never evicts the last successful verification. An omitted field takes its default; an explicit 0 keeps none, deleting each run two minutes after the operator records its verdict (or after an operator restart, whichever is later), so kubectl bloodraven verify --wait still reads it.

The operator reconciles this into a CronJob named mysql-<group>-verify-<profile> (a name over 52 characters, the longest CronJob name Kubernetes accepts, keeps its first 43 characters plus - and an 8-character hash) whose pod invokes bloodraven trigger-verification --group=<group> --profile=<profile>. That subcommand creates a MysqlBackupVerification CR, which the verification reconciler picks up.

Use concurrencyPolicy: Forbid (the default) unless you are confident two verifications of the same profile can run side by side; they each need their own node ephemeral storage and Allow just stacks failed runs.

Running a one-off verification

Use the kubectl plugin to verify the latest successful backup for a profile:

kubectl bloodraven verify-backup orders --profile nightly -n bloodraven

Add --backup <name> to select a historical backup, or --wait to block until the run reaches Succeeded or Failed. See the kubectl plugin reference.

You can also create the MysqlBackupVerification CR directly:

kubectl create -f - <<'EOF'
apiVersion: shipstream.io/v1alpha1
kind: MysqlBackupVerification
metadata:
  generateName: orders-nightly-drill-
  namespace: bloodraven
spec:
  failoverGroupRef: { name: orders }
  profileName: nightly
  # backupRef omitted → selects the latest Succeeded MysqlBackup once
  # for (group=orders, profile=nightly), then pins that source for this run.
EOF

To verify a specific historical backup:

spec:
  failoverGroupRef: { name: orders }
  profileName: nightly
  backupRef:
    name: orders-nightly-20260413-0200

backupRef.name is required when backupRef is present: an empty backupRef: {} is refused at admission rather than read as "latest".

PITR binlog replay

When the failover group has spec.backup.pitr.enabled: true, a verification can confirm that the archived binlog stream replays cleanly on top of the dump. Add spec.pointInTime:

spec:
  failoverGroupRef: { name: orders }
  profileName: nightly
  pointInTime:
    mode: Latest            # required: Latest | Timestamp
    # stopTime: "2026-04-15T09:30:00Z"    # required for Timestamp, forbidden for Latest
    # excludeGtids: "<uuid>:96-100"               # transactions never replayed
    # archiveProfileName: old-prefix   # read another profile's archive (default: spec.backup.pitr.profileName)

A verification runs the same replay as a restore (the dump-meta, pitr-download, decrypt-download and restore.py steps described under The restore Job) against an ephemeral mysqld, so a Succeeded verification proves that a restore of that backup to that target would work.

mode is required inside pointInTime; for a verification without replay, leave pointInTime out.

• mode: Latest — replay to the archive head, with the latest-mode rules: a split brain refuses PITRArchiveSplitBrain, transactions lost at a failover refuse PITRArchiveDivergentHistory until they are in excludeGtids, and nothing archived beyond the dump refuses PITRNoArchivedReplay, so an empty archive cannot count as PITR verified. The active binlog is not part of the archive.
• mode: Timestamp — restore the history of the first archived transaction committed at or after spec.pointInTime.stopTime: RFC 3339 with an explicit zone and whole seconds (a missing zone, a fractional second or a lowercase t/z is rejected at admission). stopTime is required with Timestamp and forbidden with Latest. The target must be after the dump finished (PITRBackupAfterTarget) and the archive must hold a transaction at or after it (PITRTargetNotCovered). There is no allowPartial for verification.

Which archive. pointInTime.archiveProfileName names the backup profile whose PITR archive the replay reads (a DNS label of at most 63 characters). It defaults to the group's spec.backup.pitr.profileName; with PITR disabled there is no default, and a pointInTime verification without archiveProfileName is refused PITRTargetInvalid ("... so group <g> has no active PITR archive to replay; set pointInTime.archiveProfileName to the profile of the archive to replay (a retained archive needs no writer), enable spec.backup.pitr, or verify without pointInTime"). Use it to verify history kept in a retained archive after a prefix switch (see Rotation); the replay mounts that archive's retained key snapshot, so it works even after the old profile's Secret was deleted.

Archive binding. The replay reads the archive through the group's binding for that destination (see Archive encryption bindings). A destination the group does not already have Bound gets one pitr-archive-check Job first. The verification Job mounts the binding's key snapshot and is labelled bloodraven.shipstream.io/pitr-key-secret. Until the binding is Bound, the verification waits (requeued every 15 s, with a PITRArchiveBindingNotReady message) while it is being proven or is refused for a fixable reason (KeyMismatch, ArchiveEncryptionMismatch, or a lost key snapshot); put the right passphrase in the profile's Secret and it is retried. A check that cannot list the prefix keeps the binding Pending/Authenticating with an ArchiveListFailed: ... message (grant s3:ListBucket or make the volume readable) and is retried every minute. It fails with PITRArchiveBindingNotReady when the archive can never verify (ArchiveNotFound, ArchiveDescriptorMissing, ArchiveLegacy, ArchiveVersionUnsupported) or when all 16 of the group's bindings are Bound (BindingLimitReached). While the list is full of entries that are not Bound it waits instead.

Multi-site windows. Replay reads every manifest in the selected PITR archive and plans across sites by history: a window that spans planned or emergency failovers and failbacks, with or without a reclone, verifies when the archive holds the history, and refuses with a specific token when it does not. pointInTime.sites was removed in v1.7.0, both from MysqlBackupVerification and from a profile's scheduled verification configuration.

excludeGtids. Exactly as on a restore: every archived transaction committed between the backup's newest transaction and the target (or the head) that is outside the target history must be listed, and the PITRArchiveDivergentHistory refusal prints that set. A backup taken after a lossy failover needs no exclusion; a verification of an older backup with a target after the failover does. See Target history and excludeGtids. Scheduled verifications (spec.backup.profiles[].verification.pointInTime) take the same field.

Refusals. The Verified condition's reason is the refusal token, and its message is the failing step's termination message. The tokens are listed under Restore refusals: archive identity (PITRArchiveListFailed, PITRArchiveNotFound, PITRArchiveLegacy, PITRArchiveDescriptorMissing, PITRArchiveVersionUnsupported, PITRArchiveIdentityMismatch, PITRArchiveEncryptionMismatch, PITRArchiveKeyMismatch, PITRArchiveHistoryReset), the dump (PITRBackupIdentityUnknown, PITRBackupIdentityMismatch, PITRBackupInconsistent, PITRBackupFiltered, PITRBackupIncomplete), the plan (PITRExcludeIntersectsBackup, PITRArchiveNamespaceReuse, PITRBackupAfterTarget, PITRTargetNotCovered, PITRStopAmbiguous, PITRBackupNotOnLineage, PITRArchiveDivergentHistory, PITRArchiveSplitBrain, PITRArchiveOrderInversion, PITRArchiveUnsafeHistory, PITRArchiveGap, PITRArchiveOrderConflict, PITRPlanTooLarge, PITRReplayUnsupported, PITRArchiveObjectInvalid), and the replay (PITRCoverageInvalid, PITRNoArchivedReplay, PITRReplayMismatch, PITRReplayFailed; the message of the last starts BLOODRAVEN_PITR_FAILED: ...). Both modes list the archive prefix and require list permission (S3 ListObjectsV2).

Status. A PITR verification is certified only by the version 2 verdict verify.sh publishes in the main container's termination message once the replay and the sanity check pass. The verdict must come from a Succeeded pod the Job owns, exit code 0, and match the rendered mode, target and archive. A missing or invalid verdict, or a pod that is already gone, fails with PITRCoverageInvalid. On success the verification records status.pointInTime (PointInTimeResult): the outcome (TargetCovered or ArchiveHead), the archive identity, the replay sites, segment, file and transaction counts, the newest replayed commit time, the excluded-divergent and on-history counts, and gtidExecuted, which the replay proved equal to the target history minus excludeGtids. The last segment's file is status.pointInTime.lastReplayedBinlog.name and the newest replayed commit time status.pointInTime.newestReplayedCommitTime; the former status.replayedThroughBinlog was removed. Warning Events PITRDivergentHistoryExcluded and PITROnHistoryTransactionsExcluded are emitted on the verification when they apply.

Upgrading: let in-flight PITR restores and verifications finish before upgrading, or delete their Jobs and retry afterwards. A Job rendered by an older operator writes no version 2 verdict and fails PITRCoverageInvalid. Backups taken before v1.7.0 have no stamp and refuse PITRBackupIdentityUnknown. The bloodraven_backup_verification_replay_lag_seconds gauge is published as completionTime − pointInTime.newestReplayedCommitTime for successful mode: Latest runs only: a timestamp drill's newest event is its historical target, not archive freshness. Alert on a threshold that matches your RPO target.

Sanity query

After the dump loads (and optional PITR replay succeeds), a single scalar-returning SELECT can be evaluated:

spec:
  sanityCheck:
    query: "SELECT COUNT(*) FROM orders.orders WHERE created_at > NOW() - INTERVAL 7 DAY"
    minResult: 1            # optional; fails if an integer result is below this (unset: no floor)
    timeout: 60s            # default 60s; 1s to 1h, applied in whole seconds

The verification records the outcome in status.sanityCheck:

timeout is enforced with a client-side timeout; exceeding it fails the run with reason SanityCheckTimeout, and a query error or below-floor scalar fails with SanityCheckFailed. A set minResult, 0 included, rejects any smaller integer result, negative ones too; with minResult unset no floor applies. A non-integer result is not compared with minResult.

New BLOODRAVEN_VERIFY_SANITY_COMPLETE log lines include v=2 and percent-encode result and error values. Literal underscores such as order_count remain intact in status.sanityCheck. Unversioned lines remain supported; their error text retains the older underscore-to-space decoding convention.

The query runs through the mysql client, so it must be one line, one statement, start with SELECT, WITH, or (, and contain no backslash. The backslash rule blocks mysql client commands such as \!, which would run shell commands inside the verification Job.

Lifecycle

A verification CR passes through these phases:

On the happy path the CR goes Pending → Provisioning → Restoring → Succeeded. The ephemeral Pod, Job, and Service-if-any are deleted after an optional TTL, which also releases the pod's emptyDir volumes.

The operator persists the selected backup's name, UID and a copy of its status.artifactStorage (type, location, bucket, endpoint, region, claim and dump-key reference) in status.backupRef.artifactStorage before creating the Job. The Job reads the bucket, endpoint, region and claim from that copy, never from the live profile; only the S3 credentials come from the profile, so a PVC backup needs no profile to be verified. A newer backup arriving during provisioning or restore does not change that selection. Before launch, the selected backup must still exist with the same UID; deletion or same-name replacement fails with BackupNotAvailable. Create a new verification to select a different source.

Once the Job exists, observation uses its recorded execution source and does not re-resolve the latest backup or require the source CR, current profile, or derived credential Secret to remain present. A verification's failoverGroupRef, profileName, backupRef, pointInTime and sanityCheck are immutable: admission refuses editing them, so a run is never retargeted. Create a new verification to change any of them. keepOnFailure, ttlSecondsAfterFinished and resources can still be edited.

The backup is held while it is read

A verification also holds its backup against deletion. After pinning status.backupRef, and before anything reads the artifact, the operator adds an annotation verification-hold.bloodraven.shipstream.io/<verification UID> to the MysqlBackup. While the hold is live:

• Retention skips the backup. It stays even when it falls outside the profile's keepLast and keepWithinDays windows, and gets a RetentionDeferred Event. It is pruned on the next retention pass after the hold is released.
• A manual kubectl delete waits. The CR goes Terminating, but artifact cleanup waits with ArtifactCleanup=False/HeldByVerification and an ArtifactCleanupDeferred Event.
• PITR binlogs are kept. A held backup that is being deleted still counts toward the binlog retention cutoff, so a replay that starts from it does not lose its binlogs. For a backup being deleted without a hold, the cutoff asks the API server (uncached) whether any verification still reads it, and withholds the cutoff if that lookup fails.

The hold is released only when nothing of the run can still read the backup: the Job carries a Complete or Failed condition or is gone, and no pod labelled with the verification is pending or running. A Job that is still being deleted does not count as finished, because a pod creation can still be in flight. A run that fails while its pod still runs (including the default keepOnFailure: true case) keeps the hold until the pod stops. Deleting a verification likewise waits until its Job is gone and its pods have stopped before its finalizer is removed.

A point-in-time verification also holds the archive's key snapshot: before its Job is created it stamps the same verification-hold.bloodraven.shipstream.io/<uid> annotation on the snapshot Secret, with the same optimistic lock, and releases it when the run stops. While the hold is live, releasing the binding is refused; a release that deleted the snapshot first makes the stamp fail and the verification waits.

Taking the hold and deleting the backup cannot both succeed. The hold is written with the backup's resourceVersion, and retention deletes with a resourceVersion precondition after re-reading the backup uncached. A verification never launches against a backup that is already being deleted; it fails with BackupNotAvailable, and latest-backup selection skips such backups.

Verifications started by an operator version without holds are protected too: retention and cleanup also look for any unfinished verification whose status.backupRef names the backup, and the running verification adds its hold on its next reconcile. A hold whose verification is gone stops counting once that verification's pods have stopped, and is removed from a backup being deleted, so it cannot keep the PITR cutoff pinned while cleanup is blocked. Retention logs backup prune held by verification (see the log schema).

Storage sizing

The verification datadir is an emptyDir volume on the Job's node, dedicated to each run and not size-limited. It draws on the node's ephemeral storage, so the node needs room for the restored dataset plus indexes, temp files and tablespace fragmentation; allow roughly 1.5x the referenced backup's status.sizeBytes and measure with a full-size backup before go-live. Verification wall-clock time scales roughly linearly with the node disk's write throughput.

A verification has no storage settings of its own and creates no PersistentVolumeClaim. To keep a run from filling the node's disk, set spec.resources.limits.ephemeral-storage on the verification: the kubelet evicts the Pod once its emptyDir volumes and writable layers exceed it, and the verification fails.

Failure handling and keepOnFailure

On Succeeded, the ephemeral Pod and its emptyDir volumes are deleted spec.ttlSecondsAfterFinished seconds after completion; 0 deletes them at once. When the field is unset they are kept, with the restored datadir on the node's disk, until retention deletes the verification (outside the verification.retention.keepLast window), so set 0 on schedules unless you want to inspect successful runs. A profile's verification.ttlSecondsAfterFinished is copied onto every scheduled verification, unset staying unset. With keepOnFailure: false, a Failed run is cleaned up after the TTL, or at once when it is unset.

On Failed, the default (spec.keepOnFailure: true) leaves the Pod in place, and with it the emptyDir volumes, which exist only while the Pod object does. Operators can kubectl exec into the verification instance, tail its mysqld error log, or attach an interactive mysqlsh session to inspect whatever the load got as far as. Failed verifications are still GC'd by the retention sweep once they drop out of the verification.retention.keepFailed window.

Set keepOnFailure: false on a one-off CR to force full cleanup even on failure — useful for repeated-probe use cases where the failure signal alone is what you care about.

Concurrency

The reconciler refuses to run two verifications against the same (group, profile) pair simultaneously. Newer CRs land in Failed with the BlockedByActiveVerification condition reason. Scheduled runs use concurrencyPolicy: Forbid by default so this rejection is rare; it mostly protects against kubectl create being issued right as the nightly schedule fires.

Metrics

Verification metrics mirror the bloodraven_backup_* family and share the (group, profile) label set:

Alerts

The obvious alert is staleness on the freshness gauge:

# Verification hasn't Succeeded in more than 48h:
time() - bloodraven_backup_verified_timestamp_seconds > 48 * 3600

And a failure-rate alert for fast signal:

# More than 2 failed verifications in 24h:
increase(bloodraven_backup_verification_runs_total{result="failure"}[24h]) > 2

Inspecting results

kubectl get mysqlbackupverifications -A
# Group  Profile  Phase      Ready  Started   Completed   Age
# orders nightly  Succeeded  True   12m       2m          12m

kubectl describe mysqlbackupverification orders-nightly-20260420
# ... includes status.backupRef, status.durationSeconds, conditions

The Ready condition (and column) mirrors Verified once the run has a verdict: the same status, reason and message. Before that it is False with reason InProgress.

status.backupRef identifies the backup actually selected for the run, including its UID and its recorded artifactStorage. New Jobs also record that snapshot as JSON with a schemaVersion in the immutable pod-template annotation bloodraven.shipstream.io/verification-backup. The operator checks it against the Job's input env and can restore missing or drifted status from that snapshot. A record whose schemaVersion this operator does not know proves no source: the run fails with Verified=False/MetadataVersionUnsupported and a Warning Event of the same reason; create a new verification. For encrypted sources, the decrypt init container's source prefix identifies the artifact; the main container's input points at temporary decrypted data.

Older Jobs without the annotation retain their recorded status reference only when it matches their original source inputs. An unavailable or contradictory identity fails with BackupIdentityMismatch; the operator does not certify a newly selected backup using an older Job's result.

The CR's owner references point back at the MysqlFailoverGroup, so deleting the group cascades through verifications along with the rest of its managed state.

Security

• The verification Pod runs with the same hardened pod- and container-level SecurityContext as backup / restore Jobs (RunAsNonRoot, ReadOnlyRootFilesystem, RuntimeDefault seccomp, capability drop ALL).
• Credentials flow through files mounted under /run/bloodraven/mysql-creds, never environment variables.
• The ephemeral mysqld binds 127.0.0.1 only — no Service is created, so the verification instance is not reachable from anywhere outside the Pod's network namespace.

Known limitations

• A profile's verification block has no resources field, so scheduled verifications run without an ephemeral-storage limit. Bound them with a namespace LimitRange, or create verifications yourself with spec.resources.