Operations

Upgrade and version-skew policy

What MySQL and sidecar versions Bloodraven supports, how to move between them, and what mismatches the operator tolerates during a rollout.

What MySQL and sidecar versions Bloodraven supports, how to move between them, and what mismatches the operator tolerates during a rollout.

This page is deliberately narrow. It covers the MySQL image (spec.image) and the sidecar image (spec.sidecar.image, or the operator's default when it is unset). Operator (Helm chart) upgrades and CRD-version migrations are separate topics.

Supported MySQL versions

Bloodraven supports a single steady-state MySQL baseline at a time, pinned to the default spec.image tag:

  • Current baseline: mysql:9.7. This is the default for new deployments and the version exercised in CI, the playground, and backup / restore paths.

The following are not supported:

  • MySQL 8.0 and 8.4. If you run these today, upgrade MySQL to 9.7 outside of Bloodraven before adopting the operator, or bootstrap a fresh failover group from a mysqlsh-produced dump using initFromBackup.
  • MySQL 9.8 and later Innovation releases. Bloodraven does not track Innovation releases unless and until the project explicitly adopts a later LTS baseline.
  • MySQL 5.7 and earlier (no GTID, no clone plugin).
Pin, don't floatAlways pin spec.image to an immutable tag (e.g. mysql:9.7, not mysql:9). A floating tag can drift you onto an unsupported version between pod restarts. See the Production hardening checklist.

Implementation prerequisites

These are satisfied by the supported tags and only matter if you point Bloodraven at something else: GTID-only replication (SOURCE_AUTO_POSITION=1), the MySQL clone plugin (CLONE INSTANCE FROM), and the REPLICA/SOURCE SQL vocabulary. The operator does not validate these at startup — an unsupported image surfaces as MySQL pod failures, not an operator admission error.

Rolling image upgrades

To roll a MySQL image change (e.g. patch-version bump):

  1. Confirm spec.updateStrategy.type: OrderedUpdate is set (it is the default).
  2. Bump spec.image to the new pinned tag.
  3. kubectl apply.
  4. Watch status.updatePhase and the Updating / Bootstrapping conditions on status.conditions, and confirm replication catches up before the second site rolls. Updating=True with reason OrderedUpdateStalled means a rolled pod did not become Ready and stay Ready for 30s (for example a new sidecar image that cannot start or crash-loops): the rollout stopped there, with the active site still serving, until that pod has been Ready for 30s. Each rolled site takes at least those 30s.
  5. Take a new full backup, then run a backup-verification drill against it on the new version (see Backup verification).

With OrderedUpdate, Bloodraven upgrades the standby first, then hands the primary role to it, then upgrades the old active. The handoff fences the old active and promotes the standby only once it holds every transaction the old active committed, so it loses none and pauses writes for at most about 30s (plus the promotion itself, or up to 10s to unfence on abort); if the standby cannot catch up, the old active is unfenced and the rollout retries later. This is the direction MySQL's rolling-upgrade contract requires: the replica's MySQL version must be ≥ the primary's during the rollout. See Failover → Ordered updates for the full sequence.

The handoff also refuses a standby that holds transactions the active site lacks, and the operator audits every replica for such transactions (see Divergent replicas). An upgrade can therefore stall on purpose: if a replica already carries errant GTIDs, Degraded turns True with reason DivergentTransactions on the first polls after the new operator starts, followers are still rolled, and the active site's update waits, unfenced and serving, until the replica is recloned or the primary is made to contain the set. This applies to the operator upgrade itself whenever it changes the MySQL pod template (a sidecar image bump does). Installations that ran ordered updates on v1.5.0 or earlier can carry such GTIDs from issue #202. The stall does not stop application deployments: the deployment API keeps granting leases for a replica's divergence report.

When upgrading to v1.6.0, apply the v1.6.0 MysqlFailoverGroup CRD with server-side apply before upgrading the chart (Helm does not upgrade crds/); this routine is not for v1.6.0 → v1.7.0 (see Upgrading from v1.6.0 to v1.7.0). It adds status.sites[].divergenceDetectedBy and divergenceComparedTo and the mfg short name. Without it the API server drops the two fields, and after an operator restart a replica's report reads as an old primary's: the messages say "old primary" and the deployment API treats the group as unstable until the report clears.

To find out before upgrading, audit each follower against the primary. Read the follower first, then compute the difference on the primary:

-- On the follower:
SELECT @@GLOBAL.gtid_executed;
-- On the primary, with the follower's set from the line above:
SELECT GTID_SUBTRACT('<follower gtid_executed>', @@GLOBAL.gtid_executed) AS extra;

An empty extra means the follower holds nothing the primary lacks (it says nothing about rows or lag). A non-empty one is the set to discard by reclone or to make the primary contain (see Divergent replicas for what each choice does to the replica's rows). Repeat the check if the primary changed or a server was replaced between the two reads. The bloodraven-doctor skill's gtid-audit.sh runs the same comparison for every follower.

Do not use updateStrategy.type: Recreate for a major bumpRecreate restarts both sites simultaneously. If one pod's in-place MySQL upgrade stalls or fails, you lose write availability with no healthy primary and no rollback path. Use OrderedUpdate for any MySQL image change.

For MySQL-level upgrade semantics — data-dictionary migrations, deprecated-feature removals, schema changes — defer to Oracle's own release notes. Bloodraven drives the rollout; it does not duplicate the MySQL upgrade matrix.

Cross-site version skew

Transient (during a rollout): allowed. The OrderedUpdate sequence keeps the newer MySQL on the replica side of replication throughout the bump, which is MySQL-supported. The skew window lasts from the moment the standby finishes upgrading until the old active has been restarted on the new image — roughly pod-ready time plus the failover drain. No user action is required.

Steady-state (intentional per-site version split): not supported.spec.image is a single field per MysqlFailoverGroup; SiteSpec has no per-site image override. The operator reconciles both site Deployments to the same image tag. If you edit a site's Deployment manifest out of band to pin a different tag, the next reconcile reverts it.

Sidecar ↔ operator skew

The sidecar binary and the operator binary are released together. The supported configuration is matching versions. A group that leaves spec.sidecar.image unset gets that by construction: it runs the sidecar image the operator release ships with (the chart's sidecar.image.repository and sidecar.image.tag, passed to the operator as BLOODRAVEN_SIDECAR_IMAGE; the tag defaults to the chart's appVersion). A group that pins spec.sidecar.image should pin the release of the running operator Deployment; a pin with another release tag gets a SidecarVersionSkew Warning.

Because an unpinned group follows the operator, an operator upgrade that ships a new sidecar image changes that group's pod spec, and the group rolls: an ordered update (followers first, a handoff of the primary role, then the old primary; see Ordered updates) under OrderedUpdate, both sites at once under Recreate. Each such group starts its roll on its own after the upgrade. A pinned group does not move its sidecar until the pin changes, though a release that changes the rendered pod spec in other ways rolls it too.

Bloodraven tolerates a one-minor transient skew in either direction during a rolling upgrade. Both sides of the HTTP surface between the two are additive-only:

  • Operator endpoints (polled by sidecars): /healthz, /status, /active-site, /pitr-cutoff.
  • Sidecar endpoints (polled by the operator and peer sidecars): /health, /status, /peer/ping, /peer/active-site, /archiver/status, /keyring/status, /livez.

PITR retention uses the additive coverage=gtid-v1 capability and backup GTID bounds. During skew, older sidecars receive no timestamp cutoff from the updated operator, and updated sidecars reject time-only replies from older operators. New archive retirements pause until both sides can prove content-bound coverage; persisted pending deletes can still finish. Previously complete proofs without matching proof content IDs are re-inspected in bounded batches before new retirement. Pending or Running group backups pause new retirements across all profiles. Upgrade the operator first to suppress unsafe time-only retention replies from the older protocol.

An older sidecar rewriting an ID-aware PITR manifest drops IDs, coverage, purged-base bridges, quarantine/reset evidence, divergence/acknowledgement records, and pending deletions; its pruning may drop all rows sharing one basename. Re-upgrading recovers surviving stripped IDs only from exact confined remotePath keys. It cannot recover lost rows, purged-base provenance, quarantine/reset evidence, divergence/acknowledgement records or pending deletes. Preserve a compatible recovery image for each archive generation; Use a new prefix and full backup after RESET. Obsolete upload-order guards have a verified canonical-backup acknowledgement procedure; verified namespace reuse still requires a new generation. See PITR history repair. Preserve archive objects, manifests, encryption material, and compatible recovery images before rollback. See PITR upgrade and downgrade for recovery steps.

/archiver/status gained additive storeReady and consecutiveFailedScans fields. A sidecar whose PITR archive store cannot be built (unreadable passphrase or credentials file, invalid endpoint, empty PVC mount path) now keeps serving readiness and fencing, reports enabled:true, storeReady:false and the init error, and retries the store in the background. The fix lives in the sidecar image, which the operator rolls through the ordered update: until a pod runs the new image it still exits on that error, crash-loops, and drops out of every Service. The updated operator reports such a pod as PITRArchiveReady=False with reason SidecarUnreachable once it has not answered for 5 minutes since the operator started (3 failed polls over 60 s for a sidecar that answered before), and reads a missing storeReady from an older sidecar as "not reported", never as an init failure. Without consecutiveFailedScans it debounces errors on the older sidecar's lastScanAt. An older operator ignores the field and still shows the init error in status.pitr.message.

The PVC archive store now refuses to start unless its mount path is a mount point, and writers also probe write access. A site whose pitr-archive volume was missing from the pod, or not writable, used to show PITRArchiveReady=True (writes landed in the container filesystem) or a later ArchiverError. After the sidecar rollout it shows False / ArchiveStoreInitFailed. pitr-download, pitr-quarantine and the encrypted-backup encrypt-upload / decrypt-download steps now fail at start when their PVC is not mounted at the configured path. Writers also refuse a write prefix that leaves the mounted volume through a symlink (for example binlogs linked to another volume), which used to work: the store reports ArchiveStoreInitFailed with outside the archive volume; replace the symlink with a directory on the archive volume. Unencrypted PVC backups are unchanged.

The kubelet probes the sidecar too, and the operator renders those probes. That makes the probe paths part of the same contract, so they only use endpoints every supported sidecar serves. Liveness probes /peer/ping, which answers without touching MySQL. Readiness probes /health. The newer /livez also detects a wedged fencing loop, but a sidecar that predates it answers 404, and a liveness probe on it would restart that sidecar in a loop. It becomes the liveness target once every supported sidecar serves it.

A briefly older-or-newer sidecar stays compatible with the operator while pods roll. Skew beyond one minor is not tested and not guaranteed.

When a change moves both spec.image and the sidecar image together, OrderedUpdate restarts the sidecar as part of the pod restart it already performs — the replica-first ordering applies automatically, no separate sidecar sequencing is required.

Upgrading from v1.6.0 to v1.7.0

There is no in-place upgrade from v1.6.0 to v1.7.0.Never apply the v1.7.0 CRDs to a cluster that runs v1.6.0, by any route: the "CRDs first, then the chart" routine of a normal upgrade (including the GitOps safe upgrade workflow), a platform CRD application or an Argo CD Helm release that renders the chart's crds/, the Argo CD layout in Production install examples, or the "CRDs managed separately" loop in Production install pointed at a newer release. On a v1.6.0 cluster, pin both the chart version and the CRD source in Git for as long as it runs.

Why an in-place upgrade cannot work:

  • Stored defaults change type. Every v1.6.0 MysqlFailoverGroup stores cloneTimeout: 3600 (a duration in v1.7.0) and updateStrategy: OrderedUpdate (an object in v1.7.0), and every backup profile stores retention: 7 (an object in v1.7.0). The API server prunes unknown fields but does not convert types, so the v1.7.0 operator cannot decode a stored v1.6.0 group, and one undecodable object fails its whole list of groups: it then manages none of them.
  • CRDs are cluster-scoped. Applying the v1.7.0 CRDs changes the schema for every namespace at once. From then on every read of an old object has the old field names pruned, and the still-running v1.6.0 operator persists the pruned object on its next write.
  • Stored objects can break the new name and size rules. A group stored before v1.7.0 can have a name that is not a DNS-1035 label, a group name plus site name longer than 41 characters, or an integer spec.sites[].storage.size. On Kubernetes 1.31 the API server validates status writes without ratcheting, so it refuses the operator's own status writes to such a group. An integer size cannot be decoded at all: v1.7.0 stores a size as a string.
  • Re-applying the manifests does not help. Every bootstrapped group's initFromBackup block is renamed, and v1.7.0 refuses any change to it. A manifest re-apply cannot rewrite status, and several status records cannot be rebuilt from observation.

If the v1.7.0 CRDs were applied to a v1.6.0 cluster by mistake, scale every Bloodraven operator to 0 first and write no Bloodraven object. Do not re-apply the v1.6.0 CRDs as a fix; recovery depends on whether any object was written since, and is part of the procedure below.

The supported path: a new group in a new cluster

The supported path bootstraps a new group, in a cluster that has never had the Bloodraven CRDs, from a v1.6.0 backup. The step-by-step procedure, its recovery steps and its end-to-end test are tracked in issue #243. Until it is published, this outline is its scope, not a tested procedure:

  1. Stop application writes and take a full backup on v1.6.0, on an S3 profile that the new cluster can reach.
  2. Install v1.7.0 in a cluster with no Bloodraven CRDs.
  3. Write the new manifests with the changes listed in What changes in v1.7.0. Give every backup and PITR profile a prefix that no v1.6.0 group has used; reference the old backup only as a restore source. Apply with kubectl apply --server-side --validate=strict, so a leftover old field name is refused instead of pruned.
  4. Bootstrap the group from the backup (initFromBackup, with decryption for an encrypted backup), excluding the schemas that MysqlDatabase objects manage. Then apply the MysqlDatabase objects and load each of their schemas with a per-schema restoreInPlace.
  5. Recreate the MySQL accounts that neither the operator's credentials nor a MysqlDatabase create: restores do not load users.
  6. Take a new full backup as the new group's baseline, wait for PITRArchiveReady=True when PITR is on, verify the baseline with a MysqlBackupVerification, and compare the application's data with the frozen source.
  7. Cut over, and only then retire the old group. Keep the old cluster, its backups, its passphrases and its keyring escrow Secrets for as long as you may need them.

The new group does not carry over failover history, operation records (restores, planned failovers, recoveries), MysqlBackup and MysqlBackupVerification objects, verification holds, keyrings (it mints its own) or PITR across the old archive.

There is no in-place rollback either: the v1.6.0 operator cannot decode a v1.7.0 object. Until cut-over is verified, the old group in the old cluster is the way back.

What changes in v1.7.0

Use this list to write the new group's manifests and to update alert rules, scripts and dashboards. v1.7.0 supports Kubernetes 1.31 and newer (the chart's kubeVersion is >=1.31.0-0). On 1.31 and 1.32, keep the CRDValidationRatcheting feature gate on (the default): with it off, the API server does not check at creation that spec.initFromBackup.pointInTime.archiveProfileName names a profile (see Install in production).

Manifest fields

Every site needs an explicit, unique serverID (the examples use 101, 102, ...). The restore creates new data directories, so the ids need not match the old group's 101 + index. The id is immutable once set.

Kindv1.6.0 fieldv1.7.0 field
MysqlFailoverGroupspec.credentials.{operator,app,readOnly,monitor,backup}Secretspec.credentials.{operator,app,readOnly,monitor,backup}SecretName
MysqlFailoverGroups3.credentialsSecret (profiles, initFromBackup and restoreInPlace sources)s3.credentialsSecretName
MysqlStandbyClusterspec.source.s3.credentialsSecretspec.source.s3.credentialsSecretName
MysqlFailoverGroup, MysqlStandbyClusterencryption.passphraseSecret, decryption.passphraseSecret…passphraseSecretKeyRef (same {name, key} body)
MysqlFailoverGroupspec.dragonfly.auth.{secretName, passwordKey}spec.dragonfly.auth.passwordSecretKeyRef{name, key}
MysqlFailoverGroupspec.dragonfly.snapshot.{s3Endpoint, s3UseHTTPS}spec.dragonfly.snapshot.s3EndpointURL: s3Endpoint: rustfs:9000 with s3UseHTTPS: false becomes http://rustfs:9000; an unset s3UseHTTPS means https://
MysqlFailoverGroupspec.sidecar.bloodravenAddressspec.sidecar.operatorEndpoint
MysqlFailoverGroupspec.sidecarImage (every v1.6.0 group stores the default 1.3.2), spec.sidecarResourcesspec.sidecar.image, spec.sidecar.resources. Leave image unset to run the sidecar the operator release ships with; do not copy the stored 1.3.2
MysqlFailoverGroupinitFromBackup.source.mysqlBackupRef, restoreInPlace.source.mysqlBackupRef…source.backupRef (name required)
MysqlDatabasespec.groupRefspec.failoverGroupRef
MysqlDatabasespec.collation defaulted to utf8mb4_unicode_ci (stored in every v1.6.0 object)no default: unset renders no COLLATE. Set collation: utf8mb4_unicode_ci where new schemas must match the old ones
MysqlBackupspec.sourceSiteOverridespec.sourceSite
MysqlFailoverGroupspec.sites[].lbIPspec.sites[].loadBalancerIP (an IPv4 address)
MysqlFailoverGroupprofile storage.pvc.{storageClassName, size} (an unset size meant 10Gi)storage.pvc.provisioned.{storageClassName, size} (size required); keep pvc.claimName for an existing claim. Exactly one of the two
MysqlFailoverGroupspec.sites[].role: primary-candidate, dr-only, read-only (stored in every v1.6.0 group)PrimaryCandidate, DisasterRecovery, ReadOnly
MysqlFailoverGroupspec.updateStrategy: OrderedUpdate or Recreate (stored in every v1.6.0 group)spec.updateStrategy: {type: OrderedUpdate} or {type: Recreate}; omit it for the default OrderedUpdate
MysqlFailoverGroupspec.plannedFailover.onCooldown: reject, defer; spec.dragonfly.plannedFailover.onSyncTimeout: proceed, failReject, Defer; Proceed, Fail. Lower-case values are refused
MysqlFailoverGroupspec.backup.profiles[].retention: 7 (stored in every v1.6.0 profile) and retentionPolicyretention: {keepLast, keepWithinDays, keepFailed}; see Moving from the released retention fields
MysqlFailoverGroupspec.cloneTimeout: 3600 (integer seconds, stored in every v1.6.0 group)spec.cloneTimeout: 1h (a duration, 1m to 24h); omit it for the default
MysqlFailoverGroupspec.dns.ttlspec.dns.ttlSeconds
MysqlFailoverGroupencryptionAtRest.keyring.escrowTimeoutSeconds: 600encryptionAtRest.keyring.escrowTimeout: 10m
MysqlFailoverGroupbackup.retry.{initialBackoffSeconds: 60, maxBackoffSeconds: 1800}backup.retry.{initialBackoff: 1m, maxBackoff: 30m}
MysqlFailoverGroupspec.dragonfly.maxMemoryMb: 512spec.dragonfly.maxMemory: 512Mi
MysqlFailoverGroupspec.dragonfly.proactorThreads: 0omit it (unset means the default)
MysqlFailoverGroupinitFromBackup.pointInTime.stopDatetimeinitFromBackup.pointInTime.stopTime: RFC 3339 with an explicit zone and whole seconds. A YYYY-MM-DD HH:MM:SS or zone-less value (read as UTC before) needs a Z
MysqlFailoverGroupinitFromBackup.pointInTime.{sites, allowUnverifiedCrossSite}removed; pointInTime.archiveProfileName is required instead
MysqlBackupVerification, profile verificationpointInTime.mode: none, latest, timestampLatest or Timestamp, required in the block; for none, omit the block
MysqlBackupVerification, profile verificationpointInTime.timestamppointInTime.stopTime, required for Timestamp and forbidden for Latest
MysqlBackupVerification, profile verificationpointInTime.sitesremoved
MysqlBackupVerification, profile verificationsanityCheck.expect.{minRows, maxDurationSeconds: 60}sanityCheck.{minResult, timeout: 60s}
MysqlFailoverGroupprofile verification.retentionPolicy{keepSuccessful, keepFailures}verification.retention{keepLast, keepFailed}; omit an explicit 0, which now means keep none
MysqlStandbyClusterspec.source.storage{type: S3, s3{…}}spec.source.s3{…} (prefix required)
MysqlStandbyClusterspec.source.{failoverGroupName, namespace, cluster, profileName}, spec.templateremoved; spec.transport accepts only ObjectStore

v1.7.0 removes these fields, which earlier releases accepted and ignored or refused:

  • spec.secretName (the DSN credential mode). spec.credentials is required: create an operator Secret with username, password and MYSQL_ROOT_PASSWORD (plus MYSQL_REPLICATION_USER and MYSQL_REPLICATION_PASSWORD if replication uses its own account) and set credentials.operatorSecretName. As the DSN mode did, the first-boot init script of the new group creates that replication account with REPLICATION SLAVE, REPLICATION CLIENT, BACKUP_ADMIN, CLONE_ADMIN. Express DSN tls= options through spec.tls.
  • spec.tls.issuerRef: spec.tls is secretName only.
  • spec.restoreInPlace.pointInTime, which the operator refused; use initFromBackup.pointInTime on a new group.
  • loadOptions.resetProgress, profile storage.s3.storageClass, profile verification.storage and MysqlBackupVerification spec.storage. Bound a verification's disk with spec.resources.limits.ephemeral-storage.
  • spec.triggeredBy on MysqlBackup and MysqlBackupVerification: the value moves to the annotation bloodraven.shipstream.io/triggered-by.
  • A replica-preserve-commit-order override in spec.mysqlConf: it is operator-owned and forced ON.

Some values that v1.6.0 accepted are refused, and some defaults change meaning:

  • Timers must be inside their bounds (see Timers): for example pollInterval: 0s, a failoverCooldown under 30s, or a plannedFailover.maxLagWait over 1h is refused, and so is a planned-failover annotation whose maxLagWait= is outside 1s to 1h. dragonfly.maxMemory, if set, must be a positive quantity.
  • Names and lists. S3 prefix values (profiles, restore and standby sources) are at most 512 characters with no empty (a//b), . or .. segment, and a leading / is ignored, so /prod and prod are the same prefix. Profile names are DNS labels of at most 63 characters, and spec.backup.profiles is a list-map keyed by name (at most 32). Every Secret name and every failoverGroupRef.name and backupRef.name is a DNS subdomain (lowercase, at most 253 characters), and every Secret key is at most 253 characters of [-._a-zA-Z0-9]; set an optional Secret name to nothing rather than "". MysqlBackup and MysqlBackupVerificationprofileName and schedule profileName are DNS labels, MysqlBackupsourceSite matches the site-name pattern, and spec.sidecar.operatorEndpoint is host:port. The CRDs also refuse duplicate entries in splitBrainPolicy.sitePriorities; more than one restoreInPlace.loadOptions.includeSchemas entry; duplicate names, or more than 16 entries, in extraContainers, extraInitContainers and backup.imagePullSecrets; duplicate backup.schedules names, more than 32 schedules, or a schedule name longer than 63 characters; duplicate, more than 256, or longer than 64 character names in the dump and loadOptions includeSchemas/excludeSchemas lists; more than 16 MysqlDatabase deploymentClients (now keyed by the namespace and ServiceAccount pair); and a MysqlBackup whose name ends in .bloodraven-backup.json.
  • Names that become object names. A MysqlFailoverGroup name is a DNS-1035 label (a.b and 1orders are refused: the name starts the group's Service names). A site name is a DNS label of at most 20 characters (no leading or trailing -), and the group name plus each site name is at most 41 characters. splitBrainPolicy.sitePriorities entries and MysqlBackup sourceSite follow the site rule. A MysqlBackup or MysqlBackupVerification name is at most 63 characters. A site name is part of the site's PVC and Service names, so a name that needs to change means a new group.
  • Sizes (spec.sites[].storage.size, pvc.provisioned.size) are positive quantity strings; a bare number of bytes must be quoted.
  • restoreInPlace.confirm must be an RFC 3339 timestamp with an upper-case T and Z, an offset within -23:59..+23:59 and a digits-only fraction.
  • Edits that are refused. A site cannot be removed or renamed (see Replacing a site); a site's zone, storage.storageClassName and storage.size are immutable (a size is compared by value); spec.initFromBackup cannot be added, changed or removed (a bootstrap that failed on its own block is retried by recreating the never-bootstrapped group, see A failed bootstrap restore); spec.encryptionAtRest.enabled cannot go back to false, and the keyring paths cannot change while it is true; a profile's pvc.provisioned cannot change, be added or be removed; and a MysqlBackup's failoverGroupRef, profileName and sourceSite, and a MysqlBackupVerification's failoverGroupRef, profileName, backupRef, pointInTime and sanityCheck, are immutable.
  • ttlSecondsAfterFinished on a MysqlBackupVerification and on a profile's verification no longer defaults to 0: an omitted value keeps a successful verification's Pod (and its restored datadir) until retention deletes the verification. Set ttlSecondsAfterFinished: 0 to keep the immediate cleanup.
  • An explicit 0 is kept as 0 instead of being replaced by the default: replication.maxLagSeconds: 0 requires a caught-up replica (it meant 300), and backup.backoffLimit: 0 and backup.maxLagSecondsForSource: 0 are honored.
  • MysqlBackupVerification backupRef: {} (which meant "latest") is refused: omit backupRef instead.
  • MysqlBackup spec.deletionPolicy is new. The default Delete keeps the v1.6.0 behaviour (deleting the object deletes its artifact); Retain keeps the artifact, unless it is on a claim the group provisioned, which is deleted with the group (Keeping an artifact).
  • The chart's operator environment variable BLOODRAVEN_DEFAULT_AUXILIARY_ADDRESS is now BLOODRAVEN_DEFAULT_OPERATOR_ENDPOINT; rename an extraEnv entry that overrides it.

Labels, annotations and action annotations

Every label, annotation and finalizer key moves from shipstream.io/ to bloodraven.shipstream.io/, and boolean label values are "true" or "false" (they were healthy=yes|no and dragonfly-traffic=enabled). Write Service selectors, NetworkPolicies, monitors and scripts for the new keys; the full list is in Well-known metadata.

v1.6.0v1.7.0
Node taint shipstream.io/db-readonly-<group><namespace>.db-readonly.bloodraven.shipstream.io/<group>: write tolerations for the new key. Remove the old taints from the old cluster's nodes only once its groups are retired (kubectl taint nodes --all <old key>-)
MysqlBackup annotation hold.verification.shipstream.io/<uid>verification-hold.bloodraven.shipstream.io/<uid> (operator-written)
Finalizer shipstream.io/graceful-shutdownbloodraven.shipstream.io/failover-group (operator-written)
reclone-site=<site>:confirm=<group> and reclone-site=<site>:<gtid-prefix>reclone-site=<site>:confirm=<group>:uid=<group uid>, plus :gtidPrefix=<source UUID prefix> for a diverged site. kubectl bloodraven reclone builds the value
Annotation bloodraven.shipstream.io/dragonfly-snapshot-upgrade=<image>spec.dragonfly.upgradeStrategy: SnapshotRestore, then edit spec.dragonfly.image
Annotation bloodraven.shipstream.io/encryption-adopt=confirmspec.encryptionAtRest.existingData: LeavePlaintext

PITR archive identity

v1.7.0 changes the PITR archive format and refuses archives written before it. The new group's PITR profile must use a new prefix (a v1.6.0 prefix is refused as ArchiveLegacy), and its first full backup after the archive is Bound (PITRArchiveBindingReady=True) is its first point-in-time baseline: it carries a backup stamp and status.pitrArchiveID.

  • Grant the sidecar's PITR credentials s3:ListBucket, s3:GetObject, s3:PutObject and s3:DeleteObject, and confirm the store enforces conditional writes (PITR archive requirements). Grant every backup profile's credentials s3:ListBucket, s3:GetObject and s3:PutObject on <prefix>/*: each backup Job checks that its dump location is empty, then reads its backup stamp before writing it and reads it back after. Artifact cleanup still needs s3:ListBucket and s3:DeleteObject, and deleting a failed backup reads its stamp too (Backup credential permissions).
  • v1.7.0 cannot read a v1.6.0 archive. Keep the old prefix, its backups, its passphrase and the v1.6.0 cluster if you may need a target from before the move. Do not run a pre-v1.7.0 sidecar against a v1.7.0 archive: it rewrites its manifest as version 1 and the archive is then refused as PITRArchiveLegacy.
  • A v1.6.0 backup has no backup stamp, so PITR from it refuses PITRBackupIdentityUnknown; a dump-only restore from it works. An encrypted v1.6.0 backup also recorded no dump key, and the operator never decrypts a dump with a profile's live Secret, so restoring it (initFromBackup or restoreInPlace) needs an explicit decryption naming a Secret that holds the passphrase it was written with. Keep that passphrase even after you rotate the profile's Secret.
  • The archive key is pinned by the group's archive encryption binding: pods mount an operator-made key snapshot, and in-place passphrase or encryption changes on an archive in use are refused. Key snapshot Secrets (mysql-pitr-key, and one mysql-dump-key per encrypted backup) have no owner reference and survive group deletion. See Archive encryption bindings.

See also Upgrading to archive identity.

Status, conditions and printer columns

Status is written by the new group's operator; nothing converts an old value. Update tooling that reads these fields. Among others: status.lastFailover is {time, target, promotionGtidExecuted} (the flat lastFailoverTarget and promotionGtidExecuted are gone), status.sites[].replicating is the enum status.sites[].replication, status.sites[].state and status.dragonfly.sites[].role are CamelCase, MysqlBackup status.location and storageType are status.artifactStorage.{location, type}, binlogFile and binlogPos are status.binlogPosition, encrypted and encryptionAlgorithm are status.encryption, a restore's point-in-time result is status.restore.provenance.pointInTime (the flat status.restore.pitr* fields are gone; status.restoreInPlace.provenance.pointInTime stays unset), a restore's source* and target* fields are status.restore.provenance.{source, target} (sourceSizeBytes is provenance.source.sizeBytes), a MysqlBackupVerification's point-in-time result stays status.pointInTime, and its status.sanityCheck is {outcome, result, durationMilliseconds, error} (it was {ran, resultRow, durationMs, error}). Also:

  • status.encryptionAtRest.sealed is gone: alert on the EncryptionAtRestReady condition (True/AllSitesSealed when every site is sealed).
  • status.encryptionAtRest.sites[].unsealedSince, status.restoreReclones.requestedAt and status.sites[].lastSeen are unsealTime, requestTime and lastSeenTime.
  • MysqlDatabase phase Creating is Applying.
  • Observations the operator could not read are unset instead of false or 0: status.encryptionAtRest.sites[].coverage flags and status.dragonfly.sites[].{syncInProgress, lastIOSecondsAgo}. Treat an absent value as unknown.

The CRD reference has every field.

Condition reasons, polarity and printer columns change too. Update alert rules, scripts and deployment tooling that match on them, and re-apply the chart's Argo CD health script (charts/bloodraven/argocd/) if you copied it into argocd-cm: the old script reads Ready.observedGeneration and the old Bootstrapping and Updating polarity.

Kindv1.6.0v1.7.0
MysqlFailoverGroup Readyreason TopologyPolled, True or FalseTrue/Healthy; False/NoWritablePrimary (no site writable) or False/ReplicationUnhealthy
MysqlFailoverGroup DegradedTrue/Degraded, True/AlertTrue/SiteUnhealthy, True/CrossSiteAlert; new True/BootstrapFailed while a failed clone's target does not replicate
MysqlFailoverGroup BootstrappingFalse/Done after a clone; True/Failed after a failed oneTrue only while a clone runs. False/NotBootstrapping after a clone (also before any clone); False/BootstrapFailed after a failed one, together with Degraded=True/BootstrapFailed
MysqlFailoverGroup UpdatingFalse/OrderedUpdateStalledTrue/OrderedUpdateStalled (the update is still in progress)
MysqlFailoverGroup statuscondition observedGeneration was the generation of the object writtennew status.observedGeneration, the generation the topology manager applied, carried by the topology conditions; the Argo CD health script reads it
MysqlBackup ReadyFalse/NoHealthySource while waiting for a source; a wait past 30 minutes failed with its pending reasonFalse from creation (Pending, the pending reason, Running); False/WaitingForHealthySource while waiting for a source; past 30 minutes NoHealthySource (a source wait) or PendingTimedOut (any other wait). NoHealthySource is terminal only
MysqlBackupVerification Readyabsentmirrors Verified; False/InProgress before
MysqlStandbyCluster ReadyabsentTrue/SourceDiscovered when BucketReadable and SourceConfigKnown are both True, otherwise the first one's False reason
MysqlDatabase ReadyFalse/GroupNotFound while the group does not existFalse/GroupNotYetCreated (GroupNotFound stays a terminal reason on backups and verifications)
Printer columnskubectl get msc showed BucketReadable and SourceKnownkubectl get msc shows Ready instead; kubectl get mysqlbackup and kubectl get mysqlbackupverification gain Ready
Deployment API unstableReasonPlannedFailover for a standing planned-failover automation hold; Failed for a failed cloneAutomaticFailoverHeld; BootstrapFailed. PlannedFailover means only a run in flight or its claim

Events

Event reasons follow one rule (<Subject><PastTenseVerb>, with a qualifier naming the cause), and each is recorded with one type. Update Event routing, alert rules and scripts that match on the old reasons. Status condition and status reason values are unchanged by these renames: the PITRArchiveReady condition and the DragonflySyncTimeout planned-failover reason keep their names. The full list of reasons is in Kubernetes Events.

v1.6.0 Event reasonv1.7.0
DragonflyPromotionCompletedDragonflyPromotionSucceeded. An emergency promotion that fell back to REPLICAOF NO ONE no longer also emits a Warning DragonflyPromotionCompleted; it emits DragonflySessionsLost only
DragonflyUpgradeCompletedDragonflyUpgradeSucceeded
DragonflySyncTimeout (Event)DragonflySyncTimedOut
DragonflyPromotionWaitingDragonflyPromotionDeferred
RecoveryCompleteOldPrimaryRecovered
PromotionRefusedPendingReclonePromotionRefusedReclonePending
KeyringPhase (Normal, or Warning for a move to Failed)KeyringPhaseChanged, always Normal. A site that is not escrowed within keyring.escrowTimeout gets the new Warning KeyringEscrowTimedOut; the other moves to Failed already had their own Warning
KeyringRotationRefusedKeyringRotationRejected
PITRArchiveReady (Event)PITRArchiveRecovered
PointInTimePartialPITRReplayTruncated, which also starts the partial restore's status.restore.message
PITRArchiveKeySnapshotUnreadablePITRArchiveKeySnapshotReadFailed
PITRArchiveNotStampedPITRArchiveStampSkipped
PlannedFailoverDrainingPlannedFailoverDrainStarted
PlannedFailoverLagOKPlannedFailoverTargetCaughtUp
PlannedFailoverCompletedPlannedFailoverSucceeded
PlannedFailoverHoldAcknowledgementIgnoredPlannedFailoverHoldAcknowledgmentIgnored
RestoreInPlaceCancelledRestoreInPlaceCanceled
RestoreInPlaceRejected for an invalid confirm on a finished restore (nothing runs)RestoreInPlaceConfirmIgnored. RestoreInPlaceRejected now only ends a run Failed
RestoreInPlaceFailed, twice per failed restore JobRestoreInPlaceFailed, once
RestoreInPlaceWaitingForPreviousQueueRestoreInPlaceDeferredPreviousQueue
RestorePeerReplicationIncompleteRestorePeerReplicationStoppedBehind
RestoreReclonesCompleteRestoreReclonesDrained
ArtifactCleanupProducerStuckArtifactCleanupStalledProducerStop
DatabaseDropSkipped (Warning) when another MysqlDatabase still declares the databaseDatabaseDropRefused. DatabaseDropSkipped is now only the Normal "nothing to drop" case
DatabaseDropSkipped (Warning) when another MysqlDatabase still claims the owner userOwnerUserDropSkipped
DragonflyUpgradeRejected (the snapshot-upgrade annotation was refused)removed with the annotation. A request without a snapshot directory, or with Dragonfly disabled, is refused at admission; a run that cannot start yet (no status.activeSite, a planned failover or automation hold owning the sites) emits Normal DragonflyUpgradeDeferred
ArtifactLocationRestoredremoved, with no successor: status.artifactStorage.location is the only recorded location, so there is no second field to restore from

Metrics, logs and the operator's endpoints

  • bloodraven_planned_failovers_total{result} is the kebab-case of the run's outcome (succeeded, not-started, rolled-back, source-left-fenced, promoted-unconfirmed, manual-recovery-required) instead of success, rejected, failed_timeout and failed_other. failed_timeout has no single successor: a lag timeout ends rolled-back, an admission timeout not-started; select on status.plannedFailover.result.reason for the cause.
  • A role metric label of dr-only is disaster-recovery; the other role and state label values keep their kebab-case spelling.
  • bloodraven_restore_reclone_pending{phase} values are armed, draining and held (they were Armed, Draining and Held).
  • The deployment lease revocation reasons operator_revoked and topology_changed are OperatorRevoked and TopologyChanged in the bloodraven_deploy_lease_revocations_total{reason} label, the /deploy/v1 409 revoked body's reason, and the DeploymentLeaseRevoked Event message. Other reason labels carry the renamed condition and Event reasons.
  • The operator's /status and /ws/status carry the CR values for site state and role (Writable, ReadOnly, PrimaryCandidate, ...). Both carry lastFailover{time, target, promotionGtidExecuted} in the shape of status.lastFailover; the flat /statuspromotionGtidExecuted field is gone. /ws/status also carries sites[].replication, and its updatePhase is now populated (it was always empty).
  • Structured logs: the state transition record's from and to values and the role field of the fencing and planned-failover records carry the same CamelCase values. The INFO record dragonfly snapshot-upgrade deferred: the Dragonfly master differs from the MySQL active site (fields fg, dragonflyMaster, activeSite) is now dragonfly snapshot-upgrade deferred with fields fg and reason, and covers every cause a run waits for. See the log schema.
  • /deploy/v1 operations.plannedFailover carries the planned-failover vocabulary below (Deployment API).

Planned failover in v1.7.0

The durable planned-failover claim (issue #235) is part of v1.7.0. It replaced the shape of status.plannedFailover (see the CRD reference) and its vocabulary:

  • spec.plannedFailover.onCooldown accepts Reject or Defer, and spec.dragonfly.plannedFailover.onSyncTimeout accepts Proceed or Fail; the lower-case spellings are refused.
  • Queries on the removed status fields (for example transactionsLost or durationSeconds) must move to status.plannedFailover.result and the duration histogram. The renamed Event reasons are in Events; PlannedFailoverStarted, PlannedFailoverSkipped, PlannedFailoverRejected, PlannedFailoverDeferred and PlannedFailoverFailed keep their names. The new reasons are listed in Planned failover.
  • Deployment API companions. operations.plannedFailover in /deploy/v1 carries the new vocabulary. phase is status.plannedFailover.phase: Pending and Validating became Admitting, Resuming became Publishing, and Fencing and RollingBack are new; the empty phase is reported only when there is no block. A planned-failover-terminal marker that names the run wins over a stale live phase: phase is then Succeeded or Failed with the marker's outcome and reason. reason is no longer a free-form string, and it belongs to the current phase only: admission.waitReason while Admitting, admission.deferReason while Deferred, result.reason (empty when the result has none) for Succeeded and Failed, and empty in any other phase; all are enums. The new key outcome carries result.outcome (empty until the run ends). A companion that switches on phase or reason strings must be updated before cut-over. Coordination decisions should use stable and unstableReason, which kept their meaning.

Restoring across versions

MysqlBackup.status.mysqlImage records the image tag that produced each dump. The safe path is to restore onto the same major version that produced the backup. Do not restore a dump onto an older MySQL version — MySQL does not support version downgrade of data.

Backup-restore throughput and sizing guidance live under a separate performance guide (WISHLIST #27).

What Bloodraven does not do

  • No version admission check. Setting spec.image to an unsupported tag will not produce a CRD validation error. MySQL itself rejects unsupported upgrades at pod start.
  • No skip-a-major guardrail. There is no enforcement that blocks an 8.0 → 9.7 jump or a 9.7 → 9.8+ jump at the operator layer.
  • No downgrade path. MySQL does not support version downgrade, and Bloodraven inherits that limitation.
  • No per-site MySQL version. The CRD does not expose a per-site image, so mixed-version steady state is not expressible.