Upgrade and version-skew policy
What MySQL and sidecar versions Bloodraven supports, how to move between them, and what mismatches the operator tolerates during a rollout.
This page is deliberately narrow. It covers the MySQL image
(spec.image) and the sidecar image (spec.sidecar.image, or the
operator's default when it is unset). Operator
(Helm chart) upgrades and CRD-version migrations are separate topics.
Supported MySQL versions
Bloodraven supports a single steady-state MySQL baseline at a time,
pinned to the default spec.image tag:
- Current baseline:
mysql:9.7. This is the default for new deployments and the version exercised in CI, the playground, and backup / restore paths.
The following are not supported:
- MySQL 8.0 and 8.4. If you run these today, upgrade MySQL to 9.7
outside of Bloodraven before adopting the operator, or bootstrap a
fresh failover group from a
mysqlsh-produced dump usinginitFromBackup. - MySQL 9.8 and later Innovation releases. Bloodraven does not track Innovation releases unless and until the project explicitly adopts a later LTS baseline.
- MySQL 5.7 and earlier (no GTID, no clone plugin).
spec.image to an immutable tag (e.g. mysql:9.7, not
mysql:9). A floating tag can drift you onto an unsupported version
between pod restarts. See the
Production hardening checklist.Implementation prerequisites
These are satisfied by the supported tags and only matter if you
point Bloodraven at something else: GTID-only replication
(SOURCE_AUTO_POSITION=1), the MySQL clone plugin (CLONE INSTANCE FROM), and the REPLICA/SOURCE SQL vocabulary. The operator does
not validate these at startup — an unsupported image surfaces as
MySQL pod failures, not an operator admission error.
Rolling image upgrades
To roll a MySQL image change (e.g. patch-version bump):
- Confirm
spec.updateStrategy.type: OrderedUpdateis set (it is the default). - Bump
spec.imageto the new pinned tag. kubectl apply.- Watch
status.updatePhaseand theUpdating/Bootstrappingconditions onstatus.conditions, and confirm replication catches up before the second site rolls.Updating=Truewith reasonOrderedUpdateStalledmeans a rolled pod did not becomeReadyand stayReadyfor 30s (for example a new sidecar image that cannot start or crash-loops): the rollout stopped there, with the active site still serving, until that pod has beenReadyfor 30s. Each rolled site takes at least those 30s. - Take a new full backup, then run a backup-verification drill against it on the new version (see Backup verification).
With OrderedUpdate, Bloodraven upgrades the standby first, then
hands the primary role to it, then upgrades the old active. The handoff
fences the old active and promotes the standby only once it holds every
transaction the old active committed, so it loses none and pauses writes
for at most about 30s (plus the promotion itself, or up to 10s to unfence on
abort); if the standby cannot catch up, the old active is unfenced
and the rollout retries later. This is the direction
MySQL's rolling-upgrade contract requires: the replica's MySQL version
must be ≥ the primary's during the rollout. See
Failover → Ordered updates for the
full sequence.
The handoff also refuses a standby that holds transactions the active site
lacks, and the operator audits every replica for such transactions (see
Divergent replicas). An
upgrade can therefore stall on purpose: if a replica already carries errant
GTIDs, Degraded turns True with reason DivergentTransactions on the
first polls after the new operator starts, followers are still rolled, and
the active site's update waits, unfenced and serving, until the replica is
recloned or the primary is made to contain the set. This applies to the
operator upgrade itself whenever it changes the MySQL pod template (a sidecar
image bump does). Installations that ran ordered updates on v1.5.0 or earlier
can carry such GTIDs from issue #202. The stall does not stop application deployments: the deployment API keeps granting leases for a replica's divergence report.
When upgrading to v1.6.0, apply the v1.6.0 MysqlFailoverGroup CRD with server-side apply before upgrading the chart (Helm does not upgrade crds/); this routine is not for v1.6.0 → v1.7.0 (see Upgrading from v1.6.0 to v1.7.0). It adds status.sites[].divergenceDetectedBy and divergenceComparedTo and the mfg short name. Without it the API server drops the two fields, and after an operator restart a replica's report reads as an old primary's: the messages say "old primary" and the deployment API treats the group as unstable until the report clears.
To find out before upgrading, audit each follower against the primary. Read the follower first, then compute the difference on the primary:
-- On the follower:
SELECT @@GLOBAL.gtid_executed;
-- On the primary, with the follower's set from the line above:
SELECT GTID_SUBTRACT('<follower gtid_executed>', @@GLOBAL.gtid_executed) AS extra;
An empty extra means the follower holds nothing the primary lacks (it says
nothing about rows or lag). A non-empty one is the set to discard by
reclone or to make the primary contain (see Divergent replicas for what each choice does to the replica's rows). Repeat the check if the primary
changed or a server was replaced between the two reads. The
bloodraven-doctor skill's gtid-audit.sh runs the same comparison for every
follower.
updateStrategy.type: Recreate for a major bumpRecreate restarts both sites simultaneously. If one pod's in-place
MySQL upgrade stalls or fails, you lose write availability with no
healthy primary and no rollback path. Use OrderedUpdate for any
MySQL image change.For MySQL-level upgrade semantics — data-dictionary migrations, deprecated-feature removals, schema changes — defer to Oracle's own release notes. Bloodraven drives the rollout; it does not duplicate the MySQL upgrade matrix.
Cross-site version skew
Transient (during a rollout): allowed. The OrderedUpdate
sequence keeps the newer MySQL on the replica side of replication
throughout the bump, which is MySQL-supported. The skew window lasts
from the moment the standby finishes upgrading until the old active
has been restarted on the new image — roughly pod-ready time plus
the failover drain. No user action is required.
Steady-state (intentional per-site version split): not supported.spec.image is a single field per MysqlFailoverGroup; SiteSpec
has no per-site image override. The operator reconciles both site
Deployments to the same image tag. If you edit a site's Deployment
manifest out of band to pin a different tag, the next reconcile
reverts it.
Sidecar ↔ operator skew
The sidecar binary and the operator binary are released together.
The supported configuration is matching versions. A group that leaves
spec.sidecar.image unset gets that by construction: it runs the
sidecar image the operator release ships with (the chart's
sidecar.image.repository and sidecar.image.tag, passed to the
operator as BLOODRAVEN_SIDECAR_IMAGE; the tag defaults to the chart's
appVersion). A group that pins spec.sidecar.image should pin the
release of the running operator Deployment; a pin with another release
tag gets a SidecarVersionSkew Warning.
Because an unpinned group follows the operator, an operator upgrade
that ships a new sidecar image changes that group's pod spec, and the
group rolls: an ordered update (followers first, a handoff of the
primary role, then the old primary; see
Ordered updates) under
OrderedUpdate, both sites at once under Recreate. Each such group starts its roll on its own after the
upgrade. A pinned group does not move its sidecar until the pin
changes, though a release that changes the rendered pod spec in other
ways rolls it too.
Bloodraven tolerates a one-minor transient skew in either direction during a rolling upgrade. Both sides of the HTTP surface between the two are additive-only:
- Operator endpoints (polled by sidecars):
/healthz,/status,/active-site,/pitr-cutoff. - Sidecar endpoints (polled by the operator and peer sidecars):
/health,/status,/peer/ping,/peer/active-site,/archiver/status,/keyring/status,/livez.
PITR retention uses the additive coverage=gtid-v1 capability and backup
GTID bounds. During skew, older sidecars receive no timestamp cutoff
from the updated operator, and updated sidecars reject time-only replies
from older operators. New archive retirements pause until both sides can
prove content-bound coverage; persisted pending deletes can still finish.
Previously complete proofs without matching proof content IDs are
re-inspected in bounded batches before new retirement. Pending or Running
group backups pause new retirements across all profiles. Upgrade the
operator first to suppress unsafe time-only
retention replies from the older protocol.
An older sidecar rewriting an ID-aware PITR manifest drops IDs, coverage, purged-base bridges, quarantine/reset evidence, divergence/acknowledgement records,
and pending deletions; its pruning may drop all rows sharing one basename.
Re-upgrading recovers surviving stripped IDs only from exact confined
remotePath keys. It cannot recover lost rows, purged-base provenance, quarantine/reset evidence, divergence/acknowledgement records or pending deletes. Preserve a compatible recovery image for each archive generation; Use a new prefix and full backup after RESET. Obsolete upload-order guards have a verified canonical-backup acknowledgement procedure; verified namespace reuse still requires a new generation. See PITR history repair. Preserve
archive objects, manifests, encryption material, and compatible recovery
images before rollback. See PITR upgrade and downgrade for recovery steps.
/archiver/status gained additive storeReady and
consecutiveFailedScans fields. A sidecar
whose PITR archive store cannot be built (unreadable passphrase or
credentials file, invalid endpoint, empty PVC mount path) now keeps serving
readiness and fencing, reports enabled:true, storeReady:false and
the init error, and retries the store in the background. The fix lives
in the sidecar image, which the operator rolls through the ordered
update: until a pod runs the new image it still exits on that error,
crash-loops, and drops out of every Service. The updated operator
reports such a pod as PITRArchiveReady=False with reason
SidecarUnreachable once it has not answered for 5 minutes since the
operator started (3 failed polls over 60 s for a sidecar that answered
before), and reads a missing storeReady from an older sidecar as "not
reported", never as an init failure. Without consecutiveFailedScans
it debounces errors on the older sidecar's lastScanAt. An older operator ignores the field and still shows
the init error in status.pitr.message.
The PVC archive store now refuses to start unless its mount path is a
mount point, and writers also probe write access. A site whose
pitr-archive volume was missing from the pod, or not writable, used
to show PITRArchiveReady=True (writes landed in the container
filesystem) or a later ArchiverError. After the sidecar rollout it
shows False / ArchiveStoreInitFailed. pitr-download,
pitr-quarantine and the encrypted-backup encrypt-upload /
decrypt-download steps now fail at start when their PVC is not mounted
at the configured path. Writers also refuse a write prefix that leaves
the mounted volume through a symlink (for example binlogs linked to
another volume), which used to work: the store reports
ArchiveStoreInitFailed with outside the archive volume; replace the
symlink with a directory on the archive volume. Unencrypted PVC backups
are unchanged.
The kubelet probes the sidecar too, and the operator renders those
probes. That makes the probe paths part of the same contract, so they
only use endpoints every supported sidecar serves. Liveness probes
/peer/ping, which answers without touching MySQL. Readiness probes
/health. The newer /livez also detects a wedged fencing loop, but a
sidecar that predates it answers 404, and a liveness probe on it would
restart that sidecar in a loop. It becomes the liveness target once
every supported sidecar serves it.
A briefly older-or-newer sidecar stays compatible with the operator while pods roll. Skew beyond one minor is not tested and not guaranteed.
When a change moves both spec.image and the sidecar image
together, OrderedUpdate restarts the sidecar as part of the pod
restart it already performs — the replica-first ordering applies
automatically, no separate sidecar sequencing is required.
Upgrading from v1.6.0 to v1.7.0
crds/, the Argo CD layout in
Production install examples,
or the "CRDs managed separately" loop in
Production install pointed at a
newer release. On a v1.6.0 cluster, pin both the chart version and the CRD
source in Git for as long as it runs.Why an in-place upgrade cannot work:
- Stored defaults change type. Every v1.6.0
MysqlFailoverGroupstorescloneTimeout: 3600(a duration in v1.7.0) andupdateStrategy: OrderedUpdate(an object in v1.7.0), and every backup profile storesretention: 7(an object in v1.7.0). The API server prunes unknown fields but does not convert types, so the v1.7.0 operator cannot decode a stored v1.6.0 group, and one undecodable object fails its whole list of groups: it then manages none of them. - CRDs are cluster-scoped. Applying the v1.7.0 CRDs changes the schema for every namespace at once. From then on every read of an old object has the old field names pruned, and the still-running v1.6.0 operator persists the pruned object on its next write.
- Stored objects can break the new name and size rules. A group stored
before v1.7.0 can have a name that is not a DNS-1035 label, a group name
plus site name longer than 41 characters, or an integer
spec.sites[].storage.size. On Kubernetes 1.31 the API server validates status writes without ratcheting, so it refuses the operator's own status writes to such a group. An integer size cannot be decoded at all: v1.7.0 stores a size as a string. - Re-applying the manifests does not help. Every bootstrapped group's
initFromBackupblock is renamed, and v1.7.0 refuses any change to it. A manifest re-apply cannot rewrite status, and several status records cannot be rebuilt from observation.
If the v1.7.0 CRDs were applied to a v1.6.0 cluster by mistake, scale every Bloodraven operator to 0 first and write no Bloodraven object. Do not re-apply the v1.6.0 CRDs as a fix; recovery depends on whether any object was written since, and is part of the procedure below.
The supported path: a new group in a new cluster
The supported path bootstraps a new group, in a cluster that has never had the Bloodraven CRDs, from a v1.6.0 backup. The step-by-step procedure, its recovery steps and its end-to-end test are tracked in issue #243. Until it is published, this outline is its scope, not a tested procedure:
- Stop application writes and take a full backup on v1.6.0, on an S3 profile that the new cluster can reach.
- Install v1.7.0 in a cluster with no Bloodraven CRDs.
- Write the new manifests with the changes listed in
What changes in v1.7.0. Give every backup and
PITR profile a prefix that no v1.6.0 group has used; reference the old
backup only as a restore source. Apply with
kubectl apply --server-side --validate=strict, so a leftover old field name is refused instead of pruned. - Bootstrap the group from the backup (
initFromBackup, withdecryptionfor an encrypted backup), excluding the schemas thatMysqlDatabaseobjects manage. Then apply theMysqlDatabaseobjects and load each of their schemas with a per-schemarestoreInPlace. - Recreate the MySQL accounts that neither the operator's credentials nor a
MysqlDatabasecreate: restores do not load users. - Take a new full backup as the new group's baseline, wait for
PITRArchiveReady=Truewhen PITR is on, verify the baseline with aMysqlBackupVerification, and compare the application's data with the frozen source. - Cut over, and only then retire the old group. Keep the old cluster, its backups, its passphrases and its keyring escrow Secrets for as long as you may need them.
The new group does not carry over failover history, operation records
(restores, planned failovers, recoveries), MysqlBackup and
MysqlBackupVerification objects, verification holds, keyrings (it mints
its own) or PITR across the old archive.
There is no in-place rollback either: the v1.6.0 operator cannot decode a v1.7.0 object. Until cut-over is verified, the old group in the old cluster is the way back.
What changes in v1.7.0
Use this list to write the new group's manifests and to update alert rules,
scripts and dashboards. v1.7.0 supports Kubernetes 1.31 and newer (the
chart's kubeVersion is >=1.31.0-0). On 1.31 and 1.32, keep the
CRDValidationRatcheting feature gate on (the default): with it off, the
API server does not check at creation that
spec.initFromBackup.pointInTime.archiveProfileName names a profile (see
Install in production).
Manifest fields
Every site needs an explicit, unique serverID (the examples use 101,
102, ...). The restore creates new data directories, so the ids need not
match the old group's 101 + index. The id is immutable once set.
| Kind | v1.6.0 field | v1.7.0 field |
|---|---|---|
MysqlFailoverGroup | spec.credentials.{operator,app,readOnly,monitor,backup}Secret | spec.credentials.{operator,app,readOnly,monitor,backup}SecretName |
MysqlFailoverGroup | s3.credentialsSecret (profiles, initFromBackup and restoreInPlace sources) | s3.credentialsSecretName |
MysqlStandbyCluster | spec.source.s3.credentialsSecret | spec.source.s3.credentialsSecretName |
MysqlFailoverGroup, MysqlStandbyCluster | encryption.passphraseSecret, decryption.passphraseSecret | …passphraseSecretKeyRef (same {name, key} body) |
MysqlFailoverGroup | spec.dragonfly.auth.{secretName, passwordKey} | spec.dragonfly.auth.passwordSecretKeyRef{name, key} |
MysqlFailoverGroup | spec.dragonfly.snapshot.{s3Endpoint, s3UseHTTPS} | spec.dragonfly.snapshot.s3EndpointURL: s3Endpoint: rustfs:9000 with s3UseHTTPS: false becomes http://rustfs:9000; an unset s3UseHTTPS means https:// |
MysqlFailoverGroup | spec.sidecar.bloodravenAddress | spec.sidecar.operatorEndpoint |
MysqlFailoverGroup | spec.sidecarImage (every v1.6.0 group stores the default 1.3.2), spec.sidecarResources | spec.sidecar.image, spec.sidecar.resources. Leave image unset to run the sidecar the operator release ships with; do not copy the stored 1.3.2 |
MysqlFailoverGroup | initFromBackup.source.mysqlBackupRef, restoreInPlace.source.mysqlBackupRef | …source.backupRef (name required) |
MysqlDatabase | spec.groupRef | spec.failoverGroupRef |
MysqlDatabase | spec.collation defaulted to utf8mb4_unicode_ci (stored in every v1.6.0 object) | no default: unset renders no COLLATE. Set collation: utf8mb4_unicode_ci where new schemas must match the old ones |
MysqlBackup | spec.sourceSiteOverride | spec.sourceSite |
MysqlFailoverGroup | spec.sites[].lbIP | spec.sites[].loadBalancerIP (an IPv4 address) |
MysqlFailoverGroup | profile storage.pvc.{storageClassName, size} (an unset size meant 10Gi) | storage.pvc.provisioned.{storageClassName, size} (size required); keep pvc.claimName for an existing claim. Exactly one of the two |
MysqlFailoverGroup | spec.sites[].role: primary-candidate, dr-only, read-only (stored in every v1.6.0 group) | PrimaryCandidate, DisasterRecovery, ReadOnly |
MysqlFailoverGroup | spec.updateStrategy: OrderedUpdate or Recreate (stored in every v1.6.0 group) | spec.updateStrategy: {type: OrderedUpdate} or {type: Recreate}; omit it for the default OrderedUpdate |
MysqlFailoverGroup | spec.plannedFailover.onCooldown: reject, defer; spec.dragonfly.plannedFailover.onSyncTimeout: proceed, fail | Reject, Defer; Proceed, Fail. Lower-case values are refused |
MysqlFailoverGroup | spec.backup.profiles[].retention: 7 (stored in every v1.6.0 profile) and retentionPolicy | retention: {keepLast, keepWithinDays, keepFailed}; see Moving from the released retention fields |
MysqlFailoverGroup | spec.cloneTimeout: 3600 (integer seconds, stored in every v1.6.0 group) | spec.cloneTimeout: 1h (a duration, 1m to 24h); omit it for the default |
MysqlFailoverGroup | spec.dns.ttl | spec.dns.ttlSeconds |
MysqlFailoverGroup | encryptionAtRest.keyring.escrowTimeoutSeconds: 600 | encryptionAtRest.keyring.escrowTimeout: 10m |
MysqlFailoverGroup | backup.retry.{initialBackoffSeconds: 60, maxBackoffSeconds: 1800} | backup.retry.{initialBackoff: 1m, maxBackoff: 30m} |
MysqlFailoverGroup | spec.dragonfly.maxMemoryMb: 512 | spec.dragonfly.maxMemory: 512Mi |
MysqlFailoverGroup | spec.dragonfly.proactorThreads: 0 | omit it (unset means the default) |
MysqlFailoverGroup | initFromBackup.pointInTime.stopDatetime | initFromBackup.pointInTime.stopTime: RFC 3339 with an explicit zone and whole seconds. A YYYY-MM-DD HH:MM:SS or zone-less value (read as UTC before) needs a Z |
MysqlFailoverGroup | initFromBackup.pointInTime.{sites, allowUnverifiedCrossSite} | removed; pointInTime.archiveProfileName is required instead |
MysqlBackupVerification, profile verification | pointInTime.mode: none, latest, timestamp | Latest or Timestamp, required in the block; for none, omit the block |
MysqlBackupVerification, profile verification | pointInTime.timestamp | pointInTime.stopTime, required for Timestamp and forbidden for Latest |
MysqlBackupVerification, profile verification | pointInTime.sites | removed |
MysqlBackupVerification, profile verification | sanityCheck.expect.{minRows, maxDurationSeconds: 60} | sanityCheck.{minResult, timeout: 60s} |
MysqlFailoverGroup | profile verification.retentionPolicy{keepSuccessful, keepFailures} | verification.retention{keepLast, keepFailed}; omit an explicit 0, which now means keep none |
MysqlStandbyCluster | spec.source.storage{type: S3, s3{…}} | spec.source.s3{…} (prefix required) |
MysqlStandbyCluster | spec.source.{failoverGroupName, namespace, cluster, profileName}, spec.template | removed; spec.transport accepts only ObjectStore |
v1.7.0 removes these fields, which earlier releases accepted and ignored or refused:
spec.secretName(the DSN credential mode).spec.credentialsis required: create an operator Secret withusername,passwordandMYSQL_ROOT_PASSWORD(plusMYSQL_REPLICATION_USERandMYSQL_REPLICATION_PASSWORDif replication uses its own account) and setcredentials.operatorSecretName. As the DSN mode did, the first-boot init script of the new group creates that replication account withREPLICATION SLAVE, REPLICATION CLIENT, BACKUP_ADMIN, CLONE_ADMIN. Express DSNtls=options throughspec.tls.spec.tls.issuerRef:spec.tlsissecretNameonly.spec.restoreInPlace.pointInTime, which the operator refused; useinitFromBackup.pointInTimeon a new group.loadOptions.resetProgress, profilestorage.s3.storageClass, profileverification.storageandMysqlBackupVerificationspec.storage. Bound a verification's disk withspec.resources.limits.ephemeral-storage.spec.triggeredByonMysqlBackupandMysqlBackupVerification: the value moves to the annotationbloodraven.shipstream.io/triggered-by.- A
replica-preserve-commit-orderoverride inspec.mysqlConf: it is operator-owned and forcedON.
Some values that v1.6.0 accepted are refused, and some defaults change meaning:
- Timers must be inside their bounds (see
Timers): for example
pollInterval: 0s, afailoverCooldownunder30s, or aplannedFailover.maxLagWaitover1his refused, and so is aplanned-failoverannotation whosemaxLagWait=is outside1sto1h.dragonfly.maxMemory, if set, must be a positive quantity. - Names and lists. S3
prefixvalues (profiles, restore and standby sources) are at most 512 characters with no empty (a//b),.or..segment, and a leading/is ignored, so/prodandprodare the same prefix. Profile names are DNS labels of at most 63 characters, andspec.backup.profilesis a list-map keyed byname(at most 32). Every Secret name and everyfailoverGroupRef.nameandbackupRef.nameis a DNS subdomain (lowercase, at most 253 characters), and every Secret key is at most 253 characters of[-._a-zA-Z0-9]; set an optional Secret name to nothing rather than"".MysqlBackupandMysqlBackupVerificationprofileNameand scheduleprofileNameare DNS labels,MysqlBackupsourceSitematches the site-name pattern, andspec.sidecar.operatorEndpointishost:port. The CRDs also refuse duplicate entries insplitBrainPolicy.sitePriorities; more than onerestoreInPlace.loadOptions.includeSchemasentry; duplicate names, or more than 16 entries, inextraContainers,extraInitContainersandbackup.imagePullSecrets; duplicatebackup.schedulesnames, more than 32 schedules, or a schedule name longer than 63 characters; duplicate, more than 256, or longer than 64 character names in thedumpandloadOptionsincludeSchemas/excludeSchemaslists; more than 16MysqlDatabasedeploymentClients(now keyed by the namespace and ServiceAccount pair); and aMysqlBackupwhose name ends in.bloodraven-backup.json. - Names that become object names. A
MysqlFailoverGroupname is a DNS-1035 label (a.band1ordersare refused: the name starts the group's Service names). A site name is a DNS label of at most 20 characters (no leading or trailing-), and the group name plus each site name is at most 41 characters.splitBrainPolicy.sitePrioritiesentries andMysqlBackupsourceSitefollow the site rule. AMysqlBackuporMysqlBackupVerificationname is at most 63 characters. A site name is part of the site's PVC and Service names, so a name that needs to change means a new group. - Sizes (
spec.sites[].storage.size,pvc.provisioned.size) are positive quantity strings; a bare number of bytes must be quoted. restoreInPlace.confirmmust be an RFC 3339 timestamp with an upper-caseTandZ, an offset within-23:59..+23:59and a digits-only fraction.- Edits that are refused. A site cannot be removed or renamed (see
Replacing a site); a
site's
zone,storage.storageClassNameandstorage.sizeare immutable (a size is compared by value);spec.initFromBackupcannot be added, changed or removed (a bootstrap that failed on its own block is retried by recreating the never-bootstrapped group, see A failed bootstrap restore);spec.encryptionAtRest.enabledcannot go back tofalse, and the keyring paths cannot change while it istrue; a profile'spvc.provisionedcannot change, be added or be removed; and aMysqlBackup'sfailoverGroupRef,profileNameandsourceSite, and aMysqlBackupVerification'sfailoverGroupRef,profileName,backupRef,pointInTimeandsanityCheck, are immutable. ttlSecondsAfterFinishedon aMysqlBackupVerificationand on a profile'sverificationno longer defaults to0: an omitted value keeps a successful verification's Pod (and its restored datadir) until retention deletes the verification. SetttlSecondsAfterFinished: 0to keep the immediate cleanup.- An explicit
0is kept as0instead of being replaced by the default:replication.maxLagSeconds: 0requires a caught-up replica (it meant300), andbackup.backoffLimit: 0andbackup.maxLagSecondsForSource: 0are honored. MysqlBackupVerificationbackupRef: {}(which meant "latest") is refused: omitbackupRefinstead.MysqlBackupspec.deletionPolicyis new. The defaultDeletekeeps the v1.6.0 behaviour (deleting the object deletes its artifact);Retainkeeps the artifact, unless it is on a claim the group provisioned, which is deleted with the group (Keeping an artifact).- The chart's operator environment variable
BLOODRAVEN_DEFAULT_AUXILIARY_ADDRESSis nowBLOODRAVEN_DEFAULT_OPERATOR_ENDPOINT; rename anextraEnventry that overrides it.
Labels, annotations and action annotations
Every label, annotation and finalizer key moves from shipstream.io/ to
bloodraven.shipstream.io/, and boolean label values are "true" or
"false" (they were healthy=yes|no and dragonfly-traffic=enabled).
Write Service selectors, NetworkPolicies, monitors and scripts for the new
keys; the full list is in
Well-known metadata.
| v1.6.0 | v1.7.0 |
|---|---|
Node taint shipstream.io/db-readonly-<group> | <namespace>.db-readonly.bloodraven.shipstream.io/<group>: write tolerations for the new key. Remove the old taints from the old cluster's nodes only once its groups are retired (kubectl taint nodes --all <old key>-) |
MysqlBackup annotation hold.verification.shipstream.io/<uid> | verification-hold.bloodraven.shipstream.io/<uid> (operator-written) |
Finalizer shipstream.io/graceful-shutdown | bloodraven.shipstream.io/failover-group (operator-written) |
reclone-site=<site>:confirm=<group> and reclone-site=<site>:<gtid-prefix> | reclone-site=<site>:confirm=<group>:uid=<group uid>, plus :gtidPrefix=<source UUID prefix> for a diverged site. kubectl bloodraven reclone builds the value |
Annotation bloodraven.shipstream.io/dragonfly-snapshot-upgrade=<image> | spec.dragonfly.upgradeStrategy: SnapshotRestore, then edit spec.dragonfly.image |
Annotation bloodraven.shipstream.io/encryption-adopt=confirm | spec.encryptionAtRest.existingData: LeavePlaintext |
PITR archive identity
v1.7.0 changes the PITR archive format and refuses archives written before
it. The new group's PITR profile must use a new prefix (a v1.6.0 prefix is
refused as ArchiveLegacy), and its first full backup after the archive is
Bound (PITRArchiveBindingReady=True) is its first point-in-time
baseline: it carries a backup stamp and status.pitrArchiveID.
- Grant the sidecar's PITR credentials
s3:ListBucket,s3:GetObject,s3:PutObjectands3:DeleteObject, and confirm the store enforces conditional writes (PITR archive requirements). Grant every backup profile's credentialss3:ListBucket,s3:GetObjectands3:PutObjecton<prefix>/*: each backup Job checks that its dump location is empty, then reads its backup stamp before writing it and reads it back after. Artifact cleanup still needss3:ListBucketands3:DeleteObject, and deleting a failed backup reads its stamp too (Backup credential permissions). - v1.7.0 cannot read a v1.6.0 archive. Keep the old prefix, its backups, its
passphrase and the v1.6.0 cluster if you may need a target from before the
move. Do not run a pre-v1.7.0 sidecar against a v1.7.0 archive: it
rewrites its manifest as version 1 and the archive is then refused as
PITRArchiveLegacy. - A v1.6.0 backup has no backup stamp, so PITR from it refuses
PITRBackupIdentityUnknown; a dump-only restore from it works. An encrypted v1.6.0 backup also recorded no dump key, and the operator never decrypts a dump with a profile's live Secret, so restoring it (initFromBackuporrestoreInPlace) needs an explicitdecryptionnaming a Secret that holds the passphrase it was written with. Keep that passphrase even after you rotate the profile's Secret. - The archive key is pinned by the group's archive encryption binding: pods
mount an operator-made key snapshot, and in-place passphrase or encryption
changes on an archive in use are refused. Key snapshot Secrets
(
mysql-pitr-key, and onemysql-dump-keyper encrypted backup) have no owner reference and survive group deletion. See Archive encryption bindings.
See also Upgrading to archive identity.
Status, conditions and printer columns
Status is written by the new group's operator; nothing converts an old
value. Update tooling that reads these fields. Among others:
status.lastFailover is {time, target, promotionGtidExecuted} (the flat
lastFailoverTarget and promotionGtidExecuted are gone),
status.sites[].replicating is the enum status.sites[].replication,
status.sites[].state and status.dragonfly.sites[].role are CamelCase,
MysqlBackup status.location and storageType are
status.artifactStorage.{location, type}, binlogFile and binlogPos are
status.binlogPosition, encrypted and encryptionAlgorithm are
status.encryption, a restore's point-in-time result is
status.restore.provenance.pointInTime (the flat status.restore.pitr*
fields are gone; status.restoreInPlace.provenance.pointInTime stays
unset), a restore's source* and target* fields are
status.restore.provenance.{source, target} (sourceSizeBytes is
provenance.source.sizeBytes), a MysqlBackupVerification's
point-in-time result stays status.pointInTime, and its
status.sanityCheck is {outcome, result, durationMilliseconds, error}
(it was {ran, resultRow, durationMs, error}). Also:
status.encryptionAtRest.sealedis gone: alert on theEncryptionAtRestReadycondition (True/AllSitesSealedwhen every site is sealed).status.encryptionAtRest.sites[].unsealedSince,status.restoreReclones.requestedAtandstatus.sites[].lastSeenareunsealTime,requestTimeandlastSeenTime.MysqlDatabasephaseCreatingisApplying.- Observations the operator could not read are unset instead of
falseor0:status.encryptionAtRest.sites[].coverageflags andstatus.dragonfly.sites[].{syncInProgress, lastIOSecondsAgo}. Treat an absent value as unknown.
The CRD reference has every field.
Condition reasons, polarity and printer columns change too. Update alert
rules, scripts and deployment tooling that match on them, and re-apply the
chart's Argo CD health script (charts/bloodraven/argocd/) if you copied it
into argocd-cm: the old script reads Ready.observedGeneration and the
old Bootstrapping and Updating polarity.
| Kind | v1.6.0 | v1.7.0 |
|---|---|---|
MysqlFailoverGroup Ready | reason TopologyPolled, True or False | True/Healthy; False/NoWritablePrimary (no site writable) or False/ReplicationUnhealthy |
MysqlFailoverGroup Degraded | True/Degraded, True/Alert | True/SiteUnhealthy, True/CrossSiteAlert; new True/BootstrapFailed while a failed clone's target does not replicate |
MysqlFailoverGroup Bootstrapping | False/Done after a clone; True/Failed after a failed one | True only while a clone runs. False/NotBootstrapping after a clone (also before any clone); False/BootstrapFailed after a failed one, together with Degraded=True/BootstrapFailed |
MysqlFailoverGroup Updating | False/OrderedUpdateStalled | True/OrderedUpdateStalled (the update is still in progress) |
MysqlFailoverGroup status | condition observedGeneration was the generation of the object written | new status.observedGeneration, the generation the topology manager applied, carried by the topology conditions; the Argo CD health script reads it |
MysqlBackup Ready | False/NoHealthySource while waiting for a source; a wait past 30 minutes failed with its pending reason | False from creation (Pending, the pending reason, Running); False/WaitingForHealthySource while waiting for a source; past 30 minutes NoHealthySource (a source wait) or PendingTimedOut (any other wait). NoHealthySource is terminal only |
MysqlBackupVerification Ready | absent | mirrors Verified; False/InProgress before |
MysqlStandbyCluster Ready | absent | True/SourceDiscovered when BucketReadable and SourceConfigKnown are both True, otherwise the first one's False reason |
MysqlDatabase Ready | False/GroupNotFound while the group does not exist | False/GroupNotYetCreated (GroupNotFound stays a terminal reason on backups and verifications) |
| Printer columns | kubectl get msc showed BucketReadable and SourceKnown | kubectl get msc shows Ready instead; kubectl get mysqlbackup and kubectl get mysqlbackupverification gain Ready |
Deployment API unstableReason | PlannedFailover for a standing planned-failover automation hold; Failed for a failed clone | AutomaticFailoverHeld; BootstrapFailed. PlannedFailover means only a run in flight or its claim |
Events
Event reasons follow one rule (<Subject><PastTenseVerb>, with a qualifier
naming the cause), and each is recorded with one type. Update Event routing,
alert rules and scripts that match on the old reasons. Status condition and
status reason values are unchanged by these renames: the PITRArchiveReady
condition and the DragonflySyncTimeout planned-failover reason keep their
names. The full list of reasons is in
Kubernetes Events.
| v1.6.0 Event reason | v1.7.0 |
|---|---|
DragonflyPromotionCompleted | DragonflyPromotionSucceeded. An emergency promotion that fell back to REPLICAOF NO ONE no longer also emits a Warning DragonflyPromotionCompleted; it emits DragonflySessionsLost only |
DragonflyUpgradeCompleted | DragonflyUpgradeSucceeded |
DragonflySyncTimeout (Event) | DragonflySyncTimedOut |
DragonflyPromotionWaiting | DragonflyPromotionDeferred |
RecoveryComplete | OldPrimaryRecovered |
PromotionRefusedPendingReclone | PromotionRefusedReclonePending |
KeyringPhase (Normal, or Warning for a move to Failed) | KeyringPhaseChanged, always Normal. A site that is not escrowed within keyring.escrowTimeout gets the new Warning KeyringEscrowTimedOut; the other moves to Failed already had their own Warning |
KeyringRotationRefused | KeyringRotationRejected |
PITRArchiveReady (Event) | PITRArchiveRecovered |
PointInTimePartial | PITRReplayTruncated, which also starts the partial restore's status.restore.message |
PITRArchiveKeySnapshotUnreadable | PITRArchiveKeySnapshotReadFailed |
PITRArchiveNotStamped | PITRArchiveStampSkipped |
PlannedFailoverDraining | PlannedFailoverDrainStarted |
PlannedFailoverLagOK | PlannedFailoverTargetCaughtUp |
PlannedFailoverCompleted | PlannedFailoverSucceeded |
PlannedFailoverHoldAcknowledgementIgnored | PlannedFailoverHoldAcknowledgmentIgnored |
RestoreInPlaceCancelled | RestoreInPlaceCanceled |
RestoreInPlaceRejected for an invalid confirm on a finished restore (nothing runs) | RestoreInPlaceConfirmIgnored. RestoreInPlaceRejected now only ends a run Failed |
RestoreInPlaceFailed, twice per failed restore Job | RestoreInPlaceFailed, once |
RestoreInPlaceWaitingForPreviousQueue | RestoreInPlaceDeferredPreviousQueue |
RestorePeerReplicationIncomplete | RestorePeerReplicationStoppedBehind |
RestoreReclonesComplete | RestoreReclonesDrained |
ArtifactCleanupProducerStuck | ArtifactCleanupStalledProducerStop |
DatabaseDropSkipped (Warning) when another MysqlDatabase still declares the database | DatabaseDropRefused. DatabaseDropSkipped is now only the Normal "nothing to drop" case |
DatabaseDropSkipped (Warning) when another MysqlDatabase still claims the owner user | OwnerUserDropSkipped |
DragonflyUpgradeRejected (the snapshot-upgrade annotation was refused) | removed with the annotation. A request without a snapshot directory, or with Dragonfly disabled, is refused at admission; a run that cannot start yet (no status.activeSite, a planned failover or automation hold owning the sites) emits Normal DragonflyUpgradeDeferred |
ArtifactLocationRestored | removed, with no successor: status.artifactStorage.location is the only recorded location, so there is no second field to restore from |
Metrics, logs and the operator's endpoints
bloodraven_planned_failovers_total{result}is the kebab-case of the run's outcome (succeeded,not-started,rolled-back,source-left-fenced,promoted-unconfirmed,manual-recovery-required) instead ofsuccess,rejected,failed_timeoutandfailed_other.failed_timeouthas no single successor: a lag timeout endsrolled-back, an admission timeoutnot-started; select onstatus.plannedFailover.result.reasonfor the cause.- A
rolemetric label ofdr-onlyisdisaster-recovery; the otherroleandstatelabel values keep their kebab-case spelling. bloodraven_restore_reclone_pending{phase}values arearmed,drainingandheld(they wereArmed,DrainingandHeld).- The deployment lease revocation reasons
operator_revokedandtopology_changedareOperatorRevokedandTopologyChangedin thebloodraven_deploy_lease_revocations_total{reason}label, the/deploy/v1409 revokedbody'sreason, and theDeploymentLeaseRevokedEvent message. Otherreasonlabels carry the renamed condition and Event reasons. - The operator's
/statusand/ws/statuscarry the CR values for sitestateandrole(Writable,ReadOnly,PrimaryCandidate, ...). Both carrylastFailover{time, target, promotionGtidExecuted}in the shape ofstatus.lastFailover; the flat/statuspromotionGtidExecutedfield is gone./ws/statusalso carriessites[].replication, and itsupdatePhaseis now populated (it was always empty). - Structured logs: the
state transitionrecord'sfromandtovalues and therolefield of the fencing and planned-failover records carry the same CamelCase values. The INFO recorddragonfly snapshot-upgrade deferred: the Dragonfly master differs from the MySQL active site(fieldsfg,dragonflyMaster,activeSite) is nowdragonfly snapshot-upgrade deferredwith fieldsfgandreason, and covers every cause a run waits for. See the log schema. /deploy/v1operations.plannedFailovercarries the planned-failover vocabulary below (Deployment API).
Planned failover in v1.7.0
The durable planned-failover claim (issue #235) is part of v1.7.0. It
replaced the shape of status.plannedFailover (see the
CRD reference) and
its vocabulary:
spec.plannedFailover.onCooldownacceptsRejectorDefer, andspec.dragonfly.plannedFailover.onSyncTimeoutacceptsProceedorFail; the lower-case spellings are refused.- Queries on the removed status fields (for example
transactionsLostordurationSeconds) must move tostatus.plannedFailover.resultand the duration histogram. The renamed Event reasons are in Events;PlannedFailoverStarted,PlannedFailoverSkipped,PlannedFailoverRejected,PlannedFailoverDeferredandPlannedFailoverFailedkeep their names. The new reasons are listed in Planned failover. - Deployment API companions.
operations.plannedFailoverin/deploy/v1carries the new vocabulary.phaseisstatus.plannedFailover.phase:PendingandValidatingbecameAdmitting,ResumingbecamePublishing, andFencingandRollingBackare new; the empty phase is reported only when there is no block. Aplanned-failover-terminalmarker that names the run wins over a stale live phase:phaseis thenSucceededorFailedwith the marker's outcome and reason.reasonis no longer a free-form string, and it belongs to the current phase only:admission.waitReasonwhileAdmitting,admission.deferReasonwhileDeferred,result.reason(empty when the result has none) forSucceededandFailed, and empty in any other phase; all are enums. The new keyoutcomecarriesresult.outcome(empty until the run ends). A companion that switches on phase or reason strings must be updated before cut-over. Coordination decisions should usestableandunstableReason, which kept their meaning.
Restoring across versions
MysqlBackup.status.mysqlImage records the image tag that produced
each dump. The safe path is to restore onto the same major version
that produced the backup. Do not restore a dump onto an older
MySQL version — MySQL does not support version downgrade of data.
Backup-restore throughput and sizing guidance live under a separate performance guide (WISHLIST #27).
What Bloodraven does not do
- No version admission check. Setting
spec.imageto an unsupported tag will not produce a CRD validation error. MySQL itself rejects unsupported upgrades at pod start. - No skip-a-major guardrail. There is no enforcement that blocks
an
8.0 → 9.7jump or a9.7 → 9.8+jump at the operator layer. - No downgrade path. MySQL does not support version downgrade, and Bloodraven inherits that limitation.
- No per-site MySQL version. The CRD does not expose a per-site image, so mixed-version steady state is not expressible.

