CRD Reference

Complete reference for Bloodraven custom resources (shipstream.io/v1alpha1).
CRDs at a glance
| Kind | Short names | Categories | Scope | Description |
|---|---|---|---|---|
MysqlFailoverGroup | mfg, mfgs | bloodraven, mysql | Namespaced | Core resource. Declares a multi-site MySQL HA cluster with automatic failover. |
MysqlDatabase | mydb, mydbs | bloodraven, mysql | Namespaced | Tenant database, Secret-backed users and grants, and optional deployment ServiceAccount allowlist. See Tenant databases. |
MysqlBackup | mbk, mbks | bloodraven, mysql | Namespaced | One backup run. Created ad-hoc or by a schedule CronJob. Tracks phase, location, GTID, and binlog coordinates. |
MysqlBackupVerification | mbv, mbvs | bloodraven, mysql | Namespaced | Periodic restore-of-latest-backup into a throwaway mysqld to prove a backup loads cleanly. See Backup verification. |
MysqlStandbyCluster | msc, mscs | bloodraven, mysql, dr | Namespaced | Passive DR relationship descriptor: lives on the DR cluster, continuously monitors the source archive bucket, and stamps readiness conditions. See Multi-cluster DR. |
A category lists every kind in it: kubectl get bloodraven shows all five
kinds in the namespace. The short names mbk and mbv are new in v1.7.0;
mdb is not used, because other operators claim it. There is no backup
category: kubectl expands a category before a resource name, so it would
take over kubectl get backup and kubectl delete backup from other
operators' backup resources (Velero, CloudNativePG), and a delete meant
for those would delete Bloodraven backups and their artifacts.
MysqlFailoverGroup
Complete reference for the MysqlFailoverGroup custom resource (shipstream.io/v1alpha1).
Common fields quick reference
| Field | Use most often for | Guide |
|---|---|---|
spec.sites | Site topology, placement, storage, LB IPs | Placement Contract |
spec.credentials | Per-role MySQL users | Credentials And TLS |
spec.tls | MySQL TLS Secret | Credentials And TLS |
spec.dns | external-dns hostname and TTL | Production Install |
spec.backup | Backup profiles, schedules, PITR | Backup Overview |
spec.initFromBackup | Bootstrap restore into a new group | Backup And Restore |
spec.plannedFailover | Defaults for planned failover API | Planned Failover |
spec.dragonfly | Optional Dragonfly cache/session co-management | App Integration |
spec.restoreInPlace, restore-related confirm tokens, reclone controls, and any field that changes spec.sites, spec.credentials, spec.tls, or MySQL images. Restore and reclone operations can replace data; image and credential changes can trigger restarts or failed reconnects.Spec
Names and fixed fields
The group's name is a DNS-1035 label (a lowercase letter first, then
lowercase letters, digits or -, ending in a letter or digit), because it
starts the group's Service names; a.b and 1orders are refused. The
group's name plus each site's name is at most 41 characters, so the
Dragonfly StatefulSet <group>-dragonfly-<site> fits in 52.
Some fields cannot change once set. Admission compares them by value, so
the operator's own writes, which re-encode a timestamp in UTC or drop an
explicit false, are not changes:
| Field | Rule |
|---|---|
spec.sites[] | A site cannot be removed or renamed; adding one is allowed. See Replacing a site. |
spec.sites[].serverID, zone, storage.storageClassName, storage.size | Immutable. A size is compared as a quantity: 1Gi and 1024Mi are equal. |
spec.initFromBackup | Fixed at creation: it cannot be added, changed or removed. See A failed bootstrap restore. |
spec.encryptionAtRest.enabled | Cannot go from true back to false or be removed. keyring.dataFileDir, mysqldDir and pluginDir cannot change while it is true. |
spec.backup.profiles[].storage.pvc.provisioned | Immutable, and cannot be added or removed. |
spec.backup.profiles[pitr.profileName].encryption | Cannot change while that profile archives PITR binlogs to the same storage. |
Top-level fields
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
image | string | No | mysql:9.7 | MySQL container image |
sites | SiteSpec | Yes | -- | Site definitions. At least two sites with role: PrimaryCandidate; additional DisasterRecovery and ReadOnly followers may be appended. MinItems=2, MaxItems=16. A list-map keyed by name (x-kubernetes-list-type: map): names are unique, and server-side apply merges entries by name. Every serverID must be unique in the group. |
credentials | CredentialsSpec | Yes | -- | Per-role MySQL credential management. The operator creates MySQL users with role-appropriate privileges and rotates passwords when secrets change. |
dns | DNSSpec | Yes | -- | DNS configuration for traffic steering via external-dns |
tls | TLSSpec | No | -- | TLS configuration (cert-manager integration) |
pollInterval | duration | No | 2s | 1s to 5m. Interval for polling MySQL instances. After a site reaches failureThreshold and continues failing, that site's probes back off exponentially to a 30-second cap; other sites keep this interval. |
failureThreshold | int | No | 3 | Consecutive failed polls before a site is marked unreachable |
recoveryThreshold | int | No | 2 | Consecutive successful polls before a site is marked writable |
failoverCooldown | duration | No | 5m | 30s to 24h. Minimum time between automatic failovers (the anti-flap guard; it cannot be disabled). |
connectionDrainTimeout | duration | No | 30s | 1s to 10m. Maximum operator-side retry window for evicting stale application sessions from a fenced former primary after promotion |
splitBrainPolicy | SplitBrainPolicySpec | No | -- | Opt-in automated resolution when multiple sites are writable and the operator has no prior failover history. See SplitBrainPolicySpec. |
sidecar | SidecarSpec | No | {} | Sidecar-container knobs: image, resources, leaseTimeout (default 20s, 3s to 10m, and at least 3× peerCheckInterval), peerCheckInterval (default 5s, 1s to 1m), and operatorEndpoint, the host:port of the operator's auxiliary endpoint (at most 253 characters; omitted uses the chart's auxiliary Service). Defaults to {}, so a manifest without the block still gets the timer defaults and the 3× rule. |
sidecar.image | string | No | -- | Sidecar (and init container) image. Omitted means the sidecar image the running operator release ships with: the chart's sidecar.image.repository and sidecar.image.tag (default the chart's appVersion), passed to the operator as BLOODRAVEN_SIDECAR_IMAGE. Then an operator upgrade also moves the group's sidecars: the new image changes the pod spec, which rolls the group (an ordered update under OrderedUpdate). Set it only to pin a specific image; a pin tagged with another release than the operator gets a SidecarVersionSkew Warning. No value is persisted when it is omitted. |
sidecar.resources | ResourceRequirements | No | -- | Compute resources (requests/limits) for the sidecar container |
terminationGracePeriodSeconds | int | No | 60 | Grace period for MySQL container shutdown. |
mysqlConf | mapstringstring | No | -- | MySQL configuration overrides merged into the generated my.cnf. Option keys may use underscores or hyphens; Bloodraven normalizes underscores to hyphens before merging. At most 128 entries. Admission refuses a key outside [A-Za-z0-9_.-]+ and a value with a carriage return, line feed or NUL or longer than 1024 characters, since either could inject further my.cnf lines. Operator-owned settings (GTID, binary logging, partial_revokes, encryption coverage) cannot be overridden, including through loose-, skip-, disable-, enable- or maximum- spellings; those entries are dropped with a MysqlConfIgnored warning event. To load extra plugins next to the operator's mysql_clone.so, use loose-plugin-load-add. Bloodraven defaults log-bin-trust-function-creators=1 so replicated function DDL is accepted consistently at every site. Changes trigger an ordered rolling restart when updateStrategy.type is OrderedUpdate. |
replication | ReplicationSpec | No | -- | Replication health settings |
updateStrategy | UpdateStrategy | No | {type: OrderedUpdate} | How spec changes are rolled out. type is OrderedUpdate (one site at a time, followers first, then a handoff before the old active site rolls) or Recreate (every site's Deployment at once). |
cloneTimeout | duration | No | 1h | 1m to 24h. Timeout for CLONE INSTANCE (session net_*_timeout and clone_ddl_timeout, set in whole seconds, rounded up). A change is applied on the next reconcile and used by the next clone/reclone; the topology manager does not need to restart. |
podLabels | mapstringstring | No | -- | Additional labels applied to every MySQL pod. Operator labels take precedence on conflict. |
podAnnotations | mapstringstring | No | -- | Additional annotations applied to every MySQL pod. Operator annotations take precedence on conflict. |
serviceTemplate | ServiceTemplate | No | -- | Defaults the client-facing site, primary, and replicas Services. Internal per-site Services ignore this template and remain ClusterIP. |
extraContainers | Container | No | -- | Additional containers injected into every MySQL pod (e.g. exporters). A list-map keyed by name, at most 16 entries. |
extraInitContainers | Container | No | -- | Additional init containers injected after the operator's built-in init container. A list-map keyed by name, at most 16 entries. |
backup | BackupSpec | No | -- | Backup configuration (profiles, schedules, retry, PITR, encryption, security contexts). See BackupSpec. |
initFromBackup | InitFromBackupSpec | No | -- | One-shot restore-on-first-boot before normal bootstrap completes. See InitFromBackupSpec. |
restoreInPlace | RestoreInPlaceSpec | No | -- | Re-triggerable destructive restore against the active primary. Bumping confirm to a newer RFC 3339 timestamp re-arms. See RestoreInPlaceSpec. |
plannedFailover | PlannedFailoverSpec | No | -- | Cluster-wide defaults for the graceful planned-failover API (triggered via bloodraven.shipstream.io/planned-failover=<site>). |
dragonfly | DragonflySpec | No | -- | Optional per-site Dragonfly cache/session co-management. When enabled, the operator creates one Dragonfly StatefulSet per site, an active Redis-compatible Service, replication wiring, promotion status, and an optional snapshot-restore upgrade strategy. See DragonflySpec. |
encryptionAtRest | EncryptionAtRestSpec | No | -- | MySQL InnoDB data-at-rest encryption via the GPL component_keyring_file keyring component, with the live keyring projected from a per-site Secret so no key material lands on the data PVC or a node disk. Requires spec.tls. See EncryptionAtRestSpec and Data-at-rest encryption. |
license | string | No | -- | Optional Ed25519-signed JWT recording this group's license. Public assertion, not a credential. Verified offline; never gates functionality. Overrides the operator-level license. Admission requires the compact JWS shape (three base64url segments joined by ., no whitespace) and at most 4096 characters; a well-formed token that fails verification is reported, not refused. See Licensing. |
Timers
Every timer in the spec is a duration string (30s, 5m, 1h) with a lower
and an upper bound checked at admission (must be between <min> and <max>),
so a zero or a value Go's time.Duration cannot hold is refused instead of
being rewritten to a default. Integer ...Seconds fields remain only where the
value is passed verbatim to a Kubernetes field (activeDeadlineSeconds,
startingDeadlineSeconds, terminationGracePeriodSeconds,
ttlSecondsAfterFinished) or compared with a MySQL or DNS value in seconds
(maxLagSeconds, readOnlyMaxLagSeconds, maxLagSecondsForSource,
dns.ttlSeconds).
Every string, list and map a validation rule reads has an upper bound, so each rule's estimated cost stays inside the API server's CEL budget on every supported Kubernetes version: timer strings are at most 64 characters, and the bounds of other such fields are given where they are described.
| Field | Default | Bounds |
|---|---|---|
pollInterval | 2s | 1s..5m |
connectionDrainTimeout | 30s | 1s..10m |
failoverCooldown | 5m | 30s..24h |
cloneTimeout | 1h | 1m..24h |
sidecar.peerCheckInterval | 5s | 1s..1m |
sidecar.leaseTimeout | 20s | 3s..10m, and at least 3× peerCheckInterval |
backup.pitr.archivePollInterval | 60s | 5s..1h |
backup.retry.initialBackoff | 1m | 1s..1h |
backup.retry.maxBackoff | 30m | 1s..24h, and at least initialBackoff |
encryptionAtRest.keyring.escrowTimeout | 10m | 30s..1h |
plannedFailover.maxLagWait (and the planned-failover annotation's maxLagWait=) | 5m | 1s..1h |
plannedFailover.drainTimeout | 30s | 1s..10m |
dragonfly.plannedFailover.maxSyncWait | 30s | 1s..10m |
MysqlBackupVerification.spec.sanityCheck.timeout (and a profile's verification.sanityCheck.timeout) | 60s | 1s..1h |
MysqlStandbyCluster.spec.freshness.discoveryInterval | 5m | 30s..24h |
The CRDs store each default in Go's spelling (5m0s, 1h0m0s), the form the
operator writes back, so a stored object does not change on the operator's
first update. status.plannedFailover.settings holds the effective timers a
run was accepted with; the operator writes it from these bounded inputs and it
carries no bounds of its own.
SiteSpec
Each entry in spec.sites defines one site in the failover group.
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
name | string | Yes | -- | Unique site identifier (e.g. iad, pdx), the key of spec.sites. Used in Deployment, PVC and Service names, labels, and PITR archive keys. A DNS label of at most 20 characters (^[a-z0-9]([-a-z0-9]*[a-z0-9])?$), and at most 41 characters with the group's name. Cannot be removed or renamed. |
serverID | integer | Yes | -- | The site's MySQL server_id, 1 to 4294967295. Unique within the group and immutable: it identifies the server in binary logs, the PITR archive's promotion records and backup stamps, so it does not follow the site's position in spec.sites. The examples use 101, 102, ... in declaration order. |
role | string | No | PrimaryCandidate | PrimaryCandidate can be promoted; DisasterRecovery and ReadOnly are never promoted. A reader provides a health-gated site-specific read pool. At least two PrimaryCandidate sites are required. |
zone | string | Yes | -- | Availability zone or region for pod scheduling: a Kubernetes label value of at most 63 characters (letters, digits, -, _ and ., starting and ending with a letter or digit). Immutable: the site's volume cannot follow its pods into another zone. |
taintNodeSelector | mapstringstring | Conditional | -- | Required and non-empty for PrimaryCandidate and DisasterRecovery; optional for ReadOnly. Selects nodes that receive this group's db-readonly taint. Accepted but ignored on readers, which never cause taint operations. |
loadBalancerIP | string | Conditional | -- | An IPv4 address (format ipv4). Required for PrimaryCandidate and DisasterRecovery; optional for ReadOnly. The DNS A record points at the active site's address. Accepted but ignored on readers, which are never DNS targets. Renamed from lbIP. |
storage | StorageSpec | Yes | -- | Persistent storage configuration for this site |
resources | ResourceRequirements | No | -- | Compute resources (requests/limits) for the MySQL container at this site |
mysqlConf | mapstringstring | No | -- | Site-specific MySQL options applied after normalized group overrides, with the same key and value rules. Changing this map rolls only affected sites unless other drift is pending. |
serviceTemplate | SiteServiceTemplate | No | -- | Scalar and annotation overrides for this site's client-facing MySQL Service. Does not affect sibling, primary, replicas, or internal Services. |
Reader manifests may retain legacy loadBalancerIP and taintNodeSelector values for
compatibility, but Bloodraven ignores them. Changing a reader to
PrimaryCandidate or DisasterRecovery is rejected until both fields are present.
A reader does not satisfy the two-candidate minimum and cannot appear in
splitBrainPolicy.sitePriorities.
Admission combinations
| Combination | Admitted? |
|---|---|
Two PrimaryCandidate sites, each with non-empty loadBalancerIP and taintNodeSelector | Yes |
| Two candidates plus a reader with both placement fields omitted | Yes |
| Reader with either or both legacy placement fields supplied | Yes; supplied values are ignored |
Omitted/default role, explicit candidate, or DisasterRecovery with either placement field omitted | No |
Fewer than two candidates, or a reader/DR site in sitePriorities | No |
externalTrafficPolicy on an effective ClusterIP Service | No |
Site nodePort on an effective ClusterIP Service | No |
ETP on NodePort/LoadBalancer, or site NodePort in 1-65535 on those types | Yes; Kubernetes then applies its cluster-specific NodePort range |
StorageSpec
| Field | Type | Required | Description |
|---|---|---|---|
storageClassName | string | Yes | Kubernetes StorageClass name: a DNS-1123 subdomain (lowercase letters, digits, - and .) of at most 253 characters. Immutable. |
size | string | Yes | Storage request size, a positive quantity string such as 100Gi (at most 64 characters; a bare number of bytes must be quoted). Immutable, compared by value. The operator applies it only when it creates the PVC; to grow a volume, expand the PVC (see Scaling storage). |
DNSSpec
| Field | Type | Required | Description |
|---|---|---|---|
hostname | string | Yes | The DNS hostname managed by Bloodraven (e.g. orders.az.example.com). The operator creates/updates a DNSEndpoint CR with an A record pointing this hostname to the active site's loadBalancerIP. A change is applied on the next DNS reconcile (the topology poll, default 2s) and replaces dnsName on the existing DNSEndpoint; the operator does not need to restart. Apps should CNAME their DNS to this hostname. |
ttlSeconds | int | No | TTL in seconds for the DNS A record (default: 60, minimum 1). A change is applied on the next DNS reconcile, same as hostname. |
TLSSpec
| Field | Type | Required | Description |
|---|---|---|---|
secretName | string | Yes | Secret holding tls.crt, tls.key and ca.crt. The operator does not issue certificates; create the Secret yourself or with cert-manager (see Credentials and TLS). |
ReplicationSpec
| Field | Type | Required | Description |
|---|---|---|---|
maxLagSeconds | int | No | Maximum acceptable replication lag in seconds before the replica is considered unhealthy. Default 300 when omitted; explicit 0 requires zero reported lag. |
readOnlyMaxLagSeconds | int | No | Maximum lag for a reader's client endpoint. Omitted inherits the effective maxLagSeconds; explicit 0 requires zero reported lag. |
SplitBrainPolicySpec
Opt-in configuration for how the operator should behave when more than one site is simultaneously writable and the operator cannot infer a winner from prior failover history. See Split-brain resolution for a full description of semantics and tradeoffs.
| Field | Type | Required | Description |
|---|---|---|---|
sitePriorities | string | No | Ordered list of PrimaryCandidate site names. The operator walks the list and promotes the first entry that is currently writable; every other writable site is fenced. Empty or unset falls back to manual resolution (alert only). MaxItems=16. Entries must be unique and must reference sites with role PrimaryCandidate. |
ServiceTemplate
| Field | Type | Required | Description |
|---|---|---|---|
type | string | No | Kubernetes Service type: ClusterIP (default), LoadBalancer, or NodePort |
externalTrafficPolicy | string | No | Cluster or Local. Valid only when the effective Service type is NodePort or LoadBalancer. Applied to site, primary, and replicas Services when supported. |
annotations | mapstringstring | No | Default annotations for client-facing site, primary, and replicas Services. Not applied to internal Services. |
SiteServiceTemplate (spec.sites[].serviceTemplate)
| Field | Type | Required | Description |
|---|---|---|---|
type | string | No | Site Service override: ClusterIP, LoadBalancer, or NodePort. Omitted inherits the group type. |
externalTrafficPolicy | string | No | Site override: Cluster or Local. Legal only when the effective site type is NodePort or LoadBalancer. |
nodePort | int | No | Requested NodePort for this site's named mysql port, range 1-65535. Legal only for effective type NodePort or LoadBalancer; cluster admission enforces its configured allocation range. |
annotations | mapstringstring | No | Merged over group annotations for this site only; site keys win. |
The operator preserves Kubernetes-assigned Service identity and compatible
allocated NodePorts across reconciles and external-type transitions. Moving a
site Service to ClusterIP clears external-only fields. The sidecar port is
never exposed through a client Service or assigned a NodePort.
CredentialsSpec
credentials is required. The operator manages MySQL users on the primary — creating them during first boot via an init script and updating passwords via ALTER USER when the referenced Secrets change.
| Field | Type | Required | Description |
|---|---|---|---|
operatorSecretName | string | Yes | Secret for operator and sidecar connections. Required keys: username, password, MYSQL_ROOT_PASSWORD. Optional: MYSQL_REPLICATION_USER and MYSQL_REPLICATION_PASSWORD, set together (with neither, the operator credentials replicate; a dedicated account is created on first boot with replication and clone grants). Grants: ALL PRIVILEGES WITH GRANT OPTION. |
appSecretName | string | No | Secret for application read-write connections. Keys: username, password. Grants on *.*: data and schema privileges (DML, DDL, views, routines, triggers, events, temporary tables, LOCK TABLES, SHOW DATABASES). No GRANT OPTION, SUPER, CREATE USER, or other administrative privileges. |
readOnlySecretName | string | No | Secret for application read-only connections. Keys: username, password. Grants: SELECT, SHOW VIEW, SHOW DATABASES, PROCESS. |
monitorSecretName | string | No | Secret for Prometheus exporter connections. Keys: username, password. Grants: PROCESS, REPLICATION CLIENT, SELECT on performance_schema. |
backupSecretName | string | No | Secret for backup connections (restore Jobs run as the operator account). Keys: username, password. Grants: SELECT, LOCK TABLES, SHOW VIEW, EVENT, TRIGGER, RELOAD, BACKUP_ADMIN, REPLICATION CLIENT. |
Secret references follow one convention across the five kinds: a field
that names a whole Secret ends in SecretName and is a string; a field
that selects one key of a Secret ends in SecretKeyRef and is a
{name, key} object. Every Secret name is a DNS subdomain of at most 253
characters, and every key is at most 253 characters of [-._a-zA-Z0-9].
BackupSpec
Top-level backup configuration, embedded as spec.backup. Omitting
the field disables backups for this failover group.
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
image | string | No | container-registry.oracle.com/mysql/community-server:9.7 | Image containing mysqlsh used for dump / load. |
imagePullSecrets | LocalObjectReference | No | -- | Pull secrets for the backup image. A list-map keyed by name, at most 16 entries. |
profiles | BackupProfile | No | -- | Reusable named backup configurations referenced by schedules and MysqlBackup CRs. A list-map keyed by name (x-kubernetes-list-type: map): names must be unique, and server-side apply merges entries by name, so two field managers can each own a profile. At most 32 entries. |
schedules | BackupSchedule | No | -- | Cron-driven recurring backups; each references a profile by name. Each entry becomes a Kubernetes CronJob owned by this failover group. A list-map keyed by name (names at most 63 characters), at most 32 entries. Admission refuses a profileName that names no entry of profiles ("each spec.backup.schedules.profileName must name an entry of spec.backup.profiles"), so a profile cannot be removed while a schedule names it. |
maxLagSecondsForSource | int | No | 300 | Threshold above which the backup reconciler falls back from replica-first source selection to the primary. 0 uses a replica only when it reports zero lag. |
resources | ResourceRequirements | No | -- | Resources for the backup Job's mysqlsh container. |
activeDeadlineSeconds | int | No | 7200 | Wall-clock cap on a single backup Job. |
backoffLimit | int | No | 2 | Job-level backoffLimit applied to backup Jobs. 0 runs each Job's pod once. |
retry | BackupRetrySpec | No | -- | Operator-level retries for Failed scheduled MysqlBackup CRs: maxAttempts (default 1, 1 to 10, including the original), initialBackoff (default 1m, 1s to 1h) and maxBackoff (default 30m, 1s to 24h, at least initialBackoff). Independent of Job backoffLimit. |
pitr | PITRSpec | No | -- | Continuous binlog archival for point-in-time recovery. |
podSecurityContext | PodSecurityContext | No | -- | Overrides default hardened pod security context merged onto the defaults. |
containerSecurityContext | SecurityContext | No | -- | Overrides default hardened container security context. |
stagingVolumeSizeLimit | Quantity | No | -- | Cap on the plaintext-staging emptyDir used by backup / restore / verify Jobs. Unset falls through to the node's ephemeral-storage limit; set on shared clusters to keep a large dump from triggering DiskPressure. |
BackupProfile (spec.backup.profiles[])
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
name | string | Yes | -- | Profile identifier, [a-z0-9]([-a-z0-9]*[a-z0-9])?, at most 63 characters. |
storage | BackupStorage | Yes | -- | Tagged union: type: S3 + s3 block, or type: PVC + pvc block. s3.prefix (here, in initFromBackup/restoreInPlace S3 sources and in a standby's source) is at most 512 characters; empty segments (a//b) and . or .. segments are refused at admission, and a leading or trailing / is allowed and ignored. Backups land under <prefix>/<mysqlbackup-name>/ and the PITR archive under <prefix>/binlogs/. |
dump | DumpOptions | No | -- | util.dumpInstance() tuning (threads, chunk size, compression, schema filters, consistent locking, OCIMDS checks). bytesPerChunk (default 64M) is a byte count with an optional k, K, M or G suffix, the suffixes mysqlsh reads: 512k is admitted, 64m is refused. |
retention | BackupRetention | No | {} | Which of this profile's MysqlBackup CRs are kept: keepLast, keepWithinDays, keepFailed. See BackupRetention. |
verification | VerificationSpec | No | -- | Opt-in periodic restore-of-this-profile's-latest-backup into a throwaway MySQL instance; renders a CronJob when enabled: true. Its retention (keepLast, default 30; keepFailed, default 10; 0 keeps none, deleting each run two minutes after the operator records its verdict) caps the profile's Succeeded and Failed MysqlBackupVerification CRs. Admission requires verification.schedule (a non-empty cron expression of at most 256 characters) when enabled is true, and a verification.pointInTime.archiveProfileName must name an entry of profiles. |
encryption | BackupEncryptionSpec | No | -- | Enables client-side envelope encryption (AES-256-GCM + HKDF-SHA256) for every dump file and, when this profile is also used for PITR, every archived binlog. The passphrase is yours: the operator never generates it. It snapshots the bytes into immutable operator-made Secrets (one per accepted PITR archive key, one per backup), so pods never read your Secret directly. Changing the encryption or the bytes of a profile whose archive is in use is refused (at admission while PITR is enabled, otherwise on the PITRArchiveBindingReady condition); each backup's dumps use that backup's own snapshot. See Backup encryption. |
PVC storage. pvc sets exactly one of claimName (an existing claim)
and provisioned: {storageClassName?, size} (a claim the operator creates
and owns, mysql-<group>-backup-<profile>; an unset class uses the cluster
default, and a set one is a DNS-1123 subdomain). provisioned cannot change,
be added or be removed while the profile stays in spec.backup.profiles,
also not by switching storage.type to S3 and back. A backup
whose owned claim has another class or a smaller request fails
BackupStorageMismatch; see PVC backups.
Renamed: pvc.storageClassName and pvc.size moved under
pvc.provisioned, and the 10Gi default for an unset size is gone.
Bounds. s3.bucket is at most 63 characters, s3.region at most 64,
s3.endpointURL and s3.prefix at most 512, pvc.claimName at most 253 and
pvc.subPath at most 512. They are bounded in UTF-8 bytes as well as characters (CEL
bytes(self).size()), at 63, 512, 512, 253 and 512 bytes, with messages such
as "bucket must be at most 63 bytes (UTF-8)" and "prefix must be at most 512
bytes (UTF-8)". They keep the admission rule on the MysqlFailoverGroup root
within the API server's CEL cost budget.
BackupEncryptionSpec
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
algorithm | string | No | AES-256-GCM | Encryption algorithm. Only AES-256-GCM is currently supported. |
passphraseSecretKeyRef | PassphraseSecretKeyRef | Yes | -- | Reference to a Secret in the same namespace holding the passphrase. See PassphraseSecretKeyRef. |
PassphraseSecretKeyRef
Reused by BackupEncryptionSpec, BackupDecryptionSpec,
spec.initFromBackup.decryption, spec.restoreInPlace.decryption, and the
MysqlStandbyCluster source.decryption; the bounds below apply to all of
them.
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
name | string | Yes | -- | Name of the Secret, a DNS subdomain of at most 253 characters. |
key | string | No | passphrase | Key within the Secret that holds the passphrase bytes, at most 253 characters of [-._a-zA-Z0-9]. Leading and trailing \n, \r, space and tab are stripped when the passphrase is read; other whitespace is part of the key. |
For a backup profile, the operator snapshots the passphrase (see Key snapshots); a byte change on a profile whose PITR archive is in use is refused, and each backup's dumps use that backup's own dump-key snapshot.
BackupDecryptionSpec
Used on spec.initFromBackup.decryption and
spec.restoreInPlace.decryption to supply the passphrase that decrypts
an encrypted backup. When the source is a backupRef and the field is
omitted, the operator uses the backup's own dump-key snapshot recorded in
status.artifactStorage.passphraseSecretKeyRef; it never falls back to the
profile's live Secret (PITRBackupDumpKeyMissing when neither exists).
| Field | Type | Required | Description |
|---|---|---|---|
passphraseSecretKeyRef | PassphraseSecretKeyRef | Yes | Secret holding the passphrase used at backup time. Rotating to a new value does not re-wrap existing ciphertext — it will simply fail to decrypt. |
BackupRetention
A successful MysqlBackup is kept when any rule keeps it: it is one of
the newest keepLast, or it completed within keepWithinDays. It is
deleted only when it is outside both. Failed CRs are counted separately.
A backup that a verification may be reading is kept until the run stops.
See Backup and restore → Retention,
which also maps the released retention: N and retentionPolicy fields.
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
keepLast | int | No | 7 | The newest this-many successful CRs are always kept. At least 1, so retention never deletes the last good backup. |
keepWithinDays | int | No | -- | Also keep every successful CR that completed within this many days. At least 1; unset keeps by count only. |
keepFailed | int | No | 10 | The newest this-many Failed CRs are kept. 0 keeps none, deleting each two minutes after the operator records its verdict. A Failed CR that is still the latest of a schedule in spec.backup.schedules is kept until a newer CR of that schedule exists, whatever this says: spec.backup.retry retries it and status.backupSchedules reports it. |
PITRSpec (spec.backup.pitr)
Enables continuous binary-log archival so restores can target an
arbitrary stopTime on top of any retained full dump. See
Backup and restore → Point-in-time recovery
for the end-to-end story.
| Field | Type | Required | Description |
|---|---|---|---|
enabled | bool | No | Turns archival on. Omitting the whole block is equivalent to false. |
profileName | string | Yes when enabled | Name of a spec.backup.profiles[] entry, a DNS label of at most 63 characters ([a-z0-9]([-a-z0-9]*[a-z0-9])?). Admission refuses enabled: true without a profileName that names an existing profile. Binlog objects and per-site manifests live under a binlogs/ subprefix inside that profile's storage. |
maxBinlogSize | string | No | Passed to MySQL as max_binlog_size. Controls rotation cadence (and therefore the RPO gap). Default "100M". A byte count with an optional k, m or g suffix in either case, as mysqld reads it (512m and 512M are both 512 MiB), at most 20 characters; anything else is refused. |
archivePollInterval | duration | No | Belt-and-suspenders scan cadence alongside inotify. Default "60s", 5s to 1h. |
The archive under <prefix>/binlogs/ carries a write-once descriptor
(archive.json) naming this group's UID, an archive ID derived from that UID
and the normalized destination, and the encryption mode. Manifests are
version 2 and name the archive ID. Writers need a store that enforces atomic
create. Prefixes written before v1.7.0 are refused (ArchiveLegacy): use a
new prefix and take a full backup. See
Archive identity.
The operator always owns replica-preserve-commit-order (ON, for every
group, whether or not PITR is enabled). While PITR is enabled, replication or
binlog filters, binlog_format other than ROW
or binlog_row_image other than FULL in spec.mysqlConf are refused: they
are left out of the rendered my.cnf and reported by the condition
PITRMysqlConfAccepted=False / PITRUnsafeSettingRefused and a Warning
PITRMysqlConfRefused. PITRArchiveReady=False / ArchiveUnsafeHistory
comes only from a sidecar's history-integrity marker, that is, from a change
made outside the spec.
Encryption change rule. A validation rule on the MysqlFailoverGroup root
refuses an update that, with pitr.enabled true in both the old and new object
and the same pitr.profileName, changes that profile's encryption (its
presence, algorithm, passphraseSecretKeyRef.name or key) while its storage
destination (type, bucket, prefix, endpoint, claim, sub-path) stays the same:
spec.backup.profiles[pitr.profileName].encryption (presence, algorithm, passphraseSecretKeyRef) cannot change while that profile archives PITR binlogs to the same storage; create a new profile with a new storage prefix and its own passphrase, switch spec.backup.pitr.profileName to it, wait until its archive binding is Bound, then take a full backup (see Backup encryption > Rotation)
An omitted algorithm equals AES-256-GCM and an omitted key equals
passphrase. The rule compares references, so renaming the Secret to a copy
with the same bytes is refused too. Edits with PITR disabled, across a
pitr.profileName change, or together with a destination change pass
admission and are judged by the group's
archive encryption binding.
InitFromBackupSpec (spec.initFromBackup)
One-shot restore-on-first-boot. The operator runs a restore Job against
the initial primary site before normal bootstrap is considered
complete. Once status.restore.phase == Succeeded, subsequent
reconciles skip the restore even if this field is still set.
The block is fixed at creation: admission refuses adding it to an
existing group, removing it, or changing any field in it. A bootstrap that
failed on a cause inside the block is retried by deleting the
never-bootstrapped group and creating it again (see
A failed bootstrap restore).
Every string and list in it is bounded (a pvc source's claimName at most
253 characters and subPath at most 512, s3.region at most 64) so the
rule stays within the CEL cost budget.
| Field | Type | Required | Description |
|---|---|---|---|
source | RestoreSource | Yes | Tagged reference: exactly one of backupRef, s3, or pvc. |
loadOptions | LoadOptions | No | util.loadDump() tuning (threads, skipBinlog, loadIndexes, schema filters). Restore Jobs disable load-progress tracking, so every load starts clean. |
pointInTime | PointInTimeSpec | No | Post-load binlog replay to a specific target timestamp. Requires the source group to have had spec.backup.pitr.enabled=true at backup time. A validation rule requires pointInTime.archiveProfileName whenever pointInTime is set, and when the group is created it must name an entry of spec.backup.profiles. |
decryption | BackupDecryptionSpec | No | Passphrase to decrypt the source backup's dump. When omitted for a backupRef, the operator uses the backup's recorded dump-key snapshot (status.artifactStorage.passphraseSecretKeyRef), never the profile's live Secret. Replayed binlogs are decrypted with the archive's read-binding key snapshot, not with this Secret. |
RestoreInPlaceSpec (spec.restoreInPlace)
Re-triggerable destructive restore against the currently-active
primary. Unlike initFromBackup (one-shot, greenfield), this field is
meant to be edited repeatedly: bumping confirm to a newer RFC 3339
timestamp re-arms another restore. See RestoreInPlaceSpec in the
source for full semantics (full-instance vs. per-schema restores).
| Field | Type | Required | Description |
|---|---|---|---|
confirm | string | Yes | Anti-fat-finger token. Must be an RFC 3339 timestamp strictly greater than status.restoreInPlace.confirmTokenUsed. Admission requires an upper-case T and Z, an offset within -23:59..+23:59 and a digits-only fraction, at most 64 characters (2026-04-17t14:32:00Z and +25:00 are refused). It is stored as written: the status tokens compare it byte for byte. |
source | RestoreSource | Yes | Tagged reference: one of backupRef, s3, or pvc. |
loadOptions | LoadOptions | No | util.loadDump() tuning. For per-schema restores, skipBinlog is force-set to false so the DROP+load replicates. |
decryption | BackupDecryptionSpec | No | Same semantics as InitFromBackupSpec.Decryption. |
PointInTimeSpec (spec.initFromBackup.pointInTime)
Requests binlog replay on top of the loaded dump. Requires the source
failover group to have had spec.backup.pitr.enabled=true at the
time of the full dump — otherwise there is no archive to replay.
| Field | Type | Required | Description |
|---|---|---|---|
stopTime | timestamp | Yes | The target instant: RFC 3339 with an explicit zone and whole seconds (2026-04-15T09:30:00Z, 2026-04-15T09:30:00+02:00). Pattern ^[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}(Z|[+-]([01][0-9]|2[0-3]):[0-5][0-9])$: a missing zone, a space instead of T, a lowercase t or z, a fractional second, or an offset outside -23:59..+23:59 is rejected at admission, so every stored value is one the operator's typed client can read. Commit times are compared to the microsecond. Replay restores the history of the first archived transaction committed at or after it, cut just before that transaction. |
archiveProfileName | string | Yes under initFromBackup | spec.backup.profiles entry whose archive the restore reads. DNS label, 1–63 characters (^[a-z0-9]([-a-z0-9]*[a-z0-9])?$). Required by a validation rule on initFromBackup ("pointInTime.archiveProfileName is required: name the backup profile whose PITR archive the restore reads"). It may not resolve to this group's own active PITR destination; destinations are compared normalized, so aliases are refused too, with RestoreBuildFailed / PITRArchiveDestinationShared before any Job is created. Replay considers every site manifest under that archive. |
allowPartial | bool | No | Default false. Accept an archive with no transaction at or after stopTime: the restore then replays to the archive head (status.restore.provenance.pointInTime.outcome: Partial, with a Warning Event PITRReplayTruncated). It relaxes nothing else; archive identity, lineage and continuity checks still apply. |
excludeGtids | string | No | GTIDs that are never replayed, at most 131072 characters, in MySQL's GTID set syntax (tags, line breaks and upper-case UUIDs are accepted; the Job receives the set normalized). Every entry must start with a server UUID, and the canonical set (tags spelled out per tag) must fit 64 KiB, since it reaches the Job as one environment string; a set that does not parse, an entry without a UUID, or a longer canonical set fails at reconcile with PITRTargetInvalid. They must not intersect the backup (PITRExcludeIntersectsBackup). Every archived transaction committed between the backup's newest transaction and the target that is outside the target history must be listed (PITRArchiveDivergentHistory prints that set). Excluding a transaction on the target history skips its GTID, but later row images may still carry its effects; it is reported as onHistoryExclusions. See Target history and excludeGtids. |
sites and allowUnverifiedCrossSite were removed in v1.7.0; replay
considers every manifest in the archive. See
Restore to a target timestamp.
Reconcile-time refusals set status.restore.reason and emit
RestoreBuildFailed before any Job is created:
| Token | When |
|---|---|
PITRTargetInvalid | stopTime is missing or has a fractional second (an object built outside admission), archiveProfileName is empty or names no profile in spec.backup.profiles, or excludeGtids does not parse, has an entry that does not start with a server UUID, or exceeds 64 KiB canonical. |
PITRBackupIdentityUnknown | The referenced MysqlBackup records no source failover group UID (status.failoverGroupUID is empty: a backup from before v1.7.0, which has no backup stamp), or a direct S3 or PVC source sits at the root of its store (no prefix or subPath), where no stamp can identify it. Take a new backup, or name the dump's directory in the source: spec.initFromBackup is fixed at creation, so the group is deleted and created again with it (see A failed bootstrap restore). |
PITRBackupFiltered | loadOptions.includeSchemas or excludeSchemas is combined with pointInTime. |
PITRArchiveDestinationShared | archiveProfileName resolves to this group's own active PITR destination. |
PITRBackupAfterTarget is not checked at reconcile: status.completionTime
includes the upload and would refuse valid targets. dump-meta and the
planner check the target against the dump's own end and the dumped server's
clock. A logged load (loadOptions.skipBinlog: false) is also refused. The
refusals dump-meta, pitr-download and restore.py can report are listed
in Restore refusals.
PITR completion status (status.restore)
| Field | Type | Description |
|---|---|---|
jobUID | string | Producing Job identity; a recreated Job with the same name never inherits the previous outcome. |
reason | string | The failure token (the first word of the failing step's termination message, for example PITRArchiveDivergentHistory; the replay script's BLOODRAVEN_PITR_FAILED is served as PITRReplayFailed). Only PITR tokens. Unset on success, a partial one included: pointInTime.outcome: Partial records that. |
provenance | RestoreProvenance | What the restore loaded and what the target held afterwards: source{sizeBytes, gtidExecuted, binlogPosition{file, position}} (what the referenced MysqlBackup recorded; absent when the source was not a MysqlBackup), target{gtidExecuted, binlogPosition{file, position}} (the target after the load and any replay) and pointInTime. |
provenance.pointInTime | PointInTimeResult | The certified outcome of the replay, read from the replay's verdict. Unset without pointInTime. |
status.restoreInPlace.provenance has the same type; in-place restores
have no pointInTime (see Point-in-time recovery is not supported in place),
so its pointInTime stays unset. The flat sourceSizeBytes,
sourceGtidExecuted, sourceBinlogFile, sourceBinlogPos,
targetGtidExecuted, targetBinlogFile and targetBinlogPos fields and the
top-level pointInTime are now under provenance. The flat pitrStopDatetime,
pitrReplayedBinlogFile, pitrReplayedBinlogCount, pitrNewestEventTime,
pitrPartial, pitrReplaySites and pitrCrossSiteUnverified fields of
status.restore and status.restoreInPlace were removed in v1.7.0.
PointInTimeResult
One shape for MysqlFailoverGroup.status.restore.provenance.pointInTime,
status.restoreInPlace.provenance.pointInTime and MysqlBackupVerification.status.pointInTime.
It is filled only
from a verdict the operator accepted (see
How the operator reads the outcome).
| Field | Type | Description |
|---|---|---|
mode | string | Timestamp or Latest. Required. |
stopTime | timestamp | The requested target, a whole second (commit times are compared against it to the microsecond). Unset in Latest mode. |
outcome | string | TargetCovered (an archived transaction at or after the target exists; the restore is its history cut just before it), Partial (none exists and allowPartial accepted the archive head) or ArchiveHead (latest mode). Required. |
archive | PITRArchiveIdentity | The archive the replay read: archiveID, destination, source{namespace, name, uid}, encryptionMode and algorithm. |
replaySites | string | Sites whose archived binlogs were staged, sorted. At most 16. |
replaySitesOmitted | int32 | Further staged sites the verdict's bounded site list left out; replaySites is complete only when this is zero or unset. |
segmentCount | int32 | Number of replay segments. |
replayedBinlogCount | int32 | Number of distinct archived binlogs the segments read. |
lastReplayedBinlog | {site, name} | The file of the last segment. |
replayedTransactionCount | int64 | Transactions replayed on top of the backup. |
newestReplayedCommitTime | MicroTime | Newest original commit time among the replayed transactions. Never the requested target. |
excludedDivergentTransactionCount | int64 | |D|: transactions committed between the backup's newest transaction and the target, outside the target history, that excludeGtids excluded. |
onHistoryExclusions | string | Excluded transactions on the target history that precede a replayed one. Their GTIDs were skipped, but later row images may carry their effects; the result is not certified for them. At most 2048 characters; empty when the set is longer (the count still counts it). |
onHistoryExclusionCount | int64 | Number of GTIDs in onHistoryExclusions. |
gtidExecuted | string | The target's @@global.gtid_executed after replay, which the replay proved equal to the target history minus excludeGtids. Empty when longer than 2048 characters. |
gtidExecutedSHA256 | string | Hex SHA-256 of gtidExecuted as the server printed it (newlines removed); set even when gtidExecuted is too long to record. |
PointInTimeVerificationSpec (MysqlBackupVerification.spec.pointInTime)
Also used by spec.backup.profiles[].verification.pointInTime for schedules.
On a MysqlBackupVerification, failoverGroupRef, profileName,
backupRef, pointInTime and sanityCheck are immutable (create a new
verification instead); keepOnFailure, ttlSecondsAfterFinished and
resources stay mutable. Its name is at most 63 characters, and
sanityCheck.query at most 4096.
Verification has no partial override. sites was removed in v1.7.0;
verification reads every manifest in the selected PITR archive.
| Field | Type | Default | Description |
|---|---|---|---|
mode | string | -- | Latest or Timestamp, required when pointInTime is set. Omit pointInTime for a verification without replay. Latest replays to the unique archive head and requires archived transactions beyond the loaded dump (PITRNoArchivedReplay otherwise). |
stopTime | timestamp | -- | Required for Timestamp and forbidden for Latest (a validation rule: "pointInTime.stopTime is required when mode is Timestamp and forbidden when mode is Latest"). Same form and pattern as PointInTimeSpec.stopTime (RFC 3339, an explicit zone, whole seconds). The archive must hold a transaction at or after it (PITRTargetNotCovered otherwise). |
archiveProfileName | string | spec.backup.pitr.profileName | The backup profile whose PITR archive the replay reads, a DNS label of 1 to 63 characters. Use it to verify history in a retained archive after a prefix switch. The replay mounts that archive's binding key snapshot. In spec.backup.profiles[].verification.pointInTime admission requires it to name an entry of spec.backup.profiles; on a MysqlBackupVerification the group is another object, so an unknown name fails the run when its Job is built. |
excludeGtids | string | -- | Same meaning and checks as PointInTimeSpec.excludeGtids (entries start with a server UUID; at most 64 KiB canonical), at most 131072 characters: list the archived transactions committed between the backup's newest transaction and the target (or the archive head) that are outside the target history. |
A Succeeded PITR verification records status.pointInTime
(PointInTimeResult): lastReplayedBinlog.name is the
last segment's file and newestReplayedCommitTime the newest replayed commit
time. status.replayedThroughBinlog was removed; read those two fields
instead. A latest-mode verdict that reports divergent transactions is refused
PITRCoverageInvalid. A verification without
pointInTime.archiveProfileName replays the group's own active archive, so it
needs spec.backup.pitr.enabled (PITRTargetInvalid otherwise: "... so group
<g> has no active PITR archive to replay; set pointInTime.archiveProfileName
to the profile of the archive to replay (a retained archive needs no writer),
enable spec.backup.pitr, or verify without pointInTime"); naming
archiveProfileName replays a retained archive and needs no writer. When building its Job is refused with a PITR token
(PITRTargetInvalid, PITRBackupIdentityUnknown, …), the token is the
Verified condition's reason, not BuildJobFailed.
DragonflySpec (spec.dragonfly)
Optional cache/session sidekick management. Omitting this field or setting
enabled: false preserves MySQL-only behavior and removes previously
managed Dragonfly resources.
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
enabled | bool | No | false | Creates and manages one Dragonfly StatefulSet per site when true. |
image | string | Yes when enabled | -- | Pinned Dragonfly image. :latest is rejected. Supported pin is Dragonfly v1.38.0+. Bloodraven reports whether the running image advertises REPLTAKEOVER on status.dragonfly.replTakeoverSupported; it does not admit or deny based on the tag. How a change reaches the sites is upgradeStrategy's choice; the operator never writes this field. |
upgradeStrategy | string | No | Rolling | Rolling rolls an image change like any other StatefulSet template change (replicas first, then the active site after a promotion). SnapshotRestore runs the snapshot-restore upgrade whenever a site's live StatefulSet runs an image other than image, and the ordinary render then never moves an image itself. Requires snapshot.dir (admission refuses SnapshotRestore without it, including snapshot: {}). Switching back to Rolling while a change is pending rolls it the ordinary way. |
port | int | No | 6379 | Redis-compatible client port. |
adminPort | int | No | 9999 | Admin/control port used by the operator. |
maxMemory | Quantity | No | -- | Memory budget per pod (for example 512Mi), rendered as Dragonfly --maxmemory=<bytes>. Must be greater than 0. Omitted leaves Dragonfly's default. |
proactorThreads | int | No | -- | Renders Dragonfly --proactor_threads=<value>. At least 1; omitted leaves Dragonfly's default (one per CPU). |
args | string | No | -- | Extra Dragonfly args, in order (an atomic list), at most 64 of at most 256 characters. Admission refuses the flags the operator owns, in every spelling Dragonfly reads (-flag, --flag, --flag=value, --noflag): --port, --admin_port, --bind, --dir, --maxmemory, --proactor_threads, --replicaof, --requirepass, --masterauth, --break_replication_on_master_restart, --s3_endpoint, --s3_use_https, --s3_sign_payload, and --flagfile, --fromenv and --tryfromenv, which read flags from elsewhere. Set the rendered values through the typed fields. |
auth | DragonflyAuthSpec | No | -- | Secret reference for the Dragonfly password. The operator sets --requirepass and --masterauth through the DFLY_requirepass/DFLY_masterauth env vars (never argv) and authenticates its own connections. |
resources | ResourceRequirements | No | -- | Compute resources for each Dragonfly container. |
plannedFailover | DragonflyPlannedFailoverSpec | No | {maxSyncWait: 30s, onSyncTimeout: Proceed} | Planned-failover sync and timeout behavior. |
snapshot | DragonflySnapshotSpec | No | -- | Native Dragonfly snapshot directory and optional S3-compatible settings, used by the SnapshotRestore upgrade strategy. |
DragonflyAuthSpec
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
passwordSecretKeyRef | {name, key} | Yes | -- | The Secret (name, required) and data key (key, default password) holding the Dragonfly password. |
DragonflyPlannedFailoverSpec
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
maxSyncWait | duration | No | 30s | 1s to 10m. Maximum time to wait for target Dragonfly replica catch-up. Also used as the REPLTAKEOVER timeout, which Dragonfly takes in whole seconds: it is rounded up to the next second. |
onSyncTimeout | string | No | Proceed | Proceed continues MySQL promotion and records status.plannedFailover.dragonfly.sessions: Lost; Fail rolls back before MySQL promotion. |
DragonflySnapshotSpec
| Field | Type | Required | Description |
|---|---|---|---|
dir | string | No | Passed to Dragonfly as --dir. Use s3://bucket[/prefix] for S3-compatible snapshot/restore support. |
serviceAccountName | string | No | ServiceAccount assigned to Dragonfly pods for cloud IAM access. |
credentialsSecretName | string | No | Secret projected as AWS credential environment variables for S3-compatible snapshot access. |
s3EndpointURL | string | No | S3-compatible endpoint as an http:// or https:// URL with a host (an IPv6 address in brackets) and an optional port from 1 to 65535, and no path, credentials, query or fragment (for example http://rustfs.default.svc.cluster.local:9000), at most 512 bytes. The operator passes the host and port as --s3_endpoint and the scheme as --s3_use_https (false for http://, true for https://). Omitted or empty means AWS S3 with Dragonfly's defaults. |
s3SignPayload | bool | No | Passed as --s3_sign_payload when set. |
EncryptionAtRestSpec (spec.encryptionAtRest)
Requires spec.tls: MySQL mandates a secure connection to clone
encrypted data and Bloodraven bootstraps every replica with CLONE INSTANCE. The CRD rejects enabled: true without it.
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
enabled | bool | Yes | false | Turn on data-at-rest encryption for every site. On a group that is already serving, existingData decides. Once true, it cannot be set back to false or removed, even if the adoption was refused and nothing was encrypted. |
existingData | string | No | Refuse | What happens when encryption is turned on for a group that is already serving, whose existing tablespaces MySQL does not rewrite. Refuse: encryption does not start; EncryptionAtRestReady is False with reason AdoptionRefused and a Warning EncryptionAdoptionRefused points at the supported path, a new encrypted group restored from a backup. LeavePlaintext: encryption starts and covers new data; the existing tablespaces stay plaintext until they are rebuilt, and status.encryptionAtRest.sites[].coverage.unencryptedTablespaces reports how much is still exposed. Read only when encryption is first turned on. |
coverage | EncryptionCoverageSpec | No | all true | Which encryption settings the operator enforces. See EncryptionCoverageSpec. |
keyring | KeyringSpec | No | -- | Keyring component paths and escrow lifecycle. See KeyringSpec. |
EncryptionCoverageSpec
Every field defaults to true. These are operator-owned once encryption
is enabled: a spec.mysqlConf entry naming the same option is ignored,
so a stray override cannot silently downgrade a site. Turning one off
narrows the security claim.
| Field | Type | Default | MySQL setting |
|---|---|---|---|
tables | bool | true | default_table_encryption. Applies to schemas and tables created afterwards; does not rewrite existing data. |
privilegeCheck | bool | true | table_encryption_privilege_check. Stops application users opting individual tables out of encryption. |
redoLog | bool | true | innodb_redo_log_encrypt |
undoLog | bool | true | innodb_undo_log_encrypt |
binaryLog | bool | true | binlog_encryption (covers relay logs on replicas too) |
systemTablespace | bool | true | ALTER TABLESPACE mysql ENCRYPTION='Y', run once by the sidecar on the writable site. Not covered by default_table_encryption; without it the data dictionary (schema, table, and column names) stays readable on a stolen PVC. |
KeyringSpec
mysqldDir and pluginDir are image-specific. The defaults match the
official mysql:9.x images — MySQL only reads its global component
manifest from the directory holding mysqld, and its global component
config from plugin_dir, so a wrong value shows up as InnoDB refusing to
start. The three directory fields must be absolute paths made of
A-Z a-z 0-9 . _ / - (no trailing /), because the operator embeds them in
the mysqld launcher script.
| Field | Type | Default | Description |
|---|---|---|---|
dataFileDir | string | /run/mysql-keyring | Keyring mount point. It must be outside the actual MySQL datadir. Admission rejects /var/lib/mysql and descendants; custom-image datadirs require manual verification. At most 1024 characters; immutable while encryption is enabled. |
mysqldDir | string | /usr/sbin | Directory containing the mysqld binary; mysqld.my is subPath-mounted here. At most 1024 characters; immutable while encryption is enabled. |
pluginDir | string | /usr/lib64/mysql/plugin | MySQL plugin_dir; component_keyring_file.cnf is subPath-mounted here. At most 1024 characters; immutable while encryption is enabled. |
retainVersions | int | 5 | Superseded escrow Secret versions kept per site (2--50). The version a site is currently sealed against is never pruned. |
escrowTimeout | duration | 10m | How long before a site that has not escrowed its keyring is reported Failed. 30s to 1h. The site stays unsealed and the operator keeps retrying either way. |
Full example
apiVersion: shipstream.io/v1alpha1
kind: MysqlFailoverGroup
metadata:
name: orders
spec:
image: mysql:9.7
sites:
- name: iad
serverID: 101
zone: us-east-1a
taintNodeSelector:
bloodraven.shipstream.io/failover-group.orders: "true"
bloodraven.shipstream.io/site.orders: iad
loadBalancerIP: 10.0.1.1
storage:
storageClassName: fast-ssd
size: 100Gi
resources:
requests:
cpu: "2"
memory: "16Gi"
limits:
cpu: "4"
memory: "16Gi"
- name: pdx
serverID: 102
zone: us-west-2a
taintNodeSelector:
bloodraven.shipstream.io/failover-group.orders: "true"
bloodraven.shipstream.io/site.orders: pdx
loadBalancerIP: 10.0.2.1
storage:
storageClassName: fast-ssd
size: 100Gi
resources:
requests:
cpu: "2"
memory: "16Gi"
limits:
cpu: "4"
memory: "16Gi"
- name: reader
serverID: 103
role: ReadOnly
zone: us-east-1b
mysqlConf:
innodb_buffer_pool_size: "4G"
serviceTemplate:
type: LoadBalancer
externalTrafficPolicy: Local
nodePort: 32006
annotations:
service.beta.kubernetes.io/aws-load-balancer-internal: "true"
storage:
storageClassName: fast-ssd
size: 100Gi
credentials:
operatorSecretName: mysql-operator-creds
sidecar:
resources:
requests:
cpu: "100m"
memory: "64Mi"
limits:
cpu: "200m"
memory: "128Mi"
dns:
hostname: orders.az.example.com
ttlSeconds: 60
tls:
secretName: mysql-tls
pollInterval: 2s
failureThreshold: 3
recoveryThreshold: 2
failoverCooldown: 5m
mysqlConf:
innodb_buffer_pool_size: "8G"
max_connections: "500"
innodb_flush_log_at_trx_commit: "1"
sync_binlog: "1"
replication:
maxLagSeconds: 300
readOnlyMaxLagSeconds: 30
updateStrategy:
type: OrderedUpdate
dragonfly:
enabled: true
image: docker.dragonflydb.io/dragonflydb/dragonfly:v1.38.0
plannedFailover:
maxSyncWait: 30s
onSyncTimeout: Proceed
cloneTimeout: 1h
podLabels:
cost-center: platform
podAnnotations:
prometheus.io/scrape: "true"
prometheus.io/port: "9104"
serviceTemplate:
type: LoadBalancer
externalTrafficPolicy: Local
annotations:
service.beta.kubernetes.io/aws-load-balancer-internal: "true"
extraContainers:
- name: mysqld-exporter
image: prom/mysqld-exporter:latest
env:
- name: DATA_SOURCE_NAME
valueFrom:
secretKeyRef:
name: mysql-monitor-creds
key: dsn
ports:
- containerPort: 9104
Full example (every credential role)
apiVersion: shipstream.io/v1alpha1
kind: MysqlFailoverGroup
metadata:
name: orders
spec:
image: mysql:9.7
sites:
- name: iad
serverID: 101
zone: us-east-1a
taintNodeSelector:
bloodraven.shipstream.io/failover-group.orders: "true"
bloodraven.shipstream.io/site.orders: iad
loadBalancerIP: 10.0.1.1
storage:
storageClassName: fast-ssd
size: 100Gi
- name: pdx
serverID: 102
zone: us-west-2a
taintNodeSelector:
bloodraven.shipstream.io/failover-group.orders: "true"
bloodraven.shipstream.io/site.orders: pdx
loadBalancerIP: 10.0.2.1
storage:
storageClassName: fast-ssd
size: 100Gi
credentials:
operatorSecretName: mysql-operator-creds
appSecretName: mysql-app-creds
monitorSecretName: mysql-monitor-creds
backupSecretName: mysql-backup-creds
dns:
hostname: orders.az.example.com
ttlSeconds: 60
Each referenced Secret must contain username and password keys. The operator secret additionally requires MYSQL_ROOT_PASSWORD:
apiVersion: v1
kind: Secret
metadata:
name: mysql-operator-creds
stringData:
username: bloodraven
password: <operator-password>
MYSQL_ROOT_PASSWORD: <root-password>
---
apiVersion: v1
kind: Secret
metadata:
name: mysql-app-creds
stringData:
username: app
password: <app-password>
Status
The operator writes the following status fields.
Top-level status
| Field | Type | Description |
|---|---|---|
observedGeneration | int64 | The metadata.generation the topology manager has applied: the generation its configuration was built from, or a newer one whose edit left that configuration unchanged (a backup schedule, a pod label). It never moves backwards. A spec edit is taken up when this reaches metadata.generation; the topology conditions carry the same value (see Conditions). status.encryptionAtRest.observedGeneration is the encryption reconcile's own and can differ. |
activeSite | string | Name of the site currently serving as primary |
topologyGeneration | int64 | Durable monotonic generation incremented on authoritative active-site changes, including bootstrap completion and planned, emergency, ordered-update, and restore promotions. Rehydrated after restart; changes revoke deployment leases. Not the CR's metadata.generation. |
sites | SiteStatus | Per-site status, a list-map keyed by name |
conditions | Condition | Standard Kubernetes conditions, a list-map keyed by type |
lastFailover | FailoverRecord | The most recent failover: time (when it occurred), target (the site promoted) and promotionGtidExecuted (the GTID set recorded on the candidate at the moment of the promotion, before it began accepting writes; for a planned failover, the fenced set plannedFailover.fence.gtidSet). time and target drive the anti-flap cooldown; promotionGtidExecuted is used for data-loss accounting and by the primary re-assert. Mirrored out of band in the bloodraven.shipstream.io/last-failover, last-failover-target and promotion-gtid-executed annotations. Replaces the separate lastFailover timestamp, lastFailoverTarget and promotionGtidExecuted fields. |
updatePhase | string | Current phase of an ordered update (UpdateReplica, WaitReplica, Failover, UpdateOldPrimary, WaitOldPrimary); absent when no ordered update runs. |
restore | RestoreStatus | In-flight or completed initFromBackup restore (phase, JobName, targetSite, startTime, completionTime, provenance, message). See PITR completion status. |
restoreInPlace | RestoreInPlaceStatus | In-flight or completed in-place restore (phase, JobName, jobCreating (set before the restore Job is created; a run with it set may have executed), targetSite, scope ({type: Full} for the whole instance, or {type: Schema, schema: <name>}), confirmTokenUsed, runConfirm and runID (the confirm and identity of the run in flight; empty once terminal), startTime, completionTime, provenance (source and target, as on status.restore; never pointInTime), message). Mirrors the state machine described on RestoreInPlaceSpec. Once a run is accepted this status drives it: removing spec.restoreInPlace mid-run keeps the fence until the Job finishes, and the terminal status (with confirmTokenUsed, the last confirm that ran) is kept. See Removing or editing spec.restoreInPlace mid-run. |
restoreOfflineFence | RestoreOfflineFenceStatus | The offline_mode fence a full-instance in-place restore holds on its target primary: site, confirmToken, runID (the status.restoreInPlace.runID of the run that owns it), priorOfflineMode (recorded before the operator changes it, restored afterwards), sessionsEvicted. Cleared once the restore has finished and the prior value is back. See Application sessions during a full-instance restore. |
restoreReclones | RestoreReclonesStatus | Peers a full-instance in-place restore still has to reclone from the restored primary: confirmToken (the owning restore), phase (Armed while the restore runs, Draining after it succeeded, Held when a restore ran its Job on the primary, failed and changed it, so nothing is cloned or replicated into the sites until an administrator sets Draining again or removes the queue), fencedGtidExecuted (the primary's gtid_executed when the peers' replication receivers were stopped; a failed restore that left it unchanged discards the queue instead of holding it), pending (sites still to reclone, in order), requested / requestTime (the site whose reclone is in progress, and since when), failures (latest failed attempt per site, a list-map keyed by site; a failed site is retried after a minute while the others go first), superseded (an earlier restore's queue this one replaced: put back if this restore fails, draining again only if this restore left the primary unchanged, otherwise Held together with this restore's peers; held marks a queue that was already held). Pending sites are not promoted automatically, are removed from the -replica and read-only Services, and their replication receivers are stopped (only their reclone restarts replication). Cleared once every peer is recloned; when the restore fails, discarded if the primary is unchanged and Held otherwise. See Peer replication during a full-instance restore. See Peer reclones after a full-instance restore. |
backupSchedules | BackupScheduleStatus | Per-schedule rollup, a list-map keyed by name (cronJobName, lastScheduleTime, lastSuccessfulTime, lastBackupName, lastBackupPhase, lastSuccessfulBackupName, lastRetryAttempt, nextRetryTime). |
lastBackupTime | timestamp | Completion time of the most recent successful MysqlBackup across all profiles, regardless of whether it was scheduled or on-demand. |
pitrArchiveBindings | PITRArchiveBinding | The encryption identities accepted for the PITR archive destinations the group writes or reads; at most 16, keyed by id. Written by the group reconciler. See PITRArchiveBinding. |
pitr | PITRStatus | Summary of the continuous binlog archive, populated by the operator from periodic polls of each sidecar's /archiver/status endpoint. Present when spec.backup.pitr.enabled=true. message lists per-site problems as site: error. While a sidecar does not answer, that site's last reported coverage is used and the message says the coverage is last-known for it; if a site has not answered at all since the operator started, the file count, bytes and oldest time keep their previously recorded values, newestArchivedTime and lastArchivedTime still advance with the sites that answer, and the message names the silent site. Whether archiving is running now is the PITRArchiveReady condition. archive is the archive identity the sidecars observed and identifies the archive the counters belong to; see PITRArchiveObservation. Whether PITR is enabled and which profile it uses are not echoed here: read spec.backup.pitr.enabled and spec.backup.pitr.profileName. |
plannedFailover | PlannedFailoverStatus | The most recent planned (admin-triggered) failover run. Retained until a newer annotation replaces a terminal block; how kubectl describe tells the story of a switchover after the fact. See PlannedFailoverStatus. |
plannedFailoverHold | PlannedFailoverHold | The automation hold left by a planned failover that may have written to its target and ended ManualRecoveryRequired: runID (the value the acknowledgement must name), reason (the run's result.reason), sourceSite, targetSite, ambiguousSites (the run's target and source, plus every site the failing operation found writable or could not read; a hold rebuilt from the terminal marker by an operator process that did not see the run lists every site in spec.sites, target and source first) and holdTime. It outlives the run's plannedFailover block (a refused request can replace that block) and is removed when the hold is acknowledged. The AutomaticFailoverHeld condition summarises it. |
dragonfly | DragonflyStatus | Observed Dragonfly subsystem state when spec.dragonfly.enabled=true (whether it is enabled is not echoed here; read the spec): active site, phase, per-site roles, reachability, replication health, last promotion, REPLTAKEOVER capability probe, and snapshot-upgrade status. |
encryptionAtRest | EncryptionAtRestStatus | Observed data-at-rest encryption state when spec.encryptionAtRest.enabled=true: per-site keyring phase, escrow Secret/version/digest, and observed coverage. Whether every site is sealed is the EncryptionAtRestReady condition. See EncryptionAtRestStatus. |
PITRArchiveObservation
status.pitr.archive is written by the topology runner from the sidecars' /archiver/statusarchive blocks. Only reports for the group's active PITR destination
count: a pod still on an older render reports another destination, or none.
| Field | Type | Description |
|---|---|---|
destination | string | Normalized active destination, s3://<bucket>/<prefix>/binlogs?endpoint=<endpoint> or pvc://<namespace>/<claim>/<subPath>/binlogs. At most 4096 characters. |
archiveID | string | The verified archive ID (ar1- plus 26 base32 characters); empty until a descriptor is verified. |
encryptionMode | string | None or Passphrase. |
state | string | AwaitingDescriptor (no sidecar has verified a descriptor yet: no writer has created it, or a replica waits for the writer), Verified (every site verified one archive ID) or Refused (any site refused; two different verified IDs at one destination also give Refused, with no reason). |
reason | string | Set when Refused: ArchiveForeign, ArchiveCopy, ArchiveLegacy, ArchiveDescriptorMissing, ArchiveEncryptionMismatch, ArchiveKeyMismatch, ArchiveVersionUnsupported, ArchiveStoreUnsupported or ArchiveUnsafeHistory, the first in that order that any site reports. A site with a history-integrity marker counts as ArchiveUnsafeHistory. |
keyRevision | string | The key revision the counted sites report (BLOODRAVEN_PITR_KEY_REVISION): the binding's key snapshot name, or plaintext; kept only when every counted site agrees. Empty for a sidecar on an older render, which never proves a binding. |
observedTime | MicroTime | When the observation last changed. A change of state, ID, reason or key revision is written at once. |
See Archive identity for what each reason means.
PITRArchiveBinding
status.pitrArchiveBindings[]. See
Archive encryption bindings.
Encrypted bindings that were accepted are rebuilt from their key snapshot after
a status loss; plaintext bindings are proven again.
| Field | Type | Description |
|---|---|---|
id | string | pab- and 16 hex digits of a SHA-256 over the destination (^pab-[0-9a-f]{16}$). List-map key and release subject. |
destination | string | Normalized destination (s3://… or pvc://…), 1 to 4096 characters. |
archiveID | string | The descriptor's archive ID once verified or created; empty or ar1- plus 26 base32 characters. |
source | {namespace, name, uid} | The group that created the archive, from a verified descriptor or a refusal that names it (ArchiveForeign). |
encryption | {mode, algorithm, keySecretName} | mode None or Passphrase; algorithm empty for None; keySecretName the immutable key snapshot (mysql-<group>-pitr-key-<h10>, empty for None). The accepted identity when state is Bound, otherwise the candidate; read it only when state is Bound. |
state | string | Pending, Bound or Refused. |
reason | string | Pending: Authenticating, Establishing. Bound: Bound, KeyMaterialLost. Refused: KeyMismatch, ArchiveForeign, ArchiveCopy, ArchiveLegacy, ArchiveDescriptorMissing, ArchiveNotFound, ArchiveEncryptionMismatch, ArchiveVersionUnsupported. A check that could not list the prefix stays Pending / Authenticating with an ArchiveListFailed: ... message and is retried every minute; ArchiveListFailed is still an accepted reason for a refusal stored by an older operator, which is retried. |
message | string | At most 2048 characters. |
check | {jobName, jobUID} | The pitr-archive-check Job authenticating encryption, while one runs. Only the Job with that UID is believed. |
lastTransitionTime | timestamp | Last state or reason change. |
SiteStatus
| Field | Type | Description |
|---|---|---|
name | string | Site identifier |
state | string | Current state: Writable, ReadOnly, Unreachable, or Unknown. |
lastSeenTime | timestamp | Last time the operator successfully polled this site |
replication | string | NotApplicable on the writable primary. Healthy once both replication threads have run with a source set for consecutive polls; Lagging when, in addition, secondsBehindSource exceeds spec.replication.maxLagSeconds (the same threshold as Degraded=True / ReplicationLagging). A Lagging site reads Healthy again only once its lag is a fifth of the threshold (at least one second; no band at maxLagSeconds: 0) below it, so lag that hovers around the threshold does not flip the state every poll; Degraded uses the threshold itself, so it can clear while the site still reads Lagging. Broken when replica status was read and a thread is stopped or no source is set. Unknown when replica status could not be read, the site has not been probed yet, or the healthy streak is still building. A follower is replicating when this is Healthy or Lagging; that pair replaces the former boolean replicating: true. |
secondsBehindSource | int | Replication lag in seconds (only set when replicating) |
gtidExecuted | string | GTID set executed on this site |
sourceHost | string | Source hostname reported by MySQL. Followers are expected to name the active primary's internal site Service. |
sourceConvergenceState | string | Converged, Pending, or Blocked. Generic direct-source state for every follower; independent of old-primary recovery state. |
sourceConvergenceReason | string | Stable reason: DirectSource, SourceMismatch, ProbeFailed, MutationFailed, GTIDDiverged, or RestorePeerHold (the site awaits its post-restore reclone and its replication is held stopped). Detailed errors remain in structured logs. |
recoveryState | string | Recovery state: absent (no recovery needed), RecoveryInProgress (the operator is reconfiguring or waiting for an old primary to stabilize as a replica), or RecoveryBlocked (the site holds transactions the primary lacks: it is not promoted by an ordered update or planned failover and ranks last for emergency failover; reclone it or make the primary contain the set). A replica can be RecoveryBlocked while replicating normally. |
divergentGtid | string | GTID set of transactions on this site that the primary named by divergenceComparedTo lacks. Populated when recoveryState is RecoveryBlocked. |
divergentTransactionCount | int | Number of divergent transactions. Populated when recoveryState is RecoveryBlocked. |
divergenceDetectedBy | string | OldPrimaryRecovery (a returning old primary was compared with the new primary) or ReplicaAudit (the replica GTID audit found the set on a follower). Populated when recoveryState is RecoveryBlocked. |
divergenceComparedTo | string | Site whose GTID set divergentGtid was computed against: the primary at the time of the comparison. Populated when recoveryState is RecoveryBlocked. |
PlannedFailoverStatus
Every field belongs to the run named by runID. The operator writes the block with a status update conditioned on the object's resourceVersion, one write per transition, and refuses a non-terminal block without a runID (a block an older operator wrote). The CRD schema marks no field of the block required (an older operator rolled back onto this CRD must still be able to write its own block); the fields described as "always set by the operator" are set on every block this operator writes, and their minimum length or value still applies when present. See Planned failover for the lifecycle.
| Field | Type | Description |
|---|---|---|
runID | string | Always set by the operator. Identifies the run; minted when a request is first stamped (Admitting or Deferred). It is the token of the run's claim and the value the hold acknowledgement must name. |
revision | int64 | Always set by the operator, ≥ 1. Increments on every write the run makes; settlement of an ambiguous write identifies a committed write by (runID, revision). |
phase | string | Always set by the operator. Admitting, Deferred, Fencing, Draining, WaitingForLag, WaitingForDragonflySync, PromotingDragonfly, Promoting, Publishing, RollingBack, Succeeded or Failed. |
targetSite | string | Always set by the operator. The site the admin asked to promote. |
settings | object | Always set by the operator. maxLagWait, drainTimeout and, when Dragonfly was enabled at acceptance, dragonfly.{maxSyncWait, onSyncTimeout}: the effective values snapshotted at acceptance from the annotation override and the spec. |
startTime | timestamp | Always set by the operator. When the request was first accepted. |
sourceSite | string | The site the topology manager named the unique writable primary when it granted the claim. Empty until Fencing. |
claimTime | timestamp | When the claim was granted. |
claimBaseline | FailoverRecordRef | The durable failover record at the grant (the newer of status and the out-of-band record) as {time, target}, time at second precision. A re-proof after a restart compares both by value. |
admission | object | deadlineTime (an Admitting run with no claim ends NotStarted{AdmissionTimeout} then), waitReason (why the claim is not granted yet), deferReason (CooldownActive or DeploymentHold) and retryTime (when a Deferred run retries). |
fence | object | Write-ahead record of the fence on the source: state (Intended, Applied), intentTime, sourceServerUUID (the fence is sent only to this server, and a re-run attributes super_read_only=ON to the run only while it matches), applyTime, gtidSet (the fenced set; "" is valid; also the promotion watermark). |
drain | object | startTime, completionTime, remainingConnections: the application sessions still connected to the source after the final kill pass (sessions already marked killed, server daemons and replication workers are not counted), not the number killed. Unset when a drain query failed on a source that still answered (the count is unknown). |
lagWait | object | startTime, catchUpTime. |
dragonfly | object | Present when Dragonfly was enabled at acceptance: syncWaitStartTime, sourceOffsetState (Captured, Unavailable), sourceOffset, syncWaitSeconds, takeover (state Intended/Succeeded/Failed, intentTime, completionTime, method ReplTakeover/ReplicaOfNoOne/Skipped, of which a planned run writes only ReplTakeover, targetOffset), sessions (Preserved, Lost), reason (DragonflySyncTimeout, DragonflyPromotionFailed), message. |
promotion | object | Write-ahead record of the promotion: state (Intended, Confirmed), intentTime, targetServerUUID (read on a pinned session before the first statement), confirmationTime. |
dns | object | state (Applied, Failed, Skipped), attemptTime, lastError. |
rollback | object | Write-ahead record of a rollback: reason, intentTime. |
result | object | Set when the run is terminal: outcome (Succeeded, NotStarted, RolledBack, SourceLeftFenced, PromotedUnconfirmed, ManualRecoveryRequired), reason, completionTime, warnings. See Outcomes and reasons. |
message | string | Human-readable status line. |
Removed in this release (an older operator wrote them): target (now targetSite), sourcePrimary (now sourceSite), sourceGtidAtFence (now fence.gtidSet), targetGtidAtPromotion (dropped: the gate proves the target held exactly the fenced set), completionTime and reason (now under result), durationSeconds (derive it from startTime and result.completionTime; the duration histogram is unchanged), transactionsLost (dropped: always 0 by construction), maxLagWait and drainTimeout (now under settings), drainStartTime (now drain.startTime), lagWaitStartTime (now lagWait.startTime), retryAfter (now admission.retryTime), the phases Pending, Validating and Resuming, and the Dragonfly fields enabled, sourceOffsetAtDrain, offsetCaptureFailed, targetOffsetAtPromotion, sessionsPreserved (now sessions) and promotionMethod (now takeover.method).
Conditions
The topology conditions (Ready, Degraded, Bootstrapping, Updating,
RecoveryPending, DivergentPromotion) carry the generation the topology
manager applied, the same value as status.observedGeneration, so a spec edit
the manager has not taken up yet does not read as observed. A condition
another writer owns (S3CredentialsReady, PITRMysqlConfAccepted,
PITRArchiveReady, PITRArchiveBindingReady, the encryption conditions)
carries the generation that writer evaluated.
| Type | Meaning |
|---|---|
Ready | True with reason Healthy when there is exactly one confirmed writable PrimaryCandidate and every non-reader follower has healthy direct replication. Reader reachability, lag, threads, and source state do not change this group condition. False with reason NoWritablePrimary when no site is writable, or ReplicationUnhealthy when a site is writable but replication is not healthy. False with reason InitFromBackupInProgress, or InitFromBackupFailed after a failure, while a spec.initFromBackup restore has not succeeded. Unknown with reason Initializing right after an operator start, while no site is confirmed writable and a PrimaryCandidate has not been classified yet (see Initializing after an operator start). |
Degraded | False with reason Healthy when nothing is wrong. True with the most important problem, in this order: a cross-site alert (SplitBrain, NoPrimary, TotalLoss, or CrossSiteAlert for any other cross-site alert), BootstrapFailed, SiteUnhealthy (a non-reader site is unreachable, or a writable site that cannot be promoted needs fencing), then the per-site replication reasons: ReplicationBroken, ReplicationError, ReplicationLagging (lag exceeds maxLagSeconds) and ReplicationSourceMismatch. Reader failures are excluded and remain visible per site and in metrics. BootstrapFailed is recomputed on every poll: it holds while the target of the failed clone neither replicates (replication Healthy or Lagging) nor is the writable primary, and clears when a new clone starts or the site replicates by another path (once it has, a later break of that site reports its own reason, not the old clone failure); the deployment API refuses leases while it holds. Also True with reason DivergentTransactions whenever any site, readers included, is RecoveryBlocked; that reason outranks the replication reasons, and does not by itself make the deployment API refuse leases. True with reason RestoreReclonePending while status.restoreReclones.pending lists sites and nothing else is wrong: their replication is stopped until their post-restore reclone, and the deployment API refuses leases with the same reason. True with reason WorkloadSelectorMismatch when a site's Deployment or Dragonfly StatefulSet has a selector other than the one the operator renders (for example an object left by an earlier release with other label keys): the selector is immutable, so the operator does not update that workload and does not retry in a loop; a WorkloadSelectorMismatch Warning Event names the object, and deleting it lets the operator re-create it (a MySQL site keeps its PVC). True with reason MetadataVersionUnsupported when an internal annotation payload, the planned-failover terminal marker or a deployment lease record, has a schemaVersion this operator does not know, with a MetadataVersionUnsupported Warning Event: the payload proves nothing (lease requests are refused while a lease record is unreadable), except that a terminal marker naming a run with outcome ManualRecoveryRequired still holds automatic failover. Both outrank the per-site replication reasons and RestoreReclonePending, but not the cross-site alerts, BootstrapFailed, SiteUnhealthy or DivergentTransactions, and the deployment API reports the group unstable with that reason. Unknown with reason Initializing, alongside Ready, until the operator has classified the topology after a start; ReplicationBroken, ReplicationError, ReplicationLagging, DivergentTransactions, WorkloadSelectorMismatch, MetadataVersionUnsupported and RestoreReclonePending still set it True (the last comes from the persisted status.restoreReclones, not from a topology observation, so it is known during Initializing and keeps the deployment API refusing leases). |
DivergentPromotion | Set True (reason DivergentSitePromoted) when a site carrying a divergence report became the confirmed primary, so its extra transactions are now authoritative. A later divergent promotion is appended to the message (newest last, five newest kept). The operator never clears it; remove it from status after reviewing the transactions. |
AutomaticFailoverHeld | Summarises status.plannedFailoverHold. True (reason PlannedFailoverManualRecoveryRequired) after a planned failover that may have written to its target ended ManualRecoveryRequired; the message names the run, the writable or ambiguous sites and the acknowledgement, for people only (read the run from status.plannedFailoverHold.runID). False (reason PlannedFailoverHoldAcknowledged) once the hold was acknowledged; absent on a group that was never held. Its observedGeneration is the generation of the object the hold was read from and written to (the planned failover's terminal write, or the acknowledgement), not status.observedGeneration: the hold comes from durable run evidence, not from the configuration the topology manager applied. While it is True, emergency failover, split-brain resolution, the primary re-assert, ordered updates, keyring rotations, recovery and source convergence, bootstraps and DNS reconciliation stay held for the group. Cleared by the operator after the bloodraven.shipstream.io/acknowledge-planned-failover-hold=<runID> annotation (kubectl bloodraven ack-planned-failover-hold <group>); see the hold runbook. |
RecoveryPending | True while old-primary recovery is running (RecoveryInProgress) or when an old primary has divergent transactions (DivergentTransactions). Use the reclone annotation only for DivergentTransactions. |
Bootstrapping | True only while a clone runs (fresh-deploy, auto-clone, or reclone), with the phase as its reason: Cloning, WaitingForRestart, SetupReplication. A finished clone is False with reason NotBootstrapping and the message <Bootstrap|Reclone|Auto-clone> completed successfully; NotBootstrapping is also the state before any clone. A failed clone is False with reason BootstrapFailed, which also sets Degraded=True with reason BootstrapFailed while the failure stands; Bootstrapping keeps reporting the last clone's outcome until the next clone starts. To wait for a clone you requested, wait for False/NotBootstrapping with a lastTransitionTime after the request and the target site's replication Healthy (or Lagging). |
Updating | True while an ordered update runs; the reason is the phase (UpdateReplica, WaitReplica, Failover, UpdateOldPrimary, WaitOldPrimary), also in status.updatePhase. Still True, with reason OrderedUpdateStalled, while the update is stopped on a site whose pod is not Ready, or has been Ready for less than 30s; the message names the site, the pod and the cause, and the update retries every 30s. Otherwise False with reason NotUpdating. |
EncryptionAtRestReady | True with reason AllSitesSealed when every site is sealed against a verified escrowed keyring. This is the condition to alert on. False with reason KeyringNotSealed while sites are bootstrapping, cloning, or rotating; KeyringFailed when a site could not escrow or its escrow Secret is missing or corrupt; AdoptionRefused when encryption was enabled on a group that is already serving. |
PITRArchiveReady | Present only while spec.backup.pitr.enabled=true; removed when PITR is disabled. Derived from the operator's 30 s polls of every site's /archiver/status. True / Archiving: every site answered with an enabled archiver, a ready store and no error. False / ArchiveStoreInitFailed: a sidecar cannot build its archive store (for example an unreadable passphrase or credentials file, an invalid S3 endpoint, an empty PVC mount path, or a PVC mount path that is not a mount point or not writable); immediate. False / ArchiverError: a site with a ready store reported an error from 2 consecutive archiver scans (consecutiveFailedScans ≥ 2; from an older sidecar, an error from a later lastScanAt), or the same error for 10 minutes or three scan intervals (the current archivePollInterval), whichever is longer, without a scan completing. One failed scan never sets it, even though the default 60 s scan interval leaves it visible to two polls. False / ArchiverDisabled: a sidecar reports archiving disabled on 2 consecutive polls while the spec enables it and no ordered update is pending for that site; admission refuses an empty or unknown spec.backup.pitr.profileName, so this is a sidecar that does not pick up the render, or an object stored before that rule (see the BackupPITRInvalid Event). Unknown / AwaitingRollout: a sidecar reports archiving disabled while that site's ordered update is still pending (or, right after the topology manager starts, before it has computed which sites drift), the normal state right after enabling PITR; no Event. It applies only when the PITR spec renders an archiver (an empty or unknown profileName does not) and for at most 30 minutes per site; a rollout still pending after that is False / ArchiverDisabled with the message ordered update still pending after 30m. False / SidecarOutdated: an enabled archiver reports no archive-identity block, so it predates archive identity and its manifests are unusable for restores: a spec.sidecar.image pin older than v1.7.0, or, right after an upgrade to v1.7.0, old pods whose Deployment update is deferred until the archive is switched to a new prefix (see Upgrade policy; changing spec.sidecar.image does not end that); Unknown / AwaitingRollout while that site's ordered update is pending (at most 30 minutes), otherwise False on the second poll. Set spec.sidecar.image to this operator's release, or clear it, before archiving to a new prefix. False / SidecarUnreachable: a site's poll failed 3 times in a row over at least 60 s, or over at least 5 minutes for a site whose sidecar has not answered since this operator process started (new groups, bootstraps, a crash-looping older sidecar after an operator upgrade). Unknown / AwaitingFirstPoll: this operator process has not observed every site yet; a True from before an operator restart is not kept without a fresh observation, a False is, and a replacement topology manager continues its predecessor's observations. If the operator cannot read the group uncached when a manager starts, the condition is left as persisted until it can. A single missed poll or a single failed scan keeps the current status; returning from False to True needs 2 consecutive clean poll rounds. False / ArchiveForeign, ArchiveCopy, ArchiveLegacy, ArchiveDescriptorMissing, ArchiveEncryptionMismatch, ArchiveKeyMismatch, ArchiveVersionUnsupported, ArchiveStoreUnsupported or ArchiveUnsafeHistory: a sidecar refused the archive's identity, or set its permanent history-integrity marker (see Archive identity); immediate. ArchiveVersionUnsupported also covers a damaged archive.json (undecodable, trailing data, wrong kind, a malformed field, source names outside Kubernetes bounds, or larger than 64 KiB); its message then gives the repair (restore the original object, or point the profile at a new prefix) instead of an upgrade. ArchiveUnsafeHistory comes only from a sidecar's marker, with an edge-triggered Warning PITRArchiveUnsafeHistoryDetected in addition to PITRArchiveNotReady; the condition does not preview spec.mysqlConf, whose unsafe options are refused instead (see PITRMysqlConfAccepted). When sites disagree the reason is the first of ArchiveStoreInitFailed, the archive refusals in the order above, ArchiverError, SidecarOutdated, ArchiverDisabled, SidecarUnreachable that applies, and the message lists every affected site, sorted, as site: detail. A change in status, reason or the set of affected sites is written at once; a change confined to message text (for example a new S3 request ID) at most every 5 minutes. A Warning PITRArchiveNotReady Event ("PITRArchiveReady is False (False, and a Normal PITRArchiveRecovered Event ("PITRArchiveReady is True again: every site reports a running archiver with a ready store and no error") when it returns to True; reason or message changes while False fire none. Archiving state never affects pod readiness. |
PITRArchiveBindingReady | Present while the group has archive bindings or binding targets; owned by the reconciler, observedGeneration set. The first that applies: False / BindingLimitReached (a destination in use has no binding, and 16 entries that stay exist: Bound, or used by a target); False / a refused binding reason of an entry in use, or ArchiveForeign / ArchiveCopy when a Bound read binding is made the write target; False / EncryptionChangeRefused or KeyChangeRefused (a profile in archive use differs from its Bound binding; pods keep the binding); False / KeyMaterialLost (a Bound binding's key snapshot is missing, or is present but no longer passes the check of the accepted key; the message says which, and never advises deleting a snapshot that is present); Unknown / Authenticating or Establishing; True / Bound, with a note while no full backup is stamped with the write binding's archive ID. Refusal messages name the profiles (up to 8), the destination and the procedure. See Archive encryption bindings. |
PITRMysqlConfAccepted | Present only while spec.backup.pitr.enabled=true; removed when PITR is disabled. Owned by the reconciler. True / Accepted: no spec.mysqlConf option breaks PITR replay. False / PITRUnsafeSettingRefused: group or site mysqlConf sets a replication or binlog filter (replicate-do-*, replicate-ignore-*, replicate-wild-*, replicate-rewrite-db, binlog-do-db, binlog-ignore-db), binlog_format other than ROW or binlog_row_image other than FULL; the effective group-plus-site value is judged per site, so a group value every site overrides safely is not refused. Refused options are left out of the rendered my.cnf (MySQL's safe default applies). The message lists each as <site>: <field> sets <key>[=<value>] (<why>), and a Warning PITRMysqlConfRefused fires once per refused set. replica-preserve-commit-order is operator-owned and never reaches this check. |
S3CredentialsReady | Present only when the group references S3 storage (a backup profile, initFromBackup, or restoreInPlace). The initFromBackup Secret counts only until the bootstrap restore succeeds: nothing reads it after that, so it can be deleted then. True when every referenced credentialsSecretName exists and has non-empty AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. False with reason CredentialsIncomplete otherwise, together with an S3CredentialsIncomplete warning event; backup, restore, and cleanup Jobs fail in that state because they do not fall back to IRSA, Workload Identity, or instance-profile credentials. |
EncryptionAtRestStatus
| Field | Type | Description |
|---|---|---|
sites | SiteEncryptionStatus | Per-site keyring lifecycle. |
observedGeneration | int | CR generation the encryption subsystem last reconciled. |
SiteEncryptionStatus
| Field | Type | Description |
|---|---|---|
name | string | Site identifier |
phase | string | Pending, Unsealed, Escrowed, Sealed, or Failed. The normal sequence is Pending -> Unsealed -> Escrowed -> Sealed. Escrowed means the keyring is captured and verified and the pod is rolling onto the sealed rendering. |
unsealReason | string | Why the site is unsealed: Bootstrap, Clone, or Rotation. Absent when sealed. |
keyringSecret | string | Escrow Secret the site is sealed against (or seeded from while unsealed). |
keyringVersion | int | Escrow version. Monotonic per site; each version is an immutable Secret. |
keyringDigest | string | sha256:<hex> over the escrowed keyring bytes. The operator refuses to seal until the sidecar reports the same digest for the live keyring file. |
lastEscrowTime | timestamp | When the current version was accepted. |
unsealTime | timestamp | When the site entered Unsealed. Drives keyring.escrowTimeout. |
message | string | Human-readable explanation of the current phase. |
coverage | SiteEncryptionCoverage | What was actually observed on the live instance. |
SiteEncryptionCoverage
Sampled from the live instance, not inferred from my.cnf. A site can be
Sealed and still have incomplete coverage -- for example an adopted
cluster whose pre-existing tables were never rebuilt. Every boolean flag is
optional: it is absent when the probe did not report it, and an unknown value
is never reported as false.
| Field | Type | Description |
|---|---|---|
keyringComponent | string | Component_name from performance_schema.keyring_component_status. |
keyringReadOnly | bool | The component's Read_only status. Must be true in the Sealed phase. Absent when not reported. |
systemTablespaceEncrypted | bool | Whether the mysql tablespace is encrypted. Absent when not reported. |
unencryptedTablespaces | int | User tablespaces still reporting ENCRYPTION='N'. Non-zero means part of your data is in the clear. |
redoLogEncrypted | bool | @@innodb_redo_log_encrypt. Absent when not reported. |
undoLogEncrypted | bool | @@innodb_undo_log_encrypt. Absent when not reported. |
binlogEncrypted | bool | @@binlog_encryption. Absent when not reported. |
lastCheckTime | timestamp | When coverage was last sampled. |
DragonflyStatus
| Field | Type | Description |
|---|---|---|
activeSite | string | Site currently acting as Dragonfly master. Usually matches status.activeSite; may briefly diverge during planned failover. |
phase | string | Disabled, Reconciling, ConfiguringReplication, Ready, Degraded, or Promoting. A promotion that moves activeSite sets ConfiguringReplication until the next observation of the sites. |
message | string | Human-readable status summary. |
lastPromotionTime | timestamp | Last successful Dragonfly promotion time. |
lastPromotionTarget | string | Site promoted in the last successful Dragonfly promotion. |
replTakeoverSupported | bool | true when every successfully probed reachable site advertised REPLTAKEOVER. false when at least one probed site lacks it. Omitted until the first successful probe. |
replTakeoverProbeTime | timestamp | When replTakeoverSupported or replTakeoverProbeMessage last changed. |
replTakeoverProbeMessage | string | Explains a false result (which sites lacked the command). |
sites | DragonflySiteStatus | Per-site Dragonfly observation. |
image | string | The Dragonfly image in the active Dragonfly site's StatefulSet, as last observed. An observation only: the snapshot-restore upgrade compares spec.dragonfly.image with each site's live StatefulSet, never with this field, and sites can differ while a run is in flight or after one stopped part way. |
upgrade | DragonflyUpgradeStatus | The latest snapshot-restore upgrade run: phase (Pending, SavingSnapshot, UpdatingActive, WaitingForActiveRestore, ReattachingReplicas, Succeeded, Failed), sourceImage (the active site's live image when the run started, or for a run that started after the active site had moved, the first site still on another image), targetImage (spec.dragonfly.image when it started), specDigest (sha256:<hex> over spec.dragonfly when it started), activeSite, snapshotDir, startTime, snapshotTime, completionTime, reason and message. A Failed run is not restarted while spec.dragonfly still matches specDigest, and the record is removed once no site runs an image other than spec.dragonfly.image. |
DragonflySiteStatus
| Field | Type | Description |
|---|---|---|
name | string | Site identifier. |
role | string | Master, Replica, StaleMaster, Unconfigured, Unreachable, or Unknown. |
reachable | bool | Whether the operator completed an INFO replication call on the latest poll. |
serviceName | string | Site-local Dragonfly Service name. |
podName | string | Dragonfly pod name when known. |
replicationState | string | Raw Dragonfly role value from INFO replication. |
linkStatus | string | Replica master_link_status; empty on a master. |
syncInProgress | bool | Whether Dragonfly reports a full sync in progress (master_sync_in_progress). Unset when Dragonfly did not report it. |
lastIOSecondsAgo | int | Seconds since the replica last received data from its master; -1 means never. Unset when Dragonfly did not report it. |
ready | bool | Operator-level readiness for the Dragonfly role. |
message | string | Human-readable per-site status. |
Annotations
Well-known metadata is the complete list of the labels, annotations, finalizers and the taint Bloodraven reads or writes, with each key's writer, value grammar and stability level. Action annotations share one value grammar, <subject>(:<key>=<value>)*; a destructive action on a site carries confirm=<metadata.name>:uid=<metadata.uid>, where the uid is kubectl get mfg <group> -o jsonpath='{.metadata.uid}', so a request left over for an earlier group of the same name does not apply.
| Annotation | Description |
|---|---|
bloodraven.shipstream.io/last-failover | Operator-owned out-of-band copy of status.lastFailover.time, formatted as an RFC3339 UTC timestamp at second precision. Do not edit manually. Reset tooling clears it together with status history. |
bloodraven.shipstream.io/last-failover-target | Operator-owned out-of-band copy of status.lastFailover.target. It is always written and cleared as a pair with last-failover. Do not edit manually. |
bloodraven.shipstream.io/reclone-site | Triggers a CLONE INSTANCE from the current primary to the named site. CLONE INSTANCE wipes the site's datadir, so both forms carry the group's name and uid: <site>:confirm=<group-name>:uid=<group-uid> for a cold reclone (no divergent GTID recorded), and <site>:confirm=<group-name>:uid=<group-uid>:gtidPrefix=<prefix> when status.sites[].divergentGtid is non-empty. The prefix is at least 8 hex digits or - of the source UUID of the set's first entry (never its :interval part, since a value cannot contain :); it proves the requester read the divergence and does not replace confirm=. kubectl bloodraven reclone <group> <site> builds the value. Any other form, including a bare site or the positional <site>:<prefix> of earlier releases, is rejected with a message that prints the full form. Invalid annotations are rejected with a RecloneRejected Warning Event and cleared. Valid ones emit RecloneRequested and report progress via the Bootstrapping condition. The operator also writes this annotation itself to drain status.restoreReclones after a full-instance in-place restore; it never overwrites a value someone else set. |
bloodraven.shipstream.io/planned-failover | Starts a planned failover to the named site: <site>[:maxLagWait=<duration>]; the override takes the same 1s to 1h bound as spec.plannedFailover.maxLagWait. The grammar is <subject>(:<key>=<value>)*: values use only A-Za-z0-9._/+-, each key appears once, and an empty or unknown pair is rejected. The operator removes it once the run is accepted (or kept while Deferred). When Dragonfly is enabled, the planned-failover status includes a dragonfly substatus with session-preservation details. |
bloodraven.shipstream.io/acknowledge-planned-failover-hold | Set by an administrator to the held run's runID (status.plannedFailoverHold.runID), or with kubectl bloodraven ack-planned-failover-hold <group>, to acknowledge an automation hold (AutomaticFailoverHeld=True) after resolving the split brain. The operator clears the hold and then removes the annotation. A value naming another run, a value given when no hold stands, or one beside a claim held for another run resumes nothing and gets a PlannedFailoverHoldAcknowledgmentIgnored Warning Event; when no hold stands the annotation is removed anyway. |
bloodraven.shipstream.io/planned-failover-terminal | Operator-owned: a versioned JSON payload, {"schemaVersion":1,"runID":"<runID>","outcome":"<outcome>","reason":"<reason>","warnings":["<warning>"]} (reason and warnings may be omitted). warnings are the run's terminal warnings (API warning tokens); the terminal record a marker-ended run gets keeps them and adds StatusStalled, and a marker carrying FenceNotAttributable never names the source for re-assert, also after a restart. A value that is not this payload is ignored and logged; one with a schemaVersion this operator does not know is also reported as Degraded=True/MetadataVersionUnsupported with a MetadataVersionUnsupported Warning Event; a payload the strict parser rejects (an unknown field, a newer schemaVersion, an unfamiliar reason) that is still JSON naming a run with outcome ManualRecoveryRequired is treated as a hold, and the acknowledgement can delete it. Ends a run whose status writes stalled for 2 minutes; every planned-failover check treats the named run as terminal. Removed once the terminal status record lands, except that a ManualRecoveryRequired marker is a hold record and is removed only by the acknowledgement. Do not edit. |
bloodraven.shipstream.io/planned-failover-stall-since | Operator-owned: <runID>:since=<YYYYMMDDThhmmssZ>. The durable start of the status-stall deadline. Do not edit. |
bloodraven.shipstream.io/planned-failover-manager-absent-since | Operator-owned: <runID>:since=<YYYYMMDDThhmmssZ>. The durable start of the 5-minute manager-absence deadline. Do not edit. |
bloodraven.shipstream.io/planned-failover-confirmed | Operator-owned: <runID>:since=<YYYYMMDDThhmmssZ>, the time the run's promotion was confirmed. Written after the run's promotion is confirmed, recorded and published, and before its claim is released (if the write fails the claim is kept and retried). A restart before the Publishing write commits treats the run as recorded from it: no claim is hydrated and the run publishes without one, even if a later emergency failover replaced the run's failover record. Removed when the run ends; one a finished run left behind is removed by the next pass, as are its stall-since and manager-absent-since annotations. Do not edit. |
bloodraven.shipstream.io/planned-failover-status-fence | Operator-owned: an ISO 8601 basic UTC instant with nanoseconds (20261104T100000.123456789Z) bumped, conditioned on the resourceVersion an ambiguous status write was made against, to prove that write can never commit. Each bump moves the value past the stored one (1ns past it when the clock has not advanced), and it counts as proof only if it moved the object's resourceVersion. Do not edit. |
bloodraven.shipstream.io/promotion-gtid-executed | Operator-owned out-of-band copy of status.lastFailover.promotionGtidExecuted, written with last-failover and last-failover-target and conditioned on the object's resourceVersion. Omitted when the set exceeds 64 KiB. Counts only while promotion-gtid-executed-for names the event the other two keys record. Do not edit. Reset tooling clears it together with status history. |
bloodraven.shipstream.io/promotion-gtid-executed-for | Operator-owned: <RFC3339 last-failover> <last-failover-target>, binding promotion-gtid-executed to its failover event. Written and deleted in the same patch as promotion-gtid-executed. A watermark whose binding does not name the event that last-failover and last-failover-target record (a writer that knows only those two advanced them) is ignored. Do not edit. Reset tooling clears it together with status history. |
bloodraven.shipstream.io/rotate-keyring | Rotates the named site's InnoDB master key. Refused on the active primary -- rotation is the one operation whose failure window would cost data rather than a re-clone, so the supported procedure is to rotate replicas, run a planned failover, then rotate the ex-primary. Also refused while an ordered update or planned failover is in flight (decided against the operator's live state too, not only status: also refused while a planned failover or promotion is in flight, on the site the operator currently observes as primary, and until it has observed a unique writable primary with the target read-only). These refusals keep the annotation and retry every 10s; only a request for the active primary (by status or as observed) is removed. Cleared automatically once the requested rotation completes and the site is Sealed again (an escrow repair back to Sealed keeps it). |
bloodraven.shipstream.io/release-pitr-archive-binding | Releases one archive binding and deletes its key snapshots. Value <id>:confirm=<group-name>:uid=<group-uid>, one id per annotation. The id is derived from the destination, so it recurs in a re-created group of the same name; the uid keeps a stale request from releasing the new group's binding. Refused while the binding is in use (Warning PITRArchiveBindingReleaseRefused, annotation kept, retried every minute); a malformed value or a wrong confirm or uid is rejected (Warning PITRArchiveBindingReleaseRejected) and removed. Never commit it in GitOps. See Releasing a binding. |
PITR archive binding metadata
Labels and annotations the operator puts on objects it creates for archive encryption bindings. They are internal unless marked otherwise: select on them for listing, but do not set or edit them.
| Key | On | Value |
|---|---|---|
bloodraven.shipstream.io/pitr-render | Site Deployments | JSON of the resolved PITR render (schemaVersion: 1); the group's held identity. |
app.kubernetes.io/name (label) | Key snapshot Secrets | mysql-pitr-key (archive) or mysql-dump-key (per-backup dump key). |
bloodraven.shipstream.io/failover-group-uid (label) | Archive and dump key snapshots | Group UID; snapshots have no owner reference, so this scopes listing. |
bloodraven.shipstream.io/failover-group (label) | Archive and dump key snapshots | Group name. |
bloodraven.shipstream.io/pitr-binding-id (label) | Archive key snapshots | Binding id; the release selector. |
bloodraven.shipstream.io/pitr-destination | Archive key snapshots | Normalized destination. |
bloodraven.shipstream.io/pitr-accepted | Archive key snapshots | JSON {schemaVersion, destination, archiveID, source, mode, algorithm, keyCheck, acceptedAt}, written when the binding is accepted. Operator-owned, read-only; keyCheck is a passphrase verifier. |
bloodraven.shipstream.io/mysqlbackup-uid, bloodraven.shipstream.io/mysqlbackup (labels) | Dump key snapshots | The backup's UID and name. |
bloodraven.shipstream.io/pitr-key-secret (label) | Check, restore, verification Jobs | The key snapshot the Job mounts. |
bloodraven.shipstream.io/pitr-key-secret-uid | Check Jobs | The candidate key snapshot's UID when the Job was created. A Verified verdict for a snapshot of that name with another UID is dropped and the entry re-checked. |
verification-hold.bloodraven.shipstream.io/<verification UID> | Archive key snapshots | A running verification's hold; blocks release. |
bloodraven.shipstream.io/release-pitr-archive-binding | MysqlFailoverGroup | Public action: <id>:confirm=<group-name>:uid=<group-uid> (see Annotations). |
Example status
status:
observedGeneration: 4
activeSite: iad
sites:
- name: iad
state: Writable
lastSeenTime: "2025-01-01T00:00:00Z"
replication: NotApplicable
- name: pdx
state: ReadOnly
lastSeenTime: "2025-01-01T00:00:00Z"
replication: Healthy
secondsBehindSource: 0
gtidExecuted: "uuid:1-100"
sourceHost: mysql-orders-iad-internal.orders.svc.cluster.local
sourceConvergenceState: Converged
sourceConvergenceReason: DirectSource
- name: reader
state: ReadOnly
lastSeenTime: "2025-01-01T00:00:00Z"
replication: Healthy
secondsBehindSource: 0
gtidExecuted: "uuid:1-100"
sourceHost: mysql-orders-iad-internal.orders.svc.cluster.local
sourceConvergenceState: Converged
sourceConvergenceReason: DirectSource
conditions:
- type: Ready
status: "True"
reason: Healthy
observedGeneration: 4
- type: Degraded
status: "False"
reason: Healthy
observedGeneration: 4
lastFailover:
time: "2025-01-01T00:00:00Z"
target: iad
promotionGtidExecuted: "uuid:1-90"
dragonfly:
activeSite: iad
phase: Ready
message: all sites ready
sites:
- name: iad
role: Master
reachable: true
ready: true
- name: pdx
role: Replica
reachable: true
linkStatus: up
ready: true
MysqlBackup
One backup run. Created by hand or by a schedule CronJob. See
Backup and restore. The spec names
the failover group (spec.failoverGroupRef.name), the profile
(spec.profileName, a DNS label of at most 63 characters) and
optionally spec.sourceSite, a site of that group to take the dump from.
spec.deletionPolicy (Delete by default, or Retain) decides what deleting
the CR, by hand or by retention, does to the artifact: Delete removes the
artifact from storage and then the backup's dump-key snapshot Secret; Retain
runs no cleanup, keeps both, and records a Normal ArtifactRetained Event
naming them (delete them by hand when no longer needed). Retain does not
keep the storage itself: a claim the group provisioned (pvc.provisioned)
is deleted with the group, with every artifact on it (see
Deleting a group without losing backups).
It is read when the deletion is processed. A backup re-created with the name of a retained one
fails at its backup stamp step while the retained dump is still at that
location.
failoverGroupRef.name is a DNS subdomain of at most 253 characters, and
spec.sourceSite a site name (a DNS label of at most 20 characters).
failoverGroupRef, profileName and sourceSite are immutable: to back up
something else, create a new MysqlBackup. deletionPolicy stays mutable.
The backup's own name is at most 63 characters (it is a label value on the
objects the run creates). A name ending in .bloodraven-backup.json is
refused at admission: that suffix is the
backup stamp
object's.
Status
| Field | Type | Description |
|---|---|---|
phase | string | Pending, Running, Succeeded or Failed. |
startTime, completionTime | timestamp | When the backup Job started and reached a terminal state. |
jobName | string | The producing backup Job. |
sourceSite | string | The site the dump was taken from. |
artifactStorage | ArtifactStorage | Where the producing Job wrote the artifact: storage type, location, and the bucket, endpoint and region or the claim. Restores, verifications and artifact cleanup read the destination only from here. Immutable once location is set (see below). Replaces the former status.location and status.storageType. |
size, sizeBytes | string, int | Reported artifact size. |
gtidExecuted | string | @@global.gtid_executed at dump time. |
binlogPosition | {file, position} | Binary-log position at dump time. Unset when the dump did not report one. Replaces binlogFile and binlogPos. |
encryption | {mode, algorithm} | mode is Passphrase when the producing Job encrypted the artifact (the key is artifactStorage.passphraseSecretKeyRef) and None when it wrote plaintext; algorithm is the algorithm stamped on the artifact, empty for None. Written from the owned producing Job when the Job is created and derived again from it at completion; a correction emits a Warning EncryptionAttributionCorrected. Absent when no owned producing Job attributes it. Restores and verifications use it to decide whether to decrypt. Replaces encrypted and encryptionAlgorithm. |
failoverGroupUID | string | metadata.uid of the source failover group. Pinned on the backup Job as annotation bloodraven.shipstream.io/failover-group-uid when the Job is created, and recovered from the Job if the status write is lost. The backup stamp records the same UID. |
pitrArchiveID | string | The archive ID the group's rendered PITR writers used when the Job was created (^(ar1-[a-z2-7]{26})?$): the one every site Deployment's render agrees on. A backup of the active PITR profile waits Pending with reason PITRArchiveRolloutPending while a roll changes it (some renders decode and they disagree). Any other backup never waits: when the renders disagree or one is unreadable it proceeds with this field empty, as does a backup of the PITR profile while the render is deferred (any site Deployment lacks a decodable render while the write binding is not Bound/Establishing: the pre-v1.7.0 deferral, which does not wait for a roll); either way a Normal Event PITRArchiveStampSkipped says why, and the backup is no point-in-time baseline. Empty when PITR was off. Pinned as annotation bloodraven.shipstream.io/pitr-archive-id and recovered from the Job. |
attempt | int | Retry attempt for scheduled retries. |
cleanupStartTime | timestamp | When the deletion of a deletionPolicy: Delete backup committed to removing its artifact: recorded just before the first cleanup Job is created, and removed again only if that Job cannot be created (a quota, a webhook). Once it stays, a switch to Retain is not applied. See Keeping an artifact. |
conditions | Condition | See below. |
kubectl get mysqlbackup shows Group, Profile, Phase, Ready (the
Ready condition's status), Started, Completed and Age; -o wide adds
the Location (.status.artifactStorage.location) and Size columns.
Reasons from archive encryption bindings: Pending with
PITRArchiveRolloutPending (a backup of the active PITR profile while the
sites render different archive IDs),
PassphraseSecretMissing (the encrypted profile's Secret or key is missing,
so no dump-key snapshot can be made) or DumpKeySnapshotConflict (the
dump-key snapshot name is held by another backup's Secret); Failed with
ProfileReadOnly (the profile is
read-only).
ArtifactStorage (status.artifactStorage)
Recorded from the producing Job when it is created, or recovered from that Job
for backups taken by older operators. Restores, verifications, PITR dump
metadata and cleanup read the bucket, endpoint, region and claim from here and
never follow later edits of the profile's bucket, endpoint, prefix, subPath
or claim. A PVC backup therefore needs no profile to be restored or verified.
| Field | Type | Description |
|---|---|---|
type | string | S3 or PVC, the storage backend the producing Job wrote to. Required. |
location | string | The exact artifact location the producing Job wrote: a bucket-relative prefix for plain S3 (<prefix>/<name>), s3://<bucket>/<prefix>/<name>/ for encrypted S3, or /backups[/<subPath>]/<name> for PVC. |
bucket | string | S3 bucket. |
endpointURL | string | S3 endpoint override in effect for the Job; empty for AWS. |
region | string | S3 region in effect for the Job. Cleanup signs for us-east-1 when no region is set anywhere, as mysqlsh does. |
claimName | string | PersistentVolumeClaim (PVC storage). |
passphraseSecretKeyRef | {name, key} | For an encrypted backup, the backup's own immutable dump-key snapshot (mysqlbackup-<name>-dump-key-<uid prefix>, key passphrase) that the producing Job mounted. Restore and verification decrypt the dump with it, and deleting a backup that never succeeded reads its backup stamp with it; the profile's live Secret is used only when no Secret was recorded. The snapshot is deleted after the backup's artifacts are confirmed deleted. |
Credentials are deliberately not part of it: restores, verifications and
cleanup read them from the profile's current credentialsSecretName, so
rotating keys needs no change on old backups. Cleanup also deletes the backup stamp,
<location>.bloodraven-backup.json; see
Backup stamp.
Once artifactStorage.location is recorded, a validation rule on the MysqlBackup resource
rejects any update that changes or removes status.artifactStorage, including
removing or nulling the whole status through the status subresource. The
rule is evaluated on the resource root, so it cannot be bypassed by dropping
the parent object first.
Conditions
| Type | Meaning |
|---|---|
JobCreated | The producing Job was created. |
Ready | False from the first reconcile: reason Pending until a launch attempt names a specific reason, then the pending reason while the backup waits (WaitingForHealthySource, BackupJobConflict, PITRArchiveRolloutPending, DumpKeySnapshotConflict, PassphraseSecretMissing), and Running while the backup Job runs. True with reason Succeeded once the backup succeeded with a known artifact destination, or False with the failure reason: DestinationMismatch when the Job succeeded at a destination other than the recorded status.artifactStorage, BuildJobFailed also when a replacement for a lost Job would write elsewhere (see Where cleanup deletes). A wait longer than 30 minutes fails the backup: with reason NoHealthySource after waiting for a source, or PendingTimedOut after any other wait, with the message pending for <duration> (<pending reason>): <message>; a failed backup never carries a pending reason. A Warning Event named after the pending reason fires once on entering a wait (again if the reason changes), and a NoHealthySource or PendingTimedOut Event on the timeout. |
MetadataComplete | Optional completion metadata (size, GTID, binlog coordinates) was collected; DestinationUnavailable means the artifact location is unknown. |
ArtifactCleanup | Only on a Terminating backup: why its artifact has not been removed yet. See Artifact cleanup for the reasons. Before releasing an abandoned backup (one that never succeeded, or whose group or profile is gone) the operator stops its producing backup Job and waits for the Job's pods; a lost status update after the Job was created is recovered from the Job, running or failed. False / ProducerStopStuck, with a Warning ArtifactCleanupStalledProducerStop, when a producer pod has not stopped 10 minutes after the deletion: the deletion stays held, because the pod may still write to the destination. Once its node is confirmed down, kubectl delete pod <pod> --force --grace-period=0. |
Annotations
| Annotation | Meaning |
|---|---|
verification-hold.bloodraven.shipstream.io/<verification UID> | Value is the verification name. A MysqlBackupVerification is reading this backup; retention and artifact cleanup wait while it is live. See Backup verification. |
bloodraven.shipstream.io/triggered-by | What created the backup: manual (or the --triggered-by value of kubectl bloodraven backup), schedule/<schedule>, or retry/<schedule>/attempt-<n>. A MysqlBackupVerification carries it too (manual, or schedule for a scheduled run). Informational; the operator never reads it. Replaces spec.triggeredBy, removed in v1.7.0. |
MysqlStandbyCluster
Phase 1 of WISHLIST #7. A passive DR relationship descriptor that
lives on the DR cluster and continuously monitors a source
MysqlFailoverGroup's dumps and PITR archive in a shared object store. The
controller populates status.discovered and stamps readiness conditions
on a configurable cadence. No mysqld is started, no dump is loaded and
nothing is written to the bucket.
See Multi-cluster DR for the end-to-end recovery runbook and a full explanation of what the standby does and does not provide.
Top-level spec fields
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
transport | string | No | ObjectStore | Transport mode. ObjectStore is the only value. |
source | StandbySource | Yes | -- | The source group's S3 destination and optional decryption. |
freshness | StandbyFreshnessSpec | No | -- | Bucket-discovery cadence (discoveryInterval only). |
StandbySource
| Field | Type | Required | Description |
|---|---|---|---|
s3 | StandbyS3Source | Yes | The S3-compatible destination the source group's backup profile writes to. The controller only reads it. |
decryption | BackupDecryptionSpec | No | Passphrase Secret to decrypt source artifacts. Must exist in the DR namespace. |
The standby identifies its source by the prefix alone. The archive
descriptor under <prefix>/binlogs/ names the source group (namespace, name
and UID), and the standby publishes it in status.discovered.binlogArchive.identity.
Use metadata labels for a display name.
StandbyS3Source
The S3 fields of a backup profile's storage.s3 that apply to a reader,
with prefix required. The bounds are the profile's (see
BackupProfile).
| Field | Type | Required | Description |
|---|---|---|---|
bucket | string | Yes | S3 bucket name, 1-63 bytes. |
prefix | string | Yes | The source profile's prefix, at most 512 bytes, with at least one segment. A leading or trailing / is ignored; empty, . and .. segments are refused. The standby scans <prefix>/<dump>/ and <prefix>/binlogs/. |
region | string | No | AWS region. Optional for non-AWS S3 stores. |
endpointURL | string | No | Absolute http:// or https:// endpoint override without credentials, a query or a fragment, at most 512 bytes. |
credentialsSecretName | string | Yes | Secret in this namespace with AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY (and optionally AWS_SESSION_TOKEN, AWS_REGION) for read access. |
StandbyFreshnessSpec
| Field | Type | Default | Description |
|---|---|---|---|
discoveryInterval | duration | 5m | How often the controller re-scans the bucket to refresh status.discovered. 30s to 24h. |
Restore verification and activation are not implemented, so the API has no
fields for them: no verification cadence or staleness knobs, no activation
request and no template of the group an activation would create. Unknown
fields have no effect: the API server prunes them (they are silently dropped
and never persisted), and clients using strict server-side field validation —
the default for kubectl apply/create since Kubernetes 1.25 — reject them
outright.
Status
| Field | Type | Description |
|---|---|---|
discovered | StandbyDiscovered | Result of the most recent complete bucket scan; a failed or cut-short scan leaves it unchanged. See below. |
conditions | Condition | Standard Kubernetes conditions. See below. |
status.discovered:
| Field | Type | Description |
|---|---|---|
lastScanTime | time | When the most recent complete scan finished. |
latestDump | object | The newest dump under the prefix, chosen by the completion time in its @.json: name, location (the dump directory's object key in the bucket, with a trailing /), and, when the metadata records them, completionTime, gtidExecuted and sizeBytes. Absent when the prefix holds no dump. |
binlogArchive | object | The verified source PITR archive and the window its manifests cover, published together: identity (archiveID, destination, source{namespace, name, uid}, encryptionMode, algorithm, read from archive.json and checked against every manifest), manifestCount, fileCount, sizeBytes, oldestEventTime and newestEventTime. A site whose manifest carries a history-integrity marker counts in manifestCount but adds no files. Absent when the prefix holds no archive, or while the archive is refused (see SourceConfigKnown); latestDump is still published then. |
kubectl get msc shows Dump (the latest dump's name), NewestBinlog (the
newest archived event time), Ready (the Ready condition's status) and
Age.
Conditions
| Type | Meaning |
|---|---|
Ready | Summarises the two conditions below. True with reason SourceDiscovered when BucketReadable and SourceConfigKnown are both True; otherwise False with the reason and message of the first of the two that is not True (BucketReadable first). |
BucketReadable | Bucket scan succeeded within the last discovery interval. Reason: ListSucceeded (True), ListFailed (False), ScanIncomplete (False — the List call did not complete before the scan deadline; a context deadline/cancel during List, distinct from a genuine ListFailed. The previous status.discovered is preserved), AuthFailed (False — store construction failed, e.g. bad credentials), or ConfigError (False — missing/invalid spec fields). |
SourceConfigKnown | At least one full dump and one binlog manifest were found. Reason: DumpFound (True), NoDumpFound (False), NoBinlogManifests (False — dump found, no PITR window yet: a dump exists but no binlog manifests are under <prefix>/binlogs/, so recovery is limited to the dump. This is the expected state for a dump-only source or a brand-new source, not a misconfiguration), MetadataUnreadable (False — dump @.json present but genuinely malformed/cannot be parsed), ScanIncomplete (False — the scan was cut short by a context deadline/cancel before reading every dump @.json or all manifests, or a manifest was listed but is gone when read (message manifest … was listed but is gone; rescanning, for example after retention); the partial selection is not published, the previous status.discovered is preserved as last-known-good, and no BucketScanned event is emitted), ConfigError (False — propagated from storage backend error), or an archive refusal (False — the token without its PITR prefix: ArchiveLegacy, ArchiveDescriptorMissing, ArchiveIdentityMismatch (a manifest names another archive), ArchiveEncryptionMismatch, ArchiveKeyMismatch, ArchiveVersionUnsupported (also a damaged archive.json), ArchiveObjectInvalid, ArchiveListFailed (an S3 403: grant s3:ListBucket; other failures are retried), ArchiveHistoryReset (a site manifest records a reset epoch, so no replay from the archive is possible), ArchiveUnsafeHistory (every site's manifest carries a history-integrity marker), or ArchiveUnreadable (another error without a token; a manifest refusal keeps its own token, such as ArchiveLegacy, ArchiveVersionUnsupported or ArchiveObjectInvalid). A marked site among unmarked ones is left out of the published window. latestDump is still published without binlogArchive, and a Warning PITRArchiveRefused fires when the reason changes. The standby expects Passphrase / AES-256-GCM when source.decryption is set and None otherwise). |

