Known limitations

This page is the short version of Bloodraven's current boundaries. Read it before writing production manifests so the failure and recovery model matches your expectations.

API maturity

• The CRD API version is shipstream.io/v1alpha1.
• Fields may still change before v1beta1 / v1.
• There is not yet a published CRD conversion-webhook or migration contract. Track this under the CRD version-migration wishlist item.

Replication and RPO

• Bloodraven uses asynchronous MySQL replication. Emergency failover can lose transactions that committed on the old primary but had not reached the promoted replica.
• The operator records promotion and divergent GTID sets so data loss is observable, but it does not merge divergent data automatically.
• Planned failover is the zero-RPO path: it fences the source, waits for the target's GTID_EXECUTED to cover the fenced source GTID, then promotes only a target holding exactly that set. It is strict: every PrimaryCandidate and DisasterRecovery site must answer read-only, so a down DR site blocks planned failovers until it returns or is removed from spec.sites.
• Planned-failover exclusion holds within one operator leader. Across a leader handover it holds only as far as the Lease handover does; an old leader's SQL statement already on the wire is a residual shared by every topology actor (see Operator availability).
• If you require synchronous commit semantics or quorum-based zero RPO on primary loss, Bloodraven is the wrong tool; see Why not Group Replication?.
• Read-only sites are full MySQL replicas. Filtered or partial replication, replication proxying, and query routing through an operator-managed proxy are not implemented.
• Clone and reclone copy from the active primary at full speed. Bloodraven does not throttle clone traffic, so adding or rebuilding a large reader can consume primary network, storage, and CPU capacity. Schedule the operation and monitor the donor accordingly.

Operator availability

• Leader election is enabled by the chart, but the default deployment still runs one replica. Run more than one operator replica only after validating the deployment model in your cluster.
• Sidecars preserve safety while the operator is unavailable, but new failover decisions wait for an operator to run.
• If the primary fails while the operator is down, writes remain unavailable until the operator returns and completes failover.
• Anti-flap state (the failover time and target) is persisted twice on every promotion: in the status.lastFailover record (time, target), and out of band in the failover group's own annotations (bloodraven.shipstream.io/last-failover, bloodraven.shipstream.io/last-failover-target). The two travel different API paths — the status subresource has its own RBAC rule and admission chain — so an outage on one does not take the other with it. A restarting operator rehydrates from whichever copy is newer, and both writes retry every poll until accepted. Losing the cooldown across a restart therefore requires both paths to be rejecting writes at once; while that holds, a subsequent site failure inside the intended cooldown window can promote earlier than configured (deterministic-simulation finding, CooldownViolated(restart+stateLost) class). GTID gates still bound the damage — any divergence this causes is fenced and reported. Alert on mysqlfailovergroups and mysqlfailovergroups/status write failures (the operator logs and retries both every poll) to close the window. A restart that had to fall back to the annotations logs restored lastFailover from out-of-band annotations at WARN, which is a direct signal that this group's status writes were failing. This guarantee starts with operator builds that understand those annotations. Rolling back to an older build makes the annotation copy invisible again; do not rely on cooldown continuity across such a rollback while status writes are failing.

Placement and shared nodes

• Taints and node discovery are scoped per failover group with spec.sites[].taintNodeSelector, so one physical node can advertise membership in multiple failover groups at the same site.
• Application workloads on shared nodes must tolerate other groups' readonly taints but not their own group's taint.

Backups and restore

• Archives written before v1.7.0 (version 1 manifests, or a prefix with no archive.json descriptor) are refused for both writing and reading (PITRArchiveLegacy). Point spec.backup.pitr at a new prefix and take a full backup. See PITR upgrade.
• Encrypted backups taken before v1.7.0 recorded no dump key, so a MysqlBackupVerification of one fails PITRBackupDumpKeyMissing (there is no decryption override), and restoring one needs an explicit decryption. See Upgrade policy: v1.7.0.
• Backup and PITR support is present, including backup verification, but restore-duration metrics and restore-performance guidance are still missing.
• In-place restore exists for destructive rollback of a live group. Use it carefully: full-instance restore fences writes and, after loading the dump, reclones every peer one at a time. Until a peer is recloned it holds pre-restore data and is excluded from automatic promotion and the reader Services, so a primary failure in that window does not fail over to it automatically (status.restoreReclones). Its replication receiver is stopped from before the restore drops anything until it is recloned; a restore that fails after changing the primary leaves the peers Held with their pre-restore data until an administrator decides. A full-instance restore cannot start while a peer is unreachable, and a planned failover cannot promote a held peer. Standby clusters outside spec.sites that replicate from the primary are not held, so they receive the restore's DROP statements.
• In-place restore cannot recover to a point in time. A restoreInPlace that sets pointInTime is rejected before anything is fenced, because the live primary has already executed the archived transactions and the replay would be skipped. For PITR, restore into a new failover group with spec.initFromBackup.pointInTime. See Backup and restore.
• PVC loss can be recovered by recloning from the current primary, but committed transactions that only existed on the lost PVC are gone. See Operations.
• Total cluster loss (all nodes and PVCs destroyed) requires recovering into a separate Kubernetes cluster from the source backup archive. See Multi-cluster DR for the end-to-end runbook using spec.initFromBackup with optional PITR replay.
• ReadOnly sites cannot be selected for scheduled, automatic, or source-overridden backups. Healthy DisasterRecovery followers remain eligible.

Point-in-time recovery

• Errant or lost transactions in the window block restores until excluded. Every archived transaction committed between the backup's newest transaction and the target that is outside the target history must be in excludeGtids; the PITRArchiveDivergentHistory refusal prints the set. A backup taken after them needs no exclusion. See Target history and excludeGtids.
• Unarchived tail at a failover or reclone. Only sealed files of a writable site are uploaded. A demoted primary's active binlog is archived only if that site is writable again when the file seals, and a reclone wipes unuploaded files. Restores that need such a tail refuse PITRArchiveGap ("never archived") with the safe targets and backups. To keep a writable site's tail, while that site is still writable (before a planned failover away from it, or a reclone of it): note the file SHOW BINARY LOG STATUS names, run FLUSH BINARY LOGS to seal it, and wait until it is uploaded before the handoff. The upload is confirmed when the site's manifest (manifest-<site>.json, files[].name) lists that file; for an encrypted archive, when /archiver/status reports backlogFiles: 0 from a scan that started after the flush and finished (lastScanAt has advanced twice since the flush). Flushing another server cannot save a tail: a site that is already read-only uploads nothing, so its unuploaded files are lost once it is recloned. Archiving the tail automatically is a planned follow-up.
• GTID reissue after a reclone. A recloned server keeps its server_uuid. After a lossy failover, a reclone of the old primary and a failback to it, the old primary can reissue the lost GTID numbers with different transactions; the archive then refuses every restore with PITRArchiveHistoryReset, and the operator warns with PITRArchiveReissueRiskDetected. Start a new archive prefix and take a full backup. Rotating server_uuid on reclone is a planned follow-up.
• XA transactions in the replay refuse PITRReplayUnsupported. Choose a target before the XA transaction or a backup taken after it.
• Promotion-record lag window. A site records each promotion at its first writable observation, from a role probe every PEER_CHECK_INTERVAL (default 5 s, minimum 1 s). The probe runs in its own goroutine, so a busy archive scan does not delay it. A target whose witness is archived only as a replica copy and falls in (first commit of the new era, record observedAt + δ] has no anchor and refuses PITRStopAmbiguous. δ is the fixed 5 s clock-skew margin, so the window is at most PEER_CHECK_INTERVAL + 5 s long (about 10 s by default); the message prints it with the safe targets (at or before the promotion, or more than δ after observedAt). A probe that takes longer than 1 s cannot open the record, so a slow server defers it to the next prompt probe and widens the window by that much.
• Unobserved incarnation changes. The planner trusts a replica-copy witness's anchor only when no record of the origin's site reports a new episode or server incarnation between the anchor and the witness. The sidecar records every change it observes, and at start compares the server with the site's latest record, so a reclone or reset while the sidecar is down is recorded with a window reaching back to that record. Two changes are not recorded: one that is made and reverted between two role probes, and a reset while the sidecar is not running whose new history still holds the site's latest record (a later origin-row anchor can then be used across it). Each needs a history change on the origin followed by writes that only a replica archives.
• Unobserved order inversions. The planner refuses two sites that applied the same transactions in opposite orders (PITRArchiveOrderInversion) only when both orders are in downloaded archived bytes. An inversion in bytes that were never archived or were pruned is not detected. Errant transactions are unsupported history.
• Clock synchronization. When some of a backup's transactions are no longer archived, the planner bounds the backup's newest commit with the dumped server's clock and assumes the servers' clocks agree to within 1 s (NTP). Larger skew can admit a target before a backup transaction.
• Non-empty injected copies. An empty transaction injected to repair an errant GTID is detected when the backup was taken on a different server; a copy injected with content is not, because that needs a full row-effect comparison.
• Split brain with one branch archived. In latest mode, a transaction of the head is flagged as lost at a promotion only when it committed more than δ (the fixed 5 s clock-skew margin) before that promotion record was observed, the record lacks it, it is not in excludeGtids, and it is archived on a site other than the record's. When only one branch of a split brain is archived and the other writer's promotion was observed before that branch's transactions, or at most δ after them, nothing flags them: the archived branch is restored. The archive holds no evidence otherwise. A promotion observed more than δ after them refuses PITRArchiveDivergentHistory.
• Latest mode excludes the active binlog. Only sealed, uploaded files are replayed.
• PITR archive writers need a store that enforces atomic create (If-None-Match: * on S3, link(2) on a PVC). Active-active multi-zone buckets are unsupported for archive writing. See S3 backups.
• No in-place key rotation for a PITR archive. Once an archive's encryption binding is accepted, a passphrase or encryption change on a profile that archives to it is refused (at admission while PITR is enabled, otherwise on PITRArchiveBindingReady); rotation is a new prefix and a full backup. See Rotation.
• At most 16 archive bindings per group. Only entries that stay count (Bound, or used by a target). A 17th destination (written or read) gets none until an unused binding is released; a reader gets BindingLimitReached only when all 16 are Bound, and at the limit a new active destination is never rendered (the group keeps its held render, or renders PITR disabled).
• Renaming the passphrase Secret to a same-bytes copy is refused at admission while PITR is enabled; the rule compares references, not bytes. Prefer leaving the reference alone: the pods mount the key snapshot, not your Secret, so the name changes nothing at runtime. The only workaround, disabling PITR, renaming, and enabling it again, is not free. Each toggle changes the PITR render, so it is an ordered update of every MySQL pod (two full rollouts), and nothing is archived while PITR is off: binlogs MySQL purges in that window leave a permanent gap in the archive, and point-in-time targets across it become unrestorable. If you do it, keep the window shorter than the sites' binlog retention, check the archive for a gap afterwards (a restore or verification across the window refuses PITRArchiveGap), and take a fresh full backup once PITR is back if one is found.
• Key snapshots outlive the group. Archive and dump key snapshot Secrets have no owner reference and are never pruned automatically; release bindings by id or delete snapshots by label. See Key snapshots.
• The standby reader uses live bytes. A MysqlStandbyCluster still reads its source.decryption Secret directly, not a key snapshot.
• Shared dump and PITR profile after a refused rotation. New dumps are encrypted with the new bytes while the archive keeps the accepted key, so a reader that takes a single key (the standby reader) cannot open both.

Dragonfly co-management

• Managed Dragonfly is optional and intended for cache/session continuity, not durable application state.
• Dragonfly pods use ephemeral storage unless you configure spec.dragonfly.snapshot for planned snapshot-restore maintenance. Bloodraven does not schedule Dragonfly backups as durable data backups.
• Emergency MySQL failover never blocks on Dragonfly. If Dragonfly sync or promotion fails, sessions/cache may be discarded while MySQL recovery completes.
• spec.tls applies to MySQL, not Dragonfly. Protect Dragonfly with NetworkPolicy, Dragonfly auth, and any external TLS/service-mesh controls your environment requires.

Network partitions

• The network-partition runbook covers shapes A–E, including asymmetric peer reachability (shape D). Shape D is injected in the deterministic simulator (partitionOneWay); there is still no live-cluster chaos scenario for C or D.
• Playground partition tests must use pod-level NetworkPolicy or an equivalent mechanism; host-level iptables rules do not reliably block Kubernetes Service traffic in k3d.

Observability and tooling

• Grafana dashboards and metrics are shipped, but PrometheusRule alert examples are not yet packaged as first-class chart artifacts.
• The kubectl bloodraven plugin wraps supported day-2 operations, but it remains a thin API client; the operator performs the authoritative safety checks.

Data-at-rest encryption

• spec.encryptionAtRest uses MySQL Community's component_keyring_file. It is not Oracle "MySQL Enterprise TDE", and Oracle states that file-based keyrings are not intended as regulatory-compliance solutions. Do not present it as compliance-grade TDE.
• The live keyring is a Kubernetes Secret, so etcd becomes part of your key custody. API-server encryption at rest for Secrets (ideally KMS-backed) and tight Secret RBAC are prerequisites, not recommendations. Bloodraven cannot verify either.
• Memory-backed volumes can reach swap on kernels older than 6.3, where tmpfs noswap is unsupported. Disable or encrypt swap on worker nodes.
• Enabling encryption on a group that already holds data is refused by default: MySQL only encrypts what is written afterwards, so existing tablespaces stay plaintext. The supported adoption path is replica-first (bootstrap encrypted, load, planned-failover, wipe the old PVCs); an annotation exists to accept partial coverage knowingly.
• Master-key rotation is refused on the active primary. Rotate replicas, planned-failover, then rotate the ex-primary.
• Backup, restore, and backup-verification still stage plaintext in pod emptyDir volumes. Backup artifacts are separately protected by spec.backup.profiles[].encryption; the transient staging inside the Job's own pod is not covered by live MySQL encryption.
• The mysqldDir and pluginDir defaults match the official mysql:9.x images. Other images need those fields set explicitly, and a wrong value shows up as InnoDB refusing to start.

External dependencies

• DNS steering depends on external-dns consuming DNSEndpoint objects. Bloodraven updates the CR; DNS provider propagation time and TTLs are outside the operator's control.
• Production installs need real topology-aware persistent storage. Local path / hostPath storage is acceptable for playground use only.
• The unauthenticated auxiliary and sidecar HTTP surfaces assume a trusted pod network. Use NetworkPolicy before exposing those Services broadly.