Multi-site topology

Bloodraven supports any number of sites (≥ 2) per MysqlFailoverGroup. Each site declares a role that controls its behaviour during failover:

The spec must contain at least two PrimaryCandidate sites so the operator always has a promotion target for the current primary. Any number of DisasterRecovery or ReadOnly sites may be appended. A reader does not count toward the two-candidate minimum.

Bloodraven uses these terms consistently:

Replication topology

Bloodraven uses a star topology: one site is the active primary, and every other site replicates from it. There are no replication chains.

                  primary (iad, PrimaryCandidate)
                        /        |        \
                     /           |           \
              (pdx)            (fra)            (reader)
        PrimaryCandidate DisasterRecovery       ReadOnly

During planned or emergency failover, the newly promoted primary becomes the direct source for every remaining follower, including DisasterRecovery and ReadOnly sites. Bloodraven also detects a healthy-looking follower that still points through another replica after restart or topology drift.

Before repointing, the operator requires exactly one writable promotable primary and checks that its GTID_EXECUTED contains the follower's executed set. It repeats containment after STOP REPLICA to close the applier race, then uses CHANGE REPLICATION SOURCE TO and START REPLICA without resetting replication metadata. Divergence is reported as sourceConvergenceState: Blocked with reason GTIDDiverged; Bloodraven does not silently abandon follower transactions.

Failover target selection

When the active site becomes unreachable and at least one PrimaryCandidate replica is reachable, the operator promotes the best candidate using this order:

1. GTID freshness. The operator queries @@gtid_executed on every eligible PrimaryCandidate replica in parallel and picks the one whose executed set is strictly a superset of the others. This is the primary selector: promoting the freshest replica minimises transactions lost to the async-replication RPO window.
2. Priority tiebreaker. When multiple replicas have equivalent GTID sets (common in healthy clusters), the first entry in spec.splitBrainPolicy.sitePriorities that is currently eligible wins. Entries not in the list fall through to declared site order.
3. Declared order. Final tiebreaker for replicas that share GTID sets and are not named in the priority list.

DisasterRecovery and ReadOnly sites are never auto-promoted, selected by planned failover, used as active DNS targets, or chosen as clone donors. A reader that becomes writable is fenced on every topology poll, even if it is the only writable site. Readers also never apply or remove node taints; supplied loadBalancerIP and taintNodeSelector fields are accepted for manifest compatibility but ignored.

If no PrimaryCandidate replica is reachable when the primary fails, the operator emits a NoPrimary alert and takes no action. Manual promotion of a DisasterRecovery site is deliberately outside automatic failover; convert it to PrimaryCandidate first if you intend to make it eligible for planned failover.

Split-brain resolution

spec.splitBrainPolicy.sitePriorities doubles as the split-brain tiebreaker. When more than one site is simultaneously writable and the operator cannot infer a winner from its own recent failover history (for example, after a fresh deploy or an operator restart that lost in-memory state), the first entry in sitePriorities that is currently writable and has role PrimaryCandidate is promoted; every other writable site is fenced (SET GLOBAL super_read_only=ON).

If sitePriorities is empty — or if no entry in the list is currently a writable PrimaryCandidate site — the operator alerts only and requires manual resolution. This matches the default "don't guess" behaviour of earlier Bloodraven versions.

Example: three-site cross-region DR

Two in-region PrimaryCandidate sites (active/standby HA) plus a cross-region DisasterRecovery follower:

apiVersion: shipstream.io/v1alpha1
kind: MysqlFailoverGroup
metadata:
  name: orders
spec:
  sites:
    - name: iad
      serverID: 101
      role: PrimaryCandidate
      zone: iad-1a
      taintNodeSelector:
        bloodraven.shipstream.io/failover-group.orders: "true"
        bloodraven.shipstream.io/site.orders: iad
      loadBalancerIP: 10.0.0.10
      storage: { storageClassName: gp3, size: 500Gi }
    - name: pdx
      serverID: 102
      role: PrimaryCandidate
      zone: pdx-1a
      taintNodeSelector:
        bloodraven.shipstream.io/failover-group.orders: "true"
        bloodraven.shipstream.io/site.orders: pdx
      loadBalancerIP: 10.1.0.10
      storage: { storageClassName: gp3, size: 500Gi }
    - name: fra
      serverID: 103
      role: DisasterRecovery
      zone: fra-1a
      taintNodeSelector:
        bloodraven.shipstream.io/failover-group.orders: "true"
        bloodraven.shipstream.io/site.orders: fra
      loadBalancerIP: 10.2.0.10
      storage: { storageClassName: gp3, size: 500Gi }
  splitBrainPolicy:
    sitePriorities: [iad, pdx]
  # ... rest of spec ...

Failover behaviour:

• If iad goes down, pdx is promoted (first priority, primary- candidate, reachable replica).
• If iad and pdx both go down, the operator alerts — it will not auto-promote fra because fra is DisasterRecovery. Manual promotion would use the forthcoming planned-failover API.
• If two sites become simultaneously writable (split-brain), the winner is the first sitePriorities entry currently writable; others are fenced.

Sidecar peer awareness

Each pod's sidecar is given the internal Service addresses for every non-self site, including DisasterRecovery sites and readers, via the PEER_ADDRESSES env var. The sidecar tracks per-peer liveness and only self-fences when the operator and every peer are unreachable beyond spec.sidecar.leaseTimeout. A single reachable peer is enough to keep the primary writable.

This quorum rule matters as the number of sites grows: a split where the primary can still reach at least one peer is preserved as a legitimate-writes window rather than collapsing to a self-fenced outage.

Reader peers provide connectivity and relay fresh operator-authoritative active-site observations; they gain no promotion or primary authority. A reachable peer without fresh authoritative topology can still suppress the lease-only all-peers-unreachable fence. This is retained compatibility behavior, not a quorum guarantee.

Example: two candidates and a reader

The reader omits candidate/DisasterRecovery placement fields, uses a lower endpoint lag threshold, and customizes only its own client Service and MySQL config:

spec:
  replication:
    maxLagSeconds: 300
    readOnlyMaxLagSeconds: 30
  sites:
    - name: iad
      serverID: 101
      role: PrimaryCandidate
      zone: iad-1a
      taintNodeSelector:
        bloodraven.shipstream.io/site.orders: iad
      loadBalancerIP: 10.0.0.10
      storage: {storageClassName: gp3, size: 500Gi}
    - name: pdx
      serverID: 102
      role: PrimaryCandidate
      zone: pdx-1a
      taintNodeSelector:
        bloodraven.shipstream.io/site.orders: pdx
      loadBalancerIP: 10.1.0.10
      storage: {storageClassName: gp3, size: 500Gi}
    - name: reader
      serverID: 103
      role: ReadOnly
      zone: iad-1b
      mysqlConf:
        innodb_buffer_pool_size: 4G
      serviceTemplate:
        type: LoadBalancer
        externalTrafficPolicy: Local
        annotations:
          service.beta.kubernetes.io/aws-load-balancer-internal: "true"
      storage: {storageClassName: gp3, size: 500Gi}
  splitBrainPolicy:
    sitePriorities: [iad, pdx]

Sizing and compatibility

• Minimum: 2 sites, both PrimaryCandidate; readers are additional sites.
• Maximum: 16 sites. Practical limits derive from replication cost on the primary (each replica opens an I/O thread) and from DNS/LB churn at failover time.
• Every site receives a Deployment, PVC, client MySQL Service, internal administrative Service, and per-site MySQL ConfigMap. Reader client endpoints have stricter health gating than other site Services.

Known limitations

• DisasterRecovery sites cannot be auto-promoted. Region-level loss of every PrimaryCandidate site requires manual intervention.
• Planned failover targets only PrimaryCandidate sites. A DisasterRecovery or ReadOnly site must be deliberately reclassified, with loadBalancerIP and taintNodeSelector supplied, before it can be promoted.