The bet Bloodraven makes
Two sites, asynchronous replication, and an RPO that is deliberately not zero. What that buys, what it costs, and the six things Bloodraven refuses to do.
By the end of this topic you can
- Say when Bloodraven is the right tool by naming the two-site, non-zero-RPO deployment it targets
- Name three things Bloodraven refuses to do, from its own non-goals list
- Explain why asynchronous replication with automatic promotion was chosen over Group Replication
You have a live group. iad is taking writes. pdx and reader are copying them. The
counter increments. The question this topic answers is not how any of that is wired. It is
what Bloodraven is willing to lose when iad goes away.
The deployment it is built for
Bloodraven targets one shape and is explicit about it: two or more MySQL sites, ordinary
asynchronous replication, exactly one site writable at a time, and a non-zero RPO accepted on
unplanned loss. spec.sites takes between 2 and 16 entries. playground uses three. That
is the whole design point.
Asynchronous means a write commits on the primary and is acknowledged to your application before any other site has seen it. Nothing waits for a peer. That buys you the write-latency of a standalone MySQL. It costs you exactly this:
An emergency failover can lose every transaction that committed on the dying primary but had not yet replicated to the surviving site.
RPO — recovery point objective — is how much recently committed data you accept losing. Bloodraven’s RPO on sudden primary loss is not zero.
RTO — recovery time objective — is how long you accept being unable to write. Bloodraven spends its whole engineering budget there: it promotes a survivor without waking anyone.
This is a choice, not an unfinished feature. Two sites is a legitimate topology for Bloodraven. For a quorum system it is a pathological one — lose one of two members and the survivor goes read-only, so you invent a witness node and inherit its problems.
The six things it refuses to do
Bloodraven publishes its own non-goals. Learn them before you write a manifest:
- It does not provide synchronous replication or zero RPO after sudden primary loss.
- It does not replace external-dns, cert-manager, Prometheus, Grafana, or your object store.
- It does not make application connection pools failover-aware automatically.
- It does not reconcile divergent writes for you after split-brain.
- It does not make PVC-local backups durable after cluster or storage loss.
- It does not treat Dragonfly as durable application storage; managed Dragonfly is for cache/session continuity.
Two of those bite hardest. The connection-pool one, because a pool will hand your application
a connection to a demoted primary: the node is alive, the health check passes, and only the
next INSERT fails. And the PVC one, because when a PVC is destroyed the active binlog that
lived on it is gone forever.
Against quorum
| Bloodraven — async + external promotion | Synchronous / quorum — e.g. Group Replication | |
|---|---|---|
| A site is lost. What happens to writes? | The surviving site is promoted and takes writes. Two sites is enough to tolerate one loss. | Only a majority side stays writable. Lose one of two members and the survivor is read-only. |
| What can be lost? | Every transaction that committed on the dying primary but had not yet replicated to the surviving site. | Nothing on the majority side — the cost is paid instead on every single commit, as a cross-node round trip. |
| Who decides? | An external operator arbitrates, and a sidecar on each MySQL pod fences itself when it cannot confirm it is still authoritative. | The group decides for itself by majority vote. No external decider, and no answer at all without a quorum. |
Group Replication is not the loser of that table. It is the other end of it. A quorum system trades write availability for consistency: it refuses writes rather than accept ones it might have to throw away. Bloodraven trades the other way — write availability and a mental model one person can hold at 3am.
If “we lost a second of writes” is a customer-visible failure, run Group Replication. If losing the ability to write while a cross-site link misbehaves is the worse outcome, run this.
The starting state
Nothing has changed since you stood playground up. This is the shape, excerpted:
apiVersion: shipstream.io/v1alpha1
kind: MysqlFailoverGroup
metadata:
name: playground
spec:
image: mysql:9.7 # the one MySQL baseline Bloodraven supports — pin it, never mysql:9
sites: # MinItems 2, MaxItems 16
- name: iad
role: primary-candidate
- name: pdx
role: primary-candidate
- name: reader
role: read-only
dns:
hostname: playground-db.example.local
# ... splitBrainPolicy, replication, sidecar, storage elidedThose three role: values are doing the work you named in the last topic.
What the bet buys
Against the old posture — page a human, promote by hand, guess the RPO — the bet pays out three ways.
Unplanned promotion happens with nobody logged in.
A planned switchover is RPO 0 by construction: the source is fenced, its GTID position snapshotted, and the target promoted only once its own executed set covers that snapshot.
And when transactions really are lost you get a number, not a shrug: the operator computes
the set difference, publishes the count, and records the set in status.sites[].divergentGtid.
All of it is exercised by 51 registered chaos scenarios against real clusters.
Where this leaves you
You can now say what deployment Bloodraven targets, and what it will refuse to do for you. You have a running group whose parts you can name, and a contract for what an emergency failover is allowed to lose.
Then a site stops answering. The fields you learned start moving — state to unreachable,
the reason off Healthy. The operator sees exactly what you see. What it decides to do
about that, and how long it waits first, is the next unit.
Flashcards
Asynchronous replication
A write commits and is acknowledged on the primary before any other site has seen it — nothing waits for a peer.
RPO (recovery point objective)
How much recently committed data you accept losing. Bloodraven's RPO on sudden primary loss is not zero.
RTO (recovery time objective)
How long you accept being unable to write. This is where Bloodraven spends its engineering budget.
Failover group
One MysqlFailoverGroup resource: two to sixteen MySQL sites, exactly one of them writable at a time.
Active site
The one site in the group currently allowed to accept writes; mysql-playground-primary points at it.
Split brain
Two sites accepting writes at the same time, so each accumulates transactions the other never saw.
Fencing
Forcing an instance to stop accepting writes — in MySQL, setting super_read_only=ON on it.
Why super_read_only rather than read_only for fencing?
read_only still permits updates from CONNECTION_ADMIN (or the deprecated SUPER); super_read_only blocks even those.
GTID
A global transaction identifier — a per-transaction ID that makes two servers' executed-transaction sets directly comparable.
status.sites[].divergentGtid
The set of transactions the old primary executed that the new primary never received — the exact, countable data loss.
Quiz
Show answer
Answer: Bloodraven is the wrong tool for that requirement; a quorum system such as Group Replication is what they want
Bloodraven's contract is that an emergency failover can lose every transaction that committed on the dying primary but had not yet replicated to the surviving site. Zero RPO on sudden loss is an explicit non-goal, so the requirement and the tool do not match. Tuning a lag threshold does not help: lag thresholds drive a Degraded condition, not the promotion decision — the alternative of no writable site at all is almost always worse, so Bloodraven promotes anyway. Adding a third site does not help either: Bloodraven has no quorum in the first place; the count of sites changes nothing about the async commit path. And there is no synchronous mode to enable — Bloodraven does not provide synchronous replication at all. (objectives 1, 2)
Bloodraven's contract is that an emergency failover can lose every transaction that committed on the dying primary but had not yet replicated to the surviving site. Zero RPO on sudden loss is an explicit non-goal, so the requirement and the tool do not match. Tuning a lag threshold does not help: lag thresholds drive a Degraded condition, not the promotion decision — the alternative of no writable site at all is almost always worse, so Bloodraven promotes anyway. Adding a third site does not help either: Bloodraven has no quorum in the first place; the count of sites changes nothing about the async commit path. And there is no synchronous mode to enable — Bloodraven does not provide synchronous replication at all. (objectives 1, 2)
Show answer
Answer: False
The reversal: this is exactly backwards, and it is the quorum system that has the two-site problem, not Bloodraven. Bloodraven has no quorum requirement — one site is writable and an external operator arbitrates who that is, so two sites is a legitimate topology that tolerates losing one. A majority-based system is the one where losing one of two members leaves the survivor read-only, which is why people bolt a witness node onto such deployments. The engineer holding this misconception has generalised quorum thinking from Group Replication or etcd onto a design that deliberately does not use it. (objectives 1, 3)
The reversal: this is exactly backwards, and it is the quorum system that has the two-site problem, not Bloodraven. Bloodraven has no quorum requirement — one site is writable and an external operator arbitrates who that is, so two sites is a legitimate topology that tolerates losing one. A majority-based system is the one where losing one of two members leaves the survivor read-only, which is why people bolt a witness node onto such deployments. The engineer holding this misconception has generalised quorum thinking from Group Replication or etcd onto a design that deliberately does not use it. (objectives 1, 3)
Show answer
Answer: It reports them — the divergent GTID set and its transaction count — and leaves the decision to a human
Reconciling divergent writes is a published non-goal: Bloodraven makes the loss observable, not invisible. It computes the set difference between the old primary's GTIDs and the new primary's, records the set in status.sites[].divergentGtid, publishes the count as bloodraven_divergent_transactions, and stops there. Automatic merging is the tempting wrong answer because GTIDs make the transactions identifiable — identifiable is not the same as safely mergeable, since two divergent histories can conflict semantically. Automatic rollback is equally wrong: nothing in Bloodraven throws data away on your behalf; the diverged instance is reported so you can decide. And the lag threshold has nothing to do with it — it is not a gate on anything in the recovery path. (objectives 2, 3)
Reconciling divergent writes is a published non-goal: Bloodraven makes the loss observable, not invisible. It computes the set difference between the old primary's GTIDs and the new primary's, records the set in status.sites[].divergentGtid, publishes the count as bloodraven_divergent_transactions, and stops there. Automatic merging is the tempting wrong answer because GTIDs make the transactions identifiable — identifiable is not the same as safely mergeable, since two divergent histories can conflict semantically. Automatic rollback is equally wrong: nothing in Bloodraven throws data away on your behalf; the diverged instance is reported so you can decide. And the lag threshold has nothing to do with it — it is not a gate on anything in the recovery path. (objectives 2, 3)
Show answer
Answer: Their connection pool will not become failover-aware on its own, and can serve connections to a demoted primary that pass validation
Making connection pools failover-aware is an explicit non-goal, and the failure is quiet rather than loud: a demoted primary is alive and answers a validation query perfectly well, so the pool hands the connection back and only the next write fails. That is a real, open upstream complaint against HikariCP, and the reason MySQL drivers added rejectReadOnly handling. Refusing to promote until clients disconnect is wrong and would defeat the whole point — availability is what Bloodraven is buying. Bloodraven does not replace external-dns, so it does not perform propagation and cannot bound it. And PVC-local backups are named in the non-goals as exactly the thing that is not durable after storage loss, so they close no gap at all. (objective 2)
Making connection pools failover-aware is an explicit non-goal, and the failure is quiet rather than loud: a demoted primary is alive and answers a validation query perfectly well, so the pool hands the connection back and only the next write fails. That is a real, open upstream complaint against HikariCP, and the reason MySQL drivers added rejectReadOnly handling. Refusing to promote until clients disconnect is wrong and would defeat the whole point — availability is what Bloodraven is buying. Bloodraven does not replace external-dns, so it does not perform propagation and cannot bound it. And PVC-local backups are named in the non-goals as exactly the thing that is not durable after storage loss, so they close no gap at all. (objective 2)
Show answer
Answer:
A quorum system trades write availability for consistency: it refuses writes rather than accept ones it might have to discard, so every commit pays a cross-node round trip and a minority side goes read-only. Bloodraven trades the other way — it trades consistency on unplanned loss for write availability and operational simplicity, accepting that an emergency failover can lose every transaction that committed on the dying primary but had not yet replicated to the surviving site. Both are defensible. The deciding fact is what the business does with the data: if losing a small window of committed writes is customer-visible (a ledger, a system of record), take Group Replication and pay the latency; if being unable to write while the cross-site link misbehaves is the worse outcome, take Bloodraven. The two-datacentre shape also pushes towards Bloodraven, because two members cannot form a quorum without inventing a witness.
A full-credit answer shows: A strong answer names both directions of the trade — quorum gives up write availability to protect consistency, Bloodraven gives up consistency-on-unplanned-loss to protect write availability and simplicity — and states that both are legitimate rather than declaring one wrong. It must identify the deciding input as the business tolerance for losing recently committed writes, not as a technical preference. Credit for noting that a two-member group has no majority, or that async commits do not pay a cross-site round trip. Do not give credit for an answer that treats Bloodraven's non-zero RPO as a defect to be fixed.
The point is that this is a design-point choice, not a quality ranking. Bloodraven is optimised for the two-site, geographically separated, accept-non-zero-RPO deployment; Group Replication is optimised for three-or-more nodes on a low-latency link with zero RPO. Answers that pick a winner without naming what is given up have missed the objective. (objectives 1, 3)
Sample answer
A quorum system trades write availability for consistency: it refuses writes rather than accept ones it might have to discard, so every commit pays a cross-node round trip and a minority side goes read-only. Bloodraven trades the other way — it trades consistency on unplanned loss for write availability and operational simplicity, accepting that an emergency failover can lose every transaction that committed on the dying primary but had not yet replicated to the surviving site. Both are defensible. The deciding fact is what the business does with the data: if losing a small window of committed writes is customer-visible (a ledger, a system of record), take Group Replication and pay the latency; if being unable to write while the cross-site link misbehaves is the worse outcome, take Bloodraven. The two-datacentre shape also pushes towards Bloodraven, because two members cannot form a quorum without inventing a witness.
A full-credit answer shows
A strong answer names both directions of the trade — quorum gives up write availability to protect consistency, Bloodraven gives up consistency-on-unplanned-loss to protect write availability and simplicity — and states that both are legitimate rather than declaring one wrong. It must identify the deciding input as the business tolerance for losing recently committed writes, not as a technical preference. Credit for noting that a two-member group has no majority, or that async commits do not pay a cross-site round trip. Do not give credit for an answer that treats Bloodraven's non-zero RPO as a defect to be fixed.
The point is that this is a design-point choice, not a quality ranking. Bloodraven is optimised for the two-site, geographically separated, accept-non-zero-RPO deployment; Group Replication is optimised for three-or-more nodes on a low-latency link with zero RPO. Answers that pick a winner without naming what is given up have missed the objective. (objectives 1, 3)