What it cost you
The RPO contract in one sentence, the two durability settings a tenant can silently switch off, and the GTID arithmetic that turns an outage into an exact number of lost transactions.
By the end of this topic you can
- State the RPO contract in one sentence and say which settings are guarantees and which are merely defaults
- Read
promotionGtidExecutedanddivergentGtidto get an exact lost-transaction count - Pick the right row of the per-failure-mode RPO matrix for an outage in front of you
playground is back. pdx took writes 12.0 seconds after you took iad down, and the counter
application is incrementing again. The incident review will not ask how fast that was. It will ask
how many writes you lost — and “some” is not an answer you can put in a ticket.
The contract
Memorise this sentence, because everything else in the topic is either an amplifier or a mitigation of it:
An emergency failover can lose every transaction that committed on the dying primary but had not yet replicated to the surviving site.
Replication between sites is asynchronous. Nothing holds your commit open on iad until pdx has
acknowledged it. The window is real, it is bounded only by how far behind pdx happened to be, and
no setting in the CRD closes it on the emergency path.
Guarantees versus defaults
The mitigation people reach for is durability settings, and this is where the documentation flattens
a distinction that decides whether your RPO story survives contact with a tenant. Bloodraven renders
each site’s config in three passes. First a base map of sensible settings. Then your overrides —
spec.mysqlConf, and then a site’s own mysqlConf, which beats the group’s. Then a short block of
operator-owned invariants, written after your overrides, which simply overwrite whatever you put
there. Finally skip-log-bin and disable-log-bin are deleted outright, because MySQL honours the
last occurrence and a sorted render would place either alias after log-bin.
That ordering is the whole test. Written after the overrides, it is a guarantee. Written before them, it is a default.
| Overridable defaults (written first) | Un-weakenable invariants (written last) | |
|---|---|---|
| Which settings? | sync-binlog=1, innodb-flush-log-at-trx-commit=2, binlog-expire-logs-seconds=1209600 (14 days) | gtid-mode=ON, enforce-gtid-consistency=ON, log-replica-updates=ON, log-bin, skip-replica-start=ON, plugin-load-add=mysql_clone.so |
| Can spec.mysqlConf beat it? | Yes — silently, with no warning and no admission rejection | No — the invariant block overwrites your key after your override is applied |
| What does losing it cost? | The durability and retention story the RPO documentation tells | GTID identity, binlog continuity, and the ability to clone or replicate at all |
| Where do you check? | The rendered per-site ConfigMap, not the layer diagram | The same place — the invariant should be present regardless of what you set |
So: a tenant who sets sync_binlog=0 in spec.mysqlConf gets sync_binlog=0, and your written RPO
promise quietly becomes fiction. A tenant who sets gtid_mode=OFF gets gtid_mode=ON anyway. Do not
trust the diagram — read the file playground actually rendered:
$ kubectl -n bloodraven-playground get configmap mysql-playground-iad-config \
-o jsonpath='{.data.bloodraven\.cnf}' | grep -E 'sync-binlog|gtid-mode|flush-log'
gtid-mode=ON
innodb-flush-log-at-trx-commit=2
sync-binlog=1Sharpen the middle line, because the MySQL manual is harsher than Bloodraven’s docs. With
innodb_flush_log_at_trx_commit=2, logs are written at commit but flushed once per second, and the
manual attributes the loss to any unexpected mysqld process exit — not just power loss — saying
plainly that it “can erase up to N seconds of transactions”. Its own recommendation is
innodb_flush_log_at_trx_commit=1 alongside sync_binlog=1, the setting it calls the safest, which
guarantees no transaction is lost from the binary log. Bloodraven ships 2 for throughput. That is a
defensible trade, and it is one spec.mysqlConf line from the stricter one. Just know the price: up
to a second of committed transactions, on the site that has just crashed, on top of the replication
window.
The arithmetic
Now count what you actually lost. At step 6 of the failover sequence the operator runs
SELECT @@global.gtid_executed on the candidate — before it accepts a single write — and records the
answer in status.promotionGtidExecuted. That is the high-water mark of everything pdx had
received. When the old primary comes back and is compared against it, the operator subtracts one set
from the other and publishes the difference as status.sites[].divergentGtid, with its cardinality on
the bloodraven_divergent_transactions gauge.
Two MySQL functions let you do that arithmetic by hand on any pair of sets. GTID_SUBSET(set1, set2)
returns true when every GTID in set1 is also in set2 — that is the “did it catch up” question.
GTID_SUBTRACT(set1, set2) returns only those GTIDs from set1 that are not in set2. The
subtraction is the divergence primitive; its cardinality is your lost-transaction count.
One wrinkle before you eyeball a set. MySQL 9.x GTID sets can carry user-defined tags, written
uuid:tag:interval, and a tag is treated as part of the UUID’s identity. uuid:Domain_1:1-3 and
uuid:Domain_2:1-3 are six different transactions, not three. Let MySQL do the subtraction.
The measurement
20-23 is four transactions, not an estimate.Here is a real playground status after an emergency promotion to pdx, with the two fields that matter
and everything else elided:
status:
activeSite: pdx # was iad
lastFailoverTarget: pdx
promotionGtidExecuted: |- # new since topic 1: pdx's set at promotion
a2cc879c-5f9d-11f1-9fae-8e47bc2a4544:1-19,
a3c3f9e8-5f9d-11f1-bf37-568bfb8d0365:1-7
sites:
- name: iad
divergentGtid: a2cc879c-5f9d-11f1-9fae-8e47bc2a4544:20-23
divergentTransactionCount: 4Read it off. a2cc879c… is iad’s own server UUID — the transactions it originated as primary. pdx
had 1-19 of them at the moment of promotion. The difference is 20-23, so iad had run to 1-23
and four transactions committed on iad never reached pdx. The count is 23 − 20 + 1 = 4, which is
exactly what the gauge reports (bloodraven_divergent_transactions{site="iad"} 4) and what the status
condition says in words: Old primary iad has 4 divergent transactions. Four counter increments. Not
an estimate.
kubectl -n bloodraven-playground get mysqlfailovergroup playground -o jsonpath='{.status.sites[?(@.name=="iad")].divergentGtid}'a2cc879c-5f9d-11f1-9fae-8e47bc2a4544:20-23mysql -N -e "SELECT GTID_SUBTRACT('a2cc879c-5f9d-11f1-9fae-8e47bc2a4544:1-23', 'a2cc879c-5f9d-11f1-9fae-8e47bc2a4544:1-19,a3c3f9e8-5f9d-11f1-bf37-568bfb8d0365:1-7')"a2cc879c-5f9d-11f1-9fae-8e47bc2a4544:20-23The distractor: maxLagSeconds
spec.replication.maxLagSeconds defaults to 300, and the playground manifest for playground sets 30. It
drives exactly one thing: a ReplicationLagging reason on the Degraded condition when a site’s
reported lag exceeds it. It is not a promotion gate. Nothing in candidate selection consults it.
If iad dies while pdx is 400 seconds behind, Bloodraven promotes pdx anyway — because no writable
site at all is almost always worse. If you believe maxLagSeconds bounds your RPO, your RPO is
whatever the lag happened to be at the moment of the crash. What does bound it is a true GTID-superset
test, which is why a planned switchover is RPO 0 by construction; that path is Unit 4.
Pick your row
| Failure mode | RPO | Why |
|---|---|---|
| Container/pod crash, PVC intact | 0 — no failover at all | The primary returns writable and the operator keeps it |
| Clean primary kill, replica caught up | Near zero, not guaranteed zero | The contract still applies to anything in flight |
| Primary kill with unapplied relay logs | Whatever was in flight | The 30 s drain applies what it can reach; the rest is gone |
| PVC destroyed with the primary | Worst row | The previously-active binlog lived on the destroyed PVC — PITR cannot replay a tail that was never shipped |
Given an outage, find the row before you quote a number.
You can now state the RPO contract, tell a durability guarantee from a durability default by reading a
rendered ConfigMap, and produce an exact lost-transaction count for the failover you just ran. Which
leaves the four transactions sitting on iad. iad is back, it is read-only, and it is holding
writes pdx has never seen. What happens when it tries to rejoin is the next topic.
Flashcards
State the RPO contract for an emergency failover in one sentence.
An emergency failover can lose every transaction that committed on the dying primary but had not yet replicated to the surviving site.
You are handed a rendered bloodraven.cnf. What single test tells you whether a setting in it is a guarantee or merely a default?
Ask where the operator writes it relative to your overrides: written after spec.mysqlConf it is an un-weakenable invariant, written before it is a default a tenant can beat.
Which two durability settings in a Bloodraven-rendered config are defaults rather than guarantees?
sync-binlog=1 and innodb-flush-log-at-trx-commit=2 — both live in the base map, applied before spec.mysqlConf overrides.
Name the settings Bloodraven writes after user overrides, so no spec.mysqlConf entry can weaken them.
gtid-mode=ON, enforce-gtid-consistency=ON, log-replica-updates=ON, log-bin, skip-replica-start=ON, and plugin-load-add=mysql_clone.so.
binlog-expire-logs-seconds — shipped value, and which layer is it in?
1209600 seconds (14 days), in the overridable base map — a tenant can shorten it and silently shrink your PITR reach.
GTID_SUBSET(set1, set2) — what does it return?
True when every GTID in set1 is also in set2; it answers 'has this site caught up', not 'by how much'.
GTID_SUBTRACT(set1, set2) — what does it return?
Only those GTIDs from set1 that are not in set2 — the divergence primitive, whose cardinality is the lost-transaction count.
status.promotionGtidExecuted — where does its value come from?
SELECT @@global.gtid_executed run on the candidate at promotion, before it accepts any write.
status.sites[].divergentGtid — what does it contain?
The GTID set of transactions held by that site which the current primary never received.
bloodraven_divergent_transactions — what does this gauge report?
The number of divergent transactions on a site pending recovery; 0 when healthy.
spec.replication.maxLagSeconds — default value and the one behaviour it drives.
Default 300; it sets the ReplicationLagging reason on the Degraded condition and does nothing else.
In a MySQL 9.x GTID set, how many transactions are uuid:Domain_1:1-3 and uuid:Domain_2:1-3 together?
Six — a user-defined tag is treated as part of the UUID's identity, so the two ranges are distinct transactions.
Quiz
Show answer
Answer: sync-binlog=0 and gtid-mode=ON
The two settings sit in different layers. sync-binlog=1 is in the base map, written before spec.mysqlConf, so the tenant's 0 wins and the durability story quietly changes. gtid-mode=ON is in the invariant block written after user overrides, so it is stamped back to ON. Option 0 is the common flattening of the model — it is right about sync_binlog and wrong about gtid_mode. Option 2 assumes the operator classifies settings by what they mean rather than by write order; it does not, which is exactly why sync_binlog is beatable. Option 3 imagines admission-time protection that does not exist: the override is accepted and silently overwritten at render time, with no rejection and no event. (objective 4)
The two settings sit in different layers. sync-binlog=1 is in the base map, written before spec.mysqlConf, so the tenant's 0 wins and the durability story quietly changes. gtid-mode=ON is in the invariant block written after user overrides, so it is stamped back to ON. Option 0 is the common flattening of the model — it is right about sync_binlog and wrong about gtid_mode. Option 2 assumes the operator classifies settings by what they mean rather than by write order; it does not, which is exactly why sync_binlog is beatable. Option 3 imagines admission-time protection that does not exist: the override is accepted and silently overwritten at render time, with no rejection and no event. (objective 4)
Show answer
Answer: False
The reversal: Bloodraven promotes it anyway. maxLagSeconds drives exactly one thing — the ReplicationLagging reason on the Degraded condition — and nothing in candidate selection consults it, because no writable site at all is almost always worse than a stale one. The practical consequence is the point: if you believe maxLagSeconds bounds your RPO, your RPO is whatever the lag happened to be when the primary died. A true GTID-superset test is what actually bounds loss, and that gate belongs to the planned path. (objective 4)
The reversal: Bloodraven promotes it anyway. maxLagSeconds drives exactly one thing — the ReplicationLagging reason on the Degraded condition — and nothing in candidate selection consults it, because no writable site at all is almost always worse than a stale one. The practical consequence is the point: if you believe maxLagSeconds bounds your RPO, your RPO is whatever the lag happened to be when the primary died. A true GTID-superset test is what actually bounds loss, and that gate belongs to the planned path. (objective 4)
Show answer
Answer: The cardinality of GTID_SUBTRACT(O, N)
GTID_SUBTRACT(O, N) returns only those GTIDs from O that are not in N — the transactions the dying primary committed and never shipped — and counting them gives the loss. Option 0 returns a boolean: it tells you whether divergence exists, not how much. Option 2 reverses the operands and yields what the new primary has that the old one lacks, which is normal post-promotion drift, not loss. Option 3 is the eyeball heuristic that breaks on real sets: intervals are sparse, several UUIDs can appear, and a MySQL 9.x tag makes uuid:Domain_1 and uuid:Domain_2 distinct identities, so comparing top sequence numbers can be wrong in either direction. (objective 5)
GTID_SUBTRACT(O, N) returns only those GTIDs from O that are not in N — the transactions the dying primary committed and never shipped — and counting them gives the loss. Option 0 returns a boolean: it tells you whether divergence exists, not how much. Option 2 reverses the operands and yields what the new primary has that the old one lacks, which is normal post-promotion drift, not loss. Option 3 is the eyeball heuristic that breaks on real sets: intervals are sparse, several UUIDs can appear, and a MySQL 9.x tag makes uuid:Domain_1 and uuid:Domain_2 distinct identities, so comparing top sequence numbers can be wrong in either direction. (objective 5)
Show answer
Answer: Up to a second of transactions on any unexpected mysqld process exit
The manual is blunter than the operator's docs: with a setting of 2 logs are written at commit but flushed once per second, and any unexpected mysqld process exit can erase up to N seconds of transactions. It recommends 1 alongside sync_binlog=1. Option 0 confuses 'written' with 'flushed' — that gap is the whole loss window. Option 1 is the softer framing worth unlearning: an OOM kill or a crashing mysqld costs you the same second as a power cut, and it is the site that just crashed. Option 3 describes the replication window, which is a separate loss window that adds to this one rather than replacing it. (objective 4)
The manual is blunter than the operator's docs: with a setting of 2 logs are written at commit but flushed once per second, and any unexpected mysqld process exit can erase up to N seconds of transactions. It recommends 1 alongside sync_binlog=1. Option 0 confuses 'written' with 'flushed' — that gap is the whole loss window. Option 1 is the softer framing worth unlearning: an OOM kill or a crashing mysqld costs you the same second as a power cut, and it is the site that just crashed. Option 3 describes the replication window, which is a separate loss window that adds to this one rather than replacing it. (objective 4)
Show answer
Answer:
This is the worst row — PVC destruction, not a pod crash. It is not the RPO-0 row, because that one requires the PVC to survive so the same primary comes back writable and no failover happens at all. Here the loss is everything iad committed but had not replicated to pdx: those transactions were only ever in the binlog on the destroyed PVC, they were never shipped, so they are not in the replica's binlog stream and therefore not in PITR's replay material. Backups plus PITR get you back to the last archived event, not to the tail. The exact count is read from divergentGtid — except that here the site holding it is gone, so quote the replication window, not a measured number.
A full-credit answer shows: A strong answer covers: (1) identifying the PVC-destruction row rather than the pod-crash row, and saying why the pod-crash row is RPO 0 (PVC intact, primary returns writable, no failover); (2) naming the previously-active binlog on the destroyed PVC as the thing that is gone; (3) the reason PITR cannot cover it — unshipped transactions are not in the replica's binlog stream and so are not in the replay material; (4) bonus credit for noticing that the usual divergentGtid measurement is unavailable because the diverged site no longer exists.
The discrimination is between two rows that both start with 'the primary died': with the PVC intact there is no failover and no loss, while with the PVC destroyed you lose the unreplicated tail permanently. The tempting wrong answer is that PITR closes the gap — it does not, because PITR can only replay events that were actually archived, and the tail never left the dead primary. (objective 6)
Sample answer
This is the worst row — PVC destruction, not a pod crash. It is not the RPO-0 row, because that one requires the PVC to survive so the same primary comes back writable and no failover happens at all. Here the loss is everything iad committed but had not replicated to pdx: those transactions were only ever in the binlog on the destroyed PVC, they were never shipped, so they are not in the replica's binlog stream and therefore not in PITR's replay material. Backups plus PITR get you back to the last archived event, not to the tail. The exact count is read from divergentGtid — except that here the site holding it is gone, so quote the replication window, not a measured number.
A full-credit answer shows
A strong answer covers: (1) identifying the PVC-destruction row rather than the pod-crash row, and saying why the pod-crash row is RPO 0 (PVC intact, primary returns writable, no failover); (2) naming the previously-active binlog on the destroyed PVC as the thing that is gone; (3) the reason PITR cannot cover it — unshipped transactions are not in the replica's binlog stream and so are not in the replay material; (4) bonus credit for noticing that the usual divergentGtid measurement is unavailable because the diverged site no longer exists.
The discrimination is between two rows that both start with 'the primary died': with the PVC intact there is no failover and no loss, while with the PVC destroyed you lose the unreplicated tail permanently. The tempting wrong answer is that PITR closes the gap — it does not, because PITR can only replay events that were actually archived, and the tail never left the dead primary. (objective 6)