The six-row table that decides everything
Turn a set of site states into exactly one action. Evaluation order matters, the fence-first return preempts every row below it, and the published table is missing a row you will meet in production.
By the end of this topic you can
- Turn any set of site states into the operator's action using the cross-site evaluation table
- Say why all-read-only and all-unreachable get an alert and no automatic action
- Predict what happens when a non-promotable reader turns up writable
You can name each site’s state. A site takes pollInterval × failureThreshold = 2 s × 3 = 6 s
to reach unreachable. What you cannot yet do is turn three of those labels into one decision.
That is one function: EvalCrossSite in internal/state/matrix.go. It takes the per-site
observations and returns a single action. The condition reason, the alert text, and whether a
promotion happens at all all fall out of it.
The idea: order is the behaviour
The function is a sequence of guarded early returns. A row can only fire if every row above it declined. Reading the table as an unordered list of cases is the commonest way to predict the operator wrongly.
Before any row is evaluated, one loop walks the observations and does three things. It counts coreCount, incrementing for every site whose role is not read-only. It routes any site that is writable while its role is not primary-candidate into action.FenceSites and continues — so that site never reaches a tally. And it drops role: read-only sites entirely. The consequence people get wrong: a dr-only site is not excluded. dr-only is non-promotable, but it still counts toward coreCount and it still lands in the readOnly or unreachable tally. Only role: read-only is invisible to the matrix (objective 4).
Put these in the order they happen.
- Fence-first early return — len(FenceSites) > 0 → Alert "writable non-promotable site requires fencing (…)", Reason "Degraded", return
- TotalLoss — len(unreachable) == coreCount → "TOTAL LOSS: all sites are unreachable", Reason "TotalLoss"
- SplitBrain — len(writable) > 1 → "SPLIT BRAIN: N sites are writable (…)", Reason "SplitBrain"
- Failover — len(writable)==0 AND len(unreachable)>0 AND len(readOnly)>0 → PromotionCandidates set, Reason "Degraded", no Alert
- NoPrimary — len(writable)==0 and the three conjuncts did not hold → Reason "NoPrimary", alert only
- Degraded (primary up, peer down) — exactly one writable AND len(unreachable) > 0 → "
unreachable while is primary", Reason "Degraded" - Healthy — exactly one writable, zero unreachable → Reason "Healthy"
Three details inside that sequence earn their own sentence.
The failover row needs three conjuncts at once: zero writable and at least one unreachable and at least one read-only. Drop the unreachable peer and the operator refuses to act. The comment in the source says why: “Without any unreachable peer we refuse to auto-elect a primary (all-read-only is a startup or recovery state that needs human input).” All-read-only is indistinguishable from a cluster that has not finished starting, so the operator alerts and stops. All-unreachable gets the same treatment for the opposite reason — nothing is left to promote (objective 5).
NoPrimary has two messages. Exactly two read-only sites and zero unreachable gives NO PRIMARY: both sites are read-only. Everything else gives NO PRIMARY: no writable site available.
The one row that acts is the only row that sets no alert. The failover branch fills PromotionCandidates and sets Reason = "Degraded" — it never assigns action.Alert. Every other non-healthy row sets one.
There is no Failover reason
The single most expensive misreading of this function has nothing to do with which row fires. It is what the winning row writes down.
The failover row sets Reason = "Degraded". That string is written verbatim onto the Degraded condition in status.conditions, with snap.Alert as the message. There is no Failover reason, and there never was one — the topology that is one promotion away from recovery reports the same reason as every other unhealthy shape. So a Prometheus or kubectl rule matching a condition reason of Failover does not fire rarely. It fires never, silently, through every promotion you will ever do.
Exactly five reason strings reach status.conditions:
Healthy · Degraded · SplitBrain · NoPrimary · TotalLoss
Write your rules against those, and read “a failover happened” off bloodraven_failovers_total — a counter, not a condition.
What a summary table cannot carry
The published documentation renders this function as a handful of two-column rows. Summaries lose things, and it is worth naming what this one loses, because both omissions describe states playground really sits in.
There is no row for “the primary is fine and a peer is down.” That is the shape of the entire recovery window after a site failure: one writable, at least one unreachable, Reason = "Degraded", with an alert naming the pair. It is a real branch of EvalCrossSite, and it belongs on any table you build for your own on-call.
There is no row for the fence-first early return. A summary that lists site states in columns structurally cannot express that one branch preempts every branch below it. Fence-first is not a case among cases; it is a return.
Take the habit rather than the grievance. The CRD is the contract, the code is the behaviour, and a rendered table — including the widget above — is a picture of the code rather than the code. When the two disagree, EvalCrossSite wins. A dated record of where the published page has and has not caught up is in the version appendix, section B, so this reading does not have to carry it.
Worked: two states of playground
A — iad dies. After 6 s of failed polls iad is unreachable, pdx is read-only, reader is read-only. The loop: reader is excluded, so coreCount = 2. No writable non-candidate, so FenceSites is empty. Tallies: writable = [], readOnly = [pdx], unreachable = [iad]. Row by row — fence-first declines (empty); TotalLoss declines (1 ≠ 2); SplitBrain declines (0 is not > 1); the failover row’s three conjuncts all hold, so PromotionCandidates = [pdx] and Reason = "Degraded", with no alert. The caller then runs pickFreshestCandidate over that list.
B — reader comes up writable. A freshly restarted MySQL pod is writable for a few seconds before anything fences it. Now iad is writable, pdx is read-only, reader is writable. The loop: reader is not counted in coreCount (still 2), and because it is writable while non-candidate it goes to FenceSites and is skipped — it never joins the writable tally. So writable = [iad]. Fence-first fires: alert writable non-promotable site requires fencing (reader), Reason = "Degraded", return. Note what did not happen: two sites in the group were genuinely accepting writes, and the operator did not report split brain, because len(writable) > 1 counts core sites only and the reader was already removed (objective 6).
| iad writable, pdx read-only, reader read-only | Healthy |
|---|---|
| iad writable, pdx unreachable | Degraded ("pdx unreachable while iad is primary") |
| iad unreachable, pdx read-only | Degraded (promotion candidates set, no alert) |
| iad read-only, pdx read-only, none unreachable | NoPrimary ("NO PRIMARY: both sites are read-only") |
| iad writable, pdx writable | SplitBrain |
| iad unreachable, pdx unreachable | TotalLoss |
| iad writable, reader writable | Degraded (fence-first, reader fenced) |
What the function does not do
EvalCrossSite is pure. The source comment is explicit: “The function is pure: it never considers history or policy beyond the supplied priorities.” It has no clock, no memory of prior promotions, and no access to MySQL. That is why split-brain auto-resolution, which needs history, is layered on top by the caller rather than living in the matrix.
The matrix is evaluated on every poll, not only on a state transition — “Evaluate every poll so all status snapshots carry the current topology condition. Mutating cross-site actions remain transition-driven.” So status.conditions always reflects the current topology, while the actions that change MySQL fire on transitions.
And when the table does select a failover, the matrix does not choose the winner. It hands PromotionCandidates to the caller, ordered by sitePriorities then declared order, and pickFreshestCandidate reads GTID_EXECUTED from each one. GTID freshness is the primary selector — it minimises data loss on promotion. sitePriorities only breaks ties or incomparable sets. A site listed first in sitePriorities will lose to a fresher peer.
Handoff
You can now read any set of site states for playground and name three things without running anything: the action, the alert string, and the Reason that lands in status.conditions. Example A said Reason = "Degraded" with PromotionCandidates = [pdx] — the table selected a failover. That is not the same as the operator performing one. Something between the table’s verdict and the promotion can still refuse, and it keeps its own record of what it did last.
Flashcards
Any site is writable while its role is not primary-candidate. What does EvalCrossSite do?
Fills action.FenceSites, sets Alert writable non-promotable site requires fencing (…) and Reason "Degraded", and returns immediately — before TotalLoss, SplitBrain, failover or anything else is evaluated.
Exact trigger condition for Reason "TotalLoss"
len(unreachable) == coreCount — every core site unreachable. Alert: TOTAL LOSS: all sites are unreachable.
Exact trigger condition for Reason "SplitBrain"
len(writable) > 1 among core sites — strictly more than one, so exactly one writable never qualifies.
The three conjuncts the failover row needs, all at once
len(writable) == 0 AND len(unreachable) > 0 AND len(readOnly) > 0. Only then are PromotionCandidates emitted.
Which single row of EvalCrossSite sets no action.Alert at all?
The failover row. It fills PromotionCandidates and sets Reason "Degraded" but never assigns Alert; every other non-healthy row sets one.
The two distinct NoPrimary alert messages and what selects between them
Exactly two read-only sites with zero unreachable gives NO PRIMARY: both sites are read-only; every other no-writable case gives NO PRIMARY: no writable site available.
One site writable, one or more unreachable — what is emitted?
Alert <site> unreachable while <site> is primary with Reason "Degraded". This row is absent from the published docs table.
The five Reason strings that actually reach status.conditions
Healthy, Degraded, SplitBrain, NoPrimary, TotalLoss. "Failover" is not among them.
A group's only surviving reachable site is dr-only and read-only; the primary-candidate peer is unreachable. Which tally holds the dr-only site, and what is emitted?
The readOnly tally — dr-only is not excluded, so the failover row's three conjuncts all hold; but RankPromotionCandidates keeps primary-candidates only, returns empty, and the code falls through to Reason "NoPrimary".
Selector order once the table has emitted PromotionCandidates
pickFreshestCandidate reads GTID_EXECUTED and takes the freshest set; sitePriorities only orders the list to break ties or incomparable sets.
"EvalCrossSite is pure" — what does that exclude, concretely?
Any history or policy beyond the supplied priorities: no prior-failover record, no clock, no MySQL access. Split-brain auto-resolution is layered on by the caller.
How often is the matrix evaluated, versus how often cross-site actions mutate MySQL?
The matrix runs on every poll so status always carries the current condition; the mutating cross-site actions remain transition-driven.
Quiz
Show answer
Answer: Reason "Degraded" with alert writable non-promotable site requires fencing (reader), returning before any other row is evaluated
The tally loop routes any writable site whose role is not primary-candidate into FenceSites and continues, so reader never joins the writable tally; the fence-first guard then fires and returns immediately. SplitBrain is wrong because len(writable) counts core sites only and equals 1 here — and even with two writable candidates the fence-first return would preempt it. Healthy is the trap version of the same fact: readers are excluded from coreCount and the tallies, but a writable one is not merely ignored, it is an anomaly that produces an alert. The last option invents an unreachable site that does not exist. (objectives 4, 6)
The tally loop routes any writable site whose role is not primary-candidate into FenceSites and continues, so reader never joins the writable tally; the fence-first guard then fires and returns immediately. SplitBrain is wrong because len(writable) counts core sites only and equals 1 here — and even with two writable candidates the fence-first return would preempt it. Healthy is the trap version of the same fact: readers are excluded from coreCount and the tallies, but a writable one is not merely ignored, it is an anomaly that produces an alert. The last option invents an unreachable site that does not exist. (objectives 4, 6)
Show answer
Answer: Emits Reason "NoPrimary" with alert NO PRIMARY: both sites are read-only and takes no automatic action
The failover row needs three conjuncts and one of them is missing: there is no unreachable peer. Without one, the source refuses to auto-elect because all-read-only is a startup or recovery state that needs human input. Options one and two both assume the operator elects from a fully reachable all-read-only set — it never does, and neither GTID freshness nor sitePriorities is even consulted, because PromotionCandidates is never populated. The fourth option confuses this with the failover row, which sets Reason "Degraded" and PromotionCandidates only when at least one peer is unreachable. Change pdx to unreachable and the same states produce exactly that. (objectives 4, 5)
The failover row needs three conjuncts and one of them is missing: there is no unreachable peer. Without one, the source refuses to auto-elect because all-read-only is a startup or recovery state that needs human input. Options one and two both assume the operator elects from a fully reachable all-read-only set — it never does, and neither GTID freshness nor sitePriorities is even consulted, because PromotionCandidates is never populated. The fourth option confuses this with the failover row, which sets Reason "Degraded" and PromotionCandidates only when at least one peer is unreachable. Change pdx to unreachable and the same states produce exactly that. (objectives 4, 5)
Show answer
Answer: "Degraded", with message pdx unreachable while iad is primary
One writable plus at least one unreachable hits the primary-up-peer-down row: Reason "Degraded", alert <site> unreachable while <site> is primary. "Failover" is the most dangerous distractor because the published docs table does name that outcome — but no code path emits it; the only five reasons that reach status.conditions are Healthy, Degraded, SplitBrain, NoPrimary and TotalLoss, so an alert rule matching a Failover reason never fires. "Healthy" requires exactly one writable AND zero unreachable, and the second clause fails. "NoPrimary" requires zero writable, and iad is still writable. (objective 4)
One writable plus at least one unreachable hits the primary-up-peer-down row: Reason "Degraded", alert <site> unreachable while <site> is primary. "Failover" is the most dangerous distractor because the published docs table does name that outcome — but no code path emits it; the only five reasons that reach status.conditions are Healthy, Degraded, SplitBrain, NoPrimary and TotalLoss, so an alert rule matching a Failover reason never fires. "Healthy" requires exactly one writable AND zero unreachable, and the second clause fails. "NoPrimary" requires zero writable, and iad is still writable. (objective 4)
Show answer
Answer: False
The opposite is true: only role: read-only is excluded. The tally loop increments coreCount for every site whose role is not read-only, so a dr-only site counts and still lands in the readOnly or unreachable tally. Promotability and matrix visibility are two separate properties, and conflating them is expensive: in a group of two primary-candidates plus one dr-only site, treating dr-only as invisible would make you predict len(unreachable) == coreCount at two unreachable sites when the real threshold is three, so you would expect TotalLoss a whole site early. (objective 4)
The opposite is true: only role: read-only is excluded. The tally loop increments coreCount for every site whose role is not read-only, so a dr-only site counts and still lands in the readOnly or unreachable tally. Promotability and matrix visibility are two separate properties, and conflating them is expensive: in a group of two primary-candidates plus one dr-only site, treating dr-only as invisible would make you predict len(unreachable) == coreCount at two unreachable sites when the real threshold is three, so you would expect TotalLoss a whole site early. (objective 4)
Show answer
Answer:
pdx. EvalCrossSite only orders the candidate list — it puts iad first because sitePriorities names it first — but the caller then runs pickFreshestCandidate, which reads GTID_EXECUTED from every candidate and takes the most caught-up set. GTID freshness is the primary selector because it minimises data loss on promotion; the priority list is consulted only to break ties or genuinely incomparable sets. So pdx wins despite being second in the list.
A full-credit answer shows: A strong answer names pdx, states that GTID freshness is the primary selector and gives the reason (minimising data loss on promotion), and states that sitePriorities acts only as a tiebreaker for equal or incomparable sets. Answering iad on the grounds that priority ordering is authoritative is the misconception being tested. Credit an answer that also notes EvalCrossSite itself never picks the winner — it is pure and has no MySQL access, so the GTID read happens in the caller.
Ordering the candidate list is not the same as choosing from it. The matrix is pure and cannot read GTID_EXECUTED at all, so the freshness comparison necessarily happens in the caller, after the table has spoken. (objective 4)
Sample answer
pdx. EvalCrossSite only orders the candidate list — it puts iad first because sitePriorities names it first — but the caller then runs pickFreshestCandidate, which reads GTID_EXECUTED from every candidate and takes the most caught-up set. GTID freshness is the primary selector because it minimises data loss on promotion; the priority list is consulted only to break ties or genuinely incomparable sets. So pdx wins despite being second in the list.
A full-credit answer shows
A strong answer names pdx, states that GTID freshness is the primary selector and gives the reason (minimising data loss on promotion), and states that sitePriorities acts only as a tiebreaker for equal or incomparable sets. Answering iad on the grounds that priority ordering is authoritative is the misconception being tested. Credit an answer that also notes EvalCrossSite itself never picks the winner — it is pure and has no MySQL access, so the GTID read happens in the caller.
Ordering the candidate list is not the same as choosing from it. The matrix is pure and cannot read GTID_EXECUTED at all, so the freshness comparison necessarily happens in the caller, after the table has spoken. (objective 4)
Show answer
Answer: The matrix is evaluated on every poll so status always carries the current condition, while the mutating cross-site actions remain transition-driven — a site's state changed, and that alone moves the reason
The reason string is recomputed from scratch on every poll and written to status.conditions, so it tracks the live topology whether or not anything was mutated; only the actions that change MySQL are gated on transitions. The first option inverts that relationship. The third contradicts the function's stated purity — it has no memory of prior polls and no notion of escalation. The fourth is not the mechanism: the chart ships a single replica with leader election, so there is no concurrent second evaluator to blame. (objective 4)
The reason string is recomputed from scratch on every poll and written to status.conditions, so it tracks the live topology whether or not anything was mutated; only the actions that change MySQL are gated on transitions. The first option inverts that relationship. The third contradicts the function's stated purity — it has no memory of prior polls and no notion of escalation. The fourth is not the mechanism: the chart ships a single replica with leader election, so there is no concurrent second evaluator to blame. (objective 4)