Predicting the operator
Quick check: can you compute the 6s detection delay from pollInterval and failureThreshold, name the Reason string for a group with one writable and one unreachable site, and say what the anti-flap cooldown will not stop?
Passing score: 70%. Feedback appears as you go, and only your first answer to each question counts.
Show answer
Answer: writable on the second consecutive success; after the first success it still reads unreachable
computeState zeroes failCount on any successful poll, then handles the writable answer separately: because the site is not already StateWritable it increments recoveryCount and, until that reaches RecoveryThreshold (2), it return site.state — the state the site already had. So poll one leaves pdx reading unreachable, poll two makes it writable. Option one confuses the two counters: clearing failCount stops the site being newly unreachable, it does not grant writability, which is the debounced direction. Option three invents a state the code never returns on that path — the fall-through is the previous state, not read-only; read-only is returned only for a read_only=1 answer. Option four applies failureThreshold to the wrong transition: 3 gates the way in to unreachable, 2 gates the way back to writable. (objectives 1, 3)
computeState zeroes failCount on any successful poll, then handles the writable answer separately: because the site is not already StateWritable it increments recoveryCount and, until that reaches RecoveryThreshold (2), it return site.state — the state the site already had. So poll one leaves pdx reading unreachable, poll two makes it writable. Option one confuses the two counters: clearing failCount stops the site being newly unreachable, it does not grant writability, which is the debounced direction. Option three invents a state the code never returns on that path — the fall-through is the previous state, not read-only; read-only is returned only for a read_only=1 answer. Option four applies failureThreshold to the wrong transition: 3 gates the way in to unreachable, 2 gates the way back to writable. (objectives 1, 3)
Show answer
Answer: INFO state transition from=unreachable to=read-only, on the first successful poll
A read_only=1 answer returns StateReadOnly immediately, with no counter in the way, so the return transition costs exactly one poll. The six-second gap you can read off the timestamps (12:04:10 to 12:04:16) is pollInterval 2 s × failureThreshold 3 — three consecutive failed probes — and that sum applies only to the way in to unreachable. Option two makes the failure debounce symmetric; it is not, deliberately, because read-only is the safe direction and costs nothing to believe early. Option three drags in recoveryThreshold, which gates only the transition to writable and is never consulted here. Option four is wrong about which level carries the event: state transition is an INFO event in the log schema's Event reference, with fields site, from, to, fg; it is DEBUG-level bookkeeping that stays invisible. (objectives 2, 10)
A read_only=1 answer returns StateReadOnly immediately, with no counter in the way, so the return transition costs exactly one poll. The six-second gap you can read off the timestamps (12:04:10 to 12:04:16) is pollInterval 2 s × failureThreshold 3 — three consecutive failed probes — and that sum applies only to the way in to unreachable. Option two makes the failure debounce symmetric; it is not, deliberately, because read-only is the safe direction and costs nothing to believe early. Option three drags in recoveryThreshold, which gates only the transition to writable and is never consulted here. Option four is wrong about which level carries the event: state transition is an INFO event in the log schema's Event reference, with fields site, from, to, fg; it is DEBUG-level bookkeeping that stays invisible. (objectives 2, 10)
Show answer
Answer: FenceSites = [dr], Alert writable non-promotable site requires fencing (dr), Reason = "Degraded", returned before any other row is evaluated
The pre-tally loop routes any site that is writable while its role is not primary-candidate into action.FenceSites and continues, so dr never joins a tally at all; len(FenceSites) > 0 then fires the fence-first early return, which preempts every row below it. Option one gets coreCount right — dr-only genuinely is counted, unlike role: read-only — but misses that the writable dr was diverted before the writable tally was built, so that tally holds only iad and len(writable) > 1 is false. Option two makes the opposite error, treating dr-only as invisible to the matrix; only role: read-only is excluded from coreCount, and in any case a writable non-candidate is never Healthy. Option four fails the failover row's three conjuncts: it needs zero writable and at least one unreachable and at least one read-only, and here iad is writable and nothing is unreachable. (objectives 4, 6)
The pre-tally loop routes any site that is writable while its role is not primary-candidate into action.FenceSites and continues, so dr never joins a tally at all; len(FenceSites) > 0 then fires the fence-first early return, which preempts every row below it. Option one gets coreCount right — dr-only genuinely is counted, unlike role: read-only — but misses that the writable dr was diverted before the writable tally was built, so that tally holds only iad and len(writable) > 1 is false. Option two makes the opposite error, treating dr-only as invisible to the matrix; only role: read-only is excluded from coreCount, and in any case a writable non-candidate is never Healthy. Option four fails the failover row's three conjuncts: it needs zero writable and at least one unreachable and at least one read-only, and here iad is writable and nothing is unreachable. (objectives 4, 6)
Show answer
Answer: True
Both halves hold, and they are two separate deliberate bypasses. A writable observation on a non-promotable site skips recoveryThreshold entirely — the comment in the poll loop calls it an immediate safety fact and says authority invalidation must not be debounced behind the normal recovery threshold. And fencing a writable non-promotable site runs on every poll rather than on a transition, so a failed fence is retried without waiting for the state to change again. The tempting wrong answer is that the reader must prove itself over two polls like any other site: that reasoning is right for a primary-candidate, where being slow to believe a new primary is prudence, and exactly backwards for a reader, where being slow to notice it taking writes is not. (objectives 3, 6)
Both halves hold, and they are two separate deliberate bypasses. A writable observation on a non-promotable site skips recoveryThreshold entirely — the comment in the poll loop calls it an immediate safety fact and says authority invalidation must not be debounced behind the normal recovery threshold. And fencing a writable non-promotable site runs on every poll rather than on a transition, so a failed fence is retried without waiting for the state to change again. The tempting wrong answer is that the reader must prove itself over two polls like any other site: that reasoning is right for a primary-candidate, where being slow to believe a new primary is prudence, and exactly backwards for a reader, where being slow to notice it taking writes is not. (objectives 3, 6)
Show answer
Answer:
The failover row needs three conjuncts — zero writable, at least one unreachable, at least one read-only — and the unreachable conjunct fails, so the operator refuses to auto-elect: all-read-only is indistinguishable from a cluster that has not finished starting, so it is treated as a startup or recovery state needing human input. The table falls through to NoPrimary, and with exactly two read-only core sites and zero unreachable the alert is NO PRIMARY: both sites are read-only. But the operator holds history the pure table refuses to consult, and it can still restore writability by re-asserting the fenced promoted primary — pdx, the recorded lastFailoverTarget. The 5 m cooldown does not stand in the way: the re-assert rate limit reuses the failoverCooldown duration but measures it against a separate lastReassert timer that is never compared against lastFailover, and here lastReassert is still zero. Every one of these must hold: no subsystem gate active (bootstrap blocking cross-site, ordered update, topology frozen, planned failover in flight), the re-assert rate limit satisfied with no promotion pending confirmation, every non-target peer read-only (not writable, not unreachable, not unknown), the target read-only and promotable (role: primary-candidate), and the target's GTID_EXECUTED containing both the recorded promotion GTID set and every peer's GTID_EXECUTED. On success you get WARN re-asserting fenced promoted primary: no site is writable and the last failover target is GTID-complete; restoring writability with field site, and bloodraven_primary_reassert_total{site} increments. If the recorded promotion GTID fails to parse the operator refuses rather than skipping the gate, warning primary re-assert refused: recorded promotion GTID set failed to parse — status corrupted or manually edited?.
A full-credit answer shows: A strong answer covers: (a) the failover row's missing unreachable conjunct and the design reason — all-read-only is a startup/recovery state needing human input; (b) the fall-through to NoPrimary with the two-site message NO PRIMARY: both sites are read-only; (c) that a re-assert of lastFailoverTarget can still restore writability inside the 5 m window because the rate limit uses a separate lastReassert timer never compared against lastFailover; (d) at least three of the re-assert preconditions, and credit for naming the GTID containment one (recorded promotion GTID plus every peer's GTID) since it carries the safety argument. Credit the verbatim re-assert log line or bloodraven_primary_reassert_total. Mark down an answer that says the group must wait out the cooldown, or that the operator promotes the freshest read-only site.
The point of the question is the seam between a pure function and a stateful operator. EvalCrossSite has no clock and no memory, so with nothing unreachable it can only alert — and failoverCooldown gates exactly one thing, the promotion call, which is not what is happening here anyway. The re-assert is the single path that restores writability inside the cooldown window, and it is safe precisely because it restores authority to the site the operator already made authoritative, only when that site's GTID set proves nothing can be lost. (objectives 5, 7, 9)
Sample answer
The failover row needs three conjuncts — zero writable, at least one unreachable, at least one read-only — and the unreachable conjunct fails, so the operator refuses to auto-elect: all-read-only is indistinguishable from a cluster that has not finished starting, so it is treated as a startup or recovery state needing human input. The table falls through to NoPrimary, and with exactly two read-only core sites and zero unreachable the alert is NO PRIMARY: both sites are read-only. But the operator holds history the pure table refuses to consult, and it can still restore writability by re-asserting the fenced promoted primary — pdx, the recorded lastFailoverTarget. The 5 m cooldown does not stand in the way: the re-assert rate limit reuses the failoverCooldown duration but measures it against a separate lastReassert timer that is never compared against lastFailover, and here lastReassert is still zero. Every one of these must hold: no subsystem gate active (bootstrap blocking cross-site, ordered update, topology frozen, planned failover in flight), the re-assert rate limit satisfied with no promotion pending confirmation, every non-target peer read-only (not writable, not unreachable, not unknown), the target read-only and promotable (role: primary-candidate), and the target's GTID_EXECUTED containing both the recorded promotion GTID set and every peer's GTID_EXECUTED. On success you get WARN re-asserting fenced promoted primary: no site is writable and the last failover target is GTID-complete; restoring writability with field site, and bloodraven_primary_reassert_total{site} increments. If the recorded promotion GTID fails to parse the operator refuses rather than skipping the gate, warning primary re-assert refused: recorded promotion GTID set failed to parse — status corrupted or manually edited?.
A full-credit answer shows
A strong answer covers: (a) the failover row's missing unreachable conjunct and the design reason — all-read-only is a startup/recovery state needing human input; (b) the fall-through to NoPrimary with the two-site message NO PRIMARY: both sites are read-only; (c) that a re-assert of lastFailoverTarget can still restore writability inside the 5 m window because the rate limit uses a separate lastReassert timer never compared against lastFailover; (d) at least three of the re-assert preconditions, and credit for naming the GTID containment one (recorded promotion GTID plus every peer's GTID) since it carries the safety argument. Credit the verbatim re-assert log line or bloodraven_primary_reassert_total. Mark down an answer that says the group must wait out the cooldown, or that the operator promotes the freshest read-only site.
The point of the question is the seam between a pure function and a stateful operator. EvalCrossSite has no clock and no memory, so with nothing unreachable it can only alert — and failoverCooldown gates exactly one thing, the promotion call, which is not what is happening here anyway. The re-assert is the single path that restores writability inside the cooldown window, and it is safe precisely because it restores authority to the site the operator already made authoritative, only when that site's GTID set proves nothing can be lost. (objectives 5, 7, 9)
Show answer
Answer: Source convergence repoints iad at pdx and old-primary recovery runs its STOP REPLICA / RESET REPLICA ALL / CHANGE REPLICATION SOURCE rejoin; only a second promotion is blocked
The cooldown is one if immediately before the promotion call: if lastFailover is non-zero and clock.Since(lastFailover) < failoverCooldown, log failover blocked by anti-flap cooldown and return. Nothing else consults it. Option two is the mental model the word 'cooldown' invites and it is wrong — split-brain fencing, non-promotable fencing, source convergence, old-primary recovery, reclone and DNS reconcile each run from their own poll call site with no reference to tm.failoverCooldown. Option three inverts DNS behaviour: the DNSEndpoint is an idempotent server-side apply on every poll, re-derived from live topology, which is what self-heals a rejected write. Option four invents a queue; a reclone annotation is honoured while the cooldown ticks. (objective 7)
The cooldown is one if immediately before the promotion call: if lastFailover is non-zero and clock.Since(lastFailover) < failoverCooldown, log failover blocked by anti-flap cooldown and return. Nothing else consults it. Option two is the mental model the word 'cooldown' invites and it is wrong — split-brain fencing, non-promotable fencing, source convergence, old-primary recovery, reclone and DNS reconcile each run from their own poll call site with no reference to tm.failoverCooldown. Option three inverts DNS behaviour: the DNSEndpoint is an idempotent server-side apply on every poll, re-derived from live topology, which is what self-heals a rejected write. Option four invents a queue; a reclone annotation is honoured while the cooldown ticks. (objective 7)
Show answer
Answer: False
It still has the history, and that is exactly why the record is written twice. Every promotion stamps lastFailover and lastFailoverTarget onto the CR status subresource and onto the object's own metadata as bloodraven.shipstream.io/last-failover and bloodraven.shipstream.io/last-failover-target, at RFC3339 second precision, written as a pair by JSON merge patch. Status is a subresource: its writes travel a separate API path with their own RBAC rule and their own admission chain, so one path can be denied or broken for hours while the other keeps recording the fact. On restart the operator reads both and installs the later copy — ties go to status, and a copy stamped more than FailoverClockSkewGrace (5 m) ahead of local time is discarded rather than installed, because the cooldown gate treats negative elapsed time as still active and a future-dated record would wedge promotion indefinitely. Here the status copy is missing and the annotation copy is present, so the annotation wins and the cooldown survives the restart. (objective 8)
It still has the history, and that is exactly why the record is written twice. Every promotion stamps lastFailover and lastFailoverTarget onto the CR status subresource and onto the object's own metadata as bloodraven.shipstream.io/last-failover and bloodraven.shipstream.io/last-failover-target, at RFC3339 second precision, written as a pair by JSON merge patch. Status is a subresource: its writes travel a separate API path with their own RBAC rule and their own admission chain, so one path can be denied or broken for hours while the other keeps recording the fact. On restart the operator reads both and installs the later copy — ties go to status, and a copy stamped more than FailoverClockSkewGrace (5 m) ahead of local time is discarded rather than installed, because the cooldown gate treats negative elapsed time as still active and a future-dated record would wedge promotion indefinitely. Here the status copy is missing and the annotation copy is present, so the annotation wins and the cooldown survives the restart. (objective 8)
Show answer
Answer: iad is writable and serving; pdx has gone unreachable and its 4 is a stale reading left on the series from before it went away; reader is replicating and caught up as far as MySQL can tell
bloodraven_site_state is a state-set gauge: exactly one of the four series per site is 1 and the other three are 0, so you read four series and find the 1. That gives iad writable, pdx unreachable, reader read-only — corroborated by the transition counter, which increments once per state transition line and shows pdx going read-only to unreachable. Option one reads the lag gauge as current: when a site goes unreachable the operator neither updates nor deletes its lag gauge, so the last value it published stays on the series indefinitely. A lag gauge is only as fresh as the poll loop, which is why the non-zero poll_latency_seconds_count rate matters — it is the series that tells you the loop is still completing cycles at all. Option two misplaces the sentinel: -1 means the lag was NULL and there is no replication stream; 0 means caught up as far as MySQL can tell. Option four mistakes the gauge for a counter — a state-set publishes the current state every poll. (objective 11)
bloodraven_site_state is a state-set gauge: exactly one of the four series per site is 1 and the other three are 0, so you read four series and find the 1. That gives iad writable, pdx unreachable, reader read-only — corroborated by the transition counter, which increments once per state transition line and shows pdx going read-only to unreachable. Option one reads the lag gauge as current: when a site goes unreachable the operator neither updates nor deletes its lag gauge, so the last value it published stays on the series indefinitely. A lag gauge is only as fresh as the poll loop, which is why the non-zero poll_latency_seconds_count rate matters — it is the series that tells you the loop is still completing cycles at all. Option two misplaces the sentinel: -1 means the lag was NULL and there is no replication stream; 0 means caught up as far as MySQL can tell. Option four mistakes the gauge for a counter — a state-set publishes the current state every poll. (objective 11)
Show answer
Answer: Neither breaches: pdx is judged against maxLagSeconds 300, and a nil readOnlyMaxLagSeconds inherits maxLagSeconds, so reader is judged against 300 too
readOnlyMaxLagSeconds has no default: when it is nil it inherits maxLagSeconds, so both sites are measured against 300 and both stay where they are. Option two is the trap the field is built around — an explicit 0 is meaningful and demands zero reported lag, making it the strictest possible reader gate, but unset is not 0, it is inheritance. Option three gives up too early: the metric genuinely carries only a site label, and the join against role is one you perform yourself against the group spec — that is the whole discrimination, not a reason it cannot be made. Option four invents a promotion gate; maxLagSeconds drives only the ReplicationLagging Degraded condition and is never consulted when picking a candidate, so a lagging candidate stays promotable while a lagging reader merely sheds its reader endpoint. (objective 12)
readOnlyMaxLagSeconds has no default: when it is nil it inherits maxLagSeconds, so both sites are measured against 300 and both stay where they are. Option two is the trap the field is built around — an explicit 0 is meaningful and demands zero reported lag, making it the strictest possible reader gate, but unset is not 0, it is inheritance. Option three gives up too early: the metric genuinely carries only a site label, and the join against role is one you perform yourself against the group spec — that is the whole discrimination, not a reason it cannot be made. Option four invents a promotion gate; maxLagSeconds drives only the ReplicationLagging Degraded condition and is never consulted when picking a candidate, so a lagging candidate stays promotable while a lagging reader merely sheds its reader endpoint. (objective 12)
Show answer
Answer: Reason = "Degraded" with PromotionCandidates = [iad] and no alert, plus INFO failover blocked by anti-flap cooldown with fields lastFailover and cooldown — and no promotion
The three conjuncts of the failover row hold — zero writable, one unreachable, one read-only — so the matrix fills PromotionCandidates and sets Reason = "Degraded"; it is the one acting row that sets no alert. The cooldown then bites at the single guard immediately before the promotion call and returns after logging failover blocked by anti-flap cooldown with lastFailover and cooldown. Option two invents a reason string: exactly five reach status.conditions — Healthy, Degraded, SplitBrain, NoPrimary, TotalLoss. Option three is the published docs table, and it is wrong in the code's favour: there is no Failover reason, so an alerting rule matching on one never fires. Option four contradicts the every-poll evaluation — the matrix runs on every poll so status always carries the current topology condition, and only the mutating cross-site actions are transition-driven. (objectives 4, 7)
The three conjuncts of the failover row hold — zero writable, one unreachable, one read-only — so the matrix fills PromotionCandidates and sets Reason = "Degraded"; it is the one acting row that sets no alert. The cooldown then bites at the single guard immediately before the promotion call and returns after logging failover blocked by anti-flap cooldown with lastFailover and cooldown. Option two invents a reason string: exactly five reach status.conditions — Healthy, Degraded, SplitBrain, NoPrimary, TotalLoss. Option three is the published docs table, and it is wrong in the code's favour: there is no Failover reason, so an alerting rule matching on one never fires. Option four contradicts the every-poll evaluation — the matrix runs on every poll so status always carries the current topology condition, and only the mutating cross-site actions are transition-driven. (objectives 4, 7)