When the operator is down
The operator is not on the request path, so playground keeps serving without it. What you lose is the ability to fail over — and a write that returned an error may have landed anyway.
By the end of this topic you can
- Say what keeps working and what stops when the operator pod is gone
- Explain why operator downtime costs write availability but not RPO
- Decide whether to wait for the operator or promote by hand
playground is healthy on the three-site playground: iad writable, pdx read-only and replicating, the reader tracking behind them, the counter app polling mysql-playground-primary for reads and writing on demand. Scale the operator to zero and watch what breaks. The lesson is that nothing does.
kubectl -n bloodraven-playground scale deploy/bloodraven --replicas=0deployment.apps/bloodraven scaledcurl -s localhost:8090/api/counter{"value":<n>,"dbHost":"mysql-playground-iad-…","readOnly":false}# sixty seconds later, operator still at 0
curl -s localhost:8090/api/counter{"value":<n+k, k>0>,"dbHost":"mysql-playground-iad-…","readOnly":false} # still climbingkubectl -n bloodraven-playground get mfg playground -o jsonpath='{.status.sites[*].lastSeen}'<the last poll before the scale-down — identical sixty seconds later>The counter keeps incrementing, readOnly stays false, and the only thing that changes is the custom resource: status.sites[].lastSeen freezes at the final poll. That is the whole shape of operator downtime. A healthy primary and replica keep serving reads and writes with zero operator involvement — the operator sits on the failure-detection and promotion path, not the request path.
Availability and correctness fail separately
Correctness — no split brain, no silent divergence — is preserved by the sidecar fencing layer regardless of how long the operator is gone. The sidecars are separate processes with their own timers, and a dead operator is exactly what their lease rule exists for. A MySQL pod that restarts mid-outage comes up fenced and stays fenced: the startup safety net cannot get an authoritative answer and refuses to guess.
Availability is not. If the primary dies while the operator is down, nothing promotes the replica, and playground has no writable site until the operator returns.
| Behaviour | At zero replicas |
|---|---|
| Reads and writes on a healthy primary | Unaffected — not the request path |
Replication to pdx and the reader | Unaffected — MySQL to MySQL |
| Sidecar self-fencing | Unaffected — separate processes, own timers |
| Failure detection and promotion | Stopped — nothing is authorised to promote |
Service selector, DNSEndpoint, taints | Stopped — all three are operator writes |
status.* on the CR | Frozen, and stale status is not a symptom of anything else |
Objective 11 is the line worth memorising: operator downtime costs write availability, never RPO. RPO is fixed by what had replicated at the instant the primary died — an emergency failover can lose every transaction that committed on the dying primary but had not yet reached the survivor, and that set is sealed at the moment of death. An operator that shows up two hours later promotes the same replica across the same GTID gap and loses the same transactions. The outage is longer; the data loss is not larger.
One replica, leader election, and what it does not buy
The chart ships replicaCount: 1 with leader election enabled, and engineers reach for replicaCount: 3 the first time they read the paragraph above. Be precise about what that buys.
Leader election means extra replicas are standbys, not parallel workers. One replica holds the lease and is the sole writer of status, DNS, and promotion commands; the others sit idle. Adding them shortens the gap between a crashed leader and a new one taking over. It cannot shorten the poll loop — the detection bottleneck you measured in Unit 2: pollInterval × failureThreshold, 2 s × 3 = 6 s on defaults, before promotion even begins. Three replicas do not make failover faster; they make an operator crash cheaper.
The cooldown cuts both ways
The anti-flap cooldown (spec.failoverCooldown, default 5m, 30s in the playground) fails in two opposite directions.
The rarer one is that the cooldown is not there when it should be. It is persisted twice on every promotion — in status and in out-of-band annotations — so losing it across a restart takes both paths rejecting writes at once. Then a restarted operator can promote earlier than you configured. The deterministic simulator names this CooldownViolated(restart+stateLost) and classifies it as an inherent finding class, not a bug queued for a fix: understood, bounded by the GTID gates, accepted.
The direction you meet on call is the opposite. The cooldown blocks a second failover you genuinely need, and it does not care that you believe this one is justified. Your judgement is not an input.
Wait, or promote by hand
Objective 12 turns on one fact about the break-glass tool. kubectl bloodraven promote playground pdx writes the bloodraven.shipstream.io/planned-failover annotation and returns. The plugin only writes resources the operator already reads; it never talks to MySQL. It is not a back door around the operator’s logic — it is a request to that logic, and it needs a live operator to execute it. Inside the cooldown it hits the same gate: spec.plannedFailover.onCooldown defaults to reject, so the request fails outright unless you set defer, which parks it until cooldown expiry.
The rule that falls out:
- Operator returning inside the cooldown, no writable site? Wait. It promotes within roughly 6 s of detection plus the drain once it is back, and the hand-driven path cannot start sooner.
- Operator not coming back — crashlooping image, broken RBAC, deleted namespace? Fix the operator first. That is the promotion path; annotating a group nobody is watching only queues an intent.
- Editing MySQL directly? Know what you are taking on. Clear
read_onlyonpdxwhile the operator is down and the sidecar’s topology-mismatch rule sees a writable site disagreeing with the lastactiveSiteit cached — and fences it straight back.
The error that already happened
A SET GLOBAL that returns an error may still have landed. Cancelling the context tears down the client connection; it does not roll back a write the server already applied. Bloodraven shipped a bug from exactly this: treating the returned error as a failure made the monitor re-fence a site it had just promoted.
Generalise it properly, because it is not a MySQL fact. A timeout or a cancellation tells you that you stopped waiting, not that the remote side did not act. Every retry, every rollback, and every “that failed, so I will try the other one” inherits that ambiguity. The only honest response to a cancelled mutation is to re-read the state.
And the wider echo: control plane and data plane fail separately in practice, not only in design documents. Cloudflare’s November 2023 incident kept the data plane serving for roughly two days of control-plane outage — your counter app climbing against a scaled-to-zero operator is the same phenomenon at playground scale.
Where this leaves you
You can take an operator outage deliberately now, and say which guarantees went with it. The cooldown protects you in one direction and traps you in the other; break-glass promotion is a request to the operator, not a substitute for it; a failed write is an unknown, not a rollback. What none of it tells you is what to do when the data itself is gone — when the question stops being “who should be primary” and becomes “which bytes are still recoverable”. That is the next unit.
Flashcards
The operator pod for playground is deleted while both MySQL sites are healthy. What do applications see?
Nothing. Reads and writes keep flowing — the operator is on the failure-detection and promotion path, not the request path.
Which layer preserves correctness while the operator is gone?
The sidecar fencing layer — separate processes with their own timers, so no site accepts writes it is not authorised to accept, however long the outage lasts.
Where is the RPO of an emergency failover actually decided?
At the instant the primary died — by what had already replicated to the survivor. Nothing that happens afterwards moves that number.
The single observable change on the CR during an operator outage.
status.* stops updating — status.sites[].lastSeen freezes at the last poll.
replicaCount: 3 on the Bloodraven chart buys you what?
A faster handover after the leader crashes. The extra replicas are idle standbys, not parallel workers.
The detection floor you cannot buy your way past with more operator replicas.
pollInterval × failureThreshold — 2 s × 3 = 6 s on the shipped defaults.
CooldownViolated(restart+stateLost)
The deterministic simulator's name for a restart that lost both durable anti-flap copies and promoted earlier than failoverCooldown allowed — a documented inherent finding class, not a queued bug.
The cooldown failure direction you actually meet on call.
It blocks a second failover you genuinely need, for the rest of spec.failoverCooldown (default 5m), regardless of how justified you believe this one is.
What kubectl bloodraven promote playground pdx actually does.
Writes the bloodraven.shipstream.io/planned-failover annotation the operator reads. It never touches MySQL, so it needs a live operator to execute.
Cloudflare, November 2023 — the operational lesson for a Bloodraven operator.
Control plane and data plane fail separately: the data plane kept serving for roughly two days of control-plane outage.
Quiz
Show answer
Answer: status.sites[].lastSeen on the playground CR is ten minutes stale
Status is written only by the operator, so stale lastSeen is the signature of operator absence and of nothing else. Replication is MySQL-to-MySQL — the operator never carries the data, so a stopped replica is a real replication fault you must diagnose on its own merits. Reads are served by MySQL through a Service whose endpoints already exist; a read outage during operator downtime means the replica itself is unwell. And sidecars are separate processes with their own timers: a dead operator is precisely the condition their lease rule exists for, so fencing is the one thing that does not stop. Attributing any of the last three to the operator sends you to the wrong pod (objective 10).
Status is written only by the operator, so stale lastSeen is the signature of operator absence and of nothing else. Replication is MySQL-to-MySQL — the operator never carries the data, so a stopped replica is a real replication fault you must diagnose on its own merits. Reads are served by MySQL through a Service whose endpoints already exist; a read outage during operator downtime means the replica itself is unwell. And sidecars are separate processes with their own timers: a dead operator is precisely the condition their lease rule exists for, so fencing is the one thing that does not stop. Attributing any of the last three to the operator sends you to the wrong pod (objective 10).
Show answer
Answer: False
The reversal: a longer outage does not enlarge the loss. RPO is fixed at the instant the primary died — the set of transactions that committed there but had not yet reached the survivor is sealed at that moment, because the dead primary is not replicating anything more. The operator that arrives two hours later promotes the same replica across the same GTID gap. What the two hours cost is write availability: playground had no writable site for the whole window. The intuition that drives people to answer 'true' is the correct one for a lagging replica under a live primary, where the gap grows with time — but a dead primary has stopped producing transactions to fall behind on (objectives 10, 11).
The reversal: a longer outage does not enlarge the loss. RPO is fixed at the instant the primary died — the set of transactions that committed there but had not yet reached the survivor is sealed at that moment, because the dead primary is not replicating anything more. The operator that arrives two hours later promotes the same replica across the same GTID gap. What the two hours cost is write availability: playground had no writable site for the whole window. The intuition that drives people to answer 'true' is the correct one for a lagging replica under a live primary, where the gap grows with time — but a dead primary has stopped producing transactions to fall behind on (objectives 10, 11).
Show answer
Answer: Only the gap between a crashed leader and a new one taking over; detection still costs pollInterval × failureThreshold
Leader election makes the extra replicas idle standbys, so the only thing they shorten is recovery from a leader crash. They do not poll in parallel — exactly one replica holds the lease and is the sole writer of status, DNS, and promotion commands, so detection still costs 2 s × 3 = 6 s on defaults. 'Nothing at all' is wrong in the other direction: the replicas really are scheduled and really do shorten the takeover gap, so this is a cheap availability improvement, just not a latency one. And there is no vote: promotion is a single-leader decision, not a quorum, so believing the standbys add safety review is a dangerous misreading of what leader election does (objective 10).
Leader election makes the extra replicas idle standbys, so the only thing they shorten is recovery from a leader crash. They do not poll in parallel — exactly one replica holds the lease and is the sole writer of status, DNS, and promotion commands, so detection still costs 2 s × 3 = 6 s on defaults. 'Nothing at all' is wrong in the other direction: the replicas really are scheduled and really do shorten the takeover gap, so this is a cheap availability improvement, just not a latency one. And there is no vote: promotion is a single-leader decision, not a quorum, so believing the standbys add safety review is a dangerous misreading of what leader election does (objective 10).
Show answer
Answer:
Roll the operator image back and get the operator running — that is the promotion path. kubectl bloodraven promote only writes the planned-failover annotation on the group, and the operator is the thing that reads it, so with the operator crashlooping the annotation just queues an intent and nothing promotes. Once the operator is up it will detect the dead primary in about 6 s and promote, and the cooldown does not lengthen the write outage further than its remaining four minutes; if I need the manual path to survive that window I set spec.plannedFailover.onCooldown: defer so the request is parked and retried at cooldown expiry rather than rejected outright. What I do not do is clear read_only on the replica by hand: with the operator down the sidecar still holds its last authoritative activeSite, sees a writable site that disagrees with it, and fences it straight back.
A full-credit answer shows: A strong answer covers: (1) fixing the operator is the fastest path to writes because the plugin only writes resources the operator reads; (2) the plugin is not a back door — it never talks to MySQL; (3) the cooldown gates the planned path too, with onCooldown defaulting to reject and defer as the alternative; (4) hand-editing MySQL is fought by the sidecar's topology-mismatch rule. An answer that says 'promote by hand with the plugin' without noticing that the plugin needs a live operator has missed the decision.
The decision rule is: if the operator is coming back and there is no writable site, waiting wins, because the hand-driven path cannot start any sooner; if the operator is not coming back, restoring it is the promotion. The trap is treating kubectl bloodraven promote as an operator-independent break-glass. It writes an annotation and returns — the plugin only writes resources the operator already reads and never talks to MySQL, which is what keeps it from being a back door around the operator's logic (objective 12).
Sample answer
Roll the operator image back and get the operator running — that is the promotion path. kubectl bloodraven promote only writes the planned-failover annotation on the group, and the operator is the thing that reads it, so with the operator crashlooping the annotation just queues an intent and nothing promotes. Once the operator is up it will detect the dead primary in about 6 s and promote, and the cooldown does not lengthen the write outage further than its remaining four minutes; if I need the manual path to survive that window I set spec.plannedFailover.onCooldown: defer so the request is parked and retried at cooldown expiry rather than rejected outright. What I do not do is clear read_only on the replica by hand: with the operator down the sidecar still holds its last authoritative activeSite, sees a writable site that disagrees with it, and fences it straight back.
A full-credit answer shows
A strong answer covers: (1) fixing the operator is the fastest path to writes because the plugin only writes resources the operator reads; (2) the plugin is not a back door — it never talks to MySQL; (3) the cooldown gates the planned path too, with onCooldown defaulting to reject and defer as the alternative; (4) hand-editing MySQL is fought by the sidecar's topology-mismatch rule. An answer that says 'promote by hand with the plugin' without noticing that the plugin needs a live operator has missed the decision.
The decision rule is: if the operator is coming back and there is no writable site, waiting wins, because the hand-driven path cannot start any sooner; if the operator is not coming back, restoring it is the promotion. The trap is treating kubectl bloodraven promote as an operator-independent break-glass. It writes an annotation and returns — the plugin only writes resources the operator already reads and never talks to MySQL, which is what keeps it from being a back door around the operator's logic (objective 12).
Show answer
Answer: Nothing definite — the write may already have been applied, so the state must be re-read before deciding anything
Cancelling the context tears down the client connection; it does not roll back a write the server already applied, so the only honest reading is 'unknown — go and look'. Bloodraven shipped a bug from the first option: treating the returned error as a failure made the monitor re-fence a site it had just successfully promoted. The second is wrong because nothing is queued — the statement either executed on the server or it did not, and the client no longer knows which. The fourth is the most expensive mistake, because an error tells you you stopped waiting, not that the remote side did nothing; acting on the other site while the first may have applied the write is how you turn ambiguity into divergence (objective 12).
Cancelling the context tears down the client connection; it does not roll back a write the server already applied, so the only honest reading is 'unknown — go and look'. Bloodraven shipped a bug from the first option: treating the returned error as a failure made the monitor re-fence a site it had just successfully promoted. The second is wrong because nothing is queued — the statement either executed on the server or it did not, and the client no longer knows which. The fourth is the most expensive mistake, because an error tells you you stopped waiting, not that the remote side did nothing; acting on the other site while the first may have applied the write is how you turn ambiguity into divergence (objective 12).