Planned failover: moving the primary on purpose
One annotation, twelve phases, and an RPO of zero by construction. The only path on which Bloodraven drains your application's connections — and the only phase from which it will roll back.
By the end of this topic you can
- Trigger a planned failover with the annotation and follow the phases through to
Succeeded - Explain why planned failover is RPO 0 by construction and what the lag gate actually compares
- Choose the rollback behaviour for a lag gate that never closes
playground is healthy. iad is writable, pdx and reader are read-only, and the counter app is still
writing through the -primary Service with the bounded-lifetime pool you fixed in the last topic.
Nothing is broken and nobody is paged — but iad’s nodes are being rebuilt on Tuesday and the primary
has to move. This is the one primary move you get to schedule, and the only one where Bloodraven shuts
your application’s connections down before it takes the write endpoint away.
That is why this topic sits here rather than beside emergency failover. The emergency path does try to kill application connections on the old primary — it is step 2 of the sequence — but it is best-effort, single-pass, and skipped when the old primary is unreachable, which is the failure mode that made you open this unit in the first place. An autonomous sidecar self-fence has no operator-side drain at all. Only planned failover actually drains. Nothing else does.
The trigger
One annotation, bloodraven.shipstream.io/planned-failover. Its value is a bare site name, or a site
name followed by :key=value overrides; maxLagWait is the only supported key, and an unknown key is
rejected outright so a typo cannot quietly run with defaults. The annotation is consumed and cleared on
the next reconcile, like reclone-site.
kubectl -n bloodraven-playground annotate mysqlfailovergroup playground bloodraven.shipstream.io/planned-failover=pdxmysqlfailovergroup.shipstream.io/playground annotatedkubectl -n bloodraven-playground get mysqlfailovergroup playground -o jsonpath='{.status.plannedFailover.phase}'WaitingForLagkubectl -n bloodraven-playground get mysqlfailovergroup playground -o jsonpath='{.status.plannedFailover.phase} {.status.plannedFailover.transactionsLost}'Succeeded 0Leave the counter app running while you do this. That is the point of the exercise.
The phases
The reconciler advances the state machine by one step per reconcile, so an operator restart always lands on a well-defined observable state. The complete list, in order:
"", Pending, Deferred, Validating, Draining, WaitingForLag, WaitingForDragonflySync,
PromotingDragonfly, Promoting, Resuming, Succeeded, Failed.
Deferred is only entered when the cooldown blocks you and onCooldown is defer.
WaitingForDragonflySync and PromotingDragonfly are skipped entirely when spec.dragonfly is unset or
disabled — the Dragonfly topics later in this unit take those two apart.
Why it is RPO 0 — by construction
“Planned failover is safe” is a sentence people repeat without knowing why, so be exact. It is not luck and it is not a lag threshold. It is an ordering.
At Draining the operator sets super_read_only = ON on iad and then reads iad’s
GTID_EXECUTED, recording it in status.plannedFailover.sourceGtidAtFence. Fence first, snapshot
second. A fenced primary accepts no new client writes, so the set pdx must catch up to cannot grow
underneath the gate. That is the whole argument.
At WaitingForLag the operator polls pdx’s GTID_EXECUTED and asks one question: does it contain
sourceGtidAtFence? The check is a genuine GTID-set superset test — gtidContains(targetGtid, cur.SourceGtidAtFence), which resolves to super.Contains(sub). Not seconds. Promotion runs only after
that returns true, so status.plannedFailover.transactionsLost is 0 on a successful switchover by
construction. The field exists for symmetry with the emergency path’s accounting, not because a clean
switchover can produce a number.
You met spec.replication.maxLagSeconds in Unit 2 and it looks like it belongs here. It does not. It
drives exactly one thing: the ReplicationLagging Degraded condition. It is not a promotion gate
anywhere — the emergency path promotes a replica that is past the threshold anyway, because no writable
site at all is nearly always worse — and WaitingForLag never consults it. Seconds are a bad gate on
their own merits: Seconds_Behind_Source compares last-executed against last-downloaded relay event,
so it reads 0 when the IO thread has stalled.
Draining is a deadline, not a barrier
The defaults on spec.plannedFailover: maxLagWait 5m, drainTimeout 30s, onCooldown reject.
Two things happen at Draining. The source’s primary role label is stripped to fenced, which matches
neither the -primary selector nor the -replicas selector, so the write Service sheds the endpoint and
new connections stop arriving. And the reconciler repeatedly kills the connections already open.
Then the sharp edge. When the budget runs out with connections still on the source, the operator logs
drain budget exhausted after %s with %d connection(s) remaining on %q; proceeding — and proceeds. A
stuck client is not allowed to block a switchover indefinitely. That is a defensible choice, and it is a
decision you have just inherited: the connections that outlive the drain are exactly the ones from the
last topic, open against a demoted primary, passing every validation query, serving stale reads until
something tries to write. Your drainTimeout and your pool’s maximum connection lifetime are two halves
of one setting. A pool that holds connections for ten minutes against a thirty-second drain makes the
drain decoration.
Rollback exists in exactly one phase
| Failure | Phase | Source afterwards | Who resolves it |
|---|---|---|---|
CooldownActive | Validating | Never fenced | Nobody — re-annotate later |
LagTimeout, InvalidGTID | WaitingForLag | Unfenced, still the primary | Nobody — nothing was lost |
ExecuteFailed | Promoting | Still fenced | A human |
| status write failure | Resuming | Still fenced | A human |
Draw the operational conclusion. A lag gate that never closes is the good failure: iad comes back
writable, the counter app resumes, and you have lost nothing but the fenced window. So when maxLagWait
expires against a pdx that is genuinely behind, let the rollback fire and fix the lag first. Raising
maxLagWait to force the gate closed does not make pdx catch up any faster — it only lengthens the
window in which your source is fenced and your writes are refused.
Two refusals worth recognising before you meet them. A planned failover aimed at reader is hard-refused
with only primary-candidate sites may be promoted — the promotability rule from Unit 1, enforced again
at this entry point. And the anti-flap cooldown gates planned admission just as it gates the automatic
path, with reason CooldownActive; because onCooldown defaults to reject, a planned failover
attempted soon after an emergency one is refused and the annotation cleared, not queued. Set
onCooldown: defer if you would rather it wait in Deferred and fire itself at cooldown expiry.
What you have now
You can move playground’ primary on purpose, follow status.plannedFailover.phase to Succeeded, read
transactionsLost: 0, and argue the superset gate to anyone who thinks it is a lag threshold. You can
also say which failures hand you the cluster back intact and which hand you a fenced primary and a
pager. Everything above assumed MySQL was the only thing moving. Next: what happens when there is a
Dragonfly beside it, and what Bloodraven does and does not promise about it.
Flashcards
Which annotation triggers a planned switchover, and what may its value contain?
bloodraven.shipstream.io/planned-failover. The value is a bare site name, or a site name plus :key=value overrides — maxLagWait is the only supported key, and an unknown key is rejected.
Recite the planned-failover phases in order.
"", Pending, Deferred, Validating, Draining, WaitingForLag, WaitingForDragonflySync, PromotingDragonfly, Promoting, Resuming, Succeeded, Failed.
status.plannedFailover.sourceGtidAtFence — what is it?
The source primary's GTID_EXECUTED, recorded immediately after super_read_only=ON took effect on it.
Why does the operator fence the source before snapshotting its GTID, rather than after?
Because a fenced primary accepts no new client writes, so the GTID set the target must catch up to cannot grow underneath the gate.
What does status.plannedFailover.transactionsLost read after a successful planned switchover?
0, by construction. The field is retained for symmetry with the emergency path's data-loss accounting.
spec.plannedFailover.maxLagWait — default value and what it bounds.
5m. It bounds time spent in WaitingForLag before the state machine rolls back to the source.
spec.plannedFailover.drainTimeout — default value and what it bounds.
30s. It bounds how long the fenced source gets to shed application connections during Draining.
spec.plannedFailover.onCooldown — default value and its effect.
reject: a cooldown hit at Validating stamps Failed{CooldownActive} and clears the annotation, so an admin must re-annotate after the cooldown expires.
What changes if you set onCooldown: defer?
The request enters the Deferred phase with retryAfter stamped, keeps the annotation, and validation is retried automatically at cooldown expiry.
Verbatim refusal when a planned failover targets a role: read-only site.
only primary-candidate sites may be promoted.
What does entering Draining do to the source's -primary Service endpoint?
The source's role label is stripped to fenced, which matches neither the -primary nor the -replicas selector, so the write Service sheds its endpoint.
Confusable pair: maxLagSeconds versus maxLagWait.
spec.replication.maxLagSeconds (default 300) drives only the ReplicationLagging Degraded condition; spec.plannedFailover.maxLagWait (default 5m) is the timeout on the GTID superset gate.
Quiz
Show answer
Answer: pdx's GTID_EXECUTED to contain the source's GTID_EXECUTED snapshot taken at the fence.
The gate is a true GTID-set superset test: the reconciler polls the target's GTID_EXECUTED and advances only when it contains status.plannedFailover.sourceGtidAtFence. maxLagSeconds is not it — that field drives exactly one thing, the ReplicationLagging Degraded condition, and is never a promotion gate on either the planned or the emergency path. A streak of Seconds_Behind_Source = 0 is not it either, and is a bad signal on its own merits: it compares last-executed against last-downloaded relay event, so it reads 0 when the IO thread has stalled. The relay-log drain belongs to the emergency sequence, not to planned failover's lag gate. (objective 8)
The gate is a true GTID-set superset test: the reconciler polls the target's GTID_EXECUTED and advances only when it contains status.plannedFailover.sourceGtidAtFence. maxLagSeconds is not it — that field drives exactly one thing, the ReplicationLagging Degraded condition, and is never a promotion gate on either the planned or the emergency path. A streak of Seconds_Behind_Source = 0 is not it either, and is a bad signal on its own merits: it compares last-executed against last-downloaded relay event, so it reads 0 when the IO thread has stalled. The relay-log drain belongs to the emergency sequence, not to planned failover's lag gate. (objective 8)
Show answer
Answer: False
The reversal: the drain is a deadline, not a barrier. When the budget is exhausted with connections remaining, the operator logs drain budget exhausted after %s with %d connection(s) remaining on %q; proceeding and proceeds to promotion, so a stuck client cannot block a switchover indefinitely. Nothing is unfenced and nothing aborts. That is why your pool's maximum connection lifetime has to be shorter than the drain budget — connections that outlive the drain are the ones that go on serving stale reads against a demoted primary. (objective 7)
The reversal: the drain is a deadline, not a barrier. When the budget is exhausted with connections remaining, the operator logs drain budget exhausted after %s with %d connection(s) remaining on %q; proceeding and proceeds to promotion, so a stuck client cannot block a switchover indefinitely. Nothing is unfenced and nothing aborts. That is why your pool's maximum connection lifetime has to be shorter than the drain budget — connections that outlive the drain are the ones that go on serving stale reads against a demoted primary. (objective 7)
Show answer
Answer: ExecuteFailed while in Promoting, when promotion of the target site failed outright.
Rollback — unfencing the source — exists only in WaitingForLag. Both failures reachable there, LagTimeout and InvalidGTID, unfence the source and leave it the active primary with nothing lost. CooldownActive fires at Validating, before the source was ever fenced, so there is nothing to undo. ExecuteFailed in Promoting is past that boundary: it stamps Failed without unfencing, with a message saying manual recovery is required. A failure in Resuming behaves the same way. (objective 9)
Rollback — unfencing the source — exists only in WaitingForLag. Both failures reachable there, LagTimeout and InvalidGTID, unfence the source and leave it the active primary with nothing lost. CooldownActive fires at Validating, before the source was ever fenced, so there is nothing to undo. ExecuteFailed in Promoting is past that boundary: it stamps Failed without unfencing, with a message saying manual recovery is required. A failure in Resuming behaves the same way. (objective 9)
Show answer
Answer:
Do not raise it — fix the lag first. The rollback is the good outcome: LagTimeout in WaitingForLag unfences iad, which stays the active primary with nothing lost, so both failed attempts cost only the fenced window. Raising maxLagWait to 30m does not make pdx apply relay logs any faster; it only extends the period during which iad is fenced with super_read_only=ON and the counter app's writes are being refused. Kill or wait out the batch job, confirm pdx is applying, then re-annotate with the default 5m and the gate will close on its own.
A full-credit answer shows: A strong answer covers: (1) rollback in WaitingForLag unfences the source, so the failure is safe and costs no data; (2) maxLagWait is a timeout, not a throttle — it does not accelerate catch-up; (3) the real cost of a longer wait is a longer fenced source with writes refused; (4) the correct action is to remove the source of the lag and retry.
The lag gate that never closes is the failure mode you want, because it is the only one that hands the cluster back intact. Lengthening maxLagWait trades a safe, reversible refusal for a longer write outage on a primary that is already fenced, without changing whether the target can catch up. (objective 9)
Sample answer
Do not raise it — fix the lag first. The rollback is the good outcome: LagTimeout in WaitingForLag unfences iad, which stays the active primary with nothing lost, so both failed attempts cost only the fenced window. Raising maxLagWait to 30m does not make pdx apply relay logs any faster; it only extends the period during which iad is fenced with super_read_only=ON and the counter app's writes are being refused. Kill or wait out the batch job, confirm pdx is applying, then re-annotate with the default 5m and the gate will close on its own.
A full-credit answer shows
A strong answer covers: (1) rollback in WaitingForLag unfences the source, so the failure is safe and costs no data; (2) maxLagWait is a timeout, not a throttle — it does not accelerate catch-up; (3) the real cost of a longer wait is a longer fenced source with writes refused; (4) the correct action is to remove the source of the lag and retry.
The lag gate that never closes is the failure mode you want, because it is the only one that hands the cluster back intact. Lengthening maxLagWait trades a safe, reversible refusal for a longer write outage on a primary that is already fenced, without changing whether the target can catch up. (objective 9)
Show answer
Answer: The anti-flap cooldown gates planned admission too, and onCooldown defaults to reject.
The same anti-flap cooldown that gates automatic promotion is evaluated at Validating for planned failover, against the same durable failover record. With the default onCooldown: reject the request is refused terminally and the annotation is cleared — so wait out the cooldown and re-annotate, or set onCooldown: defer to have it wait in Deferred and retry itself at expiry. The annotation was well-formed; a malformed one fails with InvalidAnnotation, not CooldownActive. A target that is unreachable, writable or not replicating is refused with TargetUnhealthy, a different reason. And nothing is queued by default — that is precisely what defer opts you into. (objective 7)
The same anti-flap cooldown that gates automatic promotion is evaluated at Validating for planned failover, against the same durable failover record. With the default onCooldown: reject the request is refused terminally and the annotation is cleared — so wait out the cooldown and re-annotate, or set onCooldown: defer to have it wait in Deferred and retry itself at expiry. The annotation was well-formed; a malformed one fails with InvalidAnnotation, not CooldownActive. A target that is unreachable, writable or not replicating is refused with TargetUnhealthy, a different reason. And nothing is queued by default — that is precisely what defer opts you into. (objective 7)