Planned failover

Planned failover is the admin-triggered graceful switchover path. It is the operational-hygiene counterpart to emergency failover: use it for maintenance windows, rolling kernel upgrades on the active site's nodes, or any time you want to move the primary role to a specific replica without relying on the operator's automatic detection.

TL;DR

kubectl annotate mysqlfailovergroup orders \
  bloodraven.shipstream.io/planned-failover=pdx

The operator claims the group's sites, fences the current primary, waits for pdx to apply the fenced source's whole GTID set, promotes it, publishes DNS, and records the failover. When spec.dragonfly.enabled=true, the same request also waits for the target Dragonfly replica to catch up and promotes it before the MySQL promotion. A successful run ends with .status.plannedFailover.phase: Succeeded and .status.plannedFailover.result.outcome: Succeeded.

Only PrimaryCandidate sites are valid targets. DisasterRecovery and ReadOnly followers are never promoted, used for active DNS, or involved in node-taint handoff. Readers remain direct followers and application read pools after the switchover.

Why not kubectl exec?

Manual promotion via three kubectl exec ... SET GLOBAL ... commands (see Operations) works but has four real problems:

1. No atomicity — a dropped connection or typo mid-sequence leaves the cluster fenced-on-both-sides or promoting a replica that has not drained its relay logs.
2. No lag gate — nothing mechanical checks that the target is caught up before you flip read_only=0.
3. No audit trail — bloodraven_failovers_total does not increment, no Event fires, no CR status reflects what you did.
4. Bypasses the anti-flap cooldown — a switchover immediately before a real emergency exposes the cluster to a window where the operator refuses to take automatic action.

The annotation API closes all four.

Strict by design

A planned failover is optional, so it is strict. It moves the primary role only when it can prove that doing so is safe. Otherwise it waits (before anything is fenced) or stops, and it never writes to the target on a guess:

• Positive read-only evidence. When the claim is granted, when the source is fenced, and again immediately before the target's super_read_only is cleared, every other PrimaryCandidate and every DisasterRecovery site must answer, and answer read-only. A site that does not answer is treated as a site that might be writable: the sidecar does not self-fence on lease expiry while it can still reach a peer, so a site the operator cannot reach may still be accepting writes. An unreadable ReadOnly reader does not block the run; it is recorded as the warning UnreachablePeers. A writable reader blocks it like any other writable site.
• A down DR site blocks planned failovers. This follows from the rule above and is the deliberate availability cost. While a PrimaryCandidate or DisasterRecovery site is down, a request waits in admission with waitReason: PeerUnreadable and ends NotStarted with reason: AdmissionTimeout after 2 minutes. Bring the site back, then re-annotate. A site cannot be removed from spec.sites; to drop one, replace the group (see Replacing a site). Emergency failover is not affected.
• No guessing after a possible write. Once super_read_only=OFF may have reached the target, a run that cannot explain what it finds does not let automatic split-brain resolution pick a winner. It ends ManualRecoveryRequired and holds automation until a human acknowledges it (see The automation hold).
• A run never ends Succeeded while a foreign writable site exists.

Lifecycle

            ┌──── Deferred (owns nothing; annotation kept) ────┐
 annotate ──┤                                                  ▼
            └──► Admitting (owns nothing) ──claim granted──► Fencing ──► Draining ──► WaitingForLag
                     │                                         │                         │
                     ▼ reject / deadline                       │            ┌────────────┴─────────────┐
               Failed{NotStarted}                              │            ▼ Dragonfly enabled        │
                                                               │   WaitingForDragonflySync             │
                                                               │            ▼                          │
                                                               │   PromotingDragonfly ─────────────────┤
                                                               │                                       ▼
                                                               │                                   Promoting
                                                               │                                       │ confirmed
                                                               ▼ nothing written to the target         ▼
                                                          RollingBack ──► Failed{RolledBack |      Publishing ──► Succeeded
                                                                          SourceLeftFenced}        (claim released)
              Promoting ── a write may have been sent; target lost or unconfirmable ──► Failed{PromotedUnconfirmed}
              Promoting ── a foreign writer, foreign transactions, or a writable target with a channel ──► Failed{ManualRecoveryRequired} + hold

WaitingForDragonflySync and PromotingDragonfly are skipped when Dragonfly was not enabled when the request was accepted.

Every transition is one durable status write, conditioned on the resourceVersion of an uncached read, and every side effect runs only after the record that announces it has committed. An operator restart therefore always lands on a record whose side effect is either done or provably not started, and the next pass confirms it live before going on: it logs planned failover: confirming recorded side effect. The block is retained until a new annotation replaces it; a new annotation replaces only a terminal block.

The claim

The claim is what a planned failover owns while it runs. It is held in one slot per group in the operator process, shared by every topology-manager generation, and granted by the topology manager against the other actors that can change the sites. While a run holds the claim (from Fencing until its promotion is confirmed, recorded and published, or a terminal record is durable), the topology manager defers:

• emergency failover, split-brain resolution and the primary re-assert (cross-site action deferred: planned failover in progress);
• ordered updates, keyring rotations and bootstraps;
• old-primary recovery, its connection drain and replication-source convergence;
• DNS reconciliation (the run publishes DNS itself before it releases);
• Dragonfly emergency and rollout promotions.

Application writes on whichever site is writable continue, and writable DisasterRecovery or ReadOnly sites are still fenced. A request that is only waiting (Admitting without a grant, or Deferred) owns nothing and defers nothing.

The claim is released only by the run that owns it, and only after one of: a promotion confirmed live, recorded in the topology manager, published to DNS and marked in the annotation bloodraven.shipstream.io/planned-failover-confirmed: <runID>:since=<confirmation time> (the Publishing write follows without the claim; if the annotation cannot be written, the claim is kept and the tail retries, logging planned failover: could not record the promotion's confirmation; keeping the claim), a durable terminal record, a status write proven not to have committed, or a durable terminal marker. An operator restart (or topology-manager replacement) before the Publishing write lands first looks at the run's planned-failover-confirmed annotation and the durable failover record. If the annotation names the run, or the record is the run's own, recognised by content (target, the fenced set as its watermark, and a time not before the promotion intent, all taken from the record the operator would restore), the promotion was confirmed and recorded and the claim had been released: it is not hydrated back, the run writes Publishing without it (planned failover: promotion already recorded; publishing without the claim), and the topology manager reconciles DNS. The annotation counts even after a later emergency failover replaced the run's failover record, and the new topology manager applies it as the run's promotion record (a newer record wins). Otherwise (the record was not durable yet) the restart hydrates the claim from the Promoting block, re-derives the confirmation from the live target, records the promotion, and releases again. Once the claim is released, Dragonfly emergency follow-ups, rollouts and snapshot upgrades no longer treat the run as the owner although status.plannedFailover.phase still reads Promoting: the claim slot decides, not the phase. Before the topology manager is registered after a restart, the status form decides instead: a run that still needs the claim, or a durable hold, owns Dragonfly. A released run whose Publishing write keeps failing is ended by the terminal marker after 10 seconds rather than 2 minutes, so a later failover that replaces the run's record cannot leave a restart holding the group for a run that already succeeded. A ManualRecoveryRequired run does not release it: the claim becomes an automation hold. The release is logged as planned failover: claim released with a why field.

After an operator restart the claim is hydrated from an uncached read when the topology manager starts, and the run re-proves its authority before its next side effect: the durable failover record still matches claimBaseline (an invalid copy, such as a malformed annotation or a target no longer in spec.sites, beside an unchanged valid copy is not a change; with no valid copy left the check fails closed), the fence is still the run's own, and no other site is writable. Within one process the claim and the leader election are re-checked immediately before every statement that changes a role, including the super_read_only=OFF right after the pre-write gate's probes. A source read that fails during the re-proof is a loss-window sample like any other (10 seconds before SourceLost), and an unreadable ReadOnly site is no lack of evidence. If the run's source or target is removed from spec.sites while it runs, the run ends at once without MySQL access (SourceLeftFenced{UnknownSite}, or PromotedUnconfirmed from Promoting) and releases the claim.

Admission

An Admitting pass first re-validates the request (a failure ends it NotStarted with that reason): the target is a known PrimaryCandidate, is not already active, is not mid-keyring-rotation (KeyringRotation), is a healthy read-only replica without a divergence report (DivergentTarget, TargetUnhealthy), is not awaiting its post-restore reclone (TargetUnhealthy), no in-place restore or ordered update is in progress (ConcurrentOperation), the cooldown has passed (CooldownActive) and the durable failover record is readable (InvalidFailoverState). A cooldown found by an Admitting run ends it CooldownActive, with one exception: a request that waited in Admitting before its first validation (AuthorityUnsettled, or a refused deployment admission reservation) and still carries its annotation becomes Deferred with CooldownActive under onCooldown: Defer, its annotation kept for the retry.

Validation is skipped while a failover is in flight, because the status it reads is not yet meaningful: a target that emergency failover is promoting has stopped replicating, and validating it would end the run TargetUnhealthy seconds before it becomes the primary. Authority is unsettled while status.activeSite is empty, or while the durable failover record is newer than the run's startTime (for a new request with no run yet, recorded within the last 2 minutes) and names a site other than status.activeSite. Meanwhile an Admitting run waits with admission.waitReason: AuthorityUnsettled, a Deferred run moves to Admitting with that wait reason and a fresh 2-minute deadline, and a new request whose validation refusal depends on status (anything but InvalidAnnotation, UnknownSite or TargetNotPromotable) is admitted into the same wait. A new request that arrives while the deployment admission reservation is refused is handled the same way: a request-only refusal ends it NotStarted at once, and anything else is admitted into Admitting with waitReason UpdateInFlight, RestoreInFlight, AuthorityUnsettled or ClaimHeld, a PlannedFailoverAdmissionDelayed Event and the 2-minute deadline. Once authority settles the run is validated: a run whose target was just promoted ends NotStarted{AlreadyActive}, and one aimed elsewhere proceeds or is refused against the new primary. If authority does not settle within the deadline the run ends AdmissionTimeout.

Then it asks for the claim. The grant probes every site live: the source must answer writable with a readable @@server_uuid; the target must answer read-only with both replication threads running, no applier error, and replicating from the source; every other PrimaryCandidate and DisasterRecovery site must answer read-only, and no site may be writable beside the source. Transient conditions wait; the latest is admission.waitReason, written (with a PlannedFailoverAdmissionDelayed Event) only when it or its detail changes (another claimant, say):

A reason that cannot pass by waiting rejects the run at once instead: UnknownSite, TargetNotPromotable, AlreadyActive, KeyringRotation (the target is mid-rotation), DivergentTarget, TargetUnhealthy (a reclone-pending target), and ManualRecoveryPending (an automation hold stands). A planned failover is not an override for a site awaiting its post-restore reclone.

While the latest run ended SourceLeftFenced with its own attributable fence and that fence has not healed (the source is not observed writable, and no failover was recorded after the run started), a new request is refused with a PlannedFailoverRejected Warning only: planned failover refused: planned failover <runID> left source "<site>" fenced and the fence has not healed yet; the topology manager re-asserts it, or unfence it by hand, then re-annotate. The annotation is removed and status.plannedFailover keeps the old run, because a restarted operator names the source for the re-assert from that block.

Spec-level defaults

A deployment hold defers admission regardless of onCooldown. admission.deferReason: DeploymentHold names the operation and instance in the message, and admission.retryTime is the earliest hold expiry. Renewal can extend the wait. A hold found while a run is already Admitting (normally impossible: a hold lease is refused while a run is active) ends it NotStarted with ConcurrentOperation. Ordered update and restore-in-place honor the same holds, while emergency failover and fencing never consult them. A hold does not relax any normal admission requirement when it expires. Use the explicit revocation procedure, not Lease deletion, when approved maintenance must interrupt a deployment.

Cluster-wide knobs live on the CR so common overrides don't have to be spelled into every kubectl annotate:

spec:
  plannedFailover:
    maxLagWait: 5m     # max time to wait for the target to catch up before rolling back
    drainTimeout: 30s  # upper bound on KillAppConnections polling after super_read_only=ON
    onCooldown: Reject # one of: Reject (default) or Defer
  dragonfly:
    plannedFailover:
      maxSyncWait: 30s      # Dragonfly target sync / REPLTAKEOVER budget
      onSyncTimeout: Proceed # one of: Proceed (default) or Fail

All fields are optional, and a zero or out-of-range timer is refused at admission rather than replaced by its default; omitting plannedFailover is equivalent to {maxLagWait: 5m, drainTimeout: 30s, onCooldown: Reject}. Omitting dragonfly.plannedFailover is equivalent to {maxSyncWait: 30s, onSyncTimeout: Proceed} when Dragonfly is enabled. The effective values are snapshotted into status.plannedFailover.settings when the request is accepted (again when a Deferred run resumes), so a spec edit does not change a running switchover.

• maxLagWait — how long WaitingForLag polls the target before rolling back, 1s to 1h. The source is fenced while the run waits, so this bound also caps the write outage. Set shorter than the default when you know the cluster is caught up and want to fail fast.
• drainTimeout — upper bound on the Draining phase's KillAppConnections loop, 1s to 10m. After super_read_only=ON, the operator kills stragglers every second until the count reaches zero or the budget expires; advancing to WaitingForLag happens either way (drain.remainingConnections records what was left) so a stuck client cannot block the switchover. Surviving clients cannot write: they get ER_OPTION_PREVENTS_STATEMENT.
• onCooldown — what happens when admission finds the anti-flap cooldown active:
  • Reject (default): stamp Failed{NotStarted, CooldownActive} and clear the annotation. The admin must re-annotate after the cooldown expires.
  • Defer: stamp Deferred, keep the annotation in place, and retry admission automatically at admission.retryTime. Useful when you want to queue a switchover immediately after an emergency event without manually waiting out the cooldown. Removing the annotation while Deferred ends the run NotStarted with Canceled.
• dragonfly.plannedFailover.maxSyncWait — how long WaitingForDragonflySync waits for the target Dragonfly replica to reach the source offset. The same duration is used as the REPLTAKEOVER timeout. Dragonfly takes that timeout in whole seconds, so it is rounded up to the next second. 1s to 10m. The operator waits up to 12s beyond it for the reply, because the target Dragonfly allows its master 10s past the budget.
• dragonfly.plannedFailover.onSyncTimeout — what happens when Dragonfly sync or promotion cannot prove session preservation (including a source offset that could not be captured, dragonfly.sourceOffsetState: Unavailable):
  • Proceed (default): continue to the MySQL promotion and record status.plannedFailover.dragonfly.sessions: Lost with dragonfly.reason DragonflySyncTimeout or DragonflyPromotionFailed.
  • Fail: roll back before the MySQL promotion (rollback.reason DragonflySyncTimeout or DragonflyPromotionFailed), release the MySQL fence, and leave status.activeSite unchanged.

Per-request override

The annotation value may carry a maxLagWait override. The value follows the one grammar of Bloodraven's one-shot action annotations, <subject>(:<key>=<value>)*: the subject is the target site, each pair splits at its first =, values use only A-Za-z0-9._/+- (so a value never contains : or =), and each key appears once:

kubectl annotate mysqlfailovergroup orders \
  bloodraven.shipstream.io/planned-failover=pdx:maxLagWait=30s

Useful when you know the target is caught up and want to fail fast rather than wait five minutes for the default timeout. The override takes the same 1s to 1h bound as spec.plannedFailover.maxLagWait. An unparseable value (an unknown or repeated key, an empty pair, a value outside the charset, whitespace inside the value, or overrides with no site such as :maxLagWait=5s) or an out-of-range maxLagWait is rejected with a PlannedFailoverRejected Event and removed; no status block is written for it. An annotation for the already-active site is removed with a PlannedFailoverSkipped Event. While a run is live (Admitting or later), a leftover copy of its own request (the same target, with the same or no maxLagWait) is removed silently, and any other request is removed with a PlannedFailoverRejected Event (planned-failover annotation ignored: a previous planned failover is still running). A Deferred run instead takes a new request as its target when it resumes. The operator removes only the annotation value it evaluated: a request written in the meantime is left for the next pass.

Status

Inspect progress:

kubectl get mysqlfailovergroup orders \
  -o jsonpath='{.status.plannedFailover}{"\n"}'
kubectl bloodraven status orders

Example Succeeded block:

status:
  plannedFailover:
    runID: 6f1c2a4e-4b7d-4a1f-9a0e-2d5b8c3f7e10
    revision: 10
    phase: Succeeded
    targetSite: pdx
    sourceSite: iad
    settings:
      maxLagWait: 5m0s
      drainTimeout: 30s
      dragonfly: {maxSyncWait: 30s, onSyncTimeout: Proceed}
    startTime: "2026-04-20T14:32:00Z"
    claimTime: "2026-04-20T14:32:01Z"
    claimBaseline: {time: "2026-04-19T09:10:44Z", target: iad}
    admission: {deadlineTime: "2026-04-20T14:34:00Z"}
    fence:
      state: Applied
      intentTime: "2026-04-20T14:32:01Z"
      sourceServerUUID: 3e11fa47-71ca-11e1-9e33-c80aa9429562
      applyTime: "2026-04-20T14:32:01Z"
      gtidSet: "3e11fa47-71ca-11e1-9e33-c80aa9429562:1-9182731"
    drain: {startTime: "2026-04-20T14:32:01Z", completionTime: "2026-04-20T14:32:03Z", remainingConnections: 0}
    lagWait: {startTime: "2026-04-20T14:32:03Z", catchUpTime: "2026-04-20T14:32:04Z"}
    dragonfly:
      syncWaitStartTime: "2026-04-20T14:32:04Z"
      sourceOffsetState: Captured
      sourceOffset: 81723
      syncWaitSeconds: 3
      takeover: {state: Succeeded, intentTime: "2026-04-20T14:32:07Z", completionTime: "2026-04-20T14:32:08Z", method: ReplTakeover, targetOffset: 81723}
      sessions: Preserved
    promotion:
      state: Confirmed
      intentTime: "2026-04-20T14:32:08Z"
      targetServerUUID: 7a1d0c22-71ca-11e1-9e33-c80aa9429562
      confirmationTime: "2026-04-20T14:32:09Z"
    dns: {state: Applied, attemptTime: "2026-04-20T14:32:09Z"}
    result: {outcome: Succeeded, completionTime: "2026-04-20T14:32:10Z"}
    message: promoted "pdx"

The duration is result.completionTime - startTime (kubectl bloodraven status prints it). Nothing is lost by construction: the gate promotes only a target that holds exactly the fenced set.

Outcomes and reasons

result.outcome says what the cluster was left in:

result.reason says why:

result.warnings qualify an outcome:

Deadlines and worst-case write outage

"Write outage" is the time during which no site accepts application writes, from the run's fence until a writable primary exists again. The figures assume the default settings and a topology-manager emergency debounce of about 6 seconds.

Rollback

RollingBack decides every step live, on each attempt, until it finishes or its 30-second deadline passes:

1. If a promotion intent exists and the run cannot prove that nothing was written to the target, the target must answer and hold nothing outside the fenced set. If it does not answer within the 10-second target-loss window (never past the rollback deadline; the source stays fenced meanwhile), the run ends PromotedUnconfirmed; if it holds other transactions, ManualRecoveryRequired{ExecuteFailed}.
2. A source that already answers writable ends RolledBack with no unfence sent, once a read-only target found at super_read_only=OFF is set back to super_read_only=ON and read back (while that fails, or the target cannot be read, the rollback retries; at its deadline it ends RolledBack with TargetSuperReadOnlyNotRestored and TargetReplicationNotRestored, the target's channel untouched, and no hold), unless the target is writable too and the run may have written to it (a promotion intent was recorded): that is two writers, and the run ends ManualRecoveryRequired{ForeignWritable} with the automation hold (message the rollback found another writable site beside the writable source). Another writable site the run never touched is logged (planned failover: rollback found another writable site beside the source) and left to split-brain handling; the run stays RolledBack.
3. If any other site answers writable, the source is left fenced (SourceLeftFenced): unfencing it would make two writable primaries. A site that merely does not answer does not stop the unfence.
4. A source lost for 10 seconds (SourceLost) is left fenced with no unfence retries; emergency failover takes over.
5. A fence that cannot be attributed to the run is not cleared (SourceLeftFenced with FenceNotAttributable). The fence is the run's when this process recorded sending it to that server, or when the durable intent names the server's current @@server_uuid.
6. Otherwise a read-only target found at super_read_only=OFF is first set back to super_read_only=ON and read back; while that fails the rollback retries with the source fenced (WARN planned failover: could not restore the target's super_read_only) and ends SourceLeftFenced at its deadline. The target's replication channel is then restored if the promotion had stopped or reset it, but re-created only toward a verified source (its pinned @@server_uuid matches, or it is the run's own writable source), and the source is unfenced on a pinned session to the same server (super_read_only=OFF, read_only=OFF) and read back. A confirmed unfence ends RolledBack.

A writable source whose @@server_uuid differs from the one the fence recorded adds the warning SourceReplaced and leaves the target for a reclone (TargetReplicationNotRestored). Every SourceLeftFenced rollback end reports TargetReplicationNotRestored when the target has no running channel.

The rollback never touches a site other than the source and the target.

Healing a fenced source

A SourceLeftFenced run leaves the source read-only. Whether that heals on its own depends on why:

The messages of the FenceNotAttributable and LegacyRunRefused ends carry this advice: check which site is writable; if none is, unfence the site with the most complete GTID set by hand (confirm no other site is writable first); re-annotate only once a site is writable. For the manual cases, check which site is writable first. If one is, leave the source fenced: old-primary recovery repoints it as a replica. If none is, confirm on every site that @@GLOBAL.read_only=1, pick the site with the most complete GTID set, and unfence it by hand (SET GLOBAL super_read_only=OFF; SET GLOBAL read_only=OFF;). Nothing else clears a fence the run could not attribute. Re-annotating a planned failover does not help here: admission needs exactly one writable primary, so with every site read-only the request waits with waitReason: AuthorityUnsettled and ends NotStarted{AdmissionTimeout} after 2 minutes while writes stay refused. Once one site is writable again, a planned failover can move the primary wherever you want it.

Promoted but unconfirmed

PromotedUnconfirmed means super_read_only=OFF may have reached the target, but the run lost the target or could not confirm the promotion. Once the terminal record (or marker) has committed, and before the claim is released, the run names the target as the topology manager's last failover target (it reaches status.lastFailover.target through the next status write), so split-brain judgement keeps it. It names the target only while the durable failover record is still the run's baseline (or its own); if a failover was recorded during the run it names nothing and logs planned failover: a failover was recorded during the run; the unconfirmed target is not named. It does not stamp status.lastFailover or a pending promotion, so the cooldown does not block emergency failover. When the run ended because no topology manager ran (ManagerUnavailable), nothing is named.

Emergency failover may then promote the fenced source. That is the same RPO trade as emergency failover of any unreachable primary: DNS never pointed at the target, so it took essentially no application writes, and the sidecar lease rule fences a target cut off from its peers. To recover:

1. Read @@GLOBAL.read_only, @@GLOBAL.super_read_only and @@GLOBAL.gtid_executed on the source and the target.
2. If the target is writable and the source still fenced, the target is the primary in all but name; the topology manager adopts it on its next polls (it is the named last failover target). Verify DNS and application writes.
3. If both are read-only, the re-assert or emergency failover decides on the next polls. If the group stays without a writable site, follow the NoPrimary row of the doctor skill's alert matrix.
4. If both are writable, it is a split brain: follow the split-brain runbook.

The automation hold

A run that may have written to the target and cannot explain the result ends ManualRecoveryRequired. In the same conditioned status write it records the hold and sets the condition that summarises it:

status:
  plannedFailoverHold:
    runID: 6f1c2a4e-4b7d-4a1f-9a0e-2d5b8c3f7e10
    reason: ForeignWritable
    sourceSite: iad
    targetSite: pdx
    ambiguousSites: [iad, pdx]
    holdTime: "2026-04-20T14:32:09Z"
  conditions:
  - type: AutomaticFailoverHeld
    status: "True"
    reason: PlannedFailoverManualRecoveryRequired
    observedGeneration: 7
    message: >-
      planned failover run 6f1c2a4e-4b7d-4a1f-9a0e-2d5b8c3f7e10 ended ManualRecoveryRequired (ForeignWritable);
      writable or ambiguous sites: iad, pdx. Automatic failover is held: resolve the split brain by hand, then annotate
      bloodraven.shipstream.io/acknowledge-planned-failover-hold=6f1c2a4e-4b7d-4a1f-9a0e-2d5b8c3f7e10

status.plannedFailoverHold is the record: read the run from plannedFailoverHold.runID, not from the message. It outlives the run's plannedFailover block, which a refused request can replace. ambiguousSites lists the run's target and source plus every site the failing step classified (a writable or unreadable peer found by the pre-write gate, say). A hold rebuilt from the terminal marker by a process that did not see the run lists every site in spec.sites, target and source first.

and the claim becomes a hold: emergency failover, split-brain resolution, the re-assert, ordered updates, rotations, recovery and convergence, bootstraps and DNS reconciliation stay held for the group, exactly as while a run holds the claim. Writes on whichever site is writable continue, and the rest of the group's reconcile keeps running: pod labels, backup and verification schedules, backup status and keyring work are not held. A new planned-failover request is refused with NotStarted{ManualRecoveryPending} (Event only, keeping the hold run's block, while that block is the latest), also when only a terminal marker carries the hold; a request submitted during the hold does not run after the acknowledgement. Dragonfly is held too, because the hold owns the active Dragonfly the way a running planned failover does: if the Dragonfly master fails, no replica is promoted until the hold is acknowledged; replicas that lose their master are not re-attached; the active site's Dragonfly rollout is not promoted through; and a snapshot-restore upgrade that spec.dragonfly.image asks for waits, with the Normal Event DragonflyUpgradeDeferred (a planned failover run or automation hold owns the sites), until the hold is acknowledged. Applications that depend on the cache for availability see a Dragonfly failure as an outage for as long as the hold stands: resolve and acknowledge the hold promptly. Once it is acknowledged the Dragonfly manager resumes and repairs and promotes as usual. The hold survives operator restarts: it is rebuilt from status.plannedFailoverHold (or from a terminal marker that carries ManualRecoveryRequired) before the topology manager starts. A True condition without plannedFailoverHold (a hand-edited status, or a CRD rolled back across this release) holds too, at runtime as well as after a restart, but names no run, so no acknowledgement can clear it: remove the condition by hand once the sites are resolved. The operator then releases the hold without a restart: a hold that no status.plannedFailoverHold, True condition, ManualRecoveryRequired terminal marker or acknowledgement backs is released (planned failover: released an automation hold no durable record backs, then planned failover: claim released with why=orphanHold).

Why a hold, and not let split-brain resolution pick. The existing split-brain path keeps the history-named site, which can be the old source, and fences the other, which may be the only holder of acknowledged writes. An optional operation that ends unexplained must not discard data automatically.

The hold is visible as the Warning Event PlannedFailoverManualRecoveryRequired when it is set, the Warning Event PlannedFailoverHoldPersisted every 10 minutes while it stands, the condition in kubectl bloodraven status (the summary shows +held), the doctor triage, the deployment API (unstableReason: AutomaticFailoverHeld, distinct from PlannedFailover for a run in flight; leases stay refused) and the chart's Argo CD health check, which reports the group Degraded while the condition is True. The condition's observedGeneration is the generation of the object the hold was read from and written to (the run's terminal write, or the acknowledgement), not status.observedGeneration: the hold comes from durable run evidence, not from the configuration the topology manager applied. To clear it, resolve the split brain by hand and then acknowledge it with the run's id, kubectl bloodraven ack-planned-failover-hold <group> (which writes bloodraven.shipstream.io/acknowledge-planned-failover-hold=<runID>); see the hold runbook. The operator then removes status.plannedFailoverHold and sets the condition False with reason PlannedFailoverHoldAcknowledged (and writes the terminal record if only the marker carried the hold), deletes the marker, releases the hold, deletes the acknowledgement and emits the Normal Event PlannedFailoverHoldAcknowledged. An acknowledgement that names another run is ignored with the Warning Event PlannedFailoverHoldAcknowledgmentIgnored. So is one that finds no standing hold (no automation hold is standing for planned failover <runID>; nothing was resumed) or a claim held for another run; the annotation is removed either way.

Status stalls and the terminal marker

If the run's status writes fail for 2 minutes in a row while it holds the claim (an RBAC 403 on status, a 422, a failing admission webhook, or a write that conflicts on all three attempts of a pass; leader election does not bound these), the run ends outside /status:

1. The first failed write records the annotation bloodraven.shipstream.io/planned-failover-stall-since: <runID>:since=<YYYYMMDDThhmmssZ> with a conditioned metadata patch. It is never moved forward for the same run, so operator restarts do not restart the 2 minutes; the first committed status write for the run removes it.
2. After 2 minutes (10 seconds for a run that already released its claim after its promotion was confirmed, recorded and published, where only the Publishing write is left) the run finishes in memory only, with no MySQL rollback: a confirmed promotion is still recorded and published (Succeeded with the warning StatusStalled), but only once its failover record (time, target and the fenced set as the watermark) reads back from the out-of-band annotations or status (until then the operator keeps retrying and logs planned failover: status stalled, but the promotion's failover record is not durable yet; retrying, because a restart restores the watermark only from a durable record); a possible write becomes PromotedUnconfirmed; a run that never sent a SET super_read_only=ON (by this process's own record of the statement, not merely an attempt) becomes NotStarted or RolledBack; otherwise the source stays fenced (SourceLeftFenced{StatusStalled}) and is named for the re-assert.
3. It writes the annotation bloodraven.shipstream.io/planned-failover-terminal: {"schemaVersion":1,"runID":"<runID>","outcome":"<outcome>","reason":"<reason>","warnings":[...]}, conditioned on metadata. warnings is optional; the terminal record rebuilt from the marker keeps those warnings plus StatusStalled, and a marker carrying FenceNotAttributable never names the source for the re-assert. One run's marker never replaces another run's ManualRecoveryRequired marker (that run's hold evidence); the stalled run then keeps retrying, and keeps its claim, as when metadata writes fail. A marker that is not this payload, or carries another schemaVersion, proves nothing: it is ignored and logged. The marker also fences every late status write of the run. Every planned-failover predicate treats the run it names as terminal.
4. It releases the claim (or keeps it as a hold for ManualRecoveryRequired), logs ERROR planned failover: status stalled; ended the run with a terminal marker and emits the Warning Event PlannedFailoverStatusStalled.
5. It keeps retrying the terminal status record every 10 seconds and deletes a non-hold marker once that record lands. If the marker is removed by hand first, the operator still writes the run's terminal record (every 10 seconds until it commits), releases any claim hydrated back for the run, and never touches MySQL for the run again. The run's per-run annotations (planned-failover-stall-since, planned-failover-manager-absent-since, planned-failover-confirmed) are removed by the next pass once the run has ended. A Succeeded record rebuilt from a marker (by a restarted operator) carries the run's confirmation time, so it still writes the status.lastFailover record (time, target and promotionGtidExecuted).

kubectl bloodraven promote --wait and kubectl bloodraven status honour a marker that names the current run: --wait returns the marker's outcome (success, or an error naming <outcome>:<reason>) even while status.plannedFailover.phase still reads a non-terminal phase, and status prints Ended by terminal marker: <outcome>:<reason>. A successful --wait also prints the run's result.warnings.

Residuals: if even metadata writes fail, the claim stays held, the Warning Event says the operator cannot write MysqlFailoverGroup <ns>/<name>, and leader election is the only bound (the operator could not record an emergency failover either). A run whose fenced set exceeds 64 KiB cannot use the marker release, because the out-of-band watermark annotation cannot hold it; it keeps its claim until a status write lands.

An ambiguous status write (a timeout, a reset connection, a 5xx) is settled before anything else happens: the operator re-reads the object, and if the write did not land it bumps the bloodraven.shipstream.io/planned-failover-status-fence annotation conditioned on the same resourceVersion, which proves the write can never commit (planned failover: status write ambiguous; settling). The new fence value always moves past the stored one (by 1ns when the clock has not moved), and the fence counts as proof only if it moved the object's resourceVersion; otherwise settlement retries, and the write stays unknown after its attempts.

Operator restarts, manager restarts, leader changes

• No topology manager. A run that holds the claim waits for one (with no manager, no other topology actor exists). After 30 seconds it emits the Warning Event PlannedFailoverStalledManagerAbsent; after 5 minutes, measured from the annotation bloodraven.shipstream.io/planned-failover-manager-absent-since: <runID>:since=<YYYYMMDDThhmmssZ>, it ends without MySQL access: SourceLeftFenced{ManagerUnavailable} (or the rollback's reason), or PromotedUnconfirmed{ManagerUnavailable} from Promoting. A promotion this process had already confirmed ends Succeeded instead (promoted "<site>"; no topology manager for 5m0s, so the failover record is applied by this terminal write and DNS is left to the next manager), with the failover record applied to status; if a failover was recorded during the run, it ends ManualRecoveryRequired{AuthorityChanged}.
• A replaced topology manager. The claim lives in a per-group slot shared by the old and new manager, so a configuration or credential change that restarts the manager does not drop it. A new request waits in admission (PredecessorBusy) while the old manager's ordered update is still exiting. A failover the old manager recorded (an ordered-update handoff that finished after it was replaced, whose status write the operator drops) is carried into the new manager, newest record wins, so its cooldown and last failover target still apply (carried a retired topology manager's failover record); a record whose target is no longer a site is not carried. The predecessor's named active site (a source named for the re-assert, say) is carried too (carried the replaced topology manager's named active site). A group deleted and recreated under the same name (another UID) does not inherit the old object's claim slot, hold included, or the retired manager's failover record (group recreated, restarting topology manager; WARN planned failover: the group was recreated; its claim slot starts empty when the old slot was held).
• Deployment leases. A deployment-lease refusal at admission (TopologyPersistencePending, ConcurrentOperation) is an admission wait (admission.waitReason AuthorityUnsettled, UpdateInFlight, RestoreInFlight or ClaimHeld), bounded by the admission deadline, not a reconcile error.
• A reconcile that returns early. When an earlier reconcile step returns an error or a requeue (credentials, encryption, ConfigMaps, Deployments, Services, the PDB, a Dragonfly snapshot upgrade run in flight, a restore Job or an in-place restore), the reconcile still runs one planned-failover pass for a run already in flight, a hold or acknowledgement, a terminal marker or a deadline annotation; a new request waits for a reconcile that reaches the regular step (planned failover: pass on an early-returning reconcile failed if that pass fails). While a run owns the sites, is Publishing, or a run's claim is held, the earlier step's error is logged (reconcile failed while a planned failover is in flight; retrying on the planned-failover cadence, key requeueAfter) and the reconcile requeues on the run's cadence instead of returning the error, whose exponential backoff would starve the run's deadlines. Only such a run (or a held run claim) skips the rest of the reconcile; an Admitting or Deferred run without a claim does not hold up label sync, Dragonfly labels or backups.
• Pass bounds. One reconcile pass, measured from before its uncached read of the group, is bounded at 15 seconds, 30 seconds for the promotion statements (each statement at 5 seconds), 45 seconds for the confirmed tail and the PromotingDragonfly bound above, plus up to 5 seconds for each durable write that needs its reserve and 30 seconds of settlement for each ambiguous write. Every durable write of a run (status records including the terminal one, deadline and marker annotations, request removal) and the stall bookkeeping keep a reserved 5-second budget even when MySQL work used up the pass's bound. The controller runs one reconcile at a time across groups, so a planned failover can delay another group's reconcile (label sync) by up to that long. Emergency failover runs in the topology manager and is not delayed.
• Leader changes. Exclusion holds within one operator leader, and across a handover only as far as controller-runtime's lease handover goes. See Operator availability.

Upgrading across this release

There is no in-place upgrade from v1.6.0 to this release (see Upgrade policy); a new group starts with no planned-failover record. The operator still refuses a block it did not write itself: a non-terminal block written by an older operator (no runID, or a removed phase such as Pending, Validating or Resuming) is refused, not resumed: no claim is hydrated, nothing is sent to MySQL, and the Warning Event PlannedFailoverLegacyRunRefused is emitted.

• A block in Pending, Validating or Deferred ends NotStarted{LegacyRunRefused}. A legacy Deferred request's annotation is left in place, so it re-runs as a new, strict request.
• A block in Draining or WaitingForLag ends SourceLeftFenced{LegacyRunRefused} with the message planned failover in phase "<phase>" was written by an older operator and is refused: check which site is writable; if none is, unfence the site with the most complete GTID set by hand (confirm no other site is writable first); re-annotate only once a site is writable.
• A block in Promoting or Resuming ends PromotedUnconfirmed{LegacyRunRefused}, because the older operator may already have made the target writable. Its message reads ... is refused: its promotion may have made the target writable; check which site is writable; ... with the same advice.
• In both cases re-annotating only works once a site is writable; with every site read-only, unfence one by hand first. The old sourcePrimary field is pruned on read, so the record no longer names the source. These cases are manual (see Healing a fenced source).

See Upgrade policy for the vocabulary changes.

Follower convergence after success

Once the target is confirmed writable and planned-failover mutation is no longer in flight, Bloodraven converges every other candidate, DisasterRecovery, and ReadOnly follower directly to the new primary. A chained or stale source is not considered healthy merely because its replication threads are running.

Before STOP REPLICA, the new primary's GTID_EXECUTED must contain the follower's executed set. Bloodraven repeats this check after STOP to close the SQL-applier race, then changes the source and starts replication without resetting replication metadata. A failure is retried boundedly on subsequent polls. GTID non-containment is reported per site as sourceConvergenceState: Blocked and sourceConvergenceReason: GTIDDiverged; the operator does not discard the follower's extra transactions or silently repoint it.

Reader client endpoints remain absent until MySQL is read-only, both threads are healthy, direct-source convergence is Converged, and known lag is within the effective reader threshold. A blocked or lagging reader does not change the successful planned-failover result or the group's shared Ready/Degraded conditions.

Anti-flap cooldown

The planned path records status.lastFailover on success (time is the promotion's confirmationTime, target the promoted site, promotionGtidExecuted the fenced set), exactly like the emergency path, and in the same status write as the Succeeded record. The record never moves to an older lastFailover.time. This means:

• A planned failover within spec.failoverCooldown of any prior failover is rejected at admission (or deferred with onCooldown: Defer).
• An emergency failover within spec.failoverCooldown of a planned failover is blocked by the same cooldown check in the topology manager.
• A PromotedUnconfirmed run does not stamp lastFailover, so it does not start a cooldown.

The default cooldown is 5 minutes; production deployments should keep this, not lower it.

Observability

Metrics

result is the kebab-case of the terminal result.outcome (RolledBack is rolled-back). The cause is not a label: a lag timeout counts as rolled-back and an admission timeout as not-started, and status.plannedFailover.result.reason names it. A request refused without a status block (an invalid annotation, a request during a hold or beside an unhealed fenced source) is counted as well, as not-started under the requested target_site. bloodraven_planned_failover_lag_wait_seconds gets exactly one sample per run, once the transition it measures has committed.

bloodraven_failovers_total (automatic only) and bloodraven_dns_flips_total retain their existing semantics. Dashboards keyed on bloodraven_failovers_total see no change.

Events

A topology change revokes deployment leases and emits DeploymentLeaseRevoked.

Logs

The planned failover: … log lines are listed in the log schema.

RPO

Planned failover is the one Bloodraven operation with an RPO of zero by construction. The pre-write gate promotes only a target that holds exactly the fenced source's GTID set, immediately before its super_read_only is cleared, with the source still fenced and every other promotable site read-only. Any scenario that would produce loss (the target falls behind, the source is lost mid-run, catch-up times out, a peer cannot be read) routes to a rollback or an outcome that leaves the decision to emergency failover or a human, instead of promoting a lagging replica. See Durability and RPO.