Observability

Log schema contract

Bloodraven emits structured JSON logs from both the operator (bloodraven) and the per-MySQL sidecar (bloodraven-sidecar). This page is the contract that downstream log pipelines key off of: which fields are stable, what the msg values are for the events you care about, and what guarantees we make about changing them.

If you only need one rule of thumb: filter on msg for the event vocabulary in the Event reference below — those strings are stable. Everything else is best-effort.

Streams

Both binaries write to stdout. There are two independent JSON streams; you can tell them apart by the presence of certain keys.

StreamSourceIdentifies asWhat's in it
Operational (slog)Operator and sidecarHas time, level, msgFailover, promotion, bootstrap, recovery, fencing, archiver, sidecar startup, divergence detection — every event a human operator or alerting pipeline would care about
Controller-runtime (zap)Operator onlyHas ts, level, msg, logger, controller, controllerKind, reconcileIDReconcile-loop bookkeeping from controller-runtime: CR fetches, status updates, watch events. Useful for debugging, not a stable interface

The contract on this page applies to the operational stream. The controller-runtime stream is emitted as-is by upstream sigs.k8s.io/controller-runtime and inherits whatever shape that library produces — we don't redefine it.

To filter to operational logs in most pipelines, key on the presence of the time field (slog) or the absence of the logger field (zap).

Common fields

Every record in the operational stream carries:

FieldTypeDescription
timeRFC3339Nano timestamp (string)Event time, normalized to UTC by the binary's slog handler regardless of pod timezone. Always ends in Z.
levelstringOne of DEBUG, INFO, WARN, ERROR.
msgstringThe event identifier. Stable for events listed in the Event reference; may change for ad-hoc debug logs.

Records emitted under a specific failover group also carry:

FieldTypeDescription
fgstringThe MysqlFailoverGroup namespaced name (namespace/name). Present on every operator log scoped to a group. On the sidecar, this is the bare group name passed via FAILOVER_GROUP.

Sidecar records additionally carry:

FieldTypeDescription
podstringThe pod name (set via MY_POD_NAME). Disambiguates per-replica logs when shipping multiple sites' sidecars to one stream.

Levels

LevelWhen
DEBUGPer-poll bookkeeping (status no-ops, transient probe errors, archiver tick events). Off by default — the operator's slog handler is set to INFO.
INFOState changes the operator deliberately took: failover, promotion, bootstrap, recovery, sidecar lifecycle. Most of the event vocabulary lives here.
WARNDegraded but not fatal: a single retry, a peer briefly unreachable, a non-critical operation that failed (connection kill, taint patch). The operator continues.
ERROROperator-affecting failure: failover failed, self-fence triggered, status update rejected by the API server, CronJob-pod startup validation failed. Always paired with an error field (a string carrying either the underlying error or, for validation failures, a description of what was missing).

DEBUG records may appear or disappear without notice. INFO/WARN/ERROR msg strings listed below are stable.

Field naming convention

  • Keys are camelCase. Common keys: site, fg, error, peer, count, source, donor, recipient.
  • The deployment API's duration_ms is an explicit contract exception: a JSON number in milliseconds, not durationMs. Its group identity is group plus namespace, rather than requiring fg.
  • Site identifiers (site, oldPrimary, newPrimary, promotedSite, donor, recipient, activeSite, authoritativeActiveSite) all carry the bare site name as defined in spec.sites[].name.
  • GTID fields (promotionGtid, divergentGtid, oldPrimaryGtid, newPrimaryGtid, followerGtid, activeGtid, extraGtid) carry MySQL GTID-set strings exactly as MySQL returns them — never parsed or canonicalised. Computed sets (divergentGtid, extraGtid) are MySQL-format sets; on the handoff, picker, ordered-update and promotion records they are truncated to 1024 characters with a trailing ….
  • Counts (count, divergentTransactions, extraTransactions, attempt, maxRetries) are JSON numbers, not strings.
  • Durations (leaseTimeout, pollInterval, delay, timeout) are emitted by slog's default time.Duration rendering — currently a string like "30s". Treat as opaque if you need to parse, prefer the metric of the same name.

Event reference

This is the stable vocabulary. msg strings here will not change without a deprecation note in the release notes.

Deployment API

Every /deploy/v1 request emits one record, including authentication and authorization refusals. Filter on the exact msg, not on HTTP paths that can contain caller-provided identifiers.

LevelmsgFieldsFired when
INFOdeploy apihandler, group, instance, namespace, operationId, status, duration_msA deployment API request completes.

handler identifies the route handler; group, instance, namespace, and operationId are strings (empty when not resolved or not applicable). instance is the MysqlDatabase resource name. status is the numeric HTTP status, and duration_ms is numeric elapsed milliseconds. No bearer token, lease token, or token hash is logged. Treat unresolved request identifiers as untrusted input, not proof of authorization.

Requests increment bloodraven_http_requests_total{server="deploy"}. Lease revocations emit Kubernetes Event DeploymentLeaseRevoked on the MFG and increment bloodraven_deploy_lease_revocations_total{group,kind,reason} with TopologyChanged or OperatorRevoked. A planned hold is visible as Deferred/DeploymentHold, PlannedFailoverDeferred, and bloodraven_planned_failovers_deferred_total{group,reason}. See Deployment API metrics for the complete lease metric family.

Failover

The four events that trace one failover, in order:

LevelmsgFieldsFired when
INFOinitiating failovercandidate, oldPrimary, fgOperator has chosen a promotion target and is about to run the promotion sequence. DNS flips only after promotion succeeds and the target is verified writable.
INFOfailover completepromotedSite, promotionGtid, fgExecute (or the ordered-update ExecuteHandoff) finished: candidate is writable. A planned failover emits it too, only once its live confirmation found the target writable and every other site read-only; a planned promotion that ends unconfirmed or held never emits it, and a planned re-run that finds the target already writable and confirms it emits it too. promotionGtid is the candidate's gtid_executed snapshot taken just before clearing super_read_only — the upper bound on data that survived. For a planned re-run that found the target already writable it is the fenced set; for one whose target already held the run's own writes, the set read immediately before the clears (the fenced set plus those writes).
INFOpromotion confirmed: site is writablesite, fgNext poll observes the promoted site is Writable. The internal post-failover guard clears here.
ERRORfailover failederror, fg, retryAtThe promotion sequence returned an error. The operator re-evaluates the topology at retryAt (10s later) even if no site changes state, and earlier if one does.
ERRORfailover picker: refusing candidate whose data regressedsite, gtid, promotionGtid, reason, fgA promotion candidate was excluded because promoting it would replace committed history: its gtid_executed is empty in a group known to hold data (a wiped datadir, for example after a failed CLONE), or it does not contain promotionGtid, the GTID set recorded at the last promotion. Reclone the site. If no candidate remains, no failover happens and the decision is re-evaluated every 10s. Page on it.
WARNfailover picker: skipping divergent candidatesite, divergentTransactions, divergentGtid, selected, fgEmergency failover preferred selected, a candidate without a divergence report, over site, which carries one (recoveryState=RecoveryBlocked) — even if site had a fresher GTID set. Transactions only site holds, legitimate ones included, are not on the new primary; site keeps its report.
ERRORfailover picker: promoting a divergent candidate; no other candidate remainssite, divergentTransactions, divergentGtid, comparedTo, fgEvery remaining candidate carries a divergence report, so emergency failover promotes the freshest of them anyway: divergence never blocks an emergency failover. Its extra transactions become authoritative; once it is the confirmed primary the operator records the DivergentPromotion condition. Page on it.
ERRORpromotion succeeded but writable confirmation failed; DNS not flippedsite, error, fgExecute returned successfully but the promoted site did not report writable within the confirmation window. DNS is not flipped and no failover state is recorded — the promotion is treated as unconfirmed and re-evaluated on the next tick.
ERRORDNS flip failed after successful promotionsite, error, fgPromotion and writable confirmation both succeeded, so the failover state (cooldown, split-brain target, promotionGtidExecuted) and bloodraven_failovers_total are already recorded; only the DNS update failed. bloodraven_dns_flips_total is left unincremented. The poll loop reconciles DNS against the current active site (see DNS reconciled to active site below), so a transient failure such as an RBAC denial self-heals once the write is permitted again — MySQL has already promoted regardless.
WARNDNS reconcile failedsite, target, hostname, error, fgThe poll-driven DNS reconcile tried to point the record at the current active site and the write was rejected. Logged once per failing episode, not once per poll: while the failure persists the retry continues silently (DEBUG DNS reconcile still failing) and MySQL is unaffected.
INFODNS publication withheld until the initFromBackup restore succeedssite, fgA new group's spec.initFromBackup restore has not succeeded (pending, running, or failed), so the group hostname is not pointed at the writable seed site. Logged once per restore; the record is written on the first poll after the restore succeeds.
INFODNS reconciled to active sitesite, target, hostname, fgThe DNS record diverged from the desired spec.dns (target IP, hostname, or TTL) and was repaired — a promotion-time flip that had failed, a hostname rename, a record left stale by an operator restart, or an out-of-band edit. bloodraven_dns_flips_total{site} increments here, and only when the A-record target IP actually changed. No promotion is re-run and MySQL is not touched.

Supporting events emitted inside Execute:

LevelmsgFields
INFOfenced old primary with super_read_only=ONfg
WARNfailed to fence old primary (may be unreachable)error, fg
INFOkilled app connections on old primarycount, fg
WARNfailed to kill app connections on old primaryerror, fg
INFOrelay log drain completefg
WARNrelay log drain did not complete cleanly, proceeding with promotionerror, fg

Ordered update handoff

Emitted by ExecuteHandoff when an OrderedUpdate rollout hands the primary role from the healthy active site to its standby. On success the handoff also emits failover complete (see Failover); the emergency-path Execute messages above are not emitted.

LevelmsgFieldsNotes
INFOmaintenance handoff: fenced sourcecandidate, sourceGtid, fgThe active site is fenced with super_read_only=ON. sourceGtid is its GTID_EXECUTED read after the fence, exactly as MySQL returned it — the set the standby must contain.
INFOmaintenance handoff: killed app connections on sourcecandidate, count, fgApplication sessions on the fenced active site were evicted.
WARNmaintenance handoff: failed to kill app connections on sourcecandidate, error, fgBest effort; the handoff continues.
INFOmaintenance handoff: candidate caught up to fenced sourcecandidate, sourceGtid, fgThe standby's GTID_EXECUTED contains sourceGtid; promotion starts.
WARNmaintenance handoff refused; source not fencedcandidate, error, fgThe handoff was refused before the active site was fenced, so no write was paused: the standby did not reach the active site's current set within 30s (the usual case: it is lagging), the active site stopped answering, the active site was read-only, or either site's GTID set could not be read. The update ends with ordered update failed and is retried later.
WARNmaintenance handoff refused; candidate has transactions the source lackscandidate, extraTransactions, extraGtid, fgBefore fencing, the standby's GTID_EXECUTED (read first) held transactions the active site's (read second) does not. Promoting it would make that history authoritative (issue #202), so the handoff stops; no write was paused. extraGtid is the extra set, truncated to 1024 characters. The update ends with ordered update failed; once the replica GTID audit records the divergence the standby is no longer chosen. See playbook 33.
WARNmaintenance handoff aborted; source unfencedcandidate, error, fgThe standby did not reach the fenced set within the 30s fenced budget, its applier failed, a GTID set could not be read or parsed, the active site stopped answering, the standby's set no longer equalled the fenced set when promotion re-checked it after STOP REPLICA or RESET REPLICA ALL (missing transactions, or transactions the source lacks: the error then says transactions the source lacks), or promotion failed before the standby became writable. The active site has super_read_only and read_only cleared and keeps serving; the standby's replication is restarted (re-created after RESET REPLICA ALL); the update ends with ordered update failed and is retried later.
ERRORmaintenance handoff: rollback failed: source may be left read-onlycandidate, cause, error, fgThe abort could not unfence the active site. Check @@super_read_only and @@read_only on it. Page on it.
WARNmaintenance handoff: failed to restart replication on candidate after abortcandidate, error, fgPromotion stopped after STOP REPLICA (or after RESET REPLICA ALL, once the channel was re-created) and the standby's replication threads could not be restarted. The standby is read-only and not replicating; see playbook 23.
INFOmaintenance handoff: restored candidate replication channelcandidate, host, fgPromotion stopped after RESET REPLICA ALL may have removed the standby's channel; the operator re-created it towards host with auto-positioning and restarted replication.
WARNmaintenance handoff: failed to restore candidate replication channel after abortcandidate, error, fgThe channel could not be re-created after an abort that may have followed RESET REPLICA ALL. The standby is read-only with no working channel and nothing re-points it automatically; restore it by hand (playbook 23).

Emitted by the ordered update itself:

LevelmsgFieldsNotes
WARNordered update: handoff promotion returned an error but the standby is writable; recording the handoffsite, error, fgThe handoff's final read_only=OFF on the standby returned an error after it may have applied, and a probe of the standby's read_only (pooled connection, up to 3 tries) found it writable. The handoff is recorded as a promotion (the status.lastFailover record, the anti-flap cooldown, DNS) before the error is reported. The source stays fenced and the old active is not rolled; the update ends with ordered update failed (error contains failover during update: maintenance handoff: promote <site>: … (source left fenced: the candidate may already accept writes)).
WARNordered update: could not read the standby's read_only after a failed handoff promotionsite, error, fgThe same failed read_only=OFF, but the probe could not read the standby's read_only. Nothing is recorded; the source stays fenced, the update ends with ordered update failed, and the next topology poll observes the standby.
WARNordered update: active-site update stalled; standby is divergentactive, site, divergentTransactions, divergentGtid, reason, fgThe active site has spec drift, but every otherwise eligible standby carries a divergence report, so none is a handoff target and nothing is fenced. reason reads <site> is divergent (N transactions the primary lacks); reclone or reconcile before the active site can be updated, the same text the update's error carries. The stall is intended; the update resumes on its own once the audit clears the report. Logged at most every 30s while it lasts. See playbook 33.
INFOordered update: deferring follower update; no drifted follower is a healthy direct replicaactive, driftSites, fgOnly non-active sites have spec drift, and none of them is a read-only replica with healthy threads pointed at the active site, so no update attempt could roll one; none is started and the drift stays queued. Typical cause: the drifted site is a returning old primary that old-primary recovery has not rejoined yet (an attempt would also hold that recovery off), or it is unreachable or replicating from the wrong source. The update starts on its own once a drifted follower replicates directly. Logged at most every 30s while it lasts.
INFOtopology manager deferring to its predecessor's ordered update; no failover, recovery or update until it exitsfgA topology manager restarted (configuration or credential change) while the manager it replaced still had an ordered update running. Until that update exits, the new manager takes no emergency failover, re-assert, old-primary recovery, connection drain, source repoint or update of its own, a new planned failover waits in admission (admission.waitReason: PredecessorBusy, nothing fenced), and status.updatePhase is empty. The old update's context is cancelled, so this normally lasts seconds.
INFOpredecessor's ordered update exited; topology manager resumingfg, heldForThe window above ended; heldFor is its length.
WARNordered update: stalled; site pod not Readysite, pod, reason, error, fgThe update stopped on site, whose pod pod (empty when there is none yet) is not Ready, which here means Ready continuously for 30s, measured from the pod's last transition to Ready and its newest container start: either the update rolled it and the new pod did not get there within 5 minutes, or the site already runs the new pod template and the update refused to roll another site or hand off the active site while it is not Ready. reason says why, for example sidecar not ready: crash looping, last exit code 1 (Error) (a container that has restarted, in whatever phase of its loop: back-off, running, briefly Ready again, or terminated), pod Ready for less than 30s, pod not scheduled: … or pod readiness unknown: … when the API server could not be read. error is the full update error. Nothing else is rolled or handed off and the active site keeps serving. Logged on every attempt, at most every 30s while the stall lasts. When the stall stops an attempt that was already running, ordered update failed is logged first, its error carrying the same text. Mirrored by the Updating condition (still True, reason OrderedUpdateStalled) and the OrderedUpdateStalled Warning Event; a retry that finds the same cause rewrites neither. See Ordered updates.
INFOordered update: stalled site pod is Readysite, fgNo update attempt was left to clear a stall (the stalled site was the last to roll, or another precondition holds the update), and the operator found the stalled site's pod Ready again, continuously for 30s, so a crash-looping pod caught between crashes does not log it. The Updating condition returns to NotUpdating.
ERRORordered update: old active has transactions the new primary lacks; not repointing itsite, newPrimary, divergentTransactions, divergentGtid, oldPrimaryGtid, newPrimaryGtid, error, fgAfter a successful handoff the rollout updated the old active site, fenced it again with super_read_only=ON, and found transactions in its set (oldPrimaryGtid) that the new primary's (newPrimaryGtid) lacks. The handoff proves the new primary holds the fenced set, so this means a bug or an out-of-band write. The old active is not repointed: it stays fenced with no replication channel, and the update ends with ordered update failed. Old-primary recovery then reports the set as divergentGtid and blocks (divergence detected with detectedBy=OldPrimaryRecovery). One exception: if the old active came back on an empty datadir, its set holds only server-init transactions under a new server_uuid. Recovery treats such a site as empty and leaves it to auto-clone, so no divergentGtid appears. error carries the same text as the ordered update failed line. Page on it. When the fence or a GTID read fails instead, this record is not logged: the update ends with ordered update failed and an error containing before repointing, and the site is left for old-primary recovery the same way.

Planned failover

Emitted by a planned failover run. Every record names the run with runId, the value of status.plannedFailover.runID (and of the hold acknowledgement). Two emitters produce them:

  • Engine records come from the reconciler that drives the run and are written through the reconciler's logger, so they carry the controller-runtime shape (ts, logger, controller, reconcileID) besides the fields below.
  • Topology records come from the group's topology manager on the operational stream. Their fg comes from the manager's logger and is namespace/name, as on engine records. (Earlier releases also appended the bare group name as a second fg key; none does now.)

Some events are logged by both, as noted. Filter on msg and runId.

LevelmsgFieldsNotes
INFOplanned failover: claim grantedfg, runId, source, targetThe topology manager granted the run its claim after the live evidence probe (topology), and the run's Fencing record committed (engine). From here until the claim is released, the topology manager defers cross-site decisions for the group.
INFOplanned failover: claim releasedfg, runId, why; engine also outcomeThe run's claim was released. why is publishing (the promotion is confirmed, recorded and published; logged before the Publishing write, which needs no claim), terminal, admissionTimeout, notCommitted (the Fencing write was proven not to have committed: nothing was fenced), statusStalled (a terminal marker ended the run), managerAbsent, zombie (the uncached block no longer names a live run), ack (an automation hold was acknowledged) or orphanHold (topology only: an automation hold that no durable record backs any more was released; see planned failover: released an automation hold no durable record backs). outcome is the terminal outcome, empty for a non-terminal release.
INFOplanned failover: released an automation hold no durable record backsfg, runIdEngine. The group's claim slot held an automation hold, but an uncached read found no status.plannedFailoverHold, no True AutomaticFailoverHeld condition, no ManualRecoveryRequired terminal marker and no acknowledgement annotation, so the hold is released (claim released with why=orphanHold) instead of holding automation with nothing to acknowledge. Typical causes: a run-less True condition removed by hand (this releases automation without an operator restart), a group deleted and recreated, or a hold hydrated by a manager start that raced an acknowledgement. runId is empty for a hold rebuilt from a run-less condition.
INFOplanned failover: confirming recorded side effectfg, runId, phase, effectThe pass found a record this process did not write (after an operator restart or a leader change), so it confirms the recorded side effect live before going on. effect is fence, drain, lagWait, dragonflySync, dragonflyTakeover, promotion or rollback. Logged once per run and phase.
INFOplanned failover: status write ambiguous; settlingfg, runId, revision, errorA status write timed out or failed with a reset connection or a 5xx. The operator re-reads the object and, if (runId, revision) did not land, bumps the planned-failover-status-fence annotation conditioned on the same resourceVersion to prove the write can never commit. The new fence value is always past the stored one (1ns past it when the clock has not moved), and the fence counts as proof only if it moved the object's resourceVersion; otherwise the settlement tries again, and reports the write Unknown once its attempts are used up. No side effect runs until it is settled.
ERRORplanned failover: cannot settle status write; holdingfg, runId, errorA run that holds the claim could not settle an ambiguous write (the re-read and the fence both failed). The claim stays held and the pass requeues after 2s; the status-stall deadline bounds how long this lasts.
WARNplanned failover: the target is no longer a primary candidate; not promoting itfg, runId, site, roleTopology. A promotion attempt found that the target's role in the topology manager's current configuration is not PrimaryCandidate (role carries the spec.sites[].role value, for example DisasterRecovery or ReadOnly; spec.sites changed during the run; the change restarts the topology manager, and the run's claim carries over without a re-proof). Nothing is sent; the run rolls back TargetNotPromotable. The pre-write gate repeats the check immediately before clearing super_read_only, and every promotion statement repeats it again (the line is logged there too).
WARNplanned failover: pre-write gate refusedfg, runId, reason, sitesTopology. Immediately before clearing super_read_only on the target, the gate found a reason not to: TargetNotPromotable (the target's role in the topology manager's current configuration is no longer PrimaryCandidate), TargetChanged or ExecuteFailed (the target is not the recorded server, not read-only, has a channel, or does not hold exactly the fenced set), SourceLost, SourceUnfenced, AuthorityChanged (the source holds transactions past the fenced set: its fence was lifted, written through and set again) or ForeignWritable (the source), ForeignWritable or PeerUnreadable (another site). sites is the list of sites at fault. Nothing was written to the target by this attempt. A source that does not answer is refused as SourceLost only once the 10s source-loss window has expired (counted from the first failed source observation in the drain, lag wait or gate); until then the attempt is retried. The run rolls back or leaves the source fenced; when an earlier attempt may already have written, it ends ManualRecoveryRequired (ForeignWritable, also for a source with history past the fenced set), or PromotedUnconfirmed for PeerUnreadable or an unreachable source (SourceLost, which names the target).
INFOplanned failover: named fenced source for re-assertfg, runId, siteA run ending SourceLeftFenced with its own, attributable fence named the source as the topology manager's last active site, so the primary re-assert can restore it once every site is reachable and read-only (engine and topology). The naming happens after the run's terminal status record (or terminal marker) has committed, before its claim is released. Also logged by a starting topology manager that re-derives the naming from the terminal record or a terminal marker (topology). A run whose result or marker carries the FenceNotAttributable warning is never named, including a run that sent a fence and then ended because authority moved.
INFOplanned failover: could not name the unconfirmed targetfg, runId, errorA PromotedUnconfirmed run could not record the target as the topology manager's last failover target.
INFOplanned failover: a failover was recorded during the run; the unconfirmed target is not namedfg, runId, targetA PromotedUnconfirmed run names its target only while the durable failover record is still the run's baseline (or its own). Another actor's failover was recorded during the run, so the target is not named and that failover's authority stands.
INFO / WARNplanned failover: automation held pending acknowledgementfg, runId, sitesA run ended ManualRecoveryRequired and its claim became an automation hold (engine at INFO; topology at WARN, also when a starting manager rebuilds the hold from status.plannedFailoverHold, a terminal marker or a run-less True AutomaticFailoverHeld condition, or when the engine installs a hold at runtime). sites is a comma-separated string: the hold's ambiguous sites (status.plannedFailoverHold.ambiguousSites) when known, empty for a hold rebuilt from a marker or a bare condition. runId is empty for a hold from a run-less condition. Automatic cross-site actions stay held until bloodraven.shipstream.io/acknowledge-planned-failover-hold=<runID>. Page on it.
INFOplanned failover: hold acknowledgedfg, runIdThe acknowledgement names the held run; the operator clears status.plannedFailoverHold (the AutomaticFailoverHeld condition turns False/PlannedFailoverHoldAcknowledged), the marker, the hold and then the acknowledgement, in that order. An acknowledgement that names another run, finds no standing hold, or finds the claim slot held for another run emits the PlannedFailoverHoldAcknowledgmentIgnored Warning Event instead, and resumes nothing; the annotation is removed when no hold stands.
ERRORplanned failover: status stalled; ended the run with a terminal markerfg, runId, outcome, reason, errorThe run's status writes failed for 2 minutes from the durable stall-since time (10 seconds for a run that already released its claim after its promotion was recorded and published: only its Publishing write was left); the planned-failover-terminal annotation now ends it (outcome/reason are the marker's). The claim is released, or kept as a hold for ManualRecoveryRequired. error is usually empty.
INFOplanned failover: status stalled, but the fenced set is too large for a durable out-of-band watermark; keeping the claimfg, runIdThe fenced set exceeds 64 KiB, so the stall release is not available: the claim stays held until a status write lands.
INFOplanned failover: promotion already recorded; publishing without the claimfg, runId, targetA process that does not hold the run's claim (after a restart or a topology-manager replacement) found a Promoting block whose own failover record (its target and fenced set as the watermark, not older than the promotion intent) is the durable record. The confirmed tail had released the claim before the Publishing write landed, so the claim is not hydrated back and nothing is deferred: the run writes Publishing and Succeeded without it, and the topology manager reconciles DNS. Logged once per run and process.
INFOplanned failover: status stalled, but the promotion's failover record is not durable yet; retryingfg, runIdThe run's promotion is confirmed but its Publishing write has failed for the stall deadline (2 minutes from the durable stall-since time, or 10 seconds when the confirmed tail had already released the claim, in which case the run no longer defers automation), and neither the out-of-band failover annotations nor status carry the run's own failover record (its time, target and fenced set as the watermark) yet. The terminal marker would end the run with no durable watermark, so the operator keeps retrying instead; the live Promoting block still records the run (and, unless the claim was already released, owns the sites), and a restart re-derives the confirmation from it. Logged once per run. Repeats mean both the status subresource and the annotation write are being rejected.
INFOplanned failover: target replaced before the promotion was recorded; rolling backfg, runId, target, targetServerUUIDA new operator process found the target pod replaced (@@server_uuid is not the run's) before any promotion statement it can prove. Because the run's own failover record is not durable (DNS never pointed at the target), the source still holds the whole fenced set under the run's fence and every other promotable site reads read-only, the run rolls back (TargetChanged) instead of ending ManualRecoveryRequired. Writes only the replaced pod could have held were made after the fence, which is the asynchronous-replication RPO.
INFOplanned failover: legacy status refusedfg, phaseA non-terminal block written by an older operator (no runId, or a removed phase) was refused with reason LegacyRunRefused: NotStarted for a block that owned nothing (Pending, Validating, Admitting, Deferred), PromotedUnconfirmed for Promoting or Resuming (the older operator's promotion may have made the target writable), SourceLeftFenced for the other phases. Nothing was hydrated or sent to MySQL.
INFOplanned failover: draining the source failedfg, runId, sourceReachable, errorA drain query (processlist or KILL) failed. The engine probes whether the source still answers (sourceReachable); only a source that does not answer counts toward the 10s source-loss window (SourceLost). A source that answers is drained up to drainTimeout and drain.remainingConnections is left unset, with the message the connections remaining on <site> are unknown (<error>).
INFOplanned failover: the fenced GTID set is invalid; rolling backfg, runId, errorstatus.plannedFailover.fence.gtidSet does not parse, so no lag probe can ever see the target catch up. The run rolls back with reason InvalidGTID at once.
INFOplanned failover: lag probe failed; counting the target as unreachablefg, runId, errorThe lag probe returned an error; the target counts as unreachable for this pass (the 10s target-loss window, then TargetLost).
WARNplanned failover: could not restore the target's super_read_onlyfg, runId, site, errorTopology. A rollback (or the Promoting re-proof's re-fence) found the target read-only but with super_read_only=OFF, and setting it back to ON failed or did not read back. The rollback retries with the source still fenced, and ends SourceLeftFenced at the rollback deadline if it keeps failing.
WARNplanned failover: rollback found another writable site beside the sourcefg, runId, sitesTopology. The rollback found the source already writable and another site writable too; sites is a comma-separated list. When that site is the target and the run may have written to it, the run ends ManualRecoveryRequired (ForeignWritable) and holds automation; any other writer is left to split-brain handling and the run stays RolledBack.
WARNplanned failover: the group was recreated; its claim slot starts emptyfg, previousUid, uidTopology runner. A MysqlFailoverGroup was deleted and recreated under the same name while the old object's claim slot was held (a run or an automation hold). The new object does not inherit that slot, nor the retired manager's failover record.
ERRORplanned failover: pass on an early-returning reconcile failederrorEngine (controller-runtime stream). An earlier reconcile step returned an error or a requeue, and the planned-failover pass that still runs for a run in flight (or a hold, acknowledgement, terminal marker or deadline annotation) failed too. The earlier step's error is returned. Such a pass never admits a new request.

Retry and bookkeeping records, all engine at INFO. Each means the pass did not complete a step and requeues; repeats for minutes deserve a look:

msgFields
planned failover: claim attempt failed; retryingfg, runId, error
planned failover: cannot confirm the claim; retryingfg, runId, error
planned failover: the claim is not held by this run; waitingfg, runId
planned failover: re-proof failed; retryingfg, runId, error
planned failover: status write did not commit; retryingfg, runId, outcome, error
planned failover: terminal status write did not commit; retryingfg, runId, outcome, error
planned failover: promotion attempt failed; retryingfg, runId, error
planned failover: recording the promotion failed; retryingfg, runId, error
planned failover: rollback attempt failed; retryingfg, runId, error
planned failover: could not remove the request annotation; a later pass retriesfg, why, outcome, error
planned failover: the request annotation changed since this pass read it; left for the next passfg, why (the engine removes only the request value it evaluated)
planned failover: could not remove a finished run's annotation; a later pass retriesfg, annotation, error (a planned-failover-stall-since, -manager-absent-since or -confirmed annotation left by a finished run)
planned failover: could not record the promotion's confirmation; keeping the claimfg, runId, outcome, error (the planned-failover-confirmed annotation; the confirmed tail keeps the claim and retries)
planned failover: could not record the admission wait; retryingfg, runId, waitReason, outcome, error
planned failover: the provisional claim's Fencing write is still unsettled; holdingfg, runId, error (once the write is proven not to have committed, the grant is released with why=notCommitted)
planned failover: could not record the stall-since annotationfg, runId, outcome, error
planned failover: could not clear the stall-since annotation; the next commit retriesfg, runId, error
planned failover: could not record the manager-absent-since annotation; retryingfg, runId, outcome, error
planned failover: could not clear the manager-absent-since annotation; retryingfg, runId, error
planned failover: could not delete the terminal marker of a finished runfg, runId, error
planned failover: could not clear the hold condition; retryingfg, runId, outcome, error
planned failover: could not delete the hold's terminal marker; retryingfg, runId, outcome, error
planned failover: could not delete the hold acknowledgement; retryingfg, runId, error
planned failover: admission waitingfg, runId, waitReason, detail (a deployment-lease refusal: an ordered update or in-place restore holds the sites, or a topology change is not persisted yet. runId is empty for a new request, which is then ended NotStarted at once for a refusal that depends on the request alone, or admitted into an Admitting wait with that admission.waitReason; a Deferred run logs its own runId and stays Deferred, recording admission.waitReason and the message deferred; admission waiting (<reason>): <detail>. An Admitting run's refusal is recorded as admission.waitReason and not logged)

outcome on these is the write outcome (Committed, Conflict, NotCommitted, Unknown, Gone), not a run outcome. A status write that conflicts on all three attempts of a pass counts toward the 2-minute status stall like an Unknown or NotCommitted one. Earlier releases logged no planned failover: … records, so none was renamed.

One reconciler record belongs to planned failover without the prefix:

LevelmsgFieldsNotes
ERRORplanned failover: pass failed; retrying on the planned-failover cadencefg, error, requeueAfterController-runtime stream. A planned-failover pass failed transiently (its uncached read of the group, a topology-facade error) and asked for a bounded retry. The reconcile requeues after requeueAfter (2s for a failed read) instead of returning the error, which would hand the retry to the controller's exponential backoff and could delay a fenced run's deadlines. A persistent read failure therefore logs this every couple of seconds.
ERRORreconcile failed while a planned failover is in flight; retrying on the planned-failover cadenceerror, requeueAfterController-runtime stream. An earlier reconcile step (credentials, encryption, ConfigMaps, Deployments, Services, PDB, Dragonfly snapshot upgrade, restore Job, in-place restore) failed while a planned failover owns the sites, is Publishing, or holds a run claim. The planned-failover pass still ran, and the reconcile requeues after requeueAfter (the run's cadence) instead of returning the error, whose exponential backoff would starve the run's deadlines.

Divergence and recovery

Fired when the operator inspects a returning old primary (after an emergency failover, or an old active site an ordered update refused to repoint), and by the replica GTID audit, which checks every read-only follower with a replication channel against the confirmed primary about every 30 seconds (and on the first poll after start, after the primary changes or is replaced under the same name, and after a completed promotion, handoff or re-assert; a refused divergent handoff candidate is audited on the next poll), whatever its replication threads report.

LevelmsgFieldsNotes
INFOinitiating old primary recoveryoldPrimary, newPrimary, fgRecovery sequence starting.
INFOno GTID divergence, auto-recovering old primary as replicasite, fgOld primary's GTID set is a subset of the new primary's — safe to attach as replica.
WARNdivergence detectedsite, divergentTransactions, divergentGtid, oldPrimaryGtid, newPrimaryGtid, detectedBy, primary, fgsite holds committed transactions the primary never had. detectedBy is OldPrimaryRecovery (a returning old primary: oldPrimaryGtid is its set, newPrimaryGtid the new primary's) or ReplicaAudit (a follower, often replicating normally: oldPrimaryGtid carries the follower's set, newPrimaryGtid the primary's). When the audit re-verifies a block it did not create (an old primary whose replication was reattached by hand), it keeps that block's origin, OldPrimaryRecovery or empty for a block restored without provenance; primary names the site compared against. The site gets recoveryState=RecoveryBlocked and is not promoted by an ordered update or a planned failover. The operator does not auto-recover — the admin must reclone the site or make the primary contain the divergent set (containment then clears it). Emitted when divergence is first detected and again whenever the periodic re-verification finds the set CHANGED — an unchanged report is not re-logged each cycle. Mirrored by the bloodraven_divergent_transactions gauge, Degraded reason DivergentTransactions, and the DataLossDetected (old primary) or DivergentReplicaDetected (replica audit) Kubernetes Event. See playbook 33 for the replica case.
INFOdivergence cleared: replica holds no transactions the primary lackssite, primary, fgA fresh follower-first audit proved the site's set is contained in the primary's (after a reclone, or after the primary was made to contain the set). The block is dropped and bloodraven_divergent_transactions{site} returns to 0. This is the only way a RecoveryBlocked follower with a replication channel is cleared; healthy threads alone never clear it. Mirrored by the DivergenceResolved Event (replica-audit reports) or OldPrimaryRecovered (old-primary reports).
WARNreplica GTID audit unverifiedsite, primary, error, fgThe audit could not confirm the primary or could not read or parse a GTID set. Nothing changes: an existing block stays and none is invented. Retried on the next audit interval. When the primary could not be confirmed, site lists every follower that was due, comma-separated.
INFOreplica GTID audit discarded: a server changed while it was readsite, primary, fg@@server_uuid of the follower or the primary changed between the audit's reads (a pod replaced by a clone or restore), so the sets do not describe one pair of servers. Retried on the next poll.
INFOreplica GTID audit discarded: authority changed while it ransite, primary, fgA promotion, handoff or update began while the audit was reading. Retried once authority settles.
INFOold primary recovery completesite, source, fgOld primary is now replicating from the new primary. source is the new primary's host.
ERRORold primary recovery failedsite, error, fgOne step of the recovery sequence (fence / GTID query / CHANGE REPLICATION SOURCE / START REPLICA) returned an error.
INFOrecovery state cleared (site is now replicating)site, fgThe site is a healthy, source-converged replica again, so its RecoveryInProgress marker is dropped and bloodraven_divergent_transactions{site} returns to 0. A RecoveryBlocked report is never cleared here: healthy threads do not prove the site holds nothing the primary lacks (see divergence cleared above).
INFOrecovery state cleared (site is writable)site, fgThe site is writable, so it is no longer "recovering as a replica". A RecoveryBlocked report is not dropped merely because the site turned writable — a rogue-writable site keeps its live divergence report (split-brain fencing re-fences it, and the periodic re-verification refreshes the report). When the site is the operator's own failover target and the unique writable primary, the report is converted into the DivergentPromotion condition (see the next row) rather than dropped.
ERRORdivergent site promoted; its extra transactions are now part of the primary's historysite, divergentTransactions, divergentGtid, comparedTo, fgA site carrying a divergence report became the confirmed primary (an emergency failover with no clean candidate, or a manual promotion). Its extra transactions are now authoritative. Recorded durably as the DivergentPromotion status condition and the DivergentSitePromoted Warning Event; the condition stays until an administrator removes it.

Kubernetes Event reasons, status conditions and planned-failover reasons for divergence:

NameKindWhen
DataLossDetectedWarning EventA returning old primary holds transactions the new primary never received (divergenceDetectedBy=OldPrimaryRecovery), on first detection and when the count changes.
DivergentReplicaDetectedWarning EventThe replica GTID audit found a follower holding transactions the primary lacks (divergenceDetectedBy=ReplicaAudit), on first detection and when the count changes.
DivergenceResolvedNormal EventA replica-audit report was cleared by a fresh containment proof.
DivergentSitePromotedWarning EventA site carrying a divergence report became the confirmed primary; the same text is the DivergentPromotion condition's message.
Degraded reason DivergentTransactionsConditionAt least one site has recoveryState=RecoveryBlocked. Outranks the per-site replication reasons (ReplicationLagging, ReplicationBroken, ReplicationError, ReplicationSourceMismatch); the cross-site alerts (SplitBrain, NoPrimary, TotalLoss, CrossSiteAlert), BootstrapFailed and SiteUnhealthy still win, while Initializing after an operator start does not. Ready is not changed, and the deployment API does not treat this reason (or a RecoveryPending=DivergentTransactions that only covers replica-audit reports) as unstable; an old primary's report still is.
DivergentPromotion reason DivergentSitePromotedConditionWritten when a divergent site is promoted and never cleared by the operator. A later divergent promotion is appended (newline-separated, newest last); the five newest records are kept, with a leading (earlier divergent promotions omitted; …) line once older ones are dropped. Every promotion also has its own DivergentSitePromoted Event and ERROR log. Remove it after reviewing the transactions: kubectl patch mysqlfailovergroup <group> --subresource=status --type=json -p '[{"op":"remove","path":"/status/conditions/<index>"}]'.
status.plannedFailover.result.reason=DivergentTargetPlanned failoverThe requested target has a divergence report (refused before fencing, outcome NotStarted), or held transactions outside the fenced source's set during the lag wait (rollback.reason=DivergentTarget, then outcome RolledBack, or SourceLeftFenced if another site was writable).

Replication source convergence

After topology changes, Bloodraven verifies that every follower replicates directly from the uniquely confirmed active primary. These events cover candidate, DisasterRecovery, and ReadOnly followers; they are separate from the old-primary recovery events above.

LevelmsgFieldsNotes
INFOreplication source convergence startedsite, activeSite, currentSource, expectedSource, fgA follower needs a source or thread-state correction and passed the initial mutation gates.
INFOreplication source convergence completesite, source, fgThe canonical source is the active primary and both replication threads are running.
WARNreplication source convergence blockedsite, activeSite, stage, followerGtid, activeGtid, fgGTID containment failed before or after stopping replication. No source change is issued.
ERRORreplication source convergence failedsite, activeSite, stage, error, fgA bounded source mutation or verification attempt failed. The next poll can retry safely.

Stable stage values include pre-stop-gtid, post-stop-gtid, stop, change-source, start, and verify. A follower awaiting its post-restore reclone (status.restoreReclones.pending) is not converged: its status reads Pending with reason RestorePeerHold and no convergence event is logged. Use the status sourceConvergenceState and sourceConvergenceReason for current state; use these logs for the detailed failure and GTID evidence.

Bootstrap and reclone

starting bootstrap is the single canonical event for "we are about to clone a replica". The source field disambiguates why:

source valueMeaning
fresh-deployInitial bootstrap of a new failover group; donor is the seed site.
auto-cloneOperator detected an empty replica during steady-state and is recovering it without an admin trigger.
recloneThe bloodraven.shipstream.io/reclone-site annotation was set (by an admin, or by the in-place restore's reclone queue), the safety interlock passed, and the operator is wiping the named site. This is the reclone-started event.
LevelmsgFields
INFOstarting bootstrapsource, donor, recipient, donorHost, fg
INFObootstrap deferred: another topology change holds the sitesrecipient, source, reason, fg
INFOcloning from primarydonor, fg
INFOclone completed successfullyreplica, fg
INFOsetting up replicationsource, fg
INFOreplication started successfullysource, fg
INFObootstrap completed successfullysource, fg
ERRORbootstrap failedsource, error, fg
INFOclone returned expected connection drop, waiting for restarterror, fg
INFOreplica already has primary data (prior clone detected), skipping clone phasefg

A bootstrap is not started while another actor holds the sites (a planned failover, an ordered update, a promotion, an in-place restore, or a replaced topology manager still changing them); it is retried on a later poll. bootstrap deferred: another topology change holds the sites appears only when that actor claimed the sites while the bootstrap was being prepared; reason names it.

clone returned expected connection drop, waiting for restart is not a success signal. The client connection drops both when a clone succeeds (the recipient restarts) and when it is killed or aborted after MySQL has already dropped the recipient's user data. Before configuring replication the operator reads the recipient's performance_schema.clone_status and requires a new Completed row and a non-empty gtid_executed; otherwise it re-fences the recipient and logs bootstrap failed with the clone error.

A reclone-only narrative is therefore: filter msg="starting bootstrap" AND source="reclone" for the trigger event, then watch for bootstrap completed successfully (source="reclone") or bootstrap failed (source="reclone"). The Bootstrapping condition mirrors them: True while the clone runs, then False with reason NotBootstrapping after bootstrap completed successfully, or False with reason BootstrapFailed after bootstrap failed, which also sets Degraded=True with reason BootstrapFailed until the site replicates or a new clone starts.

State transitions

Every per-site state change emits one record. Use this to replay the topology timeline.

LevelmsgFields
INFOstate transitionsite, from, to, fg

from and to values: Unknown, Unreachable, ReadOnly, Writable (the status.sites[].state values). Mirrored by the bloodraven_state_transitions_total counter, whose labels carry the kebab-case form (unknown, unreachable, read-only, writable).

LevelmsgFieldsNotes
WARNsite not classified after its probe budget; probes alternate between answers and errorssite, probes, fgOnce per site per topology manager: the site is still Unknown after failureThreshold + recoveryThreshold probes. It no longer holds the group in Initializing, which becomes NoPrimary naming the site. See Initializing after an operator start.

Topology decisions

LevelmsgFieldsNotes
WARNALERTmessage, fgA cross-site EvalCrossSite action returned an alert string (split brain, no primary, total loss). The same conditions emit SplitBrainDetected / NoPrimaryDetected / TotalLossDetected Kubernetes Events. A split brain whose second writable site the debounce has not confirmed yet ends its message with ; <sites> not yet confirmed. The Initializing state after an operator start has no alert and logs nothing here.
WARNsplit-brain auto-resolve: fencing non-preferred site per spec.splitBrainPolicy.sitePriorities(context)Opt-in splitBrainPolicy is fencing the lower-priority site. The poll-driven retry of a persistent split-brain re-emits this at WARN at most every 30s (DEBUG in between).
INFOfencing returning old primary (split brain after failover)site, fgA second site came back writable after a failover; the operator is fencing every writable site except the one holding live primary authority. Emitted at INFO on the first fence of a given split-brain and at DEBUG on the poll-driven retries in between.
ERRORfailed to fence returning old primarysite, error, fgThe super_read_only=ON write above failed. The next poll retries. DEBUG on retries, matching the INFO/DEBUG pairing of the fence itself.
INFOpost-fence application connections evictedsite, count, fgOperator-side recovery removed one or more sessions that survived the sidecar's best-effort fence eviction. It retries on later topology polls until an empty pass or timeout, without blocking the poll loop between passes. No pass runs while a planned failover holds its claim (or a planned-failover automation hold stands), an ordered update or an unobserved promotion owns the sites; the drain resumes afterwards with a fresh budget. A Dragonfly snapshot upgrade does not defer it.
WARNpost-fence application connection drain retry failedsite, error, fgA recovery drain pass could not enumerate or kill sessions. The operator retries within spec.connectionDrainTimeout.
INFOpost-fence application connection drain completesite, fgAn empty eviction pass proved no application sessions remain. Old-primary recovery may now complete.
WARNpost-fence application connection drain timed outsite, timeout, fgThe dedicated drain budget expired. Time spent deferred behind a planned failover's claim or hold, or an ordered update, does not count against it. Recovery proceeds because super_read_only=ON still prevents writes; a surviving session is limited to stale reads.
WARNre-asserting fenced promoted primary: no site is writable and the last failover target is GTID-complete; restoring writabilitysite, authority, fgThe last failover target was found fenced (read-only) with every site reachable and nothing writable — typically its own sidecar re-fenced it with a stale lease right after a promotion. The operator restores writability on the target. authority is lastFailoverTarget, or lastActiveSite for a group that has never failed over: the last site the operator saw as the group's unique writable primary (restored from status.activeSite after an operator restart), including a fresh-deploy seed and a primary that restarted read-only. While a spec.initFromBackup restore has not succeeded it is initFromBackupRestoreSeed: the site the restore loads into (restored from status.restore.targetSite, else status.activeSite, after an operator restart), re-asserted from any fence even with part of the load in it; its peers then only need to be read-only with no user schemas and no replication history. Mirrored by bloodraven_primary_reassert_total. Rate-limited to once per failoverCooldown.
WARNprimary re-assert refused: peer has transactions the target lacks — divergence needs human reviewsite, peerGtid, targetGtid, fgThe no-writable-site wedge was detected but restoring the last failover target would abandon peer transactions. The group stays read-only until an admin resolves the divergence. A peer's recorded divergentGtid (it carries a divergence report) is not counted: those transactions are already reported and stay on the peer; peerGtid is the peer's full set.
WARNprimary re-assert refused: the initFromBackup restore seed has replication configuredsite, fgThe restore seed reads as a replica, so it is not the seed the restore authority describes; writability is not restored.
WARNpromotion refused: the initFromBackup restore has not succeeded and only its seed holds the restored datasite, seed, fgEmergency failover or split-brain resolution would have promoted site while a spec.initFromBackup restore is unfinished. Nothing is cloned until the restore succeeds, so site is empty; promoting it would make the group serve an empty server once the restore succeeded. The group waits for seed.
WARNprimary re-assert refused: target no longer contains the recorded promotion GTID set (wiped or restored since promotion?)site, promotionGtid, targetGtid, fgThe failover history no longer describes the target's data lineage; the operator will not restore writability automatically.
WARNprimary re-assert refused: recorded promotion GTID set failed to parse — status corrupted or manually edited?site, promotionGtid, error, fgstatus.lastFailover.promotionGtidExecuted is non-empty but malformed. The operator wrote this value from MySQL itself, so a parse failure means corruption or manual tampering — the re-assert safety argument depends on it, so the operator refuses.
INFOfailover blocked by anti-flap cooldown(context), retryAtA failover decision was deferred because failoverCooldown has not elapsed since the last one. The operator re-evaluates at retryAt, when the cooldown expires, even if no site changes state.
WARNskipping promotion candidate: site is mid-keyring-rotationsite, unsealReason, fgEmergency failover chose another sealed candidate. Mirrored by Warning Event KeyringPromotionSkipped and bloodraven_keyring_promotions_blocked_total{outcome="skipped"}.
WARNpromotion refused: every remaining candidate is mid-keyring-rotationsites, fgNo sealed candidate remained. The group stays without a writable primary until rotation finishes. Mirrored by Warning Event KeyringPromotionRefused, Degraded reason NoPrimary with an UnsealReason=Rotation message, and bloodraven_keyring_promotions_blocked_total{outcome="refused"}.
WARNpromotion refused: split-brain winner is mid-keyring-rotationsite, unsealReason, fgSplit-brain auto-resolve would have picked a rotating site. Losers are not fenced.
WARNpromotion refused: candidate is mid-keyring-rotationsite, unsealReason, fgLast-line check refused failover.Execute on a rotating candidate.
WARNprimary re-assert refused: target is mid-keyring-rotationsite, unsealReason, fgThe last failover target is mid-rotation; writability is not restored.
WARNre-asserting restore offline_mode fencesite, fgA full-instance in-place restore holds offline_mode=ON on site, and the topology manager found it OFF (usually a mysqld restart mid-restore) and set it again.
ERRORfailed to re-assert restore offline_mode fencesite, error, fgThe re-assert above failed; the next poll retries. Application sessions may reach the primary until it succeeds.
WARNre-stopping restore peer replication IO threadsite, fgsite awaits its post-restore reclone and its replication receiver must stay stopped (the restore is past Fencing, or failed and left status.restoreReclones Held), but it was running or connecting (started by hand, or by an older operator). The topology manager stopped it again.
ERRORfailed to re-stop restore peer replication IO threadsite, error, fgThe re-stop above failed; the next poll retries. The receiver may pull the restore's DROP statements until it succeeds.
WARNcould not confirm a restore protection change; applying additions onlyfg, errorA restore guard change (a released hold, clone block, promotion exclusion, freeze or offline_mode fence, or any re-stop change) could not be confirmed by reading the group through the API server. Only additions from the operator's own view are applied; releases and re-stop changes wait for the next confirmed read.
WARNbootstrap refused: the recipient is held by an in-place restorerecipient, source, fgA clone into a site whose restore queue is Armed or Held, or while an in-place restore froze the topology, was refused, whoever asked (source is auto-clone, fresh-deploy or reclone), also when the restore arrived while the clone was being admitted. Logged once per site; mirrored by the BootstrapRefusedRestoreHold Warning Event.
WARNcould not read the group uncached; the new topology manager shares its predecessor's planned-failover claimfg, errorA topology manager starting with a predecessor (or with a claim already in the group's slot) could not read the group through the API server to hydrate the planned-failover claim. The claim slot is shared by every manager generation, and the predecessor's named active site (for example a source a planned failover left fenced, not yet re-asserted) is carried into the new manager (carried the replaced topology manager's named active site), so nothing is lost; a brand-new manager with an empty slot is not started and the next sync retries.
INFOcarried the replaced topology manager's named active sitefg, siteFollows the record above: the new manager's own restore named no active site, so it took the predecessor's (a site it still has).
WARNcould not read the group uncached; the new topology manager keeps its predecessor's restore protectionfg, errorA replacement topology manager could not read the group through the API server before starting, so it kept the restore holds of the manager it replaces. A brand-new manager in that situation is not started (failed to start topology manager) and the next sync retries.
ERRORread the primary's gtid_executed after a failed in-place restorefg, site, errorSettling a failed full-instance restore's reclone queue could not read the primary's gtid_executed for two minutes after the first failed read (or no topology manager ran for the group within ten minutes of the run ending). Whether the restore changed the primary is unproven, so the peers are held.
INFOholding the reclone queue: the failed restore's Job pods did not terminate in timefg, job, error (only when the pods could not be listed)A pod of a failed full-instance restore's Job was still running, or the pods could not be listed, ten minutes after the run ended. Whether the restore changed the primary is unproven (gtid_executed is not compared while a pod may still write), so the peers are held.
INFOold primary recovery deferred: replication held by an in-place restoresite, fgRecovery would have pointed a site awaiting its post-restore reclone at the primary and started its replication.
WARNpromotion refused: candidate is pending a post-restore reclonesites, fgEmergency failover, split-brain resolution or the primary re-assert would have promoted a site a full-instance in-place restore has not recloned yet (status.restoreReclones.pending). It still holds pre-restore data, so it is not promoted. Mirrored by Warning Event PromotionRefusedReclonePending and, when no other candidate remains, Degraded reason NoPrimary with a pending post-restore reclone message.
ERRORpromotion watermark too large for the out-of-band failover record; only the status copy carries itfg, target, bytesA promotion's promotionGtidExecuted exceeds 64 KiB, so the bloodraven.shipstream.io/promotion-gtid-executed annotation is not written; only status.lastFailover.promotionGtidExecuted carries it. A planned failover with such a fenced set cannot use the status-stall release.
WARNignored out-of-order local failover recordtarget, lastFailover, currentTarget, currentLastFailover, fgTwo local promotion paths reported out of timestamp order; the newer in-process anti-flap pair remains authoritative.
INFOcarried a retired topology manager's failover recordfg, target, lastFailoverA topology manager replaced by a configuration or credential change held a failover record newer than its replacement's (typically an ordered-update handoff that finished on the old manager, whose status writes are dropped once it is replaced). The record moved into the replacement under the timestamp-monotonic rule, at registration or once the old manager's update workers exited, so its cooldown and last failover target apply; the replacement's next status write and out-of-band flush persist it.
INFOnot carrying a retired topology manager's failover record: its target is no longer a sitefg, target, lastFailoverAs above, but the record's target was removed from (or renamed in) spec.sites, so the record is dropped rather than naming a site nothing can reach.
ERRORout-of-band anti-flap state write failed; retrying every pollfg, target, lastFailover, errorThe annotation write was rejected. The newest record stays pending and is retried on each poll.
INFOcross-site action deferred: in-place restore in progressfgDecisions are paused while restoreInPlace runs.
INFOcross-site action deferred: planned failover in progressfgEmergency failover, split-brain resolution and every other cross-site decision are paused while a planned failover holds its claim (from Fencing until its promotion is confirmed, recorded and published, or its terminal record is durable) or a planned-failover automation hold stands (AutomaticFailoverHeld=True). A request that is only waiting (Admitting without a grant, or Deferred) defers nothing. Logged on every poll while it lasts.

Sidecar fencing

The per-MySQL sidecar emits these in its operational stream. SELF-FENCING: is a stable prefix — msg strings that begin with it indicate the sidecar wrote super_read_only=ON to its local MySQL without operator instruction.

LevelmsgFieldsNotes
ERRORSELF-FENCING: topology mismatch — operator-authoritative active site disagrees with our site, setting super_read_only=ONsite, authoritativeActiveSite, observedAt, podThe operator (or a peer relaying the operator's view) reports a different active site than this sidecar is on. Fired even when the operator is reachable.
INFOfencing: clone in progress on this site; not fencingpodA CLONE INSTANCE is executing on this server. The operator makes a clone recipient writable because CLONE requires it, so the monitor skips both fencing rules while the clone runs and never kills the CLONE session. The recipient restarts after the clone and the startup safety net fences it. Repeats every monitor tick for the length of the clone.
ERRORSELF-FENCING: Bloodraven and every peer unreachable beyond lease timeout, setting super_read_only=ONbloodravenLastOk, latestPeerOk, peers, leaseTimeout, podBackstop rule: nothing is reachable, so we can't be sure we're still primary.
INFOSELF-FENCING: killed app connectionscount, podApplication sessions evicted after fencing succeeded. Server-internal threads are spared, so a fenced replica keeps applying from the authoritative primary.
WARNSELF-FENCING: failed to kill connections after fencingerror, count, podThe fence write landed but eviction was incomplete. count is how many sessions were killed — 0 when the process-list query itself failed, so no session was ever enumerated. error joins every cause that applied: the listing query failed, iteration ended early, some rows would not scan, some KILLs were refused, or the fence hit its 20s deadline. super_read_only=ON still holds, so surviving sessions cannot write. The sidecar does not retry because it cannot safely distinguish a pooled operator promotion session. After promotion, the operator drains the fenced former primary until a pass finds no sessions or spec.connectionDrainTimeout expires. The sidecar's /status reports self_fenced, scoped to the current sidecar process.
WARNSELF-FENCING: super_read_only write failed but the fence is in place; skipping connection evictionerror, podThe SET GLOBAL returned an error, but a follow-up @@super_read_only read proved the write landed. The fence counts as established: /status reports self_fenced=true and SELF-FENCED follows. Eviction is skipped because the fence budget is spent; operator-side recovery performs the bounded follow-up drain.
ERRORSELF-FENCING: super_read_only write outcome is unconfirmed; will probe againerror, probeError, podBoth the SET GLOBAL result and its independent confirmation probe were unavailable. The monitor retains a process-local pending marker and retries @@super_read_only on later ticks instead of getting stuck behind read_only=ON with self_fenced=false.
WARNfencing: could not resolve pending super_read_only writeerror, podA later tick still could not read @@super_read_only; the pending result remains armed for another tick.
WARNSELF-FENCING: previously unconfirmed super_read_only write is now confirmedpodA later tick proved the ambiguous write established super_read_only=ON; /status now reports self_fenced=true.
ERRORSELF-FENCING FAILED: could not set super_read_onlyerror, podThe fence write failed and a follow-up @@super_read_only read confirmed the instance was not super-fenced. The sidecar retries on the next tick.
WARNfencing: could not confirm whether the super_read_only write landederror, podThe immediate @@super_read_only confirmation read failed. The pending outcome is retained and retried on later ticks.
ERRORSELF-FENCED: super_read_only=ON has been set, only Bloodraven can restorepodFinal status; the sidecar will not unfence on its own. The next operator promotion clears it.
INFOfencing: MySQL is writable after prior self-fence; rearming monitorpodAn actor with SUPER privileges (the operator, per the restore contract) made MySQL writable again after a self-fence. The monitor re-arms with a fresh lease window — it will not re-fence until a full leaseTimeout passes with the operator and every peer unreachable again.
INFOfencing: MySQL turned writable; discarding active-site view from before the promotionpodThis sidecar's MySQL went from read-only to writable with no failed probe in between: the operator promoted the site. The monitor drops its cached active-site view, which may still name the old primary, and starts a fresh lease window, as after a self-fence restore. Without this, an operator that becomes unreachable right after a failover leaves the new primary to fence itself on the pre-promotion view.
INFOfencing: adopted active-site view from peerpeer, activeSite, observedAt, podPeer sidecar relayed a fresher view than what this sidecar had cached. Drives the topology-mismatch rule.

Safety-net events. The startup safety net runs before the fencing monitor starts, and each of its MySQL writes times out after 3 s. If its fence (or, on the active site, its clear) does not complete, the fencing monitor finishes the job on the first tick where MySQL answers. The monitor also re-arms the safety net after MySQL has been unreachable for 30 s or more. Rows marked monitor come only from that deferred path. The two rows marked both can come from either path.

LevelmsgFieldsNotes
INFOsafety net: set super_read_only=ON as precaution on startuppod
WARNsafety net: could not set initial super_read_only; the fencing monitor will retryerror, timeout, podThe startup fence failed or timed out (for example, a hung mysqld). Startup continues, and the fencing monitor applies the safety net once MySQL answers.
INFOsafety net: this is the active site, clearing super_read_onlysite, podboth
INFOsafety net: confirmed standby site, staying fencedsite, activeSite, pod
INFOsafety net: no active site reported by operator, staying fencedpod
WARNsafety net: could not query active site, staying fencederror, pod
ERRORsafety net: failed to clear super_read_only on active siteerror, podboth. Startup hands the clear to the monitor, and the monitor retries it on the next tick while the operator still confirms this site.
WARNsafety net: MySQL was unreachable long enough to have restarted; re-armingunreachableFor, threshold, podmonitor. A restarted mysqld comes back read_only=ON (the operator starts every mysqld with that flag), which stops application users but not SUPER sessions, so the monitor re-applies the startup rule and super-fences it unless the operator names this site active.
WARNsafety net: could not read super_read_only; will retryerror, podmonitor. Still armed.
INFOsafety net: operator confirms this is the active site; MySQL is already writablesite, podmonitor. Settled without a write.
INFOsafety net: operator confirms this is the active site; leaving the self-fence for the operator to clearsite, podmonitor. The safety net never lifts a fence this monitor set itself.
INFOsafety net: super_read_only=ON confirmed; staying fencedsite, activeSite, operatorAnswered, podmonitor. activeSite is the operator's answer on this tick, empty if it gave none.
INFOsafety net: set super_read_only=ON after MySQL became reachablesite, activeSite, operatorAnswered, podmonitor. Like the startup fence, this one does not evict sessions and does not set self_fenced. If the instance is writable and the authoritative view names another active site, the topology-mismatch rule fences instead, with eviction.
ERRORsafety net: could not set super_read_only; will retryerror, probeError, podmonitor. The fence write failed, and a confirming @@super_read_only read either failed or showed it did not land. Still armed.

PITR archiver

Emitted by the sidecar's BinlogArchiver and the supervisor that builds its archive store.

LevelmsgFields
ERROR / WARNfailed to init archive storeerror, attempt, retryIn, pod
INFOarchive store initialized after retriesattempts, failingFor, pod
INFObinlog archiver startingstorageType, binlogDir, binlogIndex, pollInterval, pod
INFOarchived sealed binlogscount, pod
INFOretention sweep complete(sweep stats), pod
WARNarchive binlogfile, error, pod
WARNretention: delete objectkey, error, pod
INFO / WARNretention deferredreason, pod; optional error for failed issuance
WARNfetch retention cutofferror, pod
WARNretention: coverage unavailablefile, error, pod
WARNretention: save coverageerror, pod
WARNretention: save delete acknowledgementserror, pod
WARNretention: compact retirement recordserror, pod
INFOarchive descriptor createdarchiveID, destination, site, pod
WARNarchive descriptor changedarchiveID, destination, error, site, pod
INFOpromotion recordedserverUuid, serverId, gtidExecuted, site, pod
WARNuploads wait for the promotion recorderror, site, pod
WARNserver incarnation reset recordedserverUuid, serverId, gtidExecuted, observedAt, probeInterval, site, pod
WARNserver incarnation changed while the sidecar was not runninglatestRecordAt, serverUuid, serverId, site, pod
WARNarchive history unsafeviolation, detail, site, pod
WARNpersist history integrity markererror, site, pod
WARNremove pending history integrity markerpath, error, site, pod

archive descriptor created is logged once, by the site whose atomic create wrote <prefix>/binlogs/archive.json; a sibling that lost the race verifies the winner's descriptor silently. archive descriptor changed means a re-verification before a manifest save or retention delete found a different or missing descriptor; the site stops writing and lastError carries the refusal token. promotion recorded follows each confirmed promotion record (the first writable observation of an episode); gtidExecuted is the server's set verbatim. uploads wait for the promotion record is logged on each scan of a writable site whose current episode has no saved promotion record yet (the record could not be written, or the role probe is incomplete); the site uploads nothing until it is saved, and lastError carries the error. server incarnation reset recorded follows each saved incarnation-reset record: the server's server_uuid or server_id changed, or its gtid_executed no longer contains the site's latest record's set (a reclone, reset or replaced datadir). probeInterval bounds how long before observedAt the change happened. server incarnation changed while the sidecar was not running is logged at sidecar start, before that record, when the first observation already shows such a change against the site's latest record (latestRecordAt); uploads wait until the record is saved. archive history unsafe is logged once per sidecar process for the first violation (violation is ReplicationFilter, BinlogFilter, BinlogFormat, BinlogRowImage or CommitOrder). persist history integrity marker means the marker could not be saved yet; uploads stay blocked and the save is retried every scan. A violation found while the archive is not yet verified (or is refused) is held as a pending marker in the sidecar's pod-lifetime state directory and saved at the first verified session; remove pending history integrity marker means that file (path) could not be deleted after the save. See Archive identity.

archive binlog also reports missing/divergent history and reset guards through error; /archiver/status.lastError retains the problem and bloodraven_archiver_upload_failures increases while it persists. The original message and field keys are unchanged. Audited divergent identities persist in the manifest and block affected transactions, including discovered aliases and an otherwise unchecked last row. Bootstrap/clone observations cannot establish divergence; eligible recovery on a fresh UUID can clear foreign-source flags. Known same-source loss stays unresolved on numeric return until verified or explicit canonical recovery. Reuse guards contain verified conflicting issuance; publication order does not set them. Quarantine or a canonical-backup acknowledgement provides explicit recovery. Recovery steps are in Repair PITR history safely.

failed to init archive store is logged once per failed archive store construction attempt. It used to be a single ERROR line followed by the sidecar exiting; the sidecar now keeps running and retries with jittered backoff from 5 s doubling to at most 5 minutes, so the line repeats at most once per backoff interval. It is ERROR when the error text differs from the previous attempt's (including the first) and WARN while the same cause repeats. attempt counts from 1; retryIn is the delay before the next attempt (a duration). archive store initialized after retries follows the first success after at least one failure; failingFor is the time since the first failed attempt. A first-attempt success logs neither.

The operator's archiver poller emits two lines about the PITRArchiveReady condition. A poller that cannot read the group through the uncached API reader (when its topology manager starts, or later) leaves the persisted condition as it is until it can:

LevelmsgFieldsNotes
WARNPITR archive condition read failed; leaving the condition as persistedfg, errorOnce per episode per topology manager, however long the reads keep failing. error also reports a group recreated under the same name (UID mismatch); that condition is not written by this manager.
INFOPITR archive condition read recoveredfgThe read that ends the episode; the condition is reconciled and written again from the next poll round.

Per-upload success/failure is also reflected in the bloodraven_archiver_upload_failures and bloodraven_archiver_last_upload_timestamp_seconds metrics — prefer those for alerting.

Kubernetes Event reasons for the PITR archive:

ReasonTypeObjectWhen
PITRArchiveNotReady / PITRArchiveRecoveredWarning / NormalMysqlFailoverGroupPITRArchiveReady entered False ("PITRArchiveReady is False (): "), or returned to True ("PITRArchiveReady is True again: every site reports …").
PITRArchiveUnsafeHistoryDetectedWarningMysqlFailoverGroupEdge-triggered on the transition into PITRArchiveReady=False / ArchiveUnsafeHistory, from a sidecar's history-integrity marker, in addition to PITRArchiveNotReady. spec.mysqlConf is not previewed.
PITRMysqlConfRefusedWarningMysqlFailoverGroupWhile PITR is enabled, spec.mysqlConf sets an option that breaks replay (replication or binlog filter, binlog_format≠ROW, binlog_row_image≠FULL); it was left out of my.cnf. Once per refused set; the condition PITRMysqlConfAccepted=False/PITRUnsafeSettingRefused lists each as <site>: <field> sets <key>[=<value>] (<why>).
PITRArchiveReissueRiskDetectedWarningMysqlFailoverGroupThe active site changed to a site whose last archiver status reports divergent or quarantined rows in its own manifest: "site <s> was promoted while its archive manifest holds N divergent/quarantined rows. If <s> was recloned after a lossy failover, it reissues the lost GTIDs, and every restore from this archive refuses PITRArchiveHistoryReset. Start a new archive prefix and take a full backup."
PITRArchiveRefusedWarningMysqlStandbyClusterThe standby refused the source archive (descriptor, encryption, keyCheck or a manifest); emitted when the SourceConfigKnown reason changes. The message is the refusal, token first.

PITRSharedArchivePrefix was removed with pointInTime.sites. The PITR replay Events (PITRReplayTruncated, PITRDivergentHistoryExcluded, PITROnHistoryTransactionsExcluded) are listed in Monitoring.

retention deferred logs reason="noRetainedBackups" or reason="backupInFlight" at INFO for normal waits. Bounded backfill with eligible work remaining also logs binlogGTIDCoverageUnknown at INFO while it schedules the next batch. archiveIssuanceUnverified at INFO means legacy/overlapping issuance checks are pending and new retirements wait. A failed issuance check logs the same reason at WARN with error, naming site/name/ID and the repair/quarantine remedy. The failure remains in /archiver/status.lastError throughout cooldown until verification succeeds or that identity is removed/quarantined. All new retirements for that site can remain deferred; monitor archive capacity. It does not itself schedule rapid retention retries. Other WARN reasons are backupMetadataUnknown (missing/malformed backup GTIDs or start times), backupGTIDCoverageUnknown (no compatible protocol/proof), and binlogGTIDCoverageUnknown (eligible files lack complete, content-bound transaction coverage with no immediately eligible backfill). Those coverage and backup deferrals retain files without themselves indicating an upload failure. Failed issuance audits do retain lastError and increase the archiver failure metric as described above.

retention: coverage unavailable warns once per failed inspection attempt; that identity waits the normal retention interval before retry. Healthy identities can continue in 30-second batches. fetch retention cutoff now logs lookup errors, including HTTP 429, at WARN with the existing error field. Throttling is per transport source IP; it cannot consume another source's group admission. A 429 has no delete authority and retries after 30 seconds; it is distinct from a successful safe-deferral response.

retention: delete object leaves a durable pending record for retry. retention: save delete acknowledgements means successful deletes could not be acknowledged; the next sweep safely repeats them. retention sweep complete keeps its cutoff, deleted, and manifest_pruned fields. For a sweep that only finishes older pending deletes without a current operator proof, cutoff is an empty string and manifest_pruned is zero. deleted counts successful, idempotent delete calls.

retention: compact retirement records means the manifest update that removes obsolete acknowledged records failed. Their durable records remain available for later scans; this does not re-upload retired data. Each site's maintenance can emit retention events while that site is a replica. Active archive counts exclude retirement records.

Data-at-rest encryption

Emitted when spec.encryptionAtRest.enabled=true. See Data-at-rest encryption for the lifecycle these events trace.

Operator events:

LevelmsgFieldsNotes
INFOkeyring phase transitionsite, from, to, reason, version, fgThe per-site keyring lifecycle advanced. The normal sequence is Pending → Unsealed → Escrowed → Sealed. Any transition to Unsealed from Sealed means the site is deliberately running with a writable keyring (clone or rotation).
INFOkeyring escrowednamespace, group, site, version, secret, digestThe operator accepted a sidecar escrow push and stored a new immutable Secret version. Also emitted by the sidecar with site, version, secret, digest, pod.
INFOrefusing to enable encryption at rest on a live failover groupactiveSite, fgspec.encryptionAtRest.enabled was turned on for a group that is already serving, with spec.encryptionAtRest.existingData: Refuse (the default). Rendering is unchanged; existing tablespaces would have stayed plaintext.
WARNkeyring escrow rejectednamespace, group, site, reasonA push to /keyring/escrow failed authentication. The response to the caller is deliberately vague; the reason is only here.
WARNkeyring escrow digest mismatchnamespace, group, site, claimed, computedAn escrow push carried a digest that did not match its payload. Nothing was stored.
ERRORkeyring escrow: store failednamespace, group, site, errorThe operator could not write the escrow Secret. The site stays unsealed and the sidecar keeps retrying.
ERRORkeyring rotation: cannot confirm whether held rotation stamps were persisted; keeping the promotion blockfg, sites, errorAn encryption pass ended without a successful status write, or wrote a status that would end a hold, and the uncached re-read that decides whether its (or an earlier pass's) UnsealReason=Rotation stamp landed failed. The listed sites stay promotion-blocked (emergency failover skips them, ordered updates and planned failovers will not promote them) until a later pass reaches the API server. Persistent repeats mean the operator cannot read its own CR.
ERRORkeyring rotation: cannot rule out a rotation stamp still in flight; keeping the promotion blockfg, sites, errorThe re-read showed no stamp for these sites, but the conditional write that makes an in-flight status write unable to commit (the bloodraven.shipstream.io/encryption-status-fence annotation bump) failed, typically a conflict or an API error. The sites stay promotion-blocked; the requeued pass retries.

Sidecar events:

LevelmsgFieldsNotes
INFOkeyring agent enabledpath, escrowArmed, rotate, podSidecar startup. escrowArmed=false means this pod renders a sealed keyring and only reports its digest.
INFOrotated innodb master keysite, podALTER INSTANCE ROTATE INNODB MASTER KEY succeeded.
INFOencrypted the mysql system tablespacesite, podALTER TABLESPACE mysql ENCRYPTION='Y' succeeded on the writable site.
WARNkeyring escrow push failed, will retryerror, site, digest, podThe push was rejected or the operator echoed a digest that did not match. Escrow is retried until it succeeds; the site is not sealed in the meantime.
WARNcould not encrypt the mysql system tablespaceerror, site, podRetried on the next tick.
ERRORkeyring rotation failederror, site, podThe rotation statement failed. The agent still escrows whatever is on disk, because a failed rotation may already have written a key.

Kubernetes Event reasons emitted on the MysqlFailoverGroup:

ReasonTypeWhen
KeyringPhaseChangedNormalAny keyring phase transition. A transition to Failed also has its own Warning (KeyringEscrowTimedOut, KeyringEscrowMissing, KeyringEscrowCorrupt, KeyringDigestMismatch, KeyringNotReadOnly).
KeyringEscrowTimedOutWarningA site's keyring was not escrowed within keyring.escrowTimeout; the site is Failed. The operator keeps retrying and the site stays unsealed.
KeyringUnsealedNormalA site was unsealed so a CLONE INSTANCE can rewrap tablespace keys.
KeyringRotationRejectedWarningA rotation was requested for the active primary, or while an ordered update or planned failover was in flight. Also emitted, once per reason, while a planned failover or promotion is in flight, the operator has not yet observed a unique writable primary with the target read-only, no topology manager runs for the group yet, or a manager being replaced still has an ordered update in flight. Every refusal except the active-primary one (by status or as the operator observes it) keeps the rotate-keyring annotation and retries every 10s.
KeyringPromotionSkippedWarningEmergency failover skipped a mid-rotation site and promoted another sealed candidate.
KeyringPromotionRefusedWarningEmergency failover refused to promote because every remaining candidate is mid-keyring-rotation. Finish the rotation, then the next poll can promote.
KeyringEscrowMissingWarningA sealed site's escrow Secret no longer exists. That site cannot restart. See the DR runbook.
KeyringEscrowCorruptWarningA sealed site's escrow Secret no longer hashes to the recorded digest.
KeyringDigestMismatchWarningA pod rendered sealed is running a keyring that does not match its escrow; the operator refuses to call it sealed.
KeyringNotReadOnlyWarningMySQL reports a writable keyring on a site the operator considers sealed.
EncryptionAdoptionRefusedWarningEncryption was enabled on a group that is already serving, and spec.encryptionAtRest.existingData is Refuse.

Dragonfly

Bloodraven optionally co-manages per-site Dragonfly instances and emits the following events when spec.dragonfly.enabled=true. Mirrored by the bloodraven_dragonfly_site_up gauge and the bloodraven_dragonfly_promotions_total{result} counter, plus the matching Dragonfly* Kubernetes Events on the MysqlFailoverGroup.

LevelmsgFieldsNotes
INFOdragonfly: configured replicasite, host, port, fgOperator issued REPLICAOF against a non-active site to align it with the active master.
WARNdragonfly: stale master on non-active sitesite, active, fgA site reports role=master but is not the active site. Auto-rejoin is attempted only when the stale instance has connected_slaves=0 AND master_repl_offset=0 (provably never accepted writes); otherwise the stale master is shed from the active Service via the traffic-label gate and left for human intervention.
INFOstale-master reconfigure: REPLICAOF appliedsite, host, port, fgAuto-rejoin succeeded: the stale master is now linked as a replica of the active master.
WARNstale-master reconfigure: REPLICAOF failedsite, host, port, error, fgAuto-rejoin attempt failed; the next tick retries.
INFOclient-kill: evicted clients from old mastersite, fgAfter a planned-failover Dragonfly promotion, the operator issued CLIENT KILL TYPE NORMAL against the demoted source so application clients reconnect through the active Service.
INFOdragonfly: source offset unavailable; cannot prove syncerrorA planned failover could not read the source Dragonfly's replication offset when it entered WaitingForDragonflySync (status.plannedFailover.dragonfly.sourceOffsetState: Unavailable). Session preservation cannot be proven, so the run applies onSyncTimeout at once: proceed continues with sessions: Lost, fail rolls back. Emitted through the reconciler's logger.
INFOdragonfly/mysql active-site drift: promoting Dragonfly replica to match MySQLoldSource, target, mysqlActiveSite, fgMySQL active site and Dragonfly master diverged; the manager is promoting the synced Dragonfly replica on the MySQL active site.
INFOdragonfly-only emergency: active master unreachable; promoting replicaoldSource, target, fgDragonfly master failed without a MySQL failover; the manager is promoting the single healthy replica and leaving MySQL status.activeSite unchanged.
INFOdragonfly emergency: REPLTAKEOVER succeededsite, fgAfter an emergency MySQL failover, Dragonfly was promoted with sessions preserved.
WARNdragonfly emergency: REPLTAKEOVER failed; falling backsite, error, fgEmergency promote could not preserve sessions; falling back to REPLICAOF NO ONE.
WARNdragonfly emergency: old master unreachable; skipping REPLTAKEOVERsite, oldSource, fgThe old Dragonfly master did not answer a dial and PING within 1s. A takeover cannot complete without the master, so the operator goes straight to REPLICAOF NO ONE (sessions lost).
WARNdragonfly emergency: re-dial for REPLICAOF NO ONE failedsite, error, fgAfter a failed REPLTAKEOVER, the fresh connection for the fallback could not be opened. The outcome is unconfirmed: a takeover can complete on the server after the client gives up, and a lost reply looks like a failure. The operator waits until the server can no longer be acting on the takeover (its budget plus 10s, plus 1s), then reads the target's role. One of the two promotion outcome unconfirmed / promotion failed lines below follows with the result.
INFOdragonfly emergency: target promoted via REPLICAOF NO ONE (sessions lost)site, fgEmergency promote completed via the empty-master fallback after REPLTAKEOVER failed or was skipped. Mirrored by the Warning Event DragonflySessionsLost (not DragonflyPromotionSucceeded) and by bloodraven_dragonfly_promotions_total{result="sessions_lost"}.
WARNdragonfly: REPLTAKEOVER not advertisedfgCapability probe found REPLTAKEOVER missing from the running command table.
INFOdragonfly: REPLTAKEOVER capability restoredfgA previously missing REPLTAKEOVER command is now advertised (typically after an image upgrade).
WARNdragonfly emergency: REPLICAOF NO ONE failedsite, error, fgThe fallback returned an error. A timeout or connection error is unconfirmed (the server may have applied it, or an earlier takeover may have completed), so the operator waits for the command to settle and reads the target's role. One of the two lines below follows with the result. MySQL emergency failover was not affected.
WARNdragonfly emergency: promotion outcome unconfirmed; target observed as mastersite, cause, fgA promotion whose reply was lost or whose fallback failed was resolved by observation: the target reports role=master. It is recorded as a completed promotion (labels, CLIENT KILL, source recreate, status stamp), with the Warning Event DragonflySessionsLost (not DragonflyPromotionSucceeded) and bloodraven_dragonfly_promotions_total{result="sessions_lost"}, since session continuity cannot be confirmed.
WARNdragonfly emergency: promotion failed; target not observed as mastersite, role, cause, fgThe target did not report role=master once the commands could no longer be in effect (role is the observed role, or unknown: <error>). The old source's traffic label is restored so the active Service is not left empty, and the failure is stamped (DragonflyPromotionFailed, result="failed"). The next Dragonfly manager tick promotes again if MySQL and Dragonfly still disagree.
INFOdragonfly emergency: restore old source traffic failedsite, errorRestoring the old source's traffic label after a promotion that did not happen failed. The reconciler's label sweep re-converges it on a later pass.
INFOdragonfly promotion: request coalescedsite, replacedSite, fgA MySQL emergency failover queued a Dragonfly promotion while an earlier one was still waiting to run. Only the latest request (site) is kept, since it reflects the latest MySQL decision; replacedSite was the target of the dropped one.
INFOdragonfly promotion: manager stopping; request droppedsite, fgA promotion was requested, or was about to run, while the group's Dragonfly manager was stopping (config or credential restart, group deleted, leadership lost). The replacement manager's Tick promotes if MySQL and Dragonfly still disagree.
INFOdragonfly promotion: yielding to another owner; skippingsite, fgA planned failover or rollout started waiting for the group's Dragonfly gate, or the manager began stopping, before this promotion sent any role-changing command. The promotion stops without sending one (the old source's traffic label is restored) and hands over the gate. A promotion that has already sent REPLTAKEOVER or REPLICAOF NO ONE never yields; it finishes and records its outcome first.
INFOdragonfly promotion: planned failover in flight; skippingsite, fgA queued emergency promotion found a planned failover that owns the sites or holds its claim (or an automation hold), or a Dragonfly snapshot upgrade in progress, which owns Dragonfly. During an upgrade nothing is stored for later: the upgrade reconcile re-derives the follow-up from MySQL authority.
INFOdragonfly promotion: request superseded by a newer decision; skippingsite, reason, fgA queued emergency promotion was dropped because a decision recorded after it was made points elsewhere: a planned failover claimed the group at or after the request's second (status.plannedFailover.claimTime; a run that is only waiting has decided nothing), MySQL failed over to another site (status.lastFailover.target), or Dragonfly was promoted to another site (status.dragonfly.lastPromotionTarget) while MySQL's status.activeSite does not name the request's target. reason says which. Status times have one-second resolution, so in a rare race (a promotion to the other site stamped in or after the request's second, before MySQL has published the request's target) a valid request is dropped; the Dragonfly manager's drift check promotes the MySQL site once pendingPromotionActiveSiteTTL (30s) after that promotion has passed.
INFOdragonfly rollout: promotion no longer valid; skippingsite, reason, fgA Dragonfly rollout promotion, re-checked against an uncached read once it held the gate, found its choice obsolete: a planned failover or snapshot upgrade is running, or the Dragonfly master is no longer the site it would take over from. The rollout requeues and decides again.
INFOdragonfly rollout: target no longer in sync; skippingsite, error, fgThe rollout promotion re-proves the target's sync under the gate; it was no longer in sync (or could not be checked). The rollout requeues.
INFOdragonfly rollout: site became the active Dragonfly master; holding its restartsite, fgA StatefulSet rollout was about to restart a site it had chosen as a replica, but a promotion has since made it the Dragonfly master. The restart is held; the next reconcile treats the site as the active one.
INFOdragonfly rollout: site is the planned failover's Dragonfly target; holding its restartsite, fgA StatefulSet template rollout would have restarted the Dragonfly target of a planned failover that owns Dragonfly. The restart is held while the run owns Dragonfly.
INFOdragonfly snapshot-upgrade deferredfg, reasonA snapshot-restore upgrade run that spec.dragonfly.image drift asks for (upgradeStrategy: SnapshotRestore) cannot start: a site's Dragonfly StatefulSet does not carry the current spec.dragonfly settings other than the image yet, or the active site's Dragonfly pod does not run its current template (the render rolls them out first), status.activeSite is empty, a planned failover run or automation hold owns the sites, the effective Dragonfly master is not status.activeSite (for example after a planned takeover failed once MySQL had moved, or a Dragonfly-only promotion; the run would restart a replica and re-point the real master at it), or a site's Dragonfly StatefulSet is refused for WorkloadSelectorMismatch. reason names the cause. Logged when the cause changes, not on every 10s retry; no site moves meanwhile. Mirrored by the DragonflyUpgradeDeferred Event. Replaces dragonfly snapshot-upgrade deferred: the Dragonfly master differs from the MySQL active site (fields dragonflyMaster, activeSite).
INFOdragonfly promotion: target already promoted since the request; skippingsite, fgThe queued request's target was already promoted: either a promotion to it was stamped after the request was made (for example by a Tick that saw MySQL move first), or status already names it and it still reports role=master. Nothing is repeated, so no duplicate events or metrics.
INFOdragonfly gate busy; retryingfgA planned-failover Dragonfly phase, a rollout promotion, a replica StatefulSet rollout or a snapshot-upgrade step waited 5s for the group's Dragonfly gate, which another writer (the Dragonfly manager's current Tick or promotion, or a stopping manager's last step) still held. The reconcile requeues after 1s. A planned failover also reports the wait (DragonflyPromotionDeferred).
DEBUGdragonfly label sweep skipped: gate busyfgThe reconciler's Dragonfly pod-label sweep skipped this pass because another writer held the group's Dragonfly gate. The next reconcile sweeps again.
WARNdragonfly emergency: target unreachable; skipping promotionsite, error, fgThe target could not be dialled within the 5s preliminary step budget.

Kubernetes Event reasons emitted on the MysqlFailoverGroup (visible via kubectl describe):

ReasonWhen
DragonflyPromotionStartedPlanned-failover state machine entered PromotingDragonfly.
DragonflyPromotionSucceededDragonfly target was promoted by a planned failover or by an emergency REPLTAKEOVER. The emergency REPLICAOF NO ONE fallback, and an outcome confirmed only by observation, emit DragonflySessionsLost instead.
DragonflyPromotionFailedPromotion command failed; behavior depends on spec.dragonfly.plannedFailover.onSyncTimeout (planned) or is best-effort (emergency). A planned failover also emits it with the takeover did not finish within <d> of its intent when PromotingDragonfly passed its deadline (40s + maxSyncWait + 12s from takeover.intentTime) without a takeover; no REPLTAKEOVER is sent then.
DragonflySessionsLostEmergency promotion fell back to REPLICAOF NO ONE after REPLTAKEOVER failed or was skipped, or its outcome was only confirmed by observing the target as master. Session continuity is not guaranteed. Emitted instead of DragonflyPromotionSucceeded.
DragonflyReplTakeoverUnsupportedRuntime probe found REPLTAKEOVER missing from the running Dragonfly command table.
DragonflyReplTakeoverSupportedRuntime probe transitioned from unsupported to supported (typically after an image upgrade).
DragonflyStaleMasterDetectedA non-active site reports master role. Logged + dedup'd in 5-minute windows. Auto-rejoin is attempted in reconcileReplication when connected_slaves=0 AND master_repl_offset=0.
DragonflyOldSiteReconfiguredA stale master passed the auto-rejoin gate and was attached as a replica of the active master via REPLICAOF.
DragonflySyncTimedOutWaitingForDragonflySync exhausted spec.dragonfly.plannedFailover.maxSyncWait.
DragonflyUpgradeStartedA snapshot-restore Dragonfly upgrade run started (upgradeStrategy: SnapshotRestore and a site's live StatefulSet runs an image other than spec.dragonfly.image); status.dragonfly.upgrade was initialized.
DragonflyUpgradeDeferredNormal. A needed snapshot-restore upgrade run waits: no status.activeSite, a planned failover run or automation hold (including a durable one after a restart before the topology manager registered) owns the sites, the Dragonfly master differs from the MySQL active site, or a Dragonfly StatefulSet is refused for WorkloadSelectorMismatch. Emitted when the cause changes; retried every 10s.
DragonflyUpgradeSnapshotStartedActive Dragonfly traffic was shed and the operator is about to issue SAVE.
DragonflyUpgradeSnapshotCompletedSAVE completed against the active Dragonfly master using spec.dragonfly.snapshot.dir.
DragonflyUpgradeSucceededActive and replica Dragonfly pods are on the target image, active traffic is restored, and replicas are linked.
DragonflyUpgradeFailedSnapshot-restore upgrade reached a terminal failure; the operator best-effort restored active traffic. With reason ActiveSiteChanged the upgrade was abandoned because a promotion moved the Dragonfly master away from its retained site; routing labels are left to that promotion. With reason WorkloadSelectorMismatch a site's StatefulSet was found refused before the next step, and nothing was saved, restarted or updated by that step.
DragonflyPromotionDeferredWarning. A planned failover reached PromotingDragonfly while another Dragonfly owner (usually an emergency promotion that had already sent its role change) held the group's Dragonfly gate. MySQL writes stay paused on the fenced source until it finishes: about 12-13s in practice, about 40s in the worst case. Emitted once per wait; status.plannedFailover.dragonfly.message says the same.

License

Offline license observation. These events never change failover or reconcile behavior. updatesUntil is an RFC3339 UTC timestamp string.

Per-group records (always include fg as namespace/name):

LevelmsgFieldsFired when
INFOlicense verifiedfg, organization, edition, updatesUntil, issuedFor, kidA group's resolved token newly verified.
INFOlicense updates period endedfg, organization, edition, updatesUntilA valid token's updatesUntil is in the past. Perpetual state, not an error.
WARNlicense invalidfg, reason, kidA presented token failed verification. The group continues as Community.

Process-level records for the operator default (no fg):

LevelmsgFieldsFired when
INFOoperator license verifiedorganization, edition, updatesUntil, issuedFor, kid--license / BLOODRAVEN_LICENSE verified at process start.
INFOoperator license updates period endedorganization, edition, updatesUntilOperator default token is valid but the update period has ended.
WARNoperator license invalidreason, kidOperator default token failed verification. Process still starts.

Kubernetes Event reasons emitted on the MysqlFailoverGroup:

ReasonTypeWhen
LicenseVerifiedNormalA presented token newly verified, including the perpetual ended-updates state.
LicenseInvalidWarningA presented token failed offline verification. Advisory only.

Backup stamp

Emitted by bloodraven backup-stamp, which runs as the backup-stamp init container of every backup Job and again from dump.py on plaintext Jobs: with --check before the dump starts and with --end after it. Records carry subcommand="backup-stamp", and every record after the store is built carries key (the stamp's object key or path), backup and site.

LevelmsgFieldsFired when
INFObackup stamp writtenkey, backup, site, reused, serverID, pitrArchiveID, serverClockStartThe init container wrote the stamp before the dump. reused is true when a retried pod adopted the stamp an earlier attempt wrote (its earlier serverClockStart is kept). pitrArchiveID is empty when PITR was off.
INFObackup stamp finishedkey, backup, site, serverClockStart, serverClockEndserverClock.end was added after a successful dump. On encrypted Jobs backup-encrypt-upload logs it after the upload, with its own subcommand and record fields.
ERRORread dumped server clockerrorUTC_TIMESTAMP(6) and @@global.server_id could not be read from the server the dump uses.
ERRORcheck dumped servererrorThe server answered with a server_id other than the source site's pinned one (its spec.sites[].serverID).
ERRORwrite backup stamperrorThe stamp could not be written, or a different stamp (another backup's identity, or one that already records an end) is at the key.
INFObackup stamp checkedkey, backup, site--check: the location still holds this backup's unfinished stamp, so a plaintext dump may start.
ERRORfinish backup stamperrorThe end step failed. Also logged by backup-encrypt-upload.
ERRORcheck backup stamperrorThe location no longer holds this backup's unfinished stamp (another backup took it over): backup-stamp --check before a plaintext dump, which then does not start, or backup-encrypt-upload before uploading, which uploads nothing.

Configuration failures log read backup stamp template, resolve backup stamp storage, init backup stamp store, parse serverClock.end or install backup stamp tool, also with error. Every failure fails the backup, and the container's termination message starts with backup stamp: . Times are µs RFC 3339 UTC strings.

Backup retention and verification holds

Emitted by the MysqlBackup reconciler. namespace and backup name the MysqlBackup; fg and profile are its failover group and profile names (not namespace/name, unlike the topology records above).

LevelmsgFieldsFired when
INFObackup prune held by verificationnamespace, backup, backupUID, fg, profile, verifications, verificationUIDsRetention would prune a successful backup, but one or more MysqlBackupVerifications may still be reading it. verifications and verificationUIDs are parallel lists, sorted by name. The backup is kept and gets a RetentionDeferred Event; it is pruned on the first retention pass after the runs' pods stop. Logged on every retention pass while the hold lasts.
INFOremoved dead verification holdsnamespace, backup, holdsA MysqlBackup being deleted carried verification-hold.bloodraven.shipstream.io/<uid> annotations whose holder can no longer read it: the verification is gone (for example its finalizer was removed by hand) or was replaced by a same-name one, with no pods of that name running; or it still exists but is finished, its Job has a terminal condition (or is gone) and its pods have stopped. holds lists the removed annotation keys.

PITR archive bindings

Emitted by the MysqlFailoverGroup reconciler for the group's archive encryption bindings. fg is the group's name. No record carries passphrase bytes, an unkeyed digest of them, or a keyCheck. keySecretName (mysql-<group>-pitr-key-<h10>) does end in a truncated keyed digest: h10 is an HMAC, under the group's digest key, over the destination and the passphrase bytes. Without that key it reveals nothing about the passphrase, but treat it as key-derived rather than key-free (see Key snapshots).

LevelmsgFieldsFired when
INFOpitr archive binding changedfg, bindingID, destination, archiveID, state, reason, keySecretName, causeA binding entry is created, or its state, reason, archiveID or keySecretName changes. The last two can change with state and reason unchanged: a Pending/Authenticating check that rejects one candidate key and starts the next logs a line whose keySecretName differs and whose state/reason repeat. Filter on state/reason yourself if you only want those transitions. cause is proof (a sidecar report), check (a pitr-archive-check Job result, including a rejected candidate), recovery (rebuilt from a key snapshot's acceptance annotation after a status loss), integrity (the snapshot no longer verifies), c3 (the snapshot is missing: re-created under the same name, or KeyMaterialLost) or input (any other spec, Secret or snapshot input change; for example a new target's first entry, the identity profile's key Secret or key missing, a Refused entry retried after an input change, a check or Establishing candidate superseded by an identity change, or an acceptance that could not be recorded). Every line carries a cause.
INFOpitr archive binding releasedfg, bindingID, destinationA release annotation removed a binding and deleted its snapshots.
WARNpitr render annotation ignoredfg, namespace, site, errorA site Deployment's bloodraven.shipstream.io/pitr-render annotation is malformed or has an unknown schemaVersion; it counts as absent.
INFOpitr render deferredfg, namespace, sitesWhile any site Deployment has no decodable render annotation (and the write binding is not Bound or Establishing), existing site Deployments and their ConfigMaps are not updated until a check Job authenticates the live key; sites lists their sites. A site created meanwhile renders.

The check Job (bloodraven pitr-archive-check) logs INFO pitr archive check (destination, role, state, reason, archiveID) and writes one termination line, which the operator parses:

BLOODRAVEN_PITR_ARCHIVE_CHECK v=2 state=<Absent|Verified|Refused> reason=<token or empty> archiveID=<id> sourceUID=<uid> sourceNamespace=<ns> sourceName=<name> destinationDigest=<64 hex>

role is Writer or Reader (BLOODRAVEN_PITR_CHECK_ROLE). Values are percent-encoded like the PITR verdict. A transient storage error is retried 5 times; the Job then exits non-zero without a line. A verdict that cannot be written to the termination log exits 4 with ERROR write the verdict to the termination log; the check fails (path, error); the operator then reports "check Job <job> failed: ..." and retries. The operator likewise retries a check that exits 0 with a line that does not parse ("check Job <job> reported a verdict line that does not parse: <parser error>").

Lifecycle

LevelmsgFields
INFOstarting bloodraven manager(none)
INFOstarting auxiliary HTTP serveraddr
INFOtopology manager runner starting(none)
INFOstarting topology managerfg
INFOtopology manager stoppedfg
INFOstopping topology managerfg
INFOconfig changed, restarting topology managerfg
INFOgroup recreated, restarting topology managerfg, previousUid, uid
WARNtopology manager exited; restartingfg
WARNtimed out waiting for the Dragonfly manager to stop; its last step finishes under the group's Dragonfly gatefg
WARNconnection material unreadable; keeping the running topology managerfg, error
ERRORfailed to start topology managerfg, error
INFOrestored lastFailoverTarget from CR statusfg, target
INFOrestored lastFailover from CR statusfg, lastFailover
WARNrestored lastFailoverTarget from out-of-band annotationsfg, target, statusTarget
WARNrestored lastFailover from out-of-band annotationsfg, lastFailover, statusLastFailover
ERRORout-of-band anti-flap annotation unreadable; falling back to CR statusfg, error
ERRORCR status anti-flap state unreadable; ignoring unsafe copyfg, error
INFOstarting graceful shutdownfg
INFOCR deleted — DNSEndpoint will be garbage-collected(none)
INFOsidecar image version differs from the operatoroperatorImage, sidecarImage (the effective image: the pin, or the operator's default), operatorTag, sidecarTag
WARNBLOODRAVEN_SIDECAR_IMAGE is not set; groups without spec.sidecar.image cannot render their Deployments(none)
INFOsidecar startinglistenAddr, peerAddresses, bloodravenAddress, leaseTimeout, peerCheckInterval, site, namespace, fg, pod
INFOsidecar stoppedpod
INFOreceived signal, shutting downsignal, pod

Kubernetes Event reasons emitted on the MysqlFailoverGroup:

ReasonTypeWhen
SidecarVersionSkewWarningThe group's sidecar image (a spec.sidecar.image pin, or the operator's default BLOODRAVEN_SIDECAR_IMAGE when it pins none) is tagged with a different release than the running operator. The two halves share a rendering contract, so a mismatch can break the sidecar silently — on a spec.tls group an older sidecar cannot reach MySQL at all. Advisory only: the operator reports skew but never blocks on it, because a re-tagged or locally built image is indistinguishable from a real mismatch.
TopologyManagerStartFailedWarningThe group has no running topology manager because starting one keeps failing; mirrors the ERROR failed to start topology manager. Emitted on the first failure after a running manager was torn down, otherwise once the failure outlives one 30s retry; then at most once per 5 minutes (a changed error) or every 10 minutes (unchanged), because Kubernetes throttles a group's Warning Events together, whatever their reason.
TopologyManagerStartedNormalA topology manager started after a TopologyManagerStartFailed Warning; the message gives the number of failed attempts. Failure history is kept in memory only, so a recovery after an operator restart is not announced.

Breaking changes

v1.7.0: PITR archive identity (#206)

  • Removed messages: pitr-download's WARN stop proof scan ended at unreadable successor and WARN quarantined binlog excluded, and the anchorReason texts, which belonged to the removed per-site planner. download site binlogs is kept; its error tokens changed (see PITR Job completion and failure metadata): PITRArchiveSiteOmitted, PITRCrossSiteOrder, PITRBackupBoundUnknown and PITRArchiveFilteredHistory are gone; the archive identity, dump, frontier, divergent-history, gap and order tokens are new.
  • Added Job messages: INFO dump metadata and ERROR read dump metadata (dump-meta), INFO pitr plan and the BLOODRAVEN_PITR_DIVERGENT_SET line (pitr-download). pitr-download no longer logs INFO pitr-download complete, ERROR missing required env var or ERROR parse stop-datetime; its configuration failures log invalid pitr-download config.
  • Verdict: BLOODRAVEN_PITR_VERDICT v=2, followed by BLOODRAVEN_PITR_RESULT v=1 and an optional BLOODRAVEN_PITR_ON_HISTORY_EXCLUSIONS line; v=1 is gone and is refused.
  • restore.py sentinels: added BLOODRAVEN_PITR_SEGMENT, BLOODRAVEN_PITR_RESULT v=1, BLOODRAVEN_PITR_DIVERGENT count=, BLOODRAVEN_PITR_DIVERGENT_SET gtids= and BLOODRAVEN_PITR_ON_HISTORY_EXCLUSIONS gtids=; removed BLOODRAVEN_PITR_NOOP; BLOODRAVEN_PITR_START and BLOODRAVEN_PITR_COMPLETE changed their fields; BLOODRAVEN_RESTORE_COMPLETE dropped its pitr* keys.
  • Removed sentinel and Events: BLOODRAVEN_PITR_CROSS_SITE_UNVERIFIED, the PointInTimeCrossSiteUnverified Event and reason, and the PITRSharedArchivePrefix Event.
  • Added messages: sidecar archive descriptor created, archive descriptor changed, promotion recorded, uploads wait for the promotion record, archive history unsafe and persist history integrity marker, remove pending history integrity marker, server incarnation reset recorded and server incarnation changed while the sidecar was not running (PITR archiver), and the backup-stamp records, including backup-encrypt-upload's check backup stamp (Backup stamp).
  • Removed line: verify.sh's BLOODRAVEN_VERIFY_REPLAY_COMPLETE (file, position, timestamp) is no longer printed; a verification's replay detail comes only from status.pointInTime (for example status.pointInTime.lastReplayedBinlog and status.pointInTime.newestReplayedCommitTime).
  • Added Events: PITRArchiveUnsafeHistoryDetected, PITRMysqlConfRefused, PITRArchiveReissueRiskDetected, PITRDivergentHistoryExcluded and PITROnHistoryTransactionsExcluded (restored group or verification) and, on MysqlStandbyCluster, PITRArchiveRefused.
  • Environment: restore and verification Jobs no longer render BLOODRAVEN_PITR_SITES, BLOODRAVEN_PITR_ALLOW_UNVERIFIED_CROSS_SITE or BLOODRAVEN_PITR_START_GTIDS; they render BLOODRAVEN_PITR_DESTINATION, BLOODRAVEN_PITR_ENCRYPTION_MODE, BLOODRAVEN_PITR_MODE and BLOODRAVEN_PITR_DUMP_META, BLOODRAVEN_PITR_STOP_DATETIME is always the canonical µs UTC form, and a dump-meta init container runs first. Sidecars also get BLOODRAVEN_FAILOVER_GROUP_UID and BLOODRAVEN_PITR_ARCHIVE_ID.
  • Status fields: /archiver/status gains archive, divergentRows, quarantinedRows, promotionRecordedAt and historyIntegrity. The PITRArchiveReady condition gains the archive identity reasons and SidecarOutdated, and no longer previews spec.mysqlConf; the new PITRMysqlConfAccepted condition reports refused mysqlConf options. Manifest records gain kind (Reset for incarnation-reset records), probeInterval and readOnlyGtidExecuted. status.restore.pitr* is replaced by status.restore.provenance.pointInTime, and status.restoreInPlace.pitr* is removed (status.restoreInPlace.provenance.pointInTime stays unset); verifications add status.pointInTime.

v1.7.0: PITR archive encryption bindings (#203)

Additions only; no documented msg or field was renamed or removed.

  • Added operator messages: pitr archive binding changed, pitr archive binding released, pitr render annotation ignored, pitr render deferred (PITR archive bindings).
  • Added Job line and message: BLOODRAVEN_PITR_ARCHIVE_CHECK v=2 (its last key is destinationDigest, the hex SHA-256 of the normalized destination, which bounds the line under the 4096-byte termination message limit) and INFO pitr archive check from bloodraven pitr-archive-check.
  • Added Events on the MysqlFailoverGroup: PITRArchiveBound, PITRArchiveBindingReleased, PITRArchiveEncryptionChangeRefused, PITRArchiveKeyChangeRefused, PITRArchiveBindingRefused, PITRArchiveKeySourceRemoved, PITRArchiveKeySnapshotRecreated, PITRArchiveKeyMaterialLost, PITRArchiveKeySnapshotReadFailed, PITRArchiveBindingReleaseRefused, PITRArchiveBindingReleaseRejected, PITRArchiveBindingLimitReached, BackupScheduleRefused; on a MysqlBackup, Normal PITRArchiveStampSkipped (the backup carries no archive stamp because the site renders disagree or none is readable). New condition PITRArchiveBindingReady; new MysqlBackup reasons ProfileReadOnly, PITRArchiveRolloutPending, DumpKeySnapshotConflict, PassphraseSecretMissing; new restore/verification tokens PITRArchiveBindingNotReady and PITRBackupDumpKeyMissing.
  • Environment: sidecars get BLOODRAVEN_PITR_KEY_REVISION (the key snapshot name or plaintext) and no longer BLOODRAVEN_PITR_PROFILE_NAME; the retention query sends archive= instead of profile=.

Stability commitments

WhatStability
msg strings listed in the Event referenceStable. Changes go through a deprecation note in the release notes.
Field names listed alongside a stable msgStable. New fields may be added to existing events; existing fields will not be renamed or removed without a deprecation note.
Field value shapes (strings, numbers, durations)Stable for the values listed. GTIDs are passed through verbatim from MySQL — their shape is whatever MySQL emits.
time, level, msg field names themselvesStable. Tied to log/slog defaults.
DEBUG-level recordsUnstable. May appear, disappear, or change shape without notice. Disabled by default.
Ad-hoc INFO/WARN/ERROR records not listed above (e.g. retry warnings, transient probe errors)Best-effort. Field set is intended to be useful but not contractual. Don't build alerts that key on the exact msg string.
Controller-runtime (zap) streamInherited from upstream. Bloodraven does not redefine this stream's shape.

Pipeline integration tips

Filtering operational vs. controller-runtime

Most aggregators (Loki, Elasticsearch, Vector) let you split streams by JSON shape. A reliable predicate:

$.time && $.msg   // operational (slog)
$.ts && $.logger  // controller-runtime (zap)

Per-event alerts

Because every key event has a stable msg, pipeline alerts can be expressed as exact-match filters rather than fragile regexes. Examples for Loki:

# Failover started
{app="bloodraven"} | json | msg = "initiating failover"

# Failover failed (escalate)
{app="bloodraven"} | json | level = "ERROR" and msg = "failover failed"

# Divergence requires manual reclone
{app="bloodraven"} | json | msg = "divergence detected"

# Reclone triggered (track who/what asked for it via fg + recipient)
{app="bloodraven"} | json | msg = "starting bootstrap" and source = "reclone"

# Sidecar self-fenced — page on this
{app="bloodraven-sidecar"} | json | msg =~ "^SELF-FENCING:"

# Planned failover held automatic failover — page on this
{app="bloodraven"} | json | msg = "planned failover: automation held pending acknowledgement"

Correlating with metrics and Kubernetes Events

Several stable log events are mirrored by other observable signals — when one fires, the others fire too:

Log eventMetricKubernetes Event
failover completebloodraven_failovers_total{target_site}FailoverExecuted
divergence detectedbloodraven_divergent_transactions{site} > 0DataLossDetected (old primary) or DivergentReplicaDetected (replica audit)
divergence cleared: replica holds no transactions the primary lacksbloodraven_divergent_transactions{site} returns to 0DivergenceResolved or OldPrimaryRecovered
divergent site promoted; its extra transactions are now part of the primary's history(none)DivergentSitePromoted
old primary recovery completebloodraven_divergent_transactions{site} returns to 0OldPrimaryRecovered
ordered update: stalled; site pod not Ready(none; the Updating condition carries it)OrderedUpdateStalled, once per stall
failed to start topology manager(none)TopologyManagerStartFailed, rate-limited: at once when a running manager was torn down, otherwise after one 30s retry, then at most every 5 minutes
planned failover: automation held pending acknowledgement(none; status.plannedFailoverHold carries it, summarised by the AutomaticFailoverHeld condition)PlannedFailoverManualRecoveryRequired, then PlannedFailoverHoldPersisted every 10 minutes
planned failover: status stalled; ended the run with a terminal markerbloodraven_planned_failovers_total{result}PlannedFailoverStatusStalled
planned failover: claim released (why=publishing)bloodraven_planned_failovers_total{result="succeeded"} once Succeeded landsPlannedFailoverSucceeded
state transitionbloodraven_state_transitions_total{site, from, to}(none — too noisy for events)

Prefer metrics for alert thresholds and Kubernetes Events for human notification routing; logs are richest for forensics and timeline reconstruction.

Useful structured fields to index

If your pipeline supports indexing specific fields, the high-value ones are:

  • fg — partitions everything by failover group
  • site (and oldPrimary / newPrimary / promotedSite / donor / recipient) — for per-site timelines
  • level — for severity routing
  • source — for bootstrap/reclone disambiguation
  • error — full error string from the operator's error chain

PITR Job completion and failure metadata

PITR restore and verification Jobs run dump-meta, pitr-download, optionally decrypt-download, and then restore.py (see The restore Job). The init containers log JSON records. dump-meta records also carry subcommand="dump-meta".

LevelmsgFieldsNotes
INFOdump metadatalocation, backupUID, sourceUID, pitrArchiveID, site, dumpEnddump-meta read and accepted the dump's @.json, @.done.json and stamp, and wrote dump.json. dumpEnd is µs RFC 3339 UTC; pitrArchiveID is empty when the backup was taken with PITR off.
ERRORread dump metadataerrordump-meta refused the dump (error starts with its token), or could not read it from the store.
INFOpitr planarchiveID, mode, outcome, witness, groundTruth, segments, files, sites, replayed, divergent, onHistoryExcluded, downloaded, indexedTransactionspitr-download found a plan and staged it. witness is the witness GTID and groundTruth is Origin or Anchored (both empty without a witness); sites is a comma-separated list; replayed is the replayed transaction count; divergent and onHistoryExcluded are GTID counts; downloaded (archived files fetched) and indexedTransactions size the planning work.
ERRORdownload site binlogserrorKept. pitr-download refused (error starts with its token) or failed to plan or stage.

Configuration and setup failures log, also with error: missing required env, invalid dump-meta config, resolve dump storage, init dump store and write dump metadata (dump-meta); invalid pitr-download config, invalid archive config and init archive store (pitr-download).

dump-meta and pitr-download write the same refusal, stable token first and bounded to about 3 KiB, to their termination messages, and the operator copies it into restore and verification status. The tokens and their order are in Restore refusals. A timestamp-mode PITRArchiveDivergentHistory names the transactions to exclude, up to 1 KiB, and pitr-download prints the whole set to its log as BLOODRAVEN_PITR_DIVERGENT_SET gtids=<set, up to 64 KiB>; it prints the same line on success when the plan excluded divergent transactions. coverage.json in the staging directory records the full plan, refusals included.

restore.py prints these text sentinels to its log:

SentinelWhen
BLOODRAVEN_PITR_TARGET_LOCAL gtids=<set, up to 64 KiB>Before the load, when the target already holds its own transactions (image initialization, first-boot users); they are kept and are part of the certified executed set.
BLOODRAVEN_PITR_START mode=<mode> stop=<target or none> segments=<n> transactions=<n>Before the first segment.
BLOODRAVEN_PITR_SEGMENT k=<index> file=<site>/<name> gtids=<count> transactions=<n>Before each segment (file is percent-encoded).
BLOODRAVEN_PITR_COMPLETE segments=<n>After the replay; segments=0 when the backup already is the target state.
BLOODRAVEN_PITR_FAILED: mysqlbinlog exit=<rc> segment=<k> / BLOODRAVEN_PITR_FAILED: mysql exit=<rc> segment=<k> (…)A replay process failed; on a mysqlbinlog failure mysql is killed first. The mysql form's parenthesised note says <k> is where the failure was detected (the failing statement may be from an earlier segment, since mysql reads ahead) and appends a pipe or read error; a mysql still running then is killed (exit=-9). Also written to the termination message.
BLOODRAVEN_PITR_FAILED: cannot open a session to … / BLOODRAVEN_PITR_FAILED: cannot read the replayed gtid_executed to prove the replay (R6): …The fresh session for the baseline proof (R2) or the replay proof (R6) could not be opened or read: an operational failure, not a mismatch.
BLOODRAVEN_PITR_PARTIAL requested=<target> newest=<time or none>Outcome Partial.
BLOODRAVEN_PITR_DIVERGENT count=<n> and BLOODRAVEN_PITR_DIVERGENT_SET gtids=<set, up to 64 KiB>The replay excluded transactions outside the target history. A set cut at 64 KiB ends with ....
BLOODRAVEN_PITR_ON_HISTORY_EXCLUSIONS gtids=<set, up to 64 KiB>An excluded transaction on the target history precedes a replayed one. A set cut at 64 KiB ends with ....
BLOODRAVEN_PITR_RESULT v=1 executed=<set> or BLOODRAVEN_PITR_RESULT v=1 executedSha256=<hex>The proven post-replay @@global.gtid_executed; only its digest when the set is longer than 2 KiB.

A PITR Job's outcome comes from its main container's termination message, not its log. On success restore.py (through verify.sh on a verification) writes, as its last action:

BLOODRAVEN_PITR_VERDICT v=2 mode= stop= outcome= archive= encryptionMode= sourceUID= sourceNamespace= sourceName= segments= files= transactions= lastFile=<site>/<name> newest= divergent=<n> onHistoryExcluded=<n> executedSha256= sites=
BLOODRAVEN_PITR_RESULT v=1 executed=<set>
BLOODRAVEN_PITR_ON_HISTORY_EXCLUSIONS gtids=<set>
  • Line 1, the verdict, is at most 1024 bytes. mode is timestamp or latest; stop is the µs UTC target (empty in latest mode); outcome is TargetCovered, Partial or ArchiveHead; archive and encryptionMode identify the archive and source* its source group; files counts the distinct files the segments read; transactions the replayed transactions; divergent and onHistoryExcluded are GTID counts; executedSha256 is the hex SHA-256 of the post-replay gtid_executed; sites lists at most eight staged sites plus a +N count. stop, source*, lastFile, newest and sites are percent-encoded.
  • Line 2 carries the post-replay gtid_executed, or only executedSha256=<hex> when it is longer than 2 KiB.
  • Line 3 is present only when the on-history exclusions are non-empty and short enough to fit; otherwise status records only their count.
  • The message fits the main container's share of the kubelet's termination messages (12 KiB split among the pod's containers, at most 4 KiB each), which the operator renders as BLOODRAVEN_TERMINATION_MESSAGE_LIMIT (default 3072): line 3 is dropped first, then line 2 gives way to executedSha256=<hex>. No line is ever cut; verify.sh refuses PITRCoverageInvalid: restore.py's verdict is N bytes, over … rather than publish one that does not fit.

The operator accepts the verdict only from a Succeeded pod the Job owns, exit code 0, matching the rendered mode, target, encryption and source group, with a line 2 whose set hashes to executedSha256; anything else reports PITRCoverageInvalid. It fills pointInTime in status from it. restore.py and verify.sh write their own refusals there too (PITRCoverageInvalid: …, PITRArchiveIdentityMismatch: …, PITRExcludeIntersectsBackup: …, PITRNoArchivedReplay: …, PITRReplayMismatch: …, PITRReplayUnsupported: …, BLOODRAVEN_PITR_FAILED: …, SanityCheckFailed: …, SanityCheckTimeout: …). Other failures fall back to the log tail. The failure detail recorded in status is the earliest failing container's message. The served reason for BLOODRAVEN_PITR_FAILED is PITRReplayFailed; the message keeps the sentinel.

The Python restore Job's text sentinel BLOODRAVEN_RESTORE_COMPLETE carries only the source and target GTID and binlog coordinates, which the operator reads best-effort; the pitr* keys are gone. These Job messages are not in the Event reference contract above.

Artifact cleanup Job

The finalizer's cleanup Job runs bloodraven artifact-cleanup. It logs artifact cleanup complete (objects) or artifact cleanup failed (exitCode, error), with subcommand, backup, storageType and location on every record. These Job messages are best-effort, not part of the Event reference contract. The operator reads the outcome from the container's termination message instead: BLOODRAVEN_CLEANUP_COMPLETE objects=<n> on success, or BLOODRAVEN_CLEANUP_FAILED: <reason> on failure, with exit code 64 for a refusal or configuration error and 75 for a transient storage error. An abandoned (never-succeeded) backup's cleanup that finds no stamp but objects at the location deletes nothing, exits 0 with BLOODRAVEN_CLEANUP_SKIPPED: <reason> and logs WARN artifact cleanup skipped (reason), with the same common fields.